2

The Python documentation for the import statement (link) contains the following:

The public names defined by a module are determined by checking the module’s namespace for a variable named __all__; if defined, it must be a sequence of strings which are names defined or imported by that module.

The Python documentation for modules (link) contains what is seemingly a contradictory statement:

if a package’s __init__.py code defines a list named __all__, it is taken to be the list of module names that should be imported when from package import * is encountered.

It then gives an example where an __init__.py file imports nothing, and simply defines __all__ to be some of the names of modules in that package.

I have tested both ways of using __all__, and both seem to work; indeed one can mix and match within the same __all__ value.

For example, consider the directory structure

foopkg/
  __init__.py
  foo.py

Where __init__.py contains

# Note no imports

def bar():
    print("BAR")

__all__ = ["bar", "foo"]

NOTE: I know one shouldn't define functions in an __init__.py file. I'm just doing it to illustrate that the same __all__ can export both names that do exist in the current namespace, and those which do not.

The following code runs, seemingly auto-importing the foo module:

>>> from foopkg import *
>>> dir()
[..., 'bar', 'foo']

Why does the __all__ attribute have this strange double-behaviour?

The docs seem really unclear on how it is supposed to be used, only mentioning one of its two sides in each place I linked. I understand the overall purpose is to explicitly set the names imported by a wildcard import, but am confused by the additional, seemingly auto-importing behaviour. Is this just a magic shortcut that avoids having to write the import out as well?

martineau
  • 119,623
  • 25
  • 170
  • 301
Christopher Riches
  • 769
  • 6
  • 21
  • 1
    I don't understand why you consider those statements to be contradictory. What is the contents of `foo.py`? What are the types of `foo` and `bar` once you've imported them? – Mark Ransom Dec 19 '19 at 22:07
  • @MarkRansom Perhaps contradictory was an overstatement; I merely meant that they each define `__all__` differently. The contents of `foo.py` are unimportant; once imported, the type of `foo` is `` and so anything in `foo.py` could be accessed by dot notation. `bar` is of course type ``. – Christopher Riches Dec 19 '19 at 22:17
  • They are different variables defined in different types of object. Their purpose is different, so there is no contradiction at all – Mad Physicist Dec 19 '19 at 22:42
  • @MadPhysicist please elaborate in an answer if you can. I fail to see any difference in purpose, just in behaviour. – Christopher Riches Dec 19 '19 at 22:44
  • 1
    Packages are different from modules. They behave similarly from the user point of view much of the time, but they aren't the same thing. Modules only have attributes. They can reference other modules, but they can't have sub-modules as such. – Mad Physicist Dec 19 '19 at 22:52
  • @MadPhysicist I think this is the real answer I was looking for. If `__all__` is in a package's `__init__.py`, then it can auto-import because that's part of its functionality. If it's just in a module, then it cannot, because the whole idea would be ill-formed given that modules cannot have submodules. If you could post *that* as an answer, I'll mark it as accepted. I think all this confusion originated from the simple fact that the second quote in my question doesn't acknowledge that the auto-import is in addition to, not a replacement for the normal behaviour. – Christopher Riches Dec 19 '19 at 23:03
  • Python's implementatino loop through ```__all__``` and try to import each element in ```__all__``` That's why it'll auto-import foo and also achieve white listing. https://github.com/python/cpython/blob/fee552669f21ca294f57fe0df826945edc779090/Python/ceval.c#L5152 – JxCode Dec 19 '19 at 23:23

2 Answers2

1

The documentation is a bit hard to parse because it does not mention that packages generally also have the behavior of modules, including their __all__ attribute. The behavior of packages is necessarily a superset of the behavior of modules, because packages, unlike modules, can have sub-packages and sub-modules. Behaviors not related to that feature are identical between the two as far as the end-user is concerned.

The python docs can be minimalistic at times. They did not bother to mention that

  1. Package __init__ performs all the module-like code for a package, including support for star-import for direct attributes via __all__, just like a module does.
  2. Modules support all the features of a package __init__.py, except that they can't have a sub-package or sub-module.

It goes without saying that to make a name refer to a sub-module, it has to be imported, hence the apparent, but not really double-standard.

Mad Physicist
  • 107,652
  • 25
  • 181
  • 264
  • I understand *what* is happening. I follow your example. My question is: `Why does the __all__ attribute have this strange double-behaviour?` It appears to me that the semantics of `__all__` are different depending on whether it is in a simple module, or a package's `__init__.py`. Is the auto-import functionality just a shortcut, or is there some deeper meaning that explains it? – Christopher Riches Dec 19 '19 at 22:53
  • 1
    @Chris. There is no double behavior. The behavior in packages is necessarily a superset of the behavior in modules. Modules don't have subsidiaries. – Mad Physicist Dec 19 '19 at 22:58
  • @ChristopherRiches. Updated my answer. Hopefully this version covers everything we discussed. – Mad Physicist Dec 19 '19 at 23:14
0

Update: How from M import * actually works?

The __all__ in __init__.py of folder foopkg works the same way as __all__ in foopkg.py

Why it'll auto-import foo you can see here: https://stackoverflow.com/a/54799108/12565014

The most import thing is to look at the cpython implementation: https://github.com/python/cpython/blob/fee552669f21ca294f57fe0df826945edc779090/Python/ceval.c#L5152

It basically loop through __all__ and try to import each element in __all__ That's why it'll auto-import foo and also achieve white listing

JxCode
  • 160
  • 5
  • What are the theoretical contents of this imaginary `foopkg.py` module? i.e. what does the name `foo` refer to within `foopkg.py`? I understand the *purpose* of `__all__`, just not why it seems to be able to magically auto-import modules from a given package. – Christopher Riches Dec 19 '19 at 22:42
  • https://stackoverflow.com/a/7948504/12565014 It's just how you structure your packages/modules. When you say from ```foopkg import foo``` **foopkg** can be a foopkg.py or foopkg(folder with ```__init__.py``` in it). For the auto-import issue I found that when you delete ```__all__``` in ```__init__.py``` ```foo``` won't be imported. So I guess how python handle ```import *``` under the hood is to somehow import elements in ```__all__``` one by one when ```__all__``` is set. – JxCode Dec 19 '19 at 22:54
  • Also if I add an name in ```__all__``` that doesn't exist, it'll say ```AttributeError: module 'foopkg' has no attribute 'blahblah'``` – JxCode Dec 19 '19 at 23:01