import werkzeug VS from werkzeug import security

Question

My current understanding (based on these answers: one, two, three; and Python documentation) of how import in Python works is (just in case it matters: all the code snippets are tested on Python 3.6.1):

Say we have a module mod, which has submodules sub and sub1; sub, in turn, has a function func; then we can (given that mod installed in current environment, of course):

import mod

mod.sub.func()
mod.sub1

# or
import mod.sub

mod.sub.func()
mod.sub1 # will result in "NameError: name 'mod' is not defined"

# or
from mod.sub import func

func()
mod.sub.func() # will result in "NameError: name 'mod' is not defined"
mod.sub1 # will result in "NameError: name 'mod' is not defined"

---

Recently, while playing with werkzeug.security.generate_password_hash and werkzeug.security.check_password_hash, in Python console, I have noticed that:

import werkzeug

werkzeug.security.generate_password_hash('some_password', method='pbkdf2:sha512', salt_length=25)

results in AttributeError: module 'werkzeug' has no attribute 'security'.

Though, the following works fine:

from werkzeug import security

security.generate_password_hash('some_password', method='pbkdf2:sha512', salt_length=25)

this (of course) too:

import werkzeug.security

werkzeug.security.generate_password_hash('some_password', method='pbkdf2:sha512', salt_length=25)

as well as this:

from werkzeug.security import generate_password_hash

generate_password_hash('some_password', method='pbkdf2:sha512', salt_length=25)

and, a bit surprisingly (at least for me), this one:

import werkzeug
from werkzeug import security

werkzeug.security.generate_password_hash('some_password', method='pbkdf2:sha512', salt_length=25)

My questions are:

1. Am I wrong (or lacking details) in some of my notions, concerning how import works in Python?
2. Why import werkzeug won't give me access to werkzeug.security? My understanding is — it should import werkzeug, along with all of it's submodules/attributes.
3. Why import werkzeug + from werkzeug import security allows access to werkzeug.security? My understanding: it should bind two separate names (with no connections between them), as follows: werkzeug to import werkzeug (i.e. werkzeug module) and security to from werkzeug import security (i.e. security submodule of werkzeug module.

Answer 1

I'm not sure I'm able to give a good answer to all your questions, but I found it interesting and took a look and here is my result.

In general, import mod.sub or from mod import sub assumes that sub is a sub-module in a mod package. However, it could also mean that sub is a field/variable declared in a mod module.

The presence of an __init.py__-file will denote that a folder is a package:

The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package (...).

I believe that, from werkzeug import security and import werkzeug.security both imports a module security, thus security.generate_password_hash is a known and valid attribute. Basically, from werkzeug.security import generate_password_hash imports that very attribute directly via the valid import statement.

In the Werkzeug Quickstart docs, I found the following:

Make sure to import all objects from the places the documentation suggests. It is theoretically possible in some situations to import objects from different locations but this is not supported.

Further, Werkzeug transition to 1.0 states:

Werkzeug originally had a magical import system hook that enabled everything to be imported from one module and still loading the actual implementations lazily as necessary. Unfortunately this turned out to be slow and also unreliable on alternative Python implementations and Google’s App Engine.

Starting with 0.7 we recommend against the short imports and strongly encourage starting importing from the actual implementation module. Werkzeug 1.0 will disable the magical import hook completely.

It appears that Werkzeug modifies how modules are loaded. (I speculate that this is not uncommon in big packages with contrib-content, e.g. Flask, Django; motivated by ability to lazy-load, improve performance, or manage contributed module content spread across packages.)

As you've discovered, import werkzeug does not import security from the werkzeug module, because (as far as I understand), the only submodules that will be imported as attributes are those defined on line 100 of the __init__.py:

# modules that should be imported when accessed as attributes of werkzeug
attribute_modules = frozenset(['exceptions', 'routing'])

In the same file, when looking at the Werkzeug's module(ModuleType)-class, and its __getattr__()-method:

class module(ModuleType):

    """Automatically import objects from the modules."""

    def __getattr__(self, name):
        if name in object_origins:
            module = __import__(object_origins[name], None, None, [name])
            for extra_name in all_by_module[module.__name__]:
                setattr(self, extra_name, getattr(module, extra_name))
            return getattr(module, name)
        elif name in attribute_modules:
            __import__('werkzeug.' + name)
        return ModuleType.__getattribute__(self, name)

It seems that module names in the object_origins dictionary, via definition in all_by_module, must be imported separately, and werkzeug.security is one of them.

Lastly, I think the reason for why the:

import werkzeug     
from werkzeug import security  

combination works, is that the first line does not import security, but the second one does, AND the __getattr__()-method will return modules that are explicitly imported.

Edit: this last section is not correct, tested by Filipp:

I expect that by simply doing only from werkzeug import security that still werkzeug.security.generate_password_hash() would work. (I have not tested or confirmed this)

Answer 2

TL;DR: import any attribute, contained in all_by_module dictionary, directly from werkzeug, i.e. from werkzeug import generate_password_hash.

---

Inspired by/ based on Thomas's answer, I will try to summarize answers to my own questions:

1. Am I wrong (or lacking details) in some of my notions, concerning how import works in Python?

From where I currently stand, the short answer is NO. Though, it's good to keep in mind that import rules/mechanics could be customized on package level via __init__.py.
Further reading on topic: Python import system, official docs on importlib, Importing Python Modules article.

2. Why import werkzeug won't give me access to werkzeug.security? My understanding is — it should import werkzeug, along with all of it's submodules/attributes.

As Thomas Fauskanger, correctly pointed out in his answer: import werkzeug does not import security from the werkzeug module, because the only submodules that will be imported as attributes — are those defined on line 100 of the Werkzeug's __init__.py (which are exceptions and routing). This assumption, could be verified by the following:

import werkzeug

werkzeug.routing # will return path to routing.py module
werkzeug.exceptions # will return path to exceptions.py module

werkzeug.security # AttributeError: module 'werkzeug' has no attribute 'security'

3. Why import werkzeug + from werkzeug import security allows access to werkzeug.security? My understanding: it should bind two separate names (with no connections between them), as follows: werkzeug to import werkzeug (i.e. werkzeug module) and security to from werkzeug import security (i.e. security submodule of werkzeug module.

That's a tricky one. As it is indicated in Werkzeug's __init__.py, by the docstring for module's __dir__ function:

Just show what we want to show.

That's (probably) why:

import werkzeug

dir1 = dir(werkzeug)
werkzeug.security # AttributeError: module 'werkzeug' has no attribute 'security'

from werkzeug import security

dir2 = dir(werkzeug)
werkzeug.security # will return path to security.py module
# BUT!
dir1 == dir2 # True

I think, Thomas right here as well, and:

...__getattr__() method will return modules that are explicitly imported.

---

Conclusion (or what I have learned =):

As stated in the docstring for Werkzeug's __init__.py:

...
The majority of the functions and classes provided by Werkzeug work on the HTTP and WSGI layer. There is no useful grouping for those which is why they are all importable from "werkzeug" instead of the modules where they are implemented.
...
The implementation of a lazy-loading module in this file replaces the werkzeug package when imported from within. Attribute access to the werkzeug module will then lazily import from the modules that implement the objects.

What this means is, instead of:

from werkzeug import security
security.generate_password_hash('some_password', method='pbkdf2:sha512', salt_length=25)
# OR
import werkzeug.security
werkzeug.security.generate_password_hash('some_password', method='pbkdf2:sha512', salt_length=25)
# OR
from werkzeug.security import generate_password_hash
generate_password_hash('some_password', method='pbkdf2:sha512', salt_length=25)
# OR
import werkzeug
from werkzeug import security
werkzeug.security.generate_password_hash('some_password', method='pbkdf2:sha512', salt_length=25)

You can simply do:

from werkzeug import generate_password_hash
generate_password_hash('some_password', method='pbkdf2:sha512', salt_length=25)

You can import & use any attribute contained in all_by_module dictionary, in the same fashion.

Answer 3

If you follow this syntax, it will work fine.

import werkzeug as xyz

With this, you can also do xyz.generate_hash_password() or can call any other function of xyz.