3

I am working with a project that has a user-written module called types.py buried in a second-level package (its path from the project root is package/subpackage/types.py).

This is causing problems because the Python library also has a types module. When enum.py, another Python library module, attempts to import types, the user-written version is imported instead, wreaking havoc.

What's puzzling me is that the import inside enum.py does not qualify types with any package names:

# line 10 of enum.py:
from types import MappingProxyType, DynamicClassAttribute

so why is Python selecting the user-written types which is in a two-level subpackage? It seems to me the user-written types would only be imported if one uses

# what I expect an 'import' would have to be like to access the user-written types.py
from package.subpackage.types import ...

Another possible explanation would be that sys.path contained the package/subpackage directory, but this is not the case when I print its content right before the enum.py import:

enum.py: Path:
/home/me/PycharmProjects/myproject
/home/me/anaconda3/envs/myproject/lib/python37.zip
/home/me/anaconda3/envs/myproject/lib/python3.7
/home/me/anaconda3/envs/myproject/lib/python3.7/lib-dynload
/home/me/anaconda3/envs/myproject/lib/python3.7/site-packages

So, how can the importing of the user-written types.py module be explained?

UPDATE: the first comment suggests this happens because my project's path is the first item in sys.path. However, I set up a really simple project in which a module called mymodule is in package.subpackage:

Sandbox Project

Importing from mymodule without using the package and subpackage names does not work:

# main.py
# Works:
from package.subpackage.mymodule import my_module_field

# Does not work:
# from mymodule import my_module_field

So I still do not understand why the from types import in enum.py can work find the user-written types.py without the packages names.

UPDATE 2: printing out more information, I see that when I print sys.path as soon as enum.py starts (I modified the standard library file to print it), I see that the package/subpackage directory is in sys.path, even though it was not at the beginning of execution. So this explains why the user-written typos.py is being used.

The issue now is why sys.path is appended with the package/subpackage directory. I searched all occurrences of sys.path in my code and even though the current directory is appended to it at some points, it is never the package/subpackage directory. Where can this be happening?

user118967
  • 4,895
  • 5
  • 33
  • 54
  • Because of `/home/me/PycharmProjects/myproject` being first in your path, `types` are first imported from your project, not from stdlib. – sanyassh Dec 20 '19 at 20:44
  • Thanks, @sanyash, that's interesting, but how does the import find the user-written types.py without package and subpackage? I've updated my question to show a test I wrote that illustrates this. – user118967 Dec 20 '19 at 21:16
  • use the `__init__.py` to resolve relative import issues – bcr Dec 20 '19 at 21:27
  • @sanyash - sys.path is only used *after* Python looks in the built in modules. The built in modules can be seen using `sys.builtin_module_names` – Rahul P Dec 20 '19 at 21:41
  • @user118967 - Are the two __init__.py files (that are shown in the screenshot) empty? – Rahul P Dec 20 '19 at 21:42
  • @RHP: yes, they are completely empty. And the ones in the actual project only set __version__, otherwise also empty. – user118967 Dec 20 '19 at 21:53
  • @bcr: even if I manage to resolve relative import issues using __init__, the question remains as to how and why Python is bringing in the user-written types.py even though it is inside a subpackage that is not mentioned in the import. – user118967 Dec 20 '19 at 21:55
  • 1
    Without access to the actual project files, I find this question is very hard to answer. I would recommend cloning the project, removing everything you think is not causing the problem one chunk at a time and checking when the problem disappears. You haven't provided a minimal example that still has the same issue - trying to reproduce the issue in a minimal setting is the first step towards solving the mystery (and perhaps the problem). – Grismar Dec 22 '19 at 23:43
  • 2
    Why not answer your own question (and perhaps edit it with hindsight to ask about the real problem) rather than editing the question to be an explanation? – Davis Herring Dec 22 '19 at 23:59
  • 1
    I agree with @Davis-Herring that you can answer your own question and explain in details what happened and why, how to avoid that in future. So if somebody will have situation like yours - will be able find your question and solution. – wowkin2 Dec 23 '19 at 09:20
  • Since I can't VTC a bountied question, here's a duplicate: https://stackoverflow.com/a/45448394/812183 – anthony sottile Dec 26 '19 at 00:03

1 Answers1

1

Not sure this counts as a real answer because it would not be possible to answer it based on the question information by itself (and adding all the details to the question is impractical). In any case, here's the solution.

Basically, upon greater inspection I found out that a script invokes another script as an external process, and this latter script is in the package/subpackage directory, which is added to sys.path in the new process. About this last point, I'm not sure why; I am assuming that a script's current directory is always added to sys.path.

user118967
  • 4,895
  • 5
  • 33
  • 54
  • It is somewhat strange because sys.path should (from my knowledge) be used only after python looks in the built in modules. Do you know that a types.py file exists elsewhere for sure (apart from the one in your package) ? – Rahul P Dec 24 '19 at 00:51
  • @user118967 Can you provide a minimal example that has the same issue? – aaron Dec 24 '19 at 01:41
  • Yes, the directory of a script is always added to sys.path. – Martijn Pieters Dec 25 '19 at 00:47
  • 1
    @RahulP: That is incorrect. Python uses `sys.path` to find all modules, built-in or not. – Ethan Furman Dec 28 '19 at 15:54