How to run a Python project using __pycache__ folder?

Question

I want to run a Pythonic project using Python compilation (.pyc or __pycache__). In order to do that in Python2, I haven't any problem.

---

Here is a simplified example in a Python2 project:

• Project tree:

  test2
  ├── main.py
  └── subfolder
      ├── __init__.py
      └── sub.py
  

• Compile:

  python -m compileall test2
  

• Project tree after the compile:

  test2
  ├── main.py
  ├── main.pyc
  └── subfolder
      ├── __init__.py
      ├── __init__.pyc
      ├── sub.py
      └── sub.pyc
  

• As you can see, several .pyc manually generated. Now I can run this project using main.pyc as fine, which has a relation with the sub.py:

  python main.pyc
  

  Out:

  Hi
  Bye
  

• main.py content:

  from subfolder import sub
  
  print('Bye')
  

• sub.py content:

  print('Hi')
  

---

Now I want to retry this behavior in a Python3 project.

Here is a simplified asyncio (available in Python3) project:

• Project tree:

  test3
  ├── main.py
  └── subfolder
      ├── __init__.py
      └── sub.py
  

• Compile:

  python3 -m compileall test3
  

• Project tree after the compile:

  test3
  ├── main.py
  ├── __pycache__
  │   └── main.cpython-36.pyc
  └── subfolder
      ├── __init__.py
      ├── __pycache__
      │   ├── __init__.cpython-36.pyc
      │   └── sub.cpython-36.pyc
      └── sub.py
  

• As you can see, __pycache__ folders manually generated. But I cannot run this project using main.cpython-36.pyc which has a relation with subfolder:

  cd test3/__pycache__
  python3 main.cpython-36.pyc
  

  Out (I expected that produced the Hi Bye message):

  Traceback (most recent call last):
    File "test3/main.py", line 2, in <module>
  ModuleNotFoundError: No module named 'subfolder'
  

• main.py content:

  import asyncio
  from subfolder import sub
  
  async def myCoroutine():
      print("Bye")
  
  def main():
      loop = asyncio.get_event_loop()
      loop.run_until_complete(myCoroutine())
      loop.close()
  
  main()
  

• sub.py content:

  print('Hi')
  

---

Question:

How can I run this project (above Python3 project) using __pycache__ folder?

Or

How can I run a Python3 project with the relation between subfolders using python compilation?

---

[NOTE]:

• I cannot use the python compileall (Python2 compile) in the above Python3 project due to the asyncio method.

• My Python(s) version is Python2.7 and Python3.6

Answer 1

You can enforce the same layout of pyc-files in the folders as in Python2 by using:

python3 -m compileall -b test3

The option -b triggers the output of pyc-files to their legacy-locations (i.e. the same as in Python2).

After that you can once again use the compiled cache via:

python3 main.pyc

The way the loading of modules works since PEP-3147, it is impossible to use pyc-files from __pycache__ folder in the way you intend: If there is no *.py-file, the content of the __pycache__ is never looked-up. Here is the most important part of the workflow:

   import foo
     |
     |
     -- >  [foo.py exists?]  --- NO ----> [foo.pyc exists?]  -- NO --> [ImportError] 
            |                                     |
            |                                    YES
           YES                                    |--> [load foo.pyc]
            |
            |-> [look up in __pycache__]

                               

That means, files from __pycache__ are only looked up, when a corresponding *.py-file can be found.

---

Obviously, building python scripts with a Python version 3.X in this way and trying to run the resulting pyc-files with another Python version 3.Y will not work: Different Python versions need different pyc-files, this is the whole point behind PEP-3147.