0

If I create a Python virtual environment like so:

$ python3 -m venv my_venv

... and then look at the Python binary in the bin directory like so:

$ ls -l my_env/bin/python*
lrwxrwxrwx 1 fred fred  7 Sep 12 15:57 my_env/bin/python -> python3
lrwxrwxrwx 1 fred fred 16 Sep 12 15:57 my_env/bin/python3 -> /usr/bin/python3

I see that the python is symlinked to the main global python. Therefore, what mechanism ensures that the Python packages we install after activating the virtual environment, that the packages are installed to site-packages?

Nathaniel Ford
  • 20,545
  • 20
  • 91
  • 102
felix001
  • 15,341
  • 32
  • 94
  • 121
  • The short answer is that the `site` module (created on startup) is responsible for constructing the `site-packages` directory. – chepner Sep 13 '22 at 19:09
  • 1
    I guess it should be explained in ["_PEP 405 – Python Virtual Environments_" Specification](https://peps.python.org/pep-0405/#specification) -- "_If a `pyvenv.cfg` file is found either adjacent to the Python executable or one directory above it (if the executable is a symlink, it is not dereferenced), this file is scanned for lines of the form `key = value`. If a `home` key is found, this signifies that the Python binary belongs to a virtual environment, and the value of the `home` key is the directory containing the Python executable used to create this virtual environment._" – sinoroc Sep 13 '22 at 21:13

1 Answers1

0

I was interested so I did a little searching and performed a couple experiments. A reference: How do I find the location of my Python site-packages directory?

I created a virtual environment my_venv in /tmp and did not activate it. On Linux activating a virtual environment (my_venv/bin/activate) adds the virtual environment's bin directory to the path; I found I could simulate that by using the full path; i.e., executing ./my_venv/bin/python and ./my_venv/bin/pip directly. Results of such experimentation:

> ################### system executables
> python3 -m site       # system's python
sys.path = [
    <current directory>
    '/usr/lib/python310.zip',
    '/usr/lib/python3.10',
    '/usr/lib/python3.10/lib-dynload',
    '/usr/local/lib/python3.10/dist-packages',
    '/usr/lib/python3/dist-packages',
]
> pip3 show <a package>    # system's pip
...
Location: /usr/lib/python3/dist-packages
...
> pip3 show <package installed in the venv>
WARNING: Package(s) not found

> ################### venv executables
> /tmp/my_venv/bin/python3 -m site
sys.path = [
    <current directory>
    '/usr/lib/python310.zip',
    '/usr/lib/python3.10',
    '/usr/lib/python3.10/lib-dynload',
    '/tmp/my_venv/lib/python3.10/site-packages
]
> /tmp/my_venv/bin/pip3 show <package installed in the venv>
...
Location: /tmp/my_venv/lib/python3.10/site-packages
...
> /tmp/my_venv/bin/pip3 show <package installed on the system>
WARNING: Package(s) not found

I took a peek at the activation script, /tmp/my_venv/bin/activate. It set a couple environment variables, notably one named VIRTUAL_ENV. I set this environment variable to the path to my virtual environment, as it would have been set if I'd activated the virtual environment. Then I checked paths using the system's python -m site and pip show <package>, and got the same responses I'd gotten without the environment variable set. From this information I concluded that python and pip probably don't pay attention to the environment variable.

I then symlinked the system's python3 and pip3 executables to a local directory, and got the same responses from commands using these (./python3 and ./pip3) that I'd gotten from the system executables.

I then created the following directory structure:

bin/python3    # symlink to system python3
bin/pip3       # symlink to system pip3
lib/python3.10/site-packages/

These still gave me the system paths. However, as soon as I copied the file pyvenv.cfg from a virtual environment into my directory structure, it started to use my local paths!

Note: I copied pyvenv.cfg straight from the real virtual environment without changing it. That file contained the line "home = /usr/bin/".

I concluded that in order to determine the path to site-packages, Python uses the path of the executed file by preference (even if that's a symlink), and knows to look for a virtual environment configuration file one directory up from the executable. If the path is a symlink and doesn't yield a valid path, looks like it is able to back to the path of the actual executed file.

(this agrees with sinoroc's summary of PEP 405 – Python Virtual Environments, written while I was fooling around)

This is deductive, of course. The source code would also provide the definitive answer.

DraftyHat
  • 428
  • 1
  • 2
  • 6