24

I know that virtualenv, if not passed the --no-site-packages argument when creating a new virtual environment, will link the packages in /usr/local/lib/python2.7/site-packages (for Python 2.7) with a newly-created virtual environment. On Ubuntu 12.04 LTS, I have three locations where Python 2.7 packages can be installed (using the default, Ubuntu-supplied Python 2.7 installation):

  1. /usr/lib/python2.7/dist-packages: this has my global installation of ipython, scipy, numpy, matplotlib – packages that I would find difficult and time-consuming to install individually (and all their dependences) if they were not available via the scipy stack.
  2. /usr/local/lib/python2.7/site-packages: this is empty, and I think it will stay that way on Ubuntu unless I install a package from source.
  3. /usr/local/lib/python2.7/dist-packages: this has very important local packages for astronomy, notably those related to PyRAF, STScI, etc., and they are extremely difficult and time-consuming to install individually.

Note that a global directory such as /usr/lib/python2.7/site-packages does not exist on my system. Note also that my global installation of ipython, scipy, etc. lets me use those packages on-the-fly without having to source/activate a virtual environment every time.

Naturally, I now want to use virtualenv to create one virtual environment in my user home directory which I will source/activate for my future projects. However, I would like this virtual environment, while being created, to link/copy all of my packages in locations (1) and (3) in the list above. The main reason for this is that I don't want to go through the pip install process (if it is even possible) to re-install ipython, scipy, the astro-packages, etc. for this (and maybe other) virtual environments.

Here are my questions:

  1. Is there a way for me to specify to virtualenv that I would like it to link/copy packages in these two dist-packages directories for virtual environments that are created in the future?
  2. When I eventually update my global installation of scipy, ipython, etc. in the two dist-packages directories, will this also update/change the packages that my virtual environment uses (and which it originally got during virtualenv creation)?
  3. If I ever install a package from source on Ubuntu, will it go in /usr/local/lib/python2.7/dist-packages, or /usr/local/lib/python2.7/site-packages?

Thanks in advance for your help!

Piotr Dobrogost
  • 41,292
  • 40
  • 236
  • 366
quantumflash
  • 691
  • 2
  • 5
  • 16

5 Answers5

17

This might be a legitimate use of PYTHONPATH - an environmental variable that virtualenv doesn't touch, which uses the same syntax as the environmental variable PATH, in bash PYTHONPATH=/usr/lib/python2.7/dist-packages:/usr/local/lib/python2.7/dist-packages in a .bashrc or similar. If you followed this path,

  1. You don't have to tell your virtual environment about this at all, it won't try to change it.

  2. No relinking will be required, and

  3. That will still go wherever it would have gone (pip install always uses /usr/local/lib/python2.7/dist-packages/ for my Ubuntu) if you install them outside of your virtual environment. If you install them from within your virtual environment (while it's activated) then of course it'll be put in the virtualenvironment.

Thomas
  • 6,515
  • 1
  • 31
  • 47
  • 4
    This works great, thanks! An alternative is to create a [.pth](http://docs.python.org/2/install/index.html#modifying-python-s-search-path) file in which each line is a path to be added to the default sys.path, and to place that file in the virtual environment's site-packages directory, e.g., dist-packages.pth in `/home/userX/envY/lib/python2.7/site-packages`. I've opted for this latter method, for now, since it makes more explicit why this particular virtual environment has access to the global dist-packages directories, rather than the "global" `PYTHONPATH` environment variable in .bashrc. – quantumflash Oct 07 '13 at 02:06
  • 2
    Ah that's nice too. Sometimes I add things to the activate script in a virtual environment, and you could set and unset PYTHONPATH in there, but since .pth files work that's maybe nicer. It's a good place for other things like that though. – Thomas Oct 07 '13 at 02:47
  • This works well. I wasn't able to pip install numpy from a virtualenv until I setup the PYTHONPATH the way this answer prescribed. – Will Feb 11 '15 at 04:04
  • 1
    Using a .pth file works well with configuration management as its a simple matter to copy the canned file into place during setup. – James Moore Jan 21 '16 at 19:46
  • I've got Ubuntu 14.04 and for me, /usr/lib/python2.7 is owned by root. I am trying to run use virtualenv (or whatever method necessary) to run Python unittest test with Jenkins. My jenkins system user does not have access to these directories. I'd rather not add my jenkins user to the root group. What am I doing wrong here? – siliconsmiley Mar 02 '16 at 16:23
8

I'm just getting my head around virtualenv, but there seems to be an easier way than mentioned so far.

  1. Since virtualenv 1.7 --no-site-packages has been the default behavior. Therefore using the --system-site-packages flag to virtualenv is all that is needed to get dist-packages in your path - if you use the tweaked virtualenv shipped by Ubuntu. (This answer and this one give some useful history). I've tested this and it does work.

    $ virtualenv --system-site-packages .

  2. I agree with Thomas here - I can't see any action required in virtualenv to see the effect of updates in dist-packages.

  3. Having tested that with python setup.py install, it does (again as Thomas said) still go to dist-packages. You could change that by building your own python, but that's a bit extreme.

Community
  • 1
  • 1
Danimal
  • 1,208
  • 10
  • 21
4

PYTHONPATH works for me.

vim ~/.bashrc

add this line below:

export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.7/dist-packages:/usr/local/lib/python2.7/dist-packages
source ~/.bashrc
hahakubile
  • 6,978
  • 4
  • 28
  • 18
4

In the directory site-packages, create a file dist.pth In the file dist.path, put the following: ../dist-packages

enter image description here

Now deactivate and activate your virtualenv. You should be set.

Mayank Jaiswal
  • 12,338
  • 7
  • 39
  • 41
2

What you want to achieve here is essentially add specific folder (dist-packages) to Python search path. You have a number of options for this:

  1. Use path configuration (.pth) file, entries will be appended to the system path.
  2. Modify PYTHONPATH (entries from it go to the beginning of system path).
  3. Modify sys.path directly from your Python script, i.e. append required folders to it.

I think that for this particular case (enable global dist-packages folder) third option is better, because with first option you have to create .pth file for every virtualenv you'll be working in (with some external shell script?). It's easy to forget it when you distribute your package. Second option requires run-time setup (add a envvar), which is, again, easy to miss.

And only third option doesn't require any prerequisites at configure- or run-time and can be distributed without issues (on the same-type system, of course).

You can use function like this:

def enable_global_distpackages():
    import sys
    sys.path.append('/usr/lib/python2.7/dist-packages')
    sys.path.append('/usr/local/lib/python2.7/dist-packages')

And then in __init__.py file of your package:

enable_global_distpackages()
Palasaty
  • 5,181
  • 1
  • 26
  • 22