5

Why does sys.path contain /usr/... before directories from my virtualenv?

I create the virtualenv with --system-site-packages

The sys.path looks like this at the moment:

/home/my-virtualenv/src/foo
/usr/lib/python2.7/site-packages        <--- /usr paths should be below 
/usr/lib64/python2.7/site-packages
/home/my-virtualenv/lib/python27.zip
/home/my-virtualenv/lib64/python2.7
/home/my-virtualenv/lib64/python2.7/plat-linux2
/home/my-virtualenv/lib64/python2.7/lib-tk
/home/my-virtualenv/lib64/python2.7/lib-old
/home/my-virtualenv/lib64/python2.7/lib-dynload
/usr/lib64/python2.7
/usr/lib/python2.7
/usr/lib64/python2.7/lib-tk
/home/my-virtualenv/lib/python2.7/site-packages

I want all paths outside my virtualenv (/usr...) to be below the paths of the virtualenv.

Otherwise crazy things happen: I install a package with pip. Pip tells me that the new version is installed (pip freeze | grep -i ...) but the import does use the one from /usr/lib/python2.7/site-packages

I can't use --no-site-packages in my context.

Is there a way to sort sys.path?

Why I use system-site-packages

There seems to be no straight forward way to make single libraries from the global site-packages available in the virtualenv. See this question: make some modules from global site-packages available in virtualenv

There are packages like python-gtk which are very hard to install in a virtualenv.

Community
  • 1
  • 1
guettli
  • 25,042
  • 81
  • 346
  • 663
  • 3
    "I can't use `--no-site-packages` in my context": Please elaborate. – Kevin Jan 30 '15 at 15:53
  • @Kevin I could explain it, but what would change? I can't use --no-site-packages in my context - That's the way it is. The question is about sorting sys.path :-) – guettli Jan 30 '15 at 15:58
  • 2
    Frankly, `--system-site-packages` is broken. You're nearly always better off installing everything into the virtualenv. The fact that your situation is (apparently) *different* makes it difficult for me to give you a useful answer without more information. – Kevin Jan 30 '15 at 16:06
  • 2
    Is that restriction (no `--no-site-packages`) because of some shared global packages? If so, why not duplicate them in virtualenv? – oxfn Feb 02 '15 at 08:05
  • I updated the question and added a section "Why I use system-site-packages" – guettli Dec 02 '15 at 06:45

5 Answers5

2

You can simply prepend desired path before module imports. Kinda hacky, but should solve your problem

import sys
sys.path.insert(1, '/home/my-virtualenv/lib/python2.7/site-packages')

import lalala
oxfn
  • 6,590
  • 2
  • 26
  • 34
  • In cetain situations this *is* the best way to go! May projects use this to make sure that automatic tests actually test the development code, and not some package installed in the system or anywhere else. Example: https://github.com/sphinx-doc/sphinx/blob/master/tests/run.py#L21 – Dr. Jan-Philip Gehrcke Feb 05 '15 at 17:21
  • Sorry, that is not a solution to my answer. Quoted from above: "I want all paths outside my virtualenv (/usr...) to be below the paths of the virtualenv." – guettli Feb 09 '15 at 09:25
2

Edit, after discussion:

"I want all paths outside my virtualenv (/usr...) to be below the paths of the virtualenv. [...] It needs to be sorted."

Then, just sort your sys.path before the first import happens. Given a certain path prefix corresponding to the location of your virtalenv, this approach likely is sufficient:

sys.path = sorted(sys.path, key=lambda x: x.startswith(prefix), reverse=True)

sorted()'s sorting behavior is stable: the original order is preserved for items with same sort key. Here, only two sort keys are used: True and False. You need to come up with a reliable way to set your prefix (You might want to hard-code it, or determine it based on the current working directory, I am sure you find a way).

Original answer (still valid, in general):

You did not want to elaborate on your requirements and application scenario too much, so I provide a more general answer: you may need to overthink your approach, and maybe not expect virtualenv to entirely solve your problem.

In certain situations, virtualenv is just not the perfect solution. It is a compromise, and one side of this compromise is exhaustively described in this article: https://pythonrants.wordpress.com/2013/12/06/why-i-hate-virtualenv-and-pip/

In many scenarios virtualenv serves a splendid purpose and does its job very fine! It helped me a lot, for sure, especially for development purposes. In other scenarios, it is either not a complete solution or even a bad solution.

So, there are a few different options now that I see:

  1. Use virtualenv, but take control over those things that you need to have changed. For instance, modify sys.path from within your package before doing a certain import. While some might consider this "not clean", it for sure is a very quick way to efficiently and reliably control directory search order. sys.path.insert(1, foo) is actually used by many packages and test environments. It not uncommon at all. This approach may provide you with a working solution after just one minute of work. Give it a try!
  2. If you think that virtualenv in your case does not behave as documented or clearly shows buggy behavior, then please report you findings to the project. They will surely appreciate your concise feedback.
  3. If you think that virtualenv does not provide the level of isolation or control that you need in your application case, you might want to look at other options, such as Docker or Vagrant, or fully-features virtual machines.
Dr. Jan-Philip Gehrcke
  • 33,287
  • 14
  • 85
  • 130
  • Thank you for your reply. I am very happy with virtualenv. For reasons I don't want to explain here, we need to include the global site packages. Before we used the default (not site packages). If sys.path gets ordered somehow, everything will be fine again. I think the site.py solution is a good one. – guettli Feb 05 '15 at 22:02
  • Okay, just modify `sys.path` *before* doing the import. This is really simple! You do not need to use `site.py` for that. Example: `sys.path.insert(1, "/opt/packages"); import superduper`. -- also, you might want to upvote those answers that provided some insight! – Dr. Jan-Philip Gehrcke Feb 06 '15 at 16:31
  • I think "just modify sys.path before doing the import" is no solution for me. If there twenty files where I use `import superduper` I would need to use `sys.path.insert()` twenty times. No, I don't like this. I upvoted the my up to now preferred answer. – guettli Feb 06 '15 at 17:16
  • 1
    "just modify sys.path before doing the import" actually *is* the solution of your upvoted anwer. Just think about hit: you do not need to modify `sys.path` in more than one place. It is a shared global! Modify it before the *first* import, e.g. in `__init__.py`. Or wherever your first import happens! `site.py` is also okay, I just wanted to point out that it is not needed generally. Also, generally you do *not need* a complex sorting algorithm, you just need the to have *that* directory first from where you want to import, the order of the remaining items does not matter. – Dr. Jan-Philip Gehrcke Feb 06 '15 at 17:30
  • @Jan-Phlip quoting my question: "I want all paths outside my virtualenv (/usr...) to be below the paths of the virtualenv.". There is not "that directory". I want this: "first check the virtualenv, then look at the globals" for all paths. AFAIK a single sys.path.insert() won't help. It needs to be sorted. – guettli Feb 06 '15 at 19:37
  • 1
    Given that is the case, then I can really recommend a `sorted()`-based solution, à la `sorted(sys.path, key=lambda x: x.startswith(prefix), reverse=True)`. The only tricky thing is to reliably find your `prefix`. This entirely depends on your project structure and runtime requirements. – Dr. Jan-Philip Gehrcke Feb 06 '15 at 19:41
  • I edited the answer to include the main result of this discussion here. – Dr. Jan-Philip Gehrcke Feb 07 '15 at 15:42
1

If you're using interactive shell you can set PYTHONSTARTUP, or also set a few bash aliases to modify your PYTHONPATH in the ways that you prefer. You can also modify the activate files activate.csh, activate_this.py in the bin directory of your virtualenv; or the site.py file as another response mentioned.

pip + virtualenv is - as it has been pointed out not a very friendly combo for system-wide packages, i.e. there is no default flag for say use system numpy or package x,y when building an environment. Although as has been suggested in comments I would also recommend building with --no-site-packages and then importing specifically what you need from /usr/lib

user3467349
  • 3,043
  • 4
  • 34
  • 61
0

Below straight solution resolve your problem:

1- Open my-virtualenv/lib/python2.7/site.py file for edit.

2- Append below lines of sort algorithm(or any code that resolve your problem) at the end of this file(without first line indent):

prefix = __file__.rsplit(os.sep, 3)[0]

Below code line Suggested by @Jan-Philip Gehrcke:

sys.path = sorted(sys.path, key=lambda x: x.startswith(prefix), reverse=True)

3- Run your program and enjoy

This solution helping you to sort the sys.path list without adding annoying lines to body of your program.

Note: The site module is automatically imported during initialization. The automatic import can be suppressed using the interpreter’s -S option.

M.javid
  • 6,387
  • 3
  • 41
  • 56
  • Can you elaborate what this method is doing? Why would you want to perform this rather unreadable sorting method? What is the advantage of your method over just making sure that *the* directory from where you want to import is first in `sys.path`? – Dr. Jan-Philip Gehrcke Feb 06 '15 at 17:34
  • This method move all items of `sys.path` that not starts with environment-directory path to end. **A problem generally have more than one solution** (base law of programming). – M.javid Feb 06 '15 at 17:56
  • `pre = __file__.rsplit(os.sep, 3)[0]` for sure will not work reliably, no matter what you actually want to achieve with this. I guess you somehow want to dynamically determine parts of the path of the executed python script file. If so, have a read over here: http://stackoverflow.com/questions/2632199/how-do-i-get-the-path-of-the-current-executed-file-in-python – Dr. Jan-Philip Gehrcke Feb 06 '15 at 18:06
  • Current question is not about this problem and i don't persist using this algorithm @guettli can use any algorithm that solve this problem, this introduced algorithm only is a suggest ;-) thanks for your finesse. – M.javid Feb 06 '15 at 18:13
  • 1
    I understand you basically want to sort the items in `sys.path`, based on the prefix. Your code is really not well-readable, possibly wrong and difficult to debug. It's "not pythonic". Better use `sorted()` and make use of the fact that this method is stable (original order is preserved for items with same sort key). That boils down to `s = sorted(sys.path, key=lambda x: x.startswith(prefix), reverse=True)`. Much simpler than your code, right? – Dr. Jan-Philip Gehrcke Feb 06 '15 at 18:15
  • Second step text changed to: "2- Append below lines of sort algorithm **(or any code that resolve your problem)** at the end of this file(without first line indent)" – M.javid Feb 06 '15 at 18:17
  • I know what you mean, and please, this is not meant in an offending way. You are showing Python code, and I just point out what I think is problematic about it. This is what StackOverflow is about. Do whatever you want with the information I provided to you ;-). – Dr. Jan-Philip Gehrcke Feb 06 '15 at 18:20
  • Presented code in normal used resolve current problem. if using of built-in `__file__` variable in normal use of python raising any exception i conclusion python have a bug &-( i say **special situations needs special solutions**. think more about **A problem generally have more than one solution** – M.javid Feb 06 '15 at 18:29
  • You edited your answer and said I suggested some code, but I surely did not suggest the `prefix = __file__.rsplit(os.sep, 3)[0]` part. Please clarify, thanks. – Dr. Jan-Philip Gehrcke Feb 09 '15 at 00:44
0

A easy way I found to solve this problem was just to add your virtualenv site-packages folder to the $VIRTUAL_ENV/lib/python2.7/site-packages/_virtualenv_path_extensions.pth file or just enter in shell after activate the environment:

$ add2virtualenv $VIRTUAL_ENV/lib/python2.7/site-packages

this would force to appear the folder before your systems folders as I notice it does for the others paths added.

I believe this is not the cleanest solution, but indeed virtualenv strangely order the paths.

Another observation is when I start ipython the sys.path appears in a reasonable order, but when inside the python interpreter doesn', not sure why tho.

AndreLobato
  • 170
  • 1
  • 12