1

I am pip installing a variety of packages inside a virtual environment on an EC2 instance (2018-03). Pip successfully installs some of the packages, but not all. I am using pip install --force-reinstall -r requirements.txt on the following requirements file, which was created on a system with a working instance with Python 3.7.4 on Macos:

alembic==1.1.0
bcrypt==3.1.7
boto3==1.9.156
botocore==1.12.249
certifi==2019.9.11
cffi==1.12.3
chardet==3.0.4
Click==7.0
coverage==4.5.4
docutils==0.15.2
Flask==1.0.2
Flask-Bcrypt==0.7.1
Flask-Login==0.4.1
Flask-Migrate==2.5.2
Flask-Script==2.0.6
Flask-Session==0.3.1
Flask-SQLAlchemy==2.4.0
Flask-WTF==0.14.2
idna==2.8
itsdangerous==1.1.0
Jinja2==2.10.1
jmespath==0.9.4
lxml==4.3.2
Mako==1.1.0
MarkupSafe==1.1.1
numpy==1.16.2
pandas==0.24.2
patsy==0.5.1
psycopg2==2.8.3
pycparser==2.19
python-dateutil==2.8.0
python-editor==1.0.4
pytz==2019.2
requests==2.21.0
s3transfer==0.2.1
scikit-learn==0.20.3
scipy==1.2.1
six==1.12.0
SQLAlchemy==1.3.8
statsmodels==0.9.0
trimesh==3.1.11
urllib3==1.24.3
Werkzeug==0.15.5
WTForms==2.2.1

And pip gives me the following message:

Successfully installed Click-7.0 Flask-1.0.2 Flask-Bcrypt-0.7.1 Flask-Login-0.4.1 Flask-Migrate-2.5.2 Flask-SQLAlchemy-2.4.0 Flask-Script-2.0.6 Flask-Session-0.3.1 Flask-WTF-0.14.2 Jinja2-2.10.1 Mako-1.1.0 MarkupSafe-1.1.1 SQLAlchemy-1.3.8 WTForms-2.2.1 Werkzeug-0.15.5 alembic-1.1.0 bcrypt-3.1.7 boto3-1.9.156 botocore-1.12.249 certifi-2019.9.11 cffi-1.12.3 chardet-3.0.4 coverage-4.5.4 docutils-0.15.2 idna-2.8 itsdangerous-1.1.0 jmespath-0.9.4 lxml-4.3.2 numpy-1.16.2 pandas-0.24.2 patsy-0.5.1 psycopg2-2.8.3 pycparser-2.19 python-dateutil-2.8.0 python-editor-1.0.4 pytz-2019.2 requests-2.21.0 s3transfer-0.2.1 scikit-learn-0.20.3 scipy-1.2.1 six-1.12.0 statsmodels-0.9.0 trimesh-3.1.11 urllib3-1.24.3

However when I do a pip list I only see the following packages installed. Note that coverage and psycopg2, for example, were listed as successfully installed, but don't show. Consequently, a quick python -c "import coverage" will fail.

Package          Version  
---------------- ---------
alembic          1.1.0    
boto3            1.9.156  
botocore         1.12.249 
certifi          2019.9.11
chardet          3.0.4    
Click            7.0      
docutils         0.15.2   
Flask            1.0.2    
Flask-Bcrypt     0.7.1    
Flask-Login      0.4.1    
Flask-Migrate    2.5.2    
Flask-Script     2.0.6    
Flask-Session    0.3.1    
Flask-SQLAlchemy 2.4.0    
Flask-WTF        0.14.2   
idna             2.8      
itsdangerous     1.1.0    
Jinja2           2.10.1   
jmespath         0.9.4    
Mako             1.1.0    
patsy            0.5.1    
pip              19.3.1   
pycparser        2.19     
python-dateutil  2.8.0    
python-editor    1.0.4    
pytz             2019.2   
requests         2.21.0   
s3transfer       0.2.1    
setuptools       41.6.0   
six              1.12.0   
trimesh          3.1.11   
urllib3          1.24.3   
Werkzeug         0.15.5   
wheel            0.33.6   
WTForms          2.2.1

My questions are:

  1. Are there ways to resolve with pip options other than --force-reinstall and --no-cache-dir, which I have already tried?
  2. Could my issue relate to the fact that the latest supported package of Python I can grab on EC2 is 3.6.8, but my requirements.txt file was created on a 3.7.4 system? (Yes yes, I know, next time I'll be wiser and just use docker from the start.)
mayosten
  • 634
  • 5
  • 17
  • You don't have to use `docker`, you could have also used a virtualenv which is more lightweight. `coverage==4.5.4`, for example, certainly supports Python 3.6.x. What happens if you try to specifically install that (i.e. `pip install coverage`)? – Selcuk Nov 22 '19 at 00:03
  • Also see https://github.com/nedbat/coveragepy/issues/682 – Selcuk Nov 22 '19 at 00:08
  • Thanks @Selcuk. Yes I'm using this inside `virtualenv` and yes I've tried to install/remove/reinstall the packages individually. Examining the github post, just tried running as `python -m pip install coverage`, which unfortunately didn't work. It also appears that the issue isn't isolated to that package. – mayosten Nov 22 '19 at 00:17

1 Answers1

0

After much hand-wringing and nearly doing Docker bootcamp, I decided to go back for another round and found the underlying problem. Inside my virtual environment, I noticed that pip list was only listing packages as listed in the $MYENV/lib/Python3.6/data/dist-packages, whereas the offending packages were all located in $MYENV/lib64/Python3.6/data/dist-packages. This led to some additional searching and this response, which suggests symlinking the lib64 version of dist-packages to the lib version, which did the trick and resolved my issue. I'm using pip version 19.3.1, and the referenced article is from over 2 years ago, so I will definitely be talking to some folks.

However, for those who want the quick solution, here are the steps.

  1. Fresh virtual environment: Set up a fresh virtual environment as you normally would.

  2. Symlink BEFORE pip: Do your symlinking before you start installing any packages. Generally this will look something like:

ln -s $YOURENV/lib/Python3.x/data/dist-packages $YOURENV/lib64/Python3.x/data/dist-packages
  1. Go ahead and pip: No you can pip install -r requirements.txt as you normally would. Double-check with a pip list and you should see all your packages listed.
mayosten
  • 634
  • 5
  • 17