import a python package ( which is a docker container) in python script

Question

I am running a python script in a conda enviroment (I have to do this on the HPC). one of the packages (called scrublet) that I need to import in the python script could not be installed using pip or conda therefore I decided to use its docker container. do you know how I can do it and import it in my python script? when I tried to install the package I got this error:

  ERROR: Command errored out with exit status 1:
   command: /mnt/miniconda/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-qvxac2gl/annoy_7749eb3100d143f695dfd00778678327/setup.py'"'"'; __file__='"'"'/tmp/pip-install-qvxac2gl/annoy_7749eb3100d143f695dfd00778678327/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-yeevfzda
       cwd: /tmp/pip-install-qvxac2gl/annoy_7749eb3100d143f695dfd00778678327/
  Complete output (16 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/annoy
  copying annoy/__init__.py -> build/lib.linux-x86_64-3.7/annoy
  copying annoy/__init__.pyi -> build/lib.linux-x86_64-3.7/annoy
  copying annoy/py.typed -> build/lib.linux-x86_64-3.7/annoy
  running build_ext
  building 'annoy.annoylib' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/src
  gcc -pthread -B /mnt/miniconda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/mnt/miniconda/include/python3.7m -c src/annoymodule.cc -o build/temp.linux-x86_64-3.7/src/annoymodule.o -D_CRT_SECURE_NO_WARNINGS -fpermissive -march=native -O3 -ffast-math -fno-associative-math -DANNOYLIB_MULTITHREADED_BUILD -std=c++14
  gcc: error: unrecognized command line option ‘-std=c++14’
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for annoy
  Running setup.py clean for annoy
Successfully built scrublet
Failed to build annoy
Installing collected packages: annoy, scrublet
    Running setup.py install for annoy ... error
    ERROR: Command errored out with exit status 1:
     command: /mnt/miniconda/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-qvxac2gl/annoy_7749eb3100d143f695dfd00778678327/setup.py'"'"'; __file__='"'"'/tmp/pip-install-qvxac2gl/annoy_7749eb3100d143f695dfd00778678327/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-3asrawrx/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/moumbeib/.local/include/python3.7m/annoy
         cwd: /tmp/pip-install-qvxac2gl/annoy_7749eb3100d143f695dfd00778678327/
    Complete output (16 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/annoy
    copying annoy/__init__.py -> build/lib.linux-x86_64-3.7/annoy
    copying annoy/__init__.pyi -> build/lib.linux-x86_64-3.7/annoy
    copying annoy/py.typed -> build/lib.linux-x86_64-3.7/annoy
    running build_ext
    building 'annoy.annoylib' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/src
    gcc -pthread -B /mnt/miniconda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/mnt/miniconda/include/python3.7m -c src/annoymodule.cc -o build/temp.linux-x86_64-3.7/src/annoymodule.o -D_CRT_SECURE_NO_WARNINGS -fpermissive -march=native -O3 -ffast-math -fno-associative-math -DANNOYLIB_MULTITHREADED_BUILD -std=c++14
    gcc: error: unrecognized command line option ‘-std=c++14’
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /mnt/miniconda/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-qvxac2gl/annoy_7749eb3100d143f695dfd00778678327/setup.py'"'"'; __file__='"'"'/tmp/pip-install-qvxac2gl/annoy_7749eb3100d143f695dfd00778678327/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-3asrawrx/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/moumbeib/.local/include/python3.7m/annoy Check the logs for full command output.

Answer 1

If you have Python code that is only packaged in a Docker image, you cannot import it into other Python environments.

One of the important features of Docker is filesystem isolation. Containers cannot access files in other containers or the host, and the host cannot access containers' filesystems. Something like a Python interpreter works by reading the source code of the modules it's importing off of disk, and a host Python interpreter can't see the files if they're only inside an image or container.

Note that this doesn't mean the files can't be extracted (see Docker: Copying files from Docker container to host), just that you have to do a lot of work to make a Python import statement actually function.

This makes Docker not an ideal packaging setup for this use case. Python's native packaging tools are fairly robust, and if you could embed the relevant files in a Docker image then you should consider creating a native Python package instead. The Python Packaging User Guide discusses the standard tooling and walks through the process of creating a package that can be installed with pip.