36

My ~/anaconda directory is taking up too much disk space (10GB), although I have only five environments and have run conda clean. I discovered that when I try to create a new conda environment, Anaconda displays a very long list of packages to be downloaded, which seems to include a full scientific Python stack (Python interpreter, numpy, scipy, etc.). It seems that Anaconda is installing everything independently for each environment. Is this true?

The following list contains some purely speculative ways which could potentially solve the space problem:

  • Can I create a "sub-environment" which "inherits" the packages of a "parent environment"?
  • Can Anaconda be made to share (e.g. via symbolic links on the file system) the same packages used in different environments?
  • Does the default environment have any special status in terms of package managing? I use Anaconda 2, but most of my environments use Python 3. Can I save space by switching to Anaconda 3? (This is regarding the default environment as the "parent environment" of all other environments.)
  • I normally use pip to install packages, as conda install often fails. Does conda install do some smart job to reuse packages already installed somewhere else?

(Debugging information) The sizes of my four environment directories under ~/anaconda/envs are between 1.2GB and 2.6GB. Is this normal?

user31039
  • 6,149
  • 4
  • 14
  • 9
  • What command do you normally use to create a new conda environment? If I try the minimal `conda create -n newenv python` I only get eight packages listed for installation. It sounds like you're specifying a bunch of others somehow. – nekomatic May 18 '18 at 08:07
  • 1
    Are you using Windows? – MSeifert Jul 31 '18 at 16:31
  • 2
    You have multiple questions in here, but the main one ("Can packages be shared across environments?") is answered in this duplicate: [Why are packages installed rather than just linked to a specific environment?](https://stackoverflow.com/questions/55566419/why-are-packages-installed-rather-than-just-linked-to-a-specific-environment) I'd also note that if you want a leaner environment, use Miniconda and install only what you need. – merv Sep 06 '19 at 19:40

2 Answers2

10

I believe the answer to your main question lies in the difference between Anaconda vs Miniconda. Anaconda includes a long list of packages that get installed automatically into each environment that you create. Miniconda creates barebone conda virtual environments (which don't contain many packages at all). Switching to Miniconda should substantially reduce the size/number of packages in your environments. Anaconda is about 2GB, while Miniconda is closer to 100MB.

Conda also uses hardlinks for packages installed vs conda install. A good description of hardlinks can be found here. They basically link dependencies across multiple environments like you've described above. Packages installed via pip are not hardlinked, so they cannot take advantage of the space savings that conda packages offer.

Drew
  • 101
  • 1
  • 3
  • Since different environments are likely to be created at different times, would it be accurate to say that package versions may differ between environments, thus limiting the actual sharing of files via hard links? – user2153235 May 19 '23 at 20:08
  • If two different environments required different versions of a package, then obviously both packages would need to be installed. I would be surprised if conda took into account that other versions of a package were already installed in a different environment when deciding what version to install in a new environment - you'd like dependency resolution to be reproducible, and depend only on the specifications in your current project. If your projects don't try to pin specific versions of packages unnecessarily, it ought to be possible to share many packages between your environments. – molecule Aug 05 '23 at 20:47
-1

To create environments that "inherit" packages:
You can export/import the names of the packages used with a yaml file.
Or use the --clone flag.
See this answer.

3pwd
  • 119
  • 2
  • 5