Does Conda replace the need for virtualenv?

Question

I recently discovered Conda after I was having trouble installing SciPy, specifically on a Heroku app that I am developing.

With Conda you create environments, very similar to what virtualenv does. My questions are:

If I use Conda will it replace the need for virtualenv? If not, how do I use the two together? Do I install virtualenv in Conda, or Conda in virtualenv?
Do I still need to use pip? If so, will I still be able to install packages with pip in an isolated environment?

If you're interested in using conda and pip on Heroku, see for example https://github.com/faph/conda-pip-buildpack — faph, Dec 22 '15 at 12:25
Thanks. I've noticed that there is quite a number of conda buildpacks for Heroku on github. What factors should I take into account when deciding which buildpack to use? — Kritz, Dec 22 '15 at 19:07
Note that you will still need to use pip if you want to install packages that aren't available directly from Continuum's servers. — ali_m, Dec 22 '15 at 22:38
Yes, I saw that they are still on Django 1.8 (not 1.9). At the moment I will use conda where needed (scipy and numpy) and pip for everything else - but still within conda. — Kritz, Dec 23 '15 at 05:22
Most conda Heroku buildpacks originate from the one by Kenneth Reitz I think. With people tweaking them to suit their preferences. Just check if they include both conda and pip support if that's what you need. And if they support the environment.yml file. You can always quickly look through the buildpack code to see if you like the build script, for example to see how exactly environments are created. — faph, Dec 23 '15 at 09:13

score 221 · Accepted Answer · edited Nov 26 '19 at 11:06

221

Conda replaces virtualenv. In my opinion it is better. It is not limited to Python but can be used for other languages too. In my experience it provides a much smoother experience, especially for scientific packages. The first time I got MayaVi properly installed on Mac was with conda.
You can still use pip. In fact, conda installs pip in each new environment. It knows about pip-installed packages.

For example:

conda list

lists all installed packages in your current environment. Conda-installed packages show up like this:

sphinx_rtd_theme          0.1.7                    py35_0    defaults

and the ones installed via pip have the <pip> marker:

wxpython-common           3.0.0.0                   <pip>

edited Nov 26 '19 at 11:06

Community

1
1

answered Dec 21 '15 at 15:10

Mike Müller

82,630
20
166
161

22

Is there any negative to using pip in an Anaconda environment? Is there ever a case where you would want to use pip even though a package was available through Conda? – clifgray May 12 '18 at 16:30
1

The difference is hyphen vs underscore? What if the package name has neither? How to differentiate then? – Tom Hale Oct 03 '18 at 08:01
4

The underscore or hyphen is part of the package name. This has nothing to do with pip or conda. The `` shows that it was installed with pip otherwise it is installed with conda. – Mike Müller Oct 03 '18 at 11:15
7

There's a big caveat with "conda knows about pip-installed packages". From my understanding, inside a conda env, pip acts independently, so conda can't uninstall pip installed packages for example – information_interchange Oct 20 '18 at 21:15
Also, you don't see pip-installed packages from the navigator https://github.com/ContinuumIO/anaconda-issues/issues/10634 – elachell Nov 07 '19 at 20:47
3

@clifgray - pip and conda packages having native shared libraries may install binary incompatible versions of those which will start causing all sorts of native world failures (sigsegv-s, etc.) hard to debug for someone not up to speed with a C debugger. Same for python-only packages, just that those are easy to make sense of. – bobah Dec 04 '19 at 12:21
I use conda, but after I use pip to reinstall scipy, the world began to collapse. `--> 411 from ._nnls import nnls` `ImportError: DLL load failed` – huang Jul 07 '21 at 12:15
4

What conda channel did you use ti install Python itself and other packages? I recommend to use `conda-forge` for everything that is supports and `pip` only for what is not available via conda. – Mike Müller Jul 09 '21 at 15:14
Does conda separately install gcc and stuff? – amzon-ex Mar 26 '22 at 06:49
There is a gcc available via conda-forge: https://anaconda.org/conda-forge/gcc_impl_linux-64 – Mike Müller Mar 28 '22 at 06:31

Pat Niemeyer · Answer 2 · 2021-04-10T19:55:25.843

I use both and (as of Jan, 2020) they have some superficial differences that lend themselves to different usages for me. By default Conda prefers to manage a list of environments for you in a central location, whereas virtualenv makes a folder in the current directory. The former (centralized) makes sense if you are e.g. doing machine learning and just have a couple of broad environments that you use across many projects and want to jump into them from anywhere. The latter (per project folder) makes sense if you are doing little one-off projects that have completely different sets of lib requirements that really belong more to the project itself.

The empty environment that Conda creates is about 122MB whereas the virtualenv's is about 12MB, so that's another reason you may prefer not to scatter Conda environments around everywhere.

Finally, another superficial indication that Conda prefers its centralized envs is that (again, by default) if you do create a Conda env in your own project folder and activate it the name prefix that appears in your shell is the (way too long) absolute path to the folder. You can fix that by giving it a name, but virtualenv does the right thing by default.

I expect this info to become stale rapidly as the two package managers vie for dominance, but these are the trade-offs as of today :)

EDIT: I reviewed the situation again in 04/2021 and it is unchanged. It's still awkward to make a local directory install with conda.

Good explanation! Do you have some difficulties using both of them? Have you ever used `pipenv`? — Mikhail_Sam, Jun 03 '20 at 13:53
I've been googling this for ages and this was the answer I was looking for! I switched to conda from pipenv after running into issues with conflicting binaries on a project. I like it for creating large "pre-set" environments that might be reused, but it seems heavy for one-off scripts or small projects, and I miss how pipenv could even automatically detect the env when you cd into the directory. — yuletide, Apr 04 '22 at 05:59
One HUGE advantage of conda is the opposite of what you say in your answer. "you may prefer not to scatter Conda environments around everywhere." Conda CENTRALIZES all virtual environments, pip scatters them everywhere. It is possible to override conda's centralized manager and put your environment somewhere OTHER than "Anaconda3/envs", but you have to specific this explicitly. — Rich Lysakowski PhD, Jul 16 '22 at 05:29

score 91 · Answer 3 · edited Mar 12 '19 at 16:55

Short answer is, you only need conda.

Conda effectively combines the functionality of pip and virtualenv in a single package, so you do not need virtualenv if you are using conda.
You would be surprised how many packages conda supports. If it is not enough, you can use pip under conda.

Here is a link to the conda page comparing conda, pip and virtualenv:

https://docs.conda.io/projects/conda/en/latest/commands.html#conda-vs-pip-vs-virtualenv-commands.

pylang · Answer 4 · 2023-03-14T17:04:36.243

Virtual Environments and pip

I will add that creating and removing conda environments is simple with Anaconda.

> conda create --name <envname> python=<version> <optional dependencies>

> conda remove --name <envname> --all

In an activated environment, install packages via conda or pip:

(envname)> conda install <package>

(envname)> pip install <package>

These environments are strongly tied to conda's pip-like package management, so it is simple to create environments and install both Python and non-Python packages.

Jupyter

In addition, installing ipykernel in an environment adds a new listing in the Kernels dropdown menu of Jupyter notebooks, extending reproducible environments to notebooks. As of Anaconda 4.1, nbextensions were added, adding extensions to notebooks more easily.

Reliability

In my experience, conda is faster and more reliable at installing large libraries such as numpy and pandas. Moreover, if you wish to transfer your preserved state of an environment, you can do so by sharing or cloning an env.

Comparisons

A non-exhaustive, quick look at features from each tool:

Feature	`virtualenv`	`conda`
Global	n	y
Local	y	n
PyPI	y	y
Channels	n	y
Lock File	n	n
Multi-Python	n	y

Description

virtualenv creates project-specific, local environments usually in a .venv/ folder per project. In contrast, conda's environments are global and saved in one place.
PyPI works with both tools through pip, but conda can add additional channels, which can sometimes install faster.
Sadly neither has an official lock file, so reproducing environments has not been solid with either tool. However, both have a mechanism to create a file of pinned packages.
Python is needed to install and run virtualenv, but conda already ships with Python. virtualenv creates environments using the same Python version it was installed with. conda allows you to create environments with nearly any Python version.

Differences:

License agreement. While virtualenv comes under most liberal MIT license, Conda uses 3 clause BSD license.

Conda provides you with their own package control system. This package control system often provides precompiled versions (for most popular systems) of popular non-python software, which can easy ones way getting some machine learning packages working. Namely you don't have to compile optimized C/C++ code for you system. While it is a great relief for most of us, it might affect performance of such libraries.

Unlike virtualenv, Conda duplicating some system libraries at least on Linux system. This libraries can get out of sync leading to inconsistent behavior of your programs.

Verdict:

Conda is great and should be your default choice while starting your way with machine learning. It will save you some time messing with gcc and numerous packages. Yet, Conda does not replace virtualenv. It introduces some additional complexity which might not always be desired. It comes under different license. You might want to avoid using conda on a distributed environments or on HPC hardware.

mind elaborating a bit more why "you might want to avoid using conda on a distributed environments or on HPC hardware"? @y.selivonchyk — Oliver Hu, Jun 06 '19 at 01:49
I disagree with some of these conclusions. "Inconsistent program behavior" is a result of not properly configuring your programs to use the `conda` installed software and libraries. And in HPC, `conda` is preferable in many cases, in fact it is being used by HPC Admins to replace things like `module` systems. It allows user-installed software and greater software isolation, two large issues on HPC. The only caveat I experience is that many HPC filesystems have hard-limits on the number of files in a dir, and conda creates many 1,000's of files. — user5359531, Jan 13 '20 at 17:57
You make a good point about `conda` handling the C/C++ compilation for popular non-python software Python packages that don't ship with Python wrappers or binaries of their own. AFAIK `virtualenv` does not do that. However, your conclusion about "avoid using conda on a distributed environment or HPC hardware" is flawed and unsupported by facts. I agree with the conclusion of @user5359531 that you have to learn more about configuring and using `conda` for complex HPC systems. It can solve more HPC problems than virtualenv alone. — Rich Lysakowski PhD, Jun 06 '22 at 07:09

score 18 · Answer 6 · edited Dec 02 '22 at 14:03

18

Another new option and my current preferred method of getting an environment up and running is Pipenv

It is currently the officially recommended Python packaging tool from Python.org

edited Dec 02 '22 at 14:03

huyz

2,297
3
25
34

answered Mar 03 '18 at 12:32

Kritz

7,099
12
43
73

3

This prompted "eh? what's pipenv?", which led me to https://www.reddit.com/r/Python/comments/8jd6aq/why_is_pipenv_the_recommended_packaging_tool_by/ and https://sedimental.org/the_packaging_gradient.html. I still don't know what to use but at least I'm better informed. I think. – matt wilkie Aug 26 '18 at 21:13
After watching the intro and quickly reading the introduction, *pipenv* seems not able to manage Python versions... – Neithan Max Nov 24 '18 at 11:59
@CarlesAlcolea pipenv can specify the various versions as well by: `pipenv --two` for Python2 and pipenv --three for python3 – Kurian Benoy Aug 12 '19 at 05:46
I was gung-ho about pipenv until I found that it has a menagerie of incompatibilities with things like Tensorflow, for example. That is a deal breaker for me. Otherwise, it was solid while I used it. – e.thompsy Aug 26 '20 at 03:18
**Warning:** https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/ – user76284 Dec 03 '20 at 22:55
2

conda is ALMOST ALWAYS a better way to go for machine learning, and should be used whenever possible. There can be problems mixing package installations in conda environments using pip and then conda repeatedly. pip does not do full-environment integrity checking after it installs things into a conda environment. It only checks its own dependencies, not those of every other package in the environment. So it is best to install all conda-formatted packages into a conda environment first, and then install pip packages to "dead-end" that new environment. pip won't update conda's package index. – Rich Lysakowski PhD Apr 06 '21 at 07:37
Pipenv is incredibly slow for me – Al-Baraa El-Hag Jun 27 '21 at 21:25
2

"It is currently the officially recommended Python packaging tool from Python.org" Trust, but verify. Do you have a link to the source of this claim? – Alexander Apr 27 '22 at 17:38
Actually "The Python Packaging `Authority`" is not the same as Python.org. PyPa is a separate third-party working group on Python-only packaging. For mixed-language environments, PyPa refers people to conda-forge and conda.org. – Rich Lysakowski PhD Jun 06 '22 at 07:14
Pipenv is no longer *the* recommended tool. "Use pip-tools, Pipenv, or poetry to generate the fully-specified application-specific dependencies, when developing Python applications." – huyz Dec 02 '22 at 14:03

score 7 · Answer 7 · answered Jan 12 '22 at 10:53

Conda has a better API no doubt. But, I would like to touch upon the negatives of using conda since conda has had its share of glory in the rest of the answers:

Solving environment Issue - One big thorn in the rear end of conda environments. As a remedy, you get advised to not use conda-forge channel. But, since it is the most prevalent channel and some packages (not just trivial ones, even really important ones like pyspark) are exclusively available on conda-forge you get cornered pretty fast.
Packing the environment is an issue

There are other known issues as well. virtualenv is an uphill journey but, rarely a wall on the road. conda on the other hand, IMO, has these occasional hard walls where you just have to take a deep breath and use virtualenv

score 5 · Answer 8 · answered Feb 19 '21 at 11:58

1.No, if you're using conda, you don't need to use any other tool for managing virtual environments (such as venv, virtualenv, pipenv etc). Maybe there's some edge case which conda doesn't cover but virtualenv (being more heavyweight) does, but I haven't encountered any so far.

2.Yes, not only can you still use pip, but you will probably have to. The conda package repository contains less than pip's does, so conda install will sometimes not be able to find the package you're looking for, more so if it's not a data-science package. And, if I remember correctly, conda's repository isn't updated as fast/often as pip's, so if you want to use the latest version of a package, pip might once again be your only option.

Note: if the pip command isn't available within a conda virtual environment, you will have to install it first, by hitting:

conda install pip

score 2 · Answer 9 · edited Dec 31 '18 at 13:57

2

Yes, conda is a lot easier to install than virtualenv, and pretty much replaces the latter.

edited Dec 31 '18 at 13:57

nbro

15,395
32
113
196

answered May 22 '17 at 18:48

Liang Huang

63
1

6

Why does Anaconda provide instructions for installing a virtual environment if it replaces them? – Natsfan Aug 30 '17 at 20:20
2

@jmh Anaconda doesn't replace virtual environments, it replaces the Python-specific virtual environment management tool `virtualenv` with a more general virtual environment management tool `conda`. Also, Anaconda is just a Python+ distribution that includes the Conda tool; the question (and answer) are only about Conda. – merv Apr 24 '19 at 19:54
6

This answer doesn't add anything beyond the answers that came years before it. – merv Apr 24 '19 at 19:54

score -1 · Answer 10 · answered Jan 29 '20 at 16:12

I work in corporate, behind several firewall with machine on which I have no admin acces

In my limited experience with python (2 years) i have come across few libraries (JayDeBeApi,sasl) which when installing via pip threw C++ dependency errors error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools

these installed fine with conda, hence since those days i started working with conda env. however it isnt easy to stop conda from installing dependency inside c.programfiles where i dont have write access.

Does Conda replace the need for virtualenv?

10 Answers10

Differences:

Verdict:

Linked

Related