How to make conda use its own gcc version?

Question

I am trying to run the training of stylegan2-pytorch on a remote system. The remote system has gcc (9.3.0) installed on it. I'm using conda env that has the following installed (cudatoolkit=10.2, torch=1.5.0+, and ninja=1.8.2, gcc_linux-64=7.5.0). I encounter the following error:

RuntimeError: Error building extension 'fused': [1/2] 
/home/envs/segmentation_base/bin/nvcc -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/TH -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/THC -isystem /home/envs/segmentation_base/include -isystem /home/envs/segmentation_base/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -std=c++14 -c /home/code/semanticGAN_code/models/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o 
FAILED: fused_bias_act_kernel.cuda.o 
/home/envs/segmentation_base/bin/nvcc -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/TH -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/THC -isystem /home/envs/segmentation_base/include -isystem /home/envs/segmentation_base/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -std=c++14 -c /home/code/semanticGAN_code/models/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o 
In file included from /home/envs/segmentation_base/include/cuda_runtime.h:83,
                 from <command-line>:
/home/envs/segmentation_base/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
ninja: build stopped: subcommand failed.

I would like to use the gcc of my conda env (gcc_linux-64=7.5.0) to build cuda. When I run gcc --version in my conda env, I get the system's gcc:

gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

which gcc when my conda env is active returns:

usr/bin/gcc

I'd expect it to return gcc version 7.5.0 (the one installed in the environment). I understand that conda has different names for gcc, but the environment variables should point to the installed gcc.

Running echo $CC returns

/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-cc.

Following suggested solution here, I get the following upon activating my environment, but the same issue stand:

INFO: activate-binutils_linux-64.sh made the following environmental changes:
+ADDR2LINE=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-addr2line
+AR=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ar
+AS=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-as
+CXXFILT=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-c++filt
+ELFEDIT=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-elfedit
+GPROF=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gprof
+LD_GOLD=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ld.gold
+LD=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ld
+NM=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-nm
+OBJCOPY=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-objcopy
+OBJDUMP=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-objdump
+RANLIB=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ranlib
+READELF=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-readelf
+SIZE=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-size
+STRINGS=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-strings
+STRIP=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-strip
INFO: activate-gcc_linux-64.sh made the following environmental changes:
+build_alias=x86_64-conda-linux-gnu
+BUILD=x86_64-conda-linux-gnu
+CC_FOR_BUILD=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-cc
+CC=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-cc
+CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
+CMAKE_ARGS=-DCMAKE_LINKER=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ld -DCMAKE_STRIP=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-strip -DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER -DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY -DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY -DCMAKE_FIND_ROOT_PATH=;/x86_64-conda-linux-gnu/sysroot -DCMAKE_INSTALL_PREFIX= -DCMAKE_INSTALL_LIBDIR=lib
+CMAKE_PREFIX_PATH=:/home/envs/segmentation_base/x86_64-conda-linux-gnu/sysroot/usr
+CONDA_BUILD_SYSROOT=/home/envs/segmentation_base/x86_64-conda-linux-gnu/sysroot
+_CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_linux_gnu
+CPPFLAGS=-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /include
+CPP=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-cpp
+DEBUG_CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
+DEBUG_CPPFLAGS=-D_DEBUG -D_FORTIFY_SOURCE=2 -Og -isystem /include
+GCC_AR=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gcc-ar
+GCC_NM=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gcc-nm
+GCC=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gcc
+GCC_RANLIB=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gcc-ranlib
+host_alias=x86_64-conda-linux-gnu
+HOST=x86_64-conda-linux-gnu
+LDFLAGS=-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/lib -Wl,-rpath-link,/lib -L/lib
INFO: activate-gxx_linux-64.sh made the following environmental changes:
+CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
+CXX_FOR_BUILD=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-c++
+CXX=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-c++
+DEBUG_CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
+GXX=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-g++

How could one set gcc to conda gcc instead of system gcc? I understand that should be done automatically, when activating the environment through bash scripts in activate.d.

Most of the open issues (regarding unsupported GNU version!) either require sudo permission to adjust gcc version (which I don't have) or aren't accepted in the case of conda environments. I have yet to find a clear solution to this :/

TLDR: How to force conda to use own installed gcc version instead of host system gcc?

Edit 1: Added conda list output

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             4.5                       1_gnu  
_sysroot_linux-64_curr_repodata_hack 3                   haa98f57_10  
absl-py                   1.0.0                    pypi_0    pypi
albumentations            0.5.2                    pypi_0    pypi
binutils_impl_linux-64    2.35.1               h27ae35d_9  
binutils_linux-64         2.35.1              h454624a_30  
blas                      1.0                         mkl  
ca-certificates           2021.10.26           h06a4308_2  
cachetools                4.2.4                    pypi_0    pypi
certifi                   2021.5.30        py36h06a4308_0  
charset-normalizer        2.0.9                    pypi_0    pypi
cudatoolkit               10.2.89                       3    hcc
cycler                    0.11.0                   pypi_0    pypi
decorator                 4.4.2                    pypi_0    pypi
freetype                  2.11.0               h70c0345_0  
gcc_impl_linux-64         7.5.0               h7105cf2_17  
gcc_linux-64              7.5.0               h8f34230_30  
google-auth               2.3.3                    pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
grpcio                    1.42.0                   pypi_0    pypi
gxx_impl_linux-64         7.5.0               h0a5bf11_17  
gxx_linux-64              7.5.0               hffc177d_30  
idna                      3.3                      pypi_0    pypi
imageio                   2.8.0                    pypi_0    pypi
imageio-ffmpeg            0.4.2                    pypi_0    pypi
imgaug                    0.4.0                    pypi_0    pypi
importlib-metadata        4.8.2                    pypi_0    pypi
intel-openmp              2021.4.0          h06a4308_3561  
jpeg                      9d                   h7f8727e_0  
kernel-headers_linux-64   3.10.0              h57e8cba_10  
kiwisolver                1.3.1                    pypi_0    pypi
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.35.1               h7274673_9  
libffi                    3.3                  he6710b0_2  
libgcc-devel_linux-64     7.5.0               hbbeae57_17  
libgcc-ng                 9.3.0               h5101ec6_17  
libgomp                   9.3.0               h5101ec6_17  
libpng                    1.6.37               hbc83047_0  
libstdcxx-devel_linux-64  7.5.0               hf0c5c8d_17  
libstdcxx-ng              9.3.0               hd4cf53a_17  
libtiff                   4.2.0                h85742a9_0  
libwebp-base              1.2.0                h27cfd23_0  
lmdb                      0.98                     pypi_0    pypi
lz4-c                     1.9.3                h295c915_1  
markdown                  3.3.6                    pypi_0    pypi
matplotlib                3.3.4                    pypi_0    pypi
mkl                       2020.2                      256  
mkl-service               2.3.0            py36he8ac12f_0  
mkl_fft                   1.3.0            py36h54f3939_0  
mkl_random                1.1.1            py36h0573a6f_0  
ncurses                   6.3                  h7f8727e_2  
networkx                  2.5.1                    pypi_0    pypi
ninja                     1.8.2                    pypi_0    pypi
numpy                     1.19.5                   pypi_0    pypi
numpy-base                1.19.2           py36hfa32c7d_0  
oauthlib                  3.1.1                    pypi_0    pypi
olefile                   0.46                     py36_0  
opencv-python             4.5.4.60                 pypi_0    pypi
opencv-python-headless    4.5.4.60                 pypi_0    pypi
openjpeg                  2.4.0                h3ad879b_0  
openssl                   1.1.1l               h7f8727e_0  
pillow                    8.4.0                    pypi_0    pypi
pip                       21.2.2           py36h06a4308_0  
protobuf                  3.19.1                   pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pyparsing                 3.0.6                    pypi_0    pypi
python                    3.6.13               h12debd9_1  
python-dateutil           2.8.2                    pypi_0    pypi
pytorch                   1.5.0           py3.6_cuda10.2.89_cudnn7.6.5_0    pytorch
pywavelets                1.1.1                    pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
readline                  8.1                  h27cfd23_0  
requests                  2.26.0                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rsa                       4.8                      pypi_0    pypi
scikit-image              0.17.2                   pypi_0    pypi
scipy                     1.5.0                    pypi_0    pypi
setuptools                58.0.4           py36h06a4308_0  
shapely                   1.8.0                    pypi_0    pypi
six                       1.16.0             pyhd3eb1b0_0  
sqlite                    3.36.0               hc218d9a_0  
sysroot_linux-64          2.17                h57e8cba_10  
tensorboard               2.7.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
tifffile                  2020.9.3                 pypi_0    pypi
tk                        8.6.11               h1ccaba5_0  
torchvision               0.6.0                py36_cu102    pytorch
typing-extensions         4.0.1                    pypi_0    pypi
urllib3                   1.26.7                   pypi_0    pypi
werkzeug                  2.0.2                    pypi_0    pypi
wheel                     0.37.0             pyhd3eb1b0_1  
xz                        5.2.5                h7b6447c_0  
zipp                      3.6.0                    pypi_0    pypi
zlib                      1.2.11               h7b6447c_3  
zstd                      1.4.9                haebb681_0

I see that you have only the `gcc_impl_linux-64` and not the `gcc` package unlike in the [package list from a fresh environment](https://stackoverflow.com/a/70322120/4413446). Have you tried to install the `gcc` package in your conda env? — Romain, Dec 12 '21 at 16:17
`gcc` under conda is called `gcc_linux-64,` it's available in my `conda list`. I managed to solve it by adding symbolic links to conda installed `gcc`. Thanks for your effort :) — A95, Dec 13 '21 at 13:46

Romain · Answer 1 · 2021-12-12T16:12:53.450

Just to share, not sure it will help you. However it shows that in standard conditions it is possible to use the conda gcc as described in the documentation instead of the system gcc.

# system gcc
which gcc && gcc --version
# /usr/bin/gcc
# gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

# creating a conda env with gcc
conda create -n gcc gcc
# activate the environment
conda activating gcc
which gcc && gcc --version
# /opt/conda/envs/gcc/bin/gcc
# gcc (GCC) 11.2.0

Here is the list of packages installed on a fresh environment created with only gcc.

# packages in environment at /opt/conda/envs/gcc:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
binutils_impl_linux-64    2.36.1               h193b22a_2    conda-forge
gcc                       11.2.0               h702ea55_2    conda-forge
gcc_impl_linux-64         11.2.0              h82a94d6_11    conda-forge
kernel-headers_linux-64   2.6.32              he073ed8_15    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libgcc-devel_linux-64     11.2.0              h0952999_11    conda-forge
libgcc-ng                 11.2.0              h1d223b6_11    conda-forge
libgomp                   11.2.0              h1d223b6_11    conda-forge
libsanitizer              11.2.0              he4da1e4_11    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_11    conda-forge
sysroot_linux-64          2.12                he073ed8_15    conda-forge

That's exactly why I have the issue opened, in standard conditions I should be able to access conda gcc, but I'm not able to. Running the same lines you suggested, and my gcc still points to system gcc. — A95, Dec 12 '21 at 10:25
@A95 could you please share the output of `conda list` when your environment is activated? — Romain, Dec 12 '21 at 11:44

score 0 · Accepted Answer · answered Dec 12 '21 at 12:27

In addition to the solution posted in this issue. I added symbolic-links that point to the conda installed gcc, which I was missing.

ln -s /home/envs/segmentation_base/bin/x86_64-conda_cos6-linux-gnu-cc gcc
ln -s /home/envs/segmentation_base/bin/x86_64-conda_cos6-linux-gnu-cpp g++

How to make conda use its own gcc version?

2 Answers2