Cannot install vowpalwabbit on CentOS via pip or via source code

Question

OS: CentOS Linux 7 (Core)

Python: Python 3.6.4

cmake: 3.14.4

boost: 1.70.0

I am having problems installing vowpalwabbit using 'pip install vowpalwabbit==8.7.0.post1' as well as using 'pip install vowpalwabbit==8.7.0'. The installer fails when attempting to build the wheel. I get the same error when trying to build the project from the source code following the process described in the project. The output log is included below.

pip install vowpalwabbit==8.7.0.post1
WARNING: The directory '/home/mydir/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/home/mydir/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting vowpalwabbit==8.7.0.post1
  Downloading https://files.pythonhosted.org/packages/f4/09/50efc9630809fdd19056d2fff8e7b8e374063002b5a70b016854fe5e8492/vowpalwabbit-8.7.0.post1.tar.gz (3.1MB)
     |################################| 3.1MB 6.8MB/s
Building wheels for collected packages: vowpalwabbit
  Building wheel for vowpalwabbit (setup.py) ... error
  ERROR: Complete output from command /opt/anaconda/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-2tn4k8le/vowpalwabbit/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-0tkzjwhf --python-tag cp36:
  ERROR: running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/vowpalwabbit
  copying python/vowpalwabbit/__init__.py -> build/lib.linux-x86_64-3.6/vowpalwabbit
  copying python/vowpalwabbit/pyvw.py -> build/lib.linux-x86_64-3.6/vowpalwabbit
  copying python/vowpalwabbit/sklearn_vw.py -> build/lib.linux-x86_64-3.6/vowpalwabbit
  running egg_info
  writing python/vowpalwabbit.egg-info/PKG-INFO
  writing dependency_links to python/vowpalwabbit.egg-info/dependency_links.txt
  writing top-level names to python/vowpalwabbit.egg-info/top_level.txt
  reading manifest file 'python/vowpalwabbit.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'python/*.dll' under directory 'python/bin'
  warning: no files found matching 'python/packages.config' under directory 'python/windows*'
  warning: no files found matching 'python/*.vcxproj*' under directory 'python/windows*'
  warning: no previously-included files matching '*.o' found anywhere in distribution
  warning: no previously-included files matching '*.exe' found anywhere in distribution
  warning: no previously-included files matching '*.pyc' found anywhere in distribution
  warning: no previously-included files matching '*' found under directory 'vowpalwabbit/.nuget'
  writing manifest file 'python/vowpalwabbit.egg-info/SOURCES.txt'
  copying python/vowpalwabbit/conda_install.sh -> build/lib.linux-x86_64-3.6/vowpalwabbit
  running build_ext
  creating build/temp.linux-x86_64-3.6
  cmake -DCMAKE_BUILD_TYPE=Release -DPY_VERSION=3.6 -DBUILD_PYTHON=On -DBUILD_TESTS=Off -DWARNINGS=Off -DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-install-2tn4k8le/vowpalwabbit/build/lib.linux-x86_64-3.6 /tmp/pip-install-2tn4k8le/vowpalwabbit
  CMake Warning (dev) at CMakeLists.txt:4 (set):
    implicitly converting 'TYPE' to 'STRING' type.
  This warning is for project developers.  Use -Wno-dev to suppress it.

  -- The C compiler identification is GNU 4.8.5
  -- The CXX compiler identification is GNU 4.8.5
  -- Check for working C compiler: /usr/bin/cc
  -- Check for working C compiler: /usr/bin/cc -- works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Check for working CXX compiler: /usr/bin/c++
  -- Check for working CXX compiler: /usr/bin/c++ -- works
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- VowpalWabbit Version: 8.7.0.post1
  -- Number of processors: 2
  -- Looking for pthread.h
  -- Looking for pthread.h - found
  -- Looking for pthread_create
  -- Looking for pthread_create - not found
  -- Check if compiler accepts -pthread
  -- Check if compiler accepts -pthread - yes
  -- Found Threads: TRUE
  -- Boost version: 1.70.0
  -- Found the following Boost libraries:
  --   program_options
  --   system
  --   thread
  --   unit_test_framework
  --   chrono
  --   date_time
  --   atomic
  -- Found ZLIB: /usr/lib64/libz.so (found version "1.2.7")
  -- help2man not found, please install it to generate manpages
  -- Python 3.6 will be used for VowpalWabbit Python bindings
  -- Found PythonInterp: /opt/anaconda/bin/python3.6 (found suitable version "3.6.4", minimum required is "3.6")
  -- Found PythonLibs: /opt/anaconda/lib/libpython3.6m.so (found suitable version "3.6.4", minimum required is "3.6")
  -- Boost version: 1.70.0
  -- Found the following Boost libraries:
  --   system
  --   python36
  -- Configuring done
  CMake Warning at python/CMakeLists.txt:37 (add_library):
    Cannot generate a safe runtime search path for target pylibvw because files
    in some directories may conflict with libraries in implicit directories:

      runtime library [libz.so.1] in /usr/lib64 may be hidden by files in:
        /opt/anaconda/lib

    Some of these libraries may not be found correctly.


  -- Generating done
  -- Build files have been written to: /tmp/pip-install-2tn4k8le/vowpalwabbit/build/temp.linux-x86_64-3.6
  cmake --build . --config Release -- -j8 pylibvw
  Scanning dependencies of target allreduce
  [  1%] Building CXX object vowpalwabbit/CMakeFiles/allreduce.dir/vw_exception.cc.o
  [  2%] Building CXX object vowpalwabbit/CMakeFiles/allreduce.dir/allreduce_sockets.cc.o
  [  3%] Building CXX object vowpalwabbit/CMakeFiles/allreduce.dir/allreduce_threads.cc.o
  [  5%] Linking CXX static library liballreduce.a
  [  5%] Built target allreduce
  Scanning dependencies of target vw
  [  6%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/options_boost_po.cc.o
  [  7%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/options_serializer_boost_po.cc.o
  [  8%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/global_data.cc.o
  [ 10%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/parse_regressor.cc.o
  [ 10%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/io_buf.cc.o
  [ 11%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/parse_primitives.cc.o
  [ 12%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/unique_sort.cc.o
  [ 13%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/cache.cc.o
  [ 15%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/rand48.cc.o
  [ 15%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/simple_label.cc.o
  [ 16%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/multiclass.cc.o
  [ 17%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/oaa.cc.o
  [ 18%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/multilabel_oaa.cc.o
  [ 20%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/boosting.cc.o
  [ 21%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/ect.cc.o
  [ 21%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/marginal.cc.o
  [ 22%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/autolink.cc.o
  [ 24%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/binary.cc.o
  In file included from /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/global_data.h:44:0,
                   from /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/reductions.h:14,
                   from /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/boosting.cc:22:
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/options.h: In instantiation of 'VW::config::typed_option<T>::typed_option(const string&, T&) [with T = std::basic_string<char>; std::string = std::basic_string<char>]':
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/options.h:81:40:   required from 'VW::config::typed_option<T> VW::config::make_option(std::string, T&) [with T = std::basic_string<char>; std::string = std::basic_string<char>]'
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/boosting.cc:407:39:   required from here
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/options.h:28:117: error: invalid initialization of non-const reference of type 'std::basic_string<char>&' from an rvalue of type '<brace-enclosed initializer list>'
     typed_option(const std::string& name, T& location) : base_option(name, typeid(T).hash_code()), m_location{location} {}
                                                                                                                       ^
  In file included from /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/global_data.h:44:0,
                   from /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/reductions.h:14,
                   from /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/ect.cc:17:
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/options.h: In instantiation of 'VW::config::typed_option<T>::typed_option(const string&, T&) [with T = std::basic_string<char>; std::string = std::basic_string<char>]':
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/options.h:81:40:   required from 'VW::config::typed_option<T> VW::config::make_option(std::string, T&) [with T = std::basic_string<char>; std::string = std::basic_string<char>]'
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/ect.cc:353:36:   required from here
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/options.h:28:117: error: invalid initialization of non-const reference of type 'std::basic_string<char>&' from an rvalue of type '<brace-enclosed initializer list>'
     typed_option(const std::string& name, T& location) : base_option(name, typeid(T).hash_code()), m_location{location} {}
                                                                                                                       ^
  In file included from /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/global_data.h:44:0,
                   from /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/reductions.h:14,
                   from /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/marginal.cc:2:
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/options.h: In instantiation of 'VW::config::typed_option<T>::typed_option(const string&, T&) [with T = std::basic_string<char>; std::string = std::basic_string<char>]':
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/options.h:81:40:   required from 'VW::config::typed_option<T> VW::config::make_option(std::string, T&) [with T = std::basic_string<char>; std::string = std::basic_string<char>]'
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/marginal.cc:361:56:   required from here
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/options.h:28:117: error: invalid initialization of non-const reference of type 'std::basic_string<char>&' from an rvalue of type '<brace-enclosed initializer list>'
     typed_option(const std::string& name, T& location) : base_option(name, typeid(T).hash_code()), m_location{location} {}
                                                                                                                       ^
  [ 25%] Building CXX object vowpalwabbit/CMakeFiles/vw.dir/lrq.cc.o
  gmake[3]: *** [vowpalwabbit/CMakeFiles/vw.dir/boosting.cc.o] Error 1
  gmake[3]: *** Waiting for unfinished jobs....
  gmake[3]: *** [vowpalwabbit/CMakeFiles/vw.dir/ect.cc.o] Error 1
  gmake[3]: *** [vowpalwabbit/CMakeFiles/vw.dir/marginal.cc.o] Error 1
  In file included from /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/global_data.h:44:0,
                   from /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/reductions.h:14,
                   from /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/lrq.cc:3:
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/options.h: In instantiation of 'VW::config::typed_option<T>::typed_option(const string&, T&) [with T = std::vector<std::basic_string<char> >; std::string = std::basic_string<char>]':
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/options.h:81:40:   required from 'VW::config::typed_option<T> VW::config::make_option(std::string, T&) [with T = std::vector<std::basic_string<char> >; std::string = std::basic_string<char>]'
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/lrq.cc:167:47:   required from here
  /tmp/pip-install-2tn4k8le/vowpalwabbit/vowpalwabbit/options.h:28:117: error: invalid initialization of non-const reference of type 'std::vector<std::basic_string<char> >&' from an rvalue of type '<brace-enclosed initializer list>'
     typed_option(const std::string& name, T& location) : base_option(name, typeid(T).hash_code()), m_location{location} {}
                                                                                                                       ^
  gmake[3]: *** [vowpalwabbit/CMakeFiles/vw.dir/lrq.cc.o] Error 1
  gmake[2]: *** [vowpalwabbit/CMakeFiles/vw.dir/all] Error 2
  gmake[1]: *** [python/CMakeFiles/pylibvw.dir/rule] Error 2
  gmake: *** [pylibvw] Error 2
  error: command 'cmake' failed with exit status 2
  ----------------------------------------
  ERROR: Failed building wheel for vowpalwabbit
  Running setup.py clean for vowpalwabbit

Have you installed the dependencies? My guess is `yum install cmake`. — Kent Shikama, Oct 29 '19 at 05:13
Yes. I have cmake 3.14.4 and boost 1.70.0. You can see from the log that it finds dependencies and starts the build but fails at 25% with "error: invalid initialization of non-const reference of type 'std::basic_string&' from an rvalue of type '' typed_option(const std::string& name, T& location) : base_option(name, typeid(T).hash_code()), m_location{location} {}" — Pavel Lesev, Oct 30 '19 at 06:26

score 2 · Answer 1 · answered Oct 30 '19 at 17:15

2

From the vowpalwabbit issues board:

The Python bindings on Linux currently need to be built from source and it looks as though the GCC version that is being used is too old and doesn't properly support C++11. See this stack overflow issue that looks to be related to the problem you're facing.

Please use a newer version of GCC to build the bindings. You can find an example on how to install a newer GCC on CentOS 7 in this dockerfile.

answered Oct 30 '19 at 17:15

peterchang

76
2

1

Yes. Also, looks like the OP python is coming from anaconda, rather than the system default python. Anaconda is fundamentally incompatible with any Linux (non-source) distro. It has its own eco-system (including certain versions of libraries and compilers). C++/boost header files are very picky about exact versions to interoperate. IOW one must build both `vw` and anaconda with exactly the same versions of headers/libs/compiler-tools. One can't use runtime (already compiled) libraries like `libboost` and mix them with anaconda's own runtime. – arielf Oct 30 '19 at 21:12

Cannot install vowpalwabbit on CentOS via pip or via source code

1 Answers1