How to import newly compiled python module?

Question

I have compiled lightgbm with GPU support for python from sources following this guide http://lightgbm.readthedocs.io/en/latest/GPU-Windows.html

Test usage from console was succesful:

C:\github_repos\LightGBM\examples\binary_classification>"../../lightgbm.exe" config=train.conf data=binary.train valid=binary.test objective=binary device=gpu
[LightGBM] [Warning] objective is set=binary, objective=binary will be ignored. Current value: objective=binary
[LightGBM] [Warning] data is set=binary.train, data=binary.train will be ignored. Current value: data=binary.train
[LightGBM] [Warning] valid is set=binary.test, valid_data=binary.test will be ignored. Current value: valid=binary.test
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Info] Loading weights...

Then I tried to import in Python with no luck. It import anaconda version without GPU support:

from sklearn.datasets import load_iris
iris = load_iris() 

import lightgbm as lgb 
lgtrain = lgb.Dataset(iris.data, iris.target)
lgb_clf = lgb.train(
        {
    'objective' : 'regression',
    'metric' : 'rmse',
    'num_leaves' : 350,
    #'max_depth': 14,
    'learning_rate' : 0.017,
    'feature_fraction' : 0.5,
    'bagging_fraction' : .8,
    'verbosity' : -1 ,
    'device' : 'gpu'

},
        lgtrain,
        num_boost_round=3500,
        verbose_eval=100
    )

LightGBMError: b'GPU Tree Learner was not enabled in this build. Recompile with CMake option -DUSE_GPU=1'

I believe I have to specify the location but how?

score 0 · Answer 1 · answered Jun 17 '18 at 07:38

I think this might not be specific to lightGBM, but rather a problem with Anaconda's virtual environment. When working within the Anaconda virtual env, your system paths are modified to point to Anaconda installation directories.

As you point out, this leads to Anaconda loading its own version, rather than the external version you configured, compiled and tested.

There are several ways to force Anaconda to find your package, see this related discussion.

The suggestions that involve running ln -s are only for Linux and Mac, but you can do something similar in Windows.

You could start by uninstalling the Anaconda version of lightGBM, then create a copy of the custom-compiled version within the Anaconda path. You can discover this using

import sys
sys.path

score 0 · Accepted Answer · answered Jun 18 '18 at 19:00

Remove previously installed Python package with the following command:

pip uninstall lightgbm
or

conda uninstall lightgbm

After doing that navigate to the Python package directory and install it with the library file which you've compiled:

cd LightGBM/python-package
python setup.py install --precompile

How to import newly compiled python module?

2 Answers2