KeyError when loading pickled scikit-learn model using joblib

Question

I have an object that contains within it two scikit-learn models, an IsolationForest and a RandomForestClassifier, that I would like to pickle and later unpickle and use to produce predictions. Apart from the two models, the object contains a couple of StandardScalers and a couple of Python lists.

Pickling this object using joblib is unproblematic, but when I try to unpickle it later I get the following exception:

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/home/(...)/python3.5/site-packages/joblib/numpy_pickle.py", line 578, in load
   obj = _unpickle(fobj, filename, mmap_mode)
 File "/home/(...)/python3.5/site-packages/joblib/numpy_pickle.py", line 508, in _unpickle
   obj = unpickler.load()
 File "/usr/lib/python3.5/pickle.py", line 1039, in load
   dispatch[key[0]](self)
KeyError: 0

The same application both pickles and unpickles the object, so the versions of scikit-learn, joblib and other libraries are the same. I'm not sure where to start debugging, given the vague error. Any ideas or pointers?

is the latest version of scikit is installed on your pc? – Flika205 Feb 23 '18 at 12:46 — Flika205, Feb 23 '18 at 12:46
Can you produce a minimal duplicateable code? – Vivek Kumar Feb 23 '18 at 12:52 — Vivek Kumar, Feb 23 '18 at 12:52

score 36 · Accepted Answer · answered Feb 26 '18 at 12:04

36

The solution to this was pretty banal: Without being aware of it I was using the version of joblib in sklearn.externals.joblib for the pickling, but a newer version of joblib for unpickling the object. The problem was resolved when I used the newer version of joblib for both tasks.

answered Feb 26 '18 at 12:04

haroba

2,120
4
22
37

hello @haroba I am facing same problem can you tell me which version you have to used ? i am using joblib-0.13.0 version – Jignasha Royala Nov 15 '18 at 09:21
banal, but useful;) – jibiel Jul 17 '19 at 10:08
You saved my day! Super useful. I had a very similar load error caused by numpy. Update joblib to equivalent version solved this. So - thanks! – petezurich Feb 26 '23 at 20:32

score 14 · Answer 2 · answered Jan 16 '19 at 17:17

14

With me, happened that I exported the model using from sklearn.externals import joblib and tried to load using import joblib.

answered Jan 16 '19 at 17:17

Marcos Paulo

317
3
10

score 8 · Answer 3 · answered Jul 16 '19 at 00:46

Mine was interesting. I was working with git-lfs and thus the files had been changed and joblib couldn't open them. So I needed to run git lfs pull to get actual files. So if you are using compatible joblib versions, make sure your files are not changed somehow!

score 1 · Answer 4 · answered May 15 '20 at 07:01

1

For me the same version of joblib was used to dump and load but I was saving the file under python 3.7.4 and attempting to load with python 3.7.6 which raised the same KeyError.

answered May 15 '20 at 07:01

BarefootDev

326
3
9

score 0 · Answer 5 · edited Sep 01 '20 at 19:22

In my case, I was trying to load an XGB. I found out XGB is not compatible with other sklearn models, so I did the following:

from xgboost import *
import joblib

def get_model(model_path):
    if 'xgb' in model_path:
        xgb_model = XGBClassifier()
        xgb_model.load_model(model_path)
        model = xgb_model
    else: 
        model = get_obj(model_path)
    return model 

xbg = get_model('Models/xgb_v1.pkl') # an xgb

tree = model = get_model('Models/dt_v1.pkl') # a decition tree

score 0 · Answer 6 · answered Mar 16 '23 at 17:23

I was trying to load years old joblib files, which gave multiple levels of errors, depending on the hack I used to bypass them.

With the increasing versions of joblib, the hacks stopped working and I had to create a conda environment specifically for sklearn-0.23 as such:

conda create -n outdated "scikit-learn<0.23"

Afterwards, I was able to open the files and save them differently. This sometimes this means re-saving the data with non-sklearn joblib files import joblib; sometimes this means using pickle; sometimes this meant using pandas.to_csv.

The solution was specific to the data file being re-saved for posterity.

KeyError when loading pickled scikit-learn model using joblib

6 Answers6

Linked