Load all pickled objects

Question

import pickle

ListNames = [["Name1","City1","Email1"],["Name2","City2","Number2"]]
ListNumbers = [1,2,3,4,5,6,7,8]

with open ("TestPickle.pickle","wb") as fileSaver:
    pickle.dump(ListNames,fileSaver)
    pickle.dump(ListNumbers,fileSaver)

with open ("TestPickle.pickle","rb") as fileOpener:
    print(pickle.load(fileOpener))

The output is:

[['Name1', 'City1', 'Email1'], ['Name2', 'City2', 'Number2']]

How do I get pickle to load the ListNumbers too

I know I can just print pickle.load again but what if I have an unknown number of items in my pickle file with a number of Datatypes (e.g: lists, tuples, dictionaries, strings....)

Thanks

Normally, if you want to save multiple objects, you'd pickle a list (or other container) with all the objects in it. If that's not practical (the objects are too big to simultaneously fit in memory, for example), you could store the count of objects as the first thing in the file, or just keep unpickling until you get an EOF exception. — jasonharper, Mar 13 '18 at 16:28

score 1 · Accepted Answer · answered Mar 13 '18 at 16:33

I am not sure if this i the correct approach.

import pickle

ListNames = [["Name1","City1","Email1"],["Name2","City2","Number2"]]
ListNumbers = [1,2,3,4,5,6,7,8]

with open ("TestPickle.pickle","wb") as fileSaver:
    pickle.dump(ListNames,fileSaver)
    pickle.dump(ListNumbers,fileSaver)
obj = []
with open("TestPickle.pickle","rb") as fileOpener:
    while True:
        try:
            obj.append(pickle.load(fileOpener))
        except EOFError:
            break
print obj

Output:

[[['Name1', 'City1', 'Email1'], ['Name2', 'City2', 'Number2']], [1, 2, 3, 4, 5, 6, 7, 8]]

score 0 · Answer 2 · answered Mar 13 '18 at 16:29

I know I can just print pickle.load again but what if I have an unknown number of items in my pickle file with a number of Datatypes (e.g: lists, tuples, dictionaries, strings....)

There are several ways around this:

Pack all of the items into a single list or tuple, and pickle that instead of pickling them individually. You can then unpack on the other side.
Pickle an integer describing the number of items first. That then tells you how many times to call pickle.load().
Just keep unpickling until you hit EOF.

cardamom · Answer 3 · 2018-03-13T16:36:23.520

0

Have not previously tried to put more than one thing in a pickle file like that before, but good to see it is possible. Look at this answer. If you call it twice without the with it should get everything out:

f = open("TestPickle.pickle", "r")
ListNames = pickle.load(f)
ListNumbers = pickle.load(f)
f.close()

or you can keep the with like this:

with open("TestPickle.pickle", "rb") as f:
    ListNames,ListNumbers = pickle.load(f)

Then,

what if I have an unknown number of items in my pickle file with a number of Datatypes

Is it really a smart idea to just pour something into a pickle file so you don't know anymore what is in it?

edited Mar 13 '18 at 16:36

answered Mar 13 '18 at 16:30

cardamom

6,873
11
48
102

Thanks. I was aiming to unpickle an undefined number of packed items. Below someone suggested a while loop with try/except :) – Mar 13 '18 at 16:37
Yes that should work and the `except` allows it to also hit the end without an error message. – cardamom Mar 13 '18 at 16:40

Load all pickled objects

3 Answers3

Linked