Plotting millions of data points in Python?

Question

I have written a complicated code. The code produces a set of numbers which I want to plot them. The problem is that I cannot put those numbers in a list since there are 2 700 000 000 of them. So I need to plot one point then produce second point (the first point is replaced by second point so the first one is erased because I cannot store them). These numbers are generated in different sections of the code so I need to hold (MATLAB code) the figure. For making it more conceivable to you, I write a simple code here and I want you to show me how to plot it.

import matplotlib.pyplot as plt
i=0
j=10
while i<2700000000:
    plt.stem(i, j, '-')
    i = i + 1
    j = j + 2
plt.show()

Suppose I have billions of i and j!

It's not clear what you're asking here. Which of your 2.7 billion points do you wish to plot? — Oliver Charlesworth, Jan 05 '15 at 23:29
I believe you should be able to plot 2.7 billion points. Keeping it in a list may not feasible, but you could use a generator if you are computing the values on the fly. — Dair, Jan 05 '15 at 23:30
@OliverCharlesworth suppose i starts from zero to 2.7 billion **x-axis** and j is generated numbers which has different values **y-axis** — Cristopher Van Paul, Jan 05 '15 at 23:32
@Bair, I am not computing the values on the fly. And I already tried and could not put all in the list. I do not have enough memory. — Cristopher Van Paul, Jan 05 '15 at 23:35
I wrote that simple code to give you some understanding of what I'm doing. Even that code is not being processed. Just the first point is being plotted. — Cristopher Van Paul, Jan 05 '15 at 23:36
http://stackoverflow.com/questions/5854515/large-plot-20-million-samples-gigabytes-of-data — dstromberg, Jan 05 '15 at 23:37
If you really must plot them all (why not bin them down to 2.7 million, or fewer, for example?) I would consider generating a large series of PIL images and altering the colours of individual pixels. You could stitch or scale the images afterwards if that was required. — John Lyon, Jan 05 '15 at 23:45
@jozzas actually the bin numbers are 2.7 billion but I want to sketch it with stem — Cristopher Van Paul, Jan 05 '15 at 23:47
have you tried graphlab (graphlab.org)? it should be able to handle that amount of data, and plot them. — adrianX, Jan 06 '15 at 00:00
@adrianX thanks for introducing something new. I should try that graphlab.org — Cristopher Van Paul, Jan 06 '15 at 00:49
What sort of billion X billion pixel resolution device are you going to plot these numbers on ? There is going to have to be some downsampling somewhere in the processing chain. You can either do it explicitly yourself, or you can find a better graphic library that does it for you. ( See @dstromberg 's link above, maybe. ) — Steven D. Majewski, Jan 06 '15 at 01:30

Aleksander Lidtke · Answer 1 · 2015-01-06T00:15:13.263

Hmm I'm not sure if I understood you correctly but this:

import matplotlib.pyplot as plt
i=0
j=10
fig=plt.figure()
ax=fig.gca()
while i<10000: # Fewer points for speed.
    ax.stem([i], [j]) # Need to provide iterable arguments to ax.stem
    i = i + 1
    j = j + 2
fig.show()

generates the following figure: enter image description here Isn't this what you're trying to achieve? After all the input numbers aren't stored anywhere, just added to the figure as soon as they are generated. You don't really need Matlab's hold equivalent, the figure won't be shown until you call fig.show() or plt.show() to show the current figure.

Or are you trying to overcome the problem that you can' hold the matplotlib.figure in your RAM? In which case my answer doesn't answer your question. Then you either have to save partial figures (only parts of the data) as pictures and combine them, as suggested in the comments, or think about an alternative way to show the data, as suggested in the other answer.

Plotting millions of data points in Python?

1 Answers1