1

I have around 10K rows in a dataframe. I want to copy-paste all items then apply them all to my SQL query as my filter. Here is how I convert to a list

list_a = df['col'].to_list()

When I tried to print out the list it only shows me 1k of the items. I tried to set pd.set_option('display.max_rows', None) but it seems doesn't work. It only works for pandas dataframe.

My output is,

>>> list_a
>>> ['a1',
     'a2',
     'a3',
   till 'a1000' then it ignore the rest of 9k with
     ...
]

Can I get all 10k items print out?

Jiayu Zhang
  • 719
  • 7
  • 22
  • 1
    Why would a pandas setting affect printing a list? – Barmar May 27 '22 at 23:42
  • I think python automatically limits it for general lag purposes. – Thornily May 27 '22 at 23:42
  • 1
    I just did `print(list(range(10000)))` and it printed everything. – Barmar May 27 '22 at 23:43
  • I'm not sure how it applies here, but see https://stackoverflow.com/questions/7223222/what-does-an-ellipsis-in-a-list-mean-in-python – Barmar May 27 '22 at 23:47
  • @Jiayu Zhang What does "len(list_a)" return? – Drakax May 28 '22 at 00:07
  • Why we need print all of them out ? – BENY May 28 '22 at 00:34
  • How did you verify the number of elements in the list? – Karl Knechtel May 28 '22 at 01:32
  • [related](https://stackoverflow.com/questions/33615811/printing-the-whole-list-wont-show-all-elements-from-the-txt) – 0x263A May 28 '22 at 02:00
  • @BENY When I write a SQL query, for the `WHERE` clause, usually you just type in the filter, but now I need more than 10k items as my filter so I tried to print it out in Python and then copy-paste to my query. I actually fix them already by merging all list items into one super long item. It will print out everything in Python. It looks so dumb. Do you know any other ways that can execute smarter? – Jiayu Zhang May 31 '22 at 06:04
  • 1
    @JiayuZhang df.yourid.to_clipboard(sep=',', index=False) ? and just paste there ~ – BENY May 31 '22 at 13:27

1 Answers1

1

As @Barmar points out in their comments, this print issue is not related to pandas because the pandas.series.to_list() function returns a list object.

Also @Thornily is stating that this is related to your python console trying to conserve resources. The reason python uses summarization (...) is because printing an excessive number of lines is slow and will be very difficult to visual read through the output.

Finally, I think @BENY's comment can probably help provide insight into why you need to print this many. Like @Drakax has pointed out, you can use len(list_a) to verify the list has all 10,000 items in it.

I am assuming that the OP doesn't actually need to read all 10,000 records and that they likely want to know WHY they can't override the summarization (...). Unfortunately, I am still working on the WHY answer. I have tried changing the default preferences in my console and using numpy.set_printoptions(threshold=sys.maxsize) but I still get the elipsis. I will keep trying to answer the WHY question and will update this post if I find a good answer.

For now, IF you want a HOW answer, you can use the very slow method below. Keep in mind, I do not recommend printing the output to your console for the reasons stated above. But if you ABSOLUTELY have to print to your console you can do this:

for n in list_a:
    print(n)

or you can also convert using the list function to force the object to a python list and print multiple values on each console line:

list_a = list(list_a)
print(list_a)
sog
  • 493
  • 4
  • 13