0

I Have a table in Mysql Which contains 10 Million Entries so far, and I am writing a script in python in which I want to get data from DB in chunks of 100K(Number of row's).

I have tried threw paging (https://www.citusdata.com/blog/2016/03/30/five-ways-to-paginate/) but unable to do it anyone can help me with this.

saintlyzero
  • 1,632
  • 2
  • 18
  • 26
Newton8989
  • 300
  • 1
  • 4
  • 22

1 Answers1

1

You can use LIMIT in MySQL
LIMIT clause is used with the SELECT,UPDATE or DELETE statement to restrict the number of rows in the result set.
It accepts one or two arguments which are offset and count.

Syntax

SELECT column1, column2, ...
FROM table_name
LIMIT offset, count;

offset specifies which row to start retrieving data from
count specifies the number of rows returned from a result set

Usage

SELECT * FROM STUDENTS
LIMIT 100, 50

Here, 100 specifies the offset, i.e the query will start retrieving data starting from the 100th record.
50 specifies the number of records that will be returned after the 100th record.

Eg.

In your case,
count will be 100k and the offset will be the multiples of 100k starting from 0

SELECT * FROM STUDENTS
LIMIT 0, 100000

Next query will be:

SELECT * FROM STUDENTS
LIMIT 100000, 100000

And so on..
So in your python code, count value will be same i.e 100000 whereas the offset value will change from 0, 100000, 200000, .. . offset value is essentially the multiples of 100000

For further reference, have a look here link1 link2

saintlyzero
  • 1,632
  • 2
  • 18
  • 26