I am getting a memory error while uploading a CSV file of size around 650 mb with a shape (10882101, 6). How can i upload such file in postgres using django framework.
Asked
Active
Viewed 70 times
-1
-
1Hi, please try to upload the error that you are seeing. – Diego Puente May 08 '20 at 15:13
-
1We are going to need more information than this. Please provide the entire error output, as well as the the relevant code. See [ask], [help/on-topic]. – AMC May 08 '20 at 16:03
1 Answers
0
- You haven't shared much details (error logs, which python package you are using etc).
- You might like to read Most efficient way to parse a large .csv in python?, https://medium.com/casual-inference/the-most-time-efficient-ways-to-import-csv-data-in-python-cc159b44063d
- How I will do it from Django Framework:
- I will use Celery to run the job as a background process, as waiting for the file to be uploaded completely before returning a response might give a HTTP timeout.
- Celery quickstart with Django
Anubhav Ujjawal
- 92
- 8