Stack Overflow
Stack Overflow

Questions tagged [pandas-to-sql]

Pandas DataFrame method that writes the object's records to a SQL database. Be sure to also include the [pandas] tag.

Pandas DataFrame method to_sql can be used to write its records to a SQL database.

The documentation:

144 questions
77
votes
9 answers

Speeding up pandas.DataFrame.to_sql with fast_executemany of pyODBC

I would like to send a large pandas.DataFrame to a remote server running MS SQL. The way I do it now is by converting a data_frame object to a list of tuples and then send it away with pyODBC's executemany() function. It goes something like this: …
J.K.
  • 1,574
  • 1
  • 13
  • 21
25
votes
1 answer

append the data to already existing table in pandas using to_sql

I have the following data frame ipdb> csv_data country sale date trans_factor 0 India 403171 12/01/2012 1 1 Bhutan 394096 12/01/2012 2 2 Nepal super 12/01/2012 3 3 madhya 355883 …
Shiva Krishna Bavandla
  • 25,548
  • 75
  • 193
  • 313
22
votes
1 answer

to_sql() error querying sqlite_master table when writing to MySQL

I have a question about how to save a dataframe to my local mysql. import MySQLdb import pandas as…
user7796650
  • 221
  • 1
  • 2
  • 3
14
votes
3 answers

duplicate key value violates unique constraint - postgres error when trying to create sql table from dask dataframe

Following on from this question, when I try to create a postgresql table from a dask.dataframe with more than one partition I get the following error: IntegrityError: (psycopg2.IntegrityError) duplicate key value violates unique constraint…
Ludo
  • 2,307
  • 2
  • 27
  • 58
13
votes
3 answers

How to write pandas dataframe to oracle database using to_sql?

I'm a new oracle learner. I'm trying to write a pandas dataframe into an oracle table. After I have made research online, I found the code itself is very simple, but I don't know why my code doesn't work. I have read the pandas dataframe from my…
Haven Shi
  • 457
  • 5
  • 14
  • 19
12
votes
4 answers

How to use pandas.to_sql but only add row if row doesn't exist yet

I have some experience with python but very new to the SQL thing and trying to use pandas.to_sql to add table data into my database, but when I add I want it to check if the data exists before append This are my 2 dataframes >>> df0.to_markdown() | …
Anonymous Anonymous
  • 121
  • 1
  • 5
12
votes
2 answers

Create sql table from dask dataframe using map_partitions and pd.df.to_sql

Dask doesn't have a df.to_sql() like pandas and so I am trying to replicate the functionality and create an sql table using the map_partitions method to do so. Here is my code: import dask.dataframe as dd import pandas as pd import sqlalchemy_utils…
Ludo
  • 2,307
  • 2
  • 27
  • 58
8
votes
2 answers

Python Connect to AWS Aurora Serverless MySQL Using SQLAlchemy

Is there a way to specify my connection with SQLAlchemy to an AWS RDS Aurora Serverless MySQL database instance without a Secrets Manager ARN? I have the database username, password, endpoint, ARN, etc., and ideally I would initialize an engine,…
OJT
  • 887
  • 1
  • 10
  • 26
8
votes
2 answers

insert ignore pandas dataframe into mysql

I want to "insert ignore" an entire pandas dataframe into mysql. Is there a way to do this without looping over the rows? In dataframe.to_sql I only see the option if_exists 'append' but will this still continue on duplicate unique keys?
user3605780
  • 6,542
  • 13
  • 42
  • 67
7
votes
1 answer

Pandas to_sql() to update unique values in DB?

How can I use the df.to_sql(if_exists = 'append') to append ONLY the unique values between the dataframe and the database. In other words, I would like to evaluate the duplicates between the DF and the DB and drop those duplicates before writing to…
Merv Merzoug
  • 1,149
  • 2
  • 19
  • 33
6
votes
0 answers

How to get primary keys from pandas.to_sql insert

I want to add some data to the database with pandas .to_sql command. Is there a way I can get the auto-generated primary-key of the inserted objects as I need it for creating foreign keys? So far I did this: Models: from django.db import…
MaxxiKing
  • 83
  • 6
6
votes
1 answer

Pandas to_sql make index unique

I have been readin about pandas to_sql solutions to not add duplicate records to a database. I am working with csv files of logs, each time i upload a new log file i then read the data and make some changes with pandas creating a new dataframe. Then…
Ana
  • 167
  • 3
  • 17
5
votes
1 answer

How to Ignore errors in Dataframe to_sql method

I am using Pandas DataFrame.to_SQL() to insert data from dataframe to table. Nothing gets loaded if even a single record has loading issues, for example: if a integer column has character data in one row or date format is not valid in one record…
PythonDeveloper
  • 289
  • 1
  • 4
  • 24
5
votes
1 answer

AttributeError: 'psycopg2.extensions.cursor' object has no attribute 'fast_executemany'

AttributeError: 'psycopg2.extensions.cursor' object has no attribute 'fast_executemany' to_sql() is too slow. so trying to resolve the problem. but when I run the following code I am getting :- AttributeError: 'psycopg2.extensions.cursor' object…
RAVI VERMA
  • 93
  • 1
  • 10
5
votes
2 answers

Error while insert dataframe to Sql-server DB using to_sql function in python

I am trying to insert pandas dataframe df into SQL Server DB using dataframe.to_sql function. But i getting below error: Source code: import pyodbc import sqlalchemy import urllib df #sample dataframe params = urllib.parse.quote_plus("DRIVER={SQL…
Arvinth Kumar
  • 964
  • 3
  • 15
  • 32
1
2 3
…
9 10