7

I have multiple, large, csv files, each of which has missing values in many places. When I import the csv file into SQLite, I would like to have the missing values recorded as NULL for the reason that another application expects missing data to be indicated by NULL. My current method does not produce the desired result.

An example CSV file (test.csv) is:

12|gamma|17|delta
67||19|zeta
96|eta||theta
98|iota|29|

The first line is complete; each of the other lines has (or is meant to show!) a single missing item. When I import using:

.headers on
.mode column
.nullvalue NULL
CREATE TABLE t (
  id1     INTEGER  PRIMARY KEY,
  a1      TEXT,
  n1      INTEGER,
  a2      TEXT
);
.import test.csv t
SELECT
  id1, typeof(id1),
  a1,  typeof(a1),
  n1,  typeof(n1),
  a2,  typeof(a2)
FROM t;

the result is

id1   typeof(id1)  a1      typeof(a1)  n1  typeof(n1)  a2      typeof(a2)
----  -----------  ------  ----------  --  ----------  ------  ----------
12    integer      gamma     text      17  integer     delta   text                      
67    integer                text      19  integer     zeta    text                      
96    integer      eta       text          text        theta   text                      
98    integer      iota      text      29  integer             text

so the missing values have become text. I would appreciate some guidance on how to ensure that all missing values become NULL.

CrimsonDark
  • 661
  • 9
  • 20

2 Answers2

7

sqlite3 imports values as text and there does not seem to be a way to make it treat empty values as nulls.

However, you can update the tables yourself after import, setting empty strings to nulls, like

UPDATE t SET a1=NULL WHERE a1='';

Repeat for each column.

You can also create a trigger for such updates:

CREATE TRIGGER trig_a1 AFTER INSERT ON t WHEN new.a1='' BEGIN
  UPDATE t SET a1=NULL WHERE rowid=new.rowid;
END;
laalto
  • 150,114
  • 66
  • 286
  • 303
  • Good idea, does not work on INTEGER fields though as in these, the empty string is automatically converted to 0. – fuz Feb 16 '16 at 23:10
0

For the cases where you cannot update after import because the import will fail when the empty string (text columns) or 0 (integer columns) is inserted instead of NULL, see my answer to this other stackoverflow question

Ebow Halm
  • 321
  • 2
  • 5