Best way to store python datetime.time in a sqlite3 column?

Question

I'm trying to replace my use of SAS with python + sqlite3; I'm trying to move my data from SAS datasets to SQLite databases. I have many time fields that are properly represented in python as datetime.time objects. Since SQLite is "lightly typed", I'm looking for advice about what format to use to store times in columns. (I know I have to write python adapters etc. to read and write the objects to and from the column.) These are the features I need to consider:

• SQLite's ability to deal with the column in queries. (Eg will I be able to select out the rows that occur between two times?)
• Size of the field. (My tables are often hundreds of millions of rows.)
• Human readability. (I'm considering storing the time as an integer: microseconds since midnight. But this makes eyeballing the data harder.)

Has anyone solved this problem to their satisfaction?

Answer 1

There is a general recipe for storing any serializable Python object in an sqlite table.

• Use sqlite3.register_adapter to register a function for converting the Python object to int, long, float, str (UTF-8 encoded), unicode or buffer.
• Use sqlite3.register_converter to register a function for converting text to the Python object. The input is always text because internally, sqlite stores everything as text.

Here is what the code look might look like for datetime.time objects:

import sqlite3
import datetime as DT

def adapt_timeobj(timeobj):
    return ((3600*timeobj.hour + 60*timeobj.minute + timeobj.second)*10**6 
            + timeobj.microsecond)

def convert_timeobj(val):
    val = int(val)
    hour, val = divmod(val, 3600*10**6)
    minute, val = divmod(val, 60*10**6)
    second, val = divmod(val, 10**6)
    microsecond = int(val)
    return DT.time(hour, minute, second, microsecond)


# Converts DT.time to TEXT when inserting
sqlite3.register_adapter(DT.time, adapt_timeobj)

# Converts TEXT to DT.time when selecting
sqlite3.register_converter("timeobj", convert_timeobj)

con = sqlite3.connect(":memory:", detect_types=sqlite3.PARSE_DECLTYPES)
cur = con.cursor()

# declare timecol to be of type timeobj
cur.execute("create table test (timecol timeobj)")

cur.executemany("insert into test (timecol) values (?)", 
                [(DT.time(1,2,3,4), ), (DT.time(5,6,7,8),) ])

You can use inequalities in the SQL, but note that the values being compared are those returned by adapt_timeobj, not the datetime.time objects. Fortunately, if the adapt_timeobj function returns integers that are orderable in the same order as the corresponding datetime.time objects (as they do above), then inequalities in the SQL will work as desired.

cur.execute("select timecol from test where timecol < ?",
            [DT.time(4,5,6)])
print(cur.fetchall())
# [(datetime.time(1, 2, 3, 4),)]

cur.execute("select timecol from test where timecol < ?",
            [DT.time(8,0,0)])
print(cur.fetchall())
# [(datetime.time(1, 2, 3, 4),), (datetime.time(5, 6, 7, 8),)]

con.commit()
cur.close()
con.close()

Note: If you look in the edit history, you'll see a simpler alternative for adapt_timeobj and convert_timeobj that stores the data as a str instead of as a int. It is simpler, but storing the data as a int is faster and more memory efficient.

Answer 2

I really like the answer by @unutbu but here's a simple way to store a timestamp.

RFC 3339 is a very unambiguous timestamp format, easy for computers to parse and easy for humans to read. You could store timestamps as strings.

One nice property of RFC 3339: a simple ASCII sort also sorts chronologically.

But you don't really need the spec because it is so simple. Here's an example:

2014-12-24T23:59:59.9999-08:00

That is the last fraction of a second before Christmas day in my time zone, which is 8 hours behind UTC (thus the -08:00 part). Year, month, date, the string T, hour, minute, seconds, optional fractional second, timezone.

The timezone may also be Z which indicates UTC time. But it's probably more convenient to store the times in the local time zone so you can read them more easily.