Save Python dictionary to file, and update it periodically

Question

I feel like this is a very simple problem and there are a lot of very similar questions to it here, but I still can't figure out how to get what I want. I am working with remote devices that I can connect to when they are online. I want to have a file that records the most recent uptime of a device, like this:

# device ID ......... last seen online
{'deviceID1':'Wed Nov 08 2017 06:11:27 PM',
'deviceID2':'Wed Nov 08 2017 06:11:27 PM',
'deviceID3':'Tues Nov 07 2017 03:47:01 PM'}

etc. I've gotten really close by storing this in a json file and doing json.dumps to store the data and json.load to view it. My code follows the steps: 'ping all device IDs', 'check output', and 'write result to file'. But every time I do this, the values get overwritten for devices that were online and are now not online. As in, instead of the above, I get something like:

# device ID ......... last seen online
{'deviceID1':'Wed Nov 08 2017 06:11:27 PM',
'deviceID2':'Wed Nov 08 2017 06:11:27 PM',
'deviceID3':''}

when I check at Wed Nov 08 2017 06:11:27 PM and deviceID3 is not online. But I want to preserve that value while updating the values of the devices I do see online. How can I essentially keep this dictionary / data in a file and update it for the same set of unique device IDs every time? This question gets the closest, but that's about appending entries, and I want to update the values of keys that are already there. Thanks.

Relevant code:

def write_to_file(data):
    with open(STATUS_FILE, 'w') as file:
        file.write(json.dumps(data))

def create_device_dictionary(deviceIDs):
    devices = {}
    for i in range(0, len(deviceIDs)):
        devices[deviceIDs[i]] = []
    return devices

def check_ping_output(cmd_output_lines,devices):

    for i, line in enumerate(cmd_output_lines):
        device_id = line.strip().strip(':')
        # if the device pinged back...
        if 'did not return' not in cmd_output_lines[i+1]:
            #current UNIX time
            human_readable_time = time.strftime(
                '%a %b %d %Y %I:%M:%S %p',
                time.gmtime(time.time())
            )
            devices[device_id].append(human_readable_time)
        else:
        #    do something here? I want the device ID to be in the file even if it's never appeared online
             pass

    return devices

Answer 1

Here's a basic example I came up with (removing the snippets parts that you have already addressed, such as time conversion, etc)

import json

# Suppose this is your existing json file (I'm keeping it as a string for the sake of the example):
devices_str = '{"deviceIDa":"Wed Nov 08 2017 06:11:27 PM", "deviceIDb":"Wed Nov 08 2017 06:11:27 PM", "deviceIDc":"Tues Nov 07 2017 03:47:01 PM"}'

cmd_output_lines = [
    'deviceIDa:True',
    'deviceIDb:Minion did not return. [No response]',
]
# Generate a dictionary of the devices for current update
devices = dict(
    line.strip().split(':') for line in cmd_output_lines
)
# Filter out the ones currently online, using whatever criteria needed
# In my example, I'm just checking if the response contained a True
online_devices = dict(
    (device_id, resp) for (device_id, resp) in devices.iteritems() if 'True' in resp
#               ^ of course in your case that resp would be current/last seen time
)

# Load existing entries
existing_devices = json.loads(devices_str)

# Update them only overwriting online devices
existing_devices.update(online_devices)

# Final result
print json.dumps(existing_devices)

This outputs:

"deviceIDb": "Wed Nov 08 2017 06:11:27 PM", "deviceIDc": "Tues Nov 07 2017 03:47:01 PM", "deviceIDa": "True"}

As you can see, deviceIDa is the only entry that got updated (deviceIDb still has the last seen from the existing entries)

Edit:

Taking this a step further, if you want to log the last 5 online times, you can either use a defaultdict(list), or get away with plain dictionaries like so:

>>> d = {1: [2, 3, 4, 5, 6,]}
>>> new = {1: 7, 2: 4} # only one current online time anyway (no need for list here)
>>> for _id, new_online in new.items():
...     d[_id] = (d.get(_id, []) + [new_online])[-5:] # only take the last 5 elements
...     
>>> d
{1: [3, 4, 5, 6, 7], 2: [4]}
>>> 

Answer 2

Instead of writing the created data directly to the file, I would suggest you follow the following steps:

1. Read the last-seen file, and convert it to a dictionary.
2. Check every deviceID in the dictionary, and UPDATE the value in the dictionary if the device is online.
3. Check any devices that ARE NOT IN the last seen dictionary, and add timestamps for them to the dictionary.
4. Replace the contents of the file with the result of running json.dumps with the current modified dictionary.

Answer 3

Basically, JSON is not a good deal for what you want. For one - you cant just append new values to a JSON data file, since it requires closing the structures.

Next, although you can build JSON data resusing a key multiple times, when converting this to a plain Python dict, only the last value will be visible anyway (or you will get an exception, if you tune your parser ok).

So, just a plain file where each record is a new line, think "CSV" - would work better for you. Or, you may opt to move everything to a SQL DB, like sqlite at once.