List dictionary in Python to group by in key's value

Question

I have this list dictionary in python

accidents_dict = [{'date': 'new Date (2017,1,1)', 'Fall': 5}, 
{'date': 'new Date (2017,1,1)', 'Vehicular Accident': 127}, 
{'date': 'new Date (2017,1,1)', 'Medical': 129}, 
{'date': 'new Date (2017,1,1)', 'OB': 10}, 
{'date': 'new Date (2017,1,1)', 'Mauling': 9}, 
{'date': 'new Date (2017,2,1)', 'Fall': 7}, 
{'date': 'new Date (2017,2,1)', 'Vehicular Accident': 113}, 
{'date': 'new Date (2017,2,1)', 'Mauling': 5}, 
{'date': 'new Date (2017,2,1)', 'OB': 9}, 
{'date': 'new Date (2017,2,1)', 'Medical': 79}, 
{'date': 'new Date (2017,3,1)', 'Medical': 112}, 
{'date': 'new Date (2017,3,1)', 'Mauling': 5}, 
{'date': 'new Date (2017,3,1)', 'OB': 11}, 
{'date': 'new Date (2017,3,1)', 'Vehicular Accident': 119}, {'date': 'new Date (2017,3,1)', 'Fall': 8}]

My desired output is to group them by date and to remove the single quote in the date value. This is my desired output:

accidents_dict =[{
      "date": new Date(2017, 1, 1),
      "Vehicular Accident": 127,
      "Medical': 129,
      "OB": 10,
      "Mauling": 9
    }, {
      "date": new Date(2017, 2, 1),
      "Fall": 7,
      "Vehicular Accident': 113,
      "Mauling": 5,
      "OB": 9,
      "Medical": 79
    }, {
      "date": new Date(2017, 3, 1),
      "Medical": 112,
      "Mauling": 5,
      "OB": 11,
      "Vehicular Accident": 119,
      "Fall": 8
    }]

I tried this:

from itertools import groupby
from operator import itemgetter

for key, value in groupby(accidents_dict, key = itemgetter('date')):
  print(key)

  for k in value:
    print(k)

This was the result:

new Date (2017,1,1)
{'date': 'new Date (2017,1,1)', 'Fall': 5}
{'date': 'new Date (2017,1,1)', 'Vehicular Accident': 127}
{'date': 'new Date (2017,1,1)', 'Medical': 129}
{'date': 'new Date (2017,1,1)', 'OB': 10}
{'date': 'new Date (2017,1,1)', 'Mauling': 9}
new Date (2017,2,1)
{'date': 'new Date (2017,2,1)', 'Fall': 7}
{'date': 'new Date (2017,2,1)', 'Vehicular Accident': 113}
{'date': 'new Date (2017,2,1)', 'Mauling': 5}
{'date': 'new Date (2017,2,1)', 'OB': 9}
{'date': 'new Date (2017,2,1)', 'Medical': 79}
new Date (2017,3,1)
{'date': 'new Date (2017,3,1)', 'Medical': 112}
{'date': 'new Date (2017,3,1)', 'Mauling': 5}
{'date': 'new Date (2017,3,1)', 'OB': 11}
{'date': 'new Date (2017,3,1)', 'Vehicular Accident': 119}
{'date': 'new Date (2017,3,1)', 'Fall': 8}

I tried to iterate the list dictionary and get the value of accident key, but no luck. I also tried replace("''",'') in date value but still, it outputs a string. Hope you can help me. Thanks a lot.

the date value. I will convert this to json then pass it to javascript. — peter, Dec 30 '21 at 15:02
What do you mean by "remove the single quote"? The single quote isn't part of the value, it's just the way strings are shown. — Barmar, Dec 30 '21 at 15:49
After you use `itertools.groupby()`, merge all the dictionaries in each nested list. — Barmar, Dec 30 '21 at 15:51
JSON is data interchange format like csv, xml and not Language like python, javascript, json only support string, int, float, bool, null, it can't support datetime while passing from different language — Yuvaraja, Dec 30 '21 at 16:02
There is only way you need to format from date to String or int in python and parse into date in javascript or you use eval function javascript — Yuvaraja, Dec 30 '21 at 16:04
As far as I can see, there is no "single quote" within `new Date (2017, 3, 1)` (only commas). What is the __purpose of the list-transformation__, where is the desired or printed output consumed ? (For example: Do you want valid JSON to copy/paste or pipe-on ?) — hc_dev, Dec 31 '21 at 02:33

hc_dev · Accepted Answer · 2021-12-31T02:51:24.350

What you did

You managed to group the dictionaries by a key (date's value). Then, inside the iteration, you printed the key and the resulting grouped list of dictionaries. Good start!

What is missing

Inside the group-by iteration:

add the key to a new dictionary, e.g. as entry 'date': 'new Date (2017,1,1)'.
Loop through each dictionary in the grouper (term or generator used by itertools). For each dictionary:

exclude the entry with date key (because it already is added to the new)
but add all accident-summaries with their variable <accident-category> key and a number as value. See following illustration what needs to be added as key-value pairs:

{.. , 'Fall': 5}
{.. , 'Vehicular Accident': 127}
{.. , 'Medical': 129}
{.. , 'OB': 10}
{.. , 'Mauling': 9}

At the end, in the group-by iteration, add the new dictionary to a list (of course the list needs to be defined empty before the group-by block).

Solution

Warning: Lambda used instead operator.itemgetter

The key-extractor passed to argument key to function groupby can also be a lambda. I know this might be an advanced topic to get familiar with. Though, for a simple key-extraction and to get rid of the additional itemgetter dependency, it could be useful. Read more about lambda's in the tutorial below.

from itertools import groupby

accidents_stats = []
for js_date, grouper in groupby(accidents_dict, key=lambda element: element['date']):
  accidents_per_date = {date: js_date}  # this starts the new dict
  for d in grouper:  # here we iterate over each accident in group, as dict 
      for (k,v) in d.items():  # for each (key,value) pair in dict entries 
          if k != 'date':  # add only non-date entries
              accidents_per_date[k] = v
  accidents_stats.append(accidents_per_date)  # then add the new dict to a list

print(accidents_stats)

Prints:

[{'date': 'new Date (2017,1,1)', 'Fall': 5, 'Vehicular Accident': 127, 'Medical': 129, 'OB': 10, 'Mauling': 9}, {'date': 'new Date (2017,2,1)', 'Fall': 7, 'Vehicular Accident': 113, 'Mauling': 5, 'OB': 9, 'Medical': 79}, {'date': 'new Date (2017,3,1)', 'Medical': 112, 'Mauling': 5, 'OB': 11, 'Vehicular Accident': 119, 'Fall': 8}]

Bonus: a universal way to represent time in JSON

JSON is an acronym for "JavaScript Object Notation". As such JSON can be evaluated by JavaScript.

To represent timestamps, date or time in JSON we often use

Strings that follow a standard-format like ISO and are human-readable, but not automatically readable by all languages
Numbers that represent the milliseconds in (UNIX-based) epoch format

Since the number can be easily read by JavaScript, and other languages we can parse and convert the JS-constructor statement to epoch integer, using:

regular expression to extract the numbers (3 capture-groups within parentheses)
datetime.timestamp() function to construct a date in Python and convert to epoch in milliseconds (as required by JavaScript)

import re
import datetime

# JavaScript Date objects contain a Number that represents milliseconds since 1 January 1970 UTC. Also called (UNIX-) epoch format.
def js_code_to_millis(js_date_constructor):
    result = re.search(r"new Date\s+\((\d{4}),(\d{1,2}),(\d{1,2})\)", js_date_constructor)
    (year, month, day) = result.groups()
    epoch_seconds = datetime.datetime(int(year), int(month), int(day)).timestamp()
    return int(epoch_seconds * 1000)

Want valid JSON as output?

Then follow 4 adjustments

Add the js_code_to_millis function (including imports)
Include the JSON-compliant timestamp to new dictionary as entry 'date' : js_code_to_millis(js_date).
Import built-in JSON module as import json.
After or instead of printing the list, dump it as JSON-array using json.dumps(accidents_stats).

Wow. The solution works. Thanks for the guide @hc_dev, this is very helpful for me. — peter, Dec 31 '21 at 02:53

score 0 · Answer 2 · answered Dec 30 '21 at 16:17

You can try like this way using eval()

accidents_dict = [{
    'date': 'new Date (2017,1,1)',
    'Fall': 5
  },
  {
    'date': 'new Date (2017,1,1)',
    'Vehicular Accident': 127
  },
  {
    'date': 'new Date (2017,1,1)',
    'Medical': 129
  },
  {
    'date': 'new Date (2017,1,1)',
    'OB': 10
  },
  {
    'date': 'new Date (2017,1,1)',
    'Mauling': 9
  },
  {
    'date': 'new Date (2017,2,1)',
    'Fall': 7
  },
  {
    'date': 'new Date (2017,2,1)',
    'Vehicular Accident': 113
  },
  {
    'date': 'new Date (2017,2,1)',
    'Mauling': 5
  },
  {
    'date': 'new Date (2017,2,1)',
    'OB': 9
  },
  {
    'date': 'new Date (2017,2,1)',
    'Medical': 79
  },
  {
    'date': 'new Date (2017,3,1)',
    'Medical': 112
  },
  {
    'date': 'new Date (2017,3,1)',
    'Mauling': 5
  },
  {
    'date': 'new Date (2017,3,1)',
    'OB': 11
  },
  {
    'date': 'new Date (2017,3,1)',
    'Vehicular Accident': 119
  }, {
    'date': 'new Date (2017,3,1)',
    'Fall': 8
  }
];

output = accidents_dict.map((e) => {
  return { ...e,
    date: eval(e.date),
  };
});

console.log(output);

/*

[
  {
    "date": "2017-01-31T18:30:00.000Z",
    "Fall": 5
  },
  {
    "date": "2017-01-31T18:30:00.000Z",
    "Vehicular Accident": 127
  },
  {
    "date": "2017-01-31T18:30:00.000Z",
    "Medical": 129
  },
  {
    "date": "2017-01-31T18:30:00.000Z",
    "OB": 10
  },
  {
    "date": "2017-01-31T18:30:00.000Z",
    "Mauling": 9
  },
  {
    "date": "2017-02-28T18:30:00.000Z",
    "Fall": 7
  },
  {
    "date": "2017-02-28T18:30:00.000Z",
    "Vehicular Accident": 113
  },
  {
    "date": "2017-02-28T18:30:00.000Z",
    "Mauling": 5
  },
  {
    "date": "2017-02-28T18:30:00.000Z",
    "OB": 9
  },
  {
    "date": "2017-02-28T18:30:00.000Z",
    "Medical": 79
  },
  {
    "date": "2017-03-31T18:30:00.000Z",
    "Medical": 112
  },
  {
    "date": "2017-03-31T18:30:00.000Z",
    "Mauling": 5
  },
  {
    "date": "2017-03-31T18:30:00.000Z",
    "OB": 11
  },
  {
    "date": "2017-03-31T18:30:00.000Z",
    "Vehicular Accident": 119
  },
  {
    "date": "2017-03-31T18:30:00.000Z",
    "Fall": 8
  }
]
*/

Nice, it works. Now, I only need to merge them. Thanks Yuvaraja =) — peter, Dec 31 '21 at 02:31