data aggregation by time window with JS

Question

I built a simple NodeJS server that emits client events. An event is a JSON object that looks like this:

{
  "date": "2019-12-12T09:55:05.679Z",
  "clientId": "Client01",
  "ip": "10.1.1.1",
  "event": "someEvent"
}

An event can be emitted anytime. I get many messages with different timestamps (date).

Currently, I store all the data in memory.

How I can aggregate the client events by date?

let's say in 15-minute bulks, so I end up with a total number of client events for each time frame, for example:

const timestamps = ["2019-12-12T09:00:00.000Z", "2019-12-12T09:15:00.000Z", "2019-12-12T09:30:00.000Z"];
const totalNumEvents = [50, 27, 82];
const clientIds = ["Client01", "Client02", "Client03"];

Sorry if the question is too generic, I tried to look for direction by googling but couldn't find any solution without using a framework / DB like MongoDB.

So far, what I did is, creating an object for each clientId and push the event into it. (I have a callback for each event)

const onEventArrived = (client) => {
      if (!clients[client.clientId]) {
        console.log(`[New] Client Added - ${client.clientId}`);
        clients[client.clientId] = [{ date, client}];
      } else {
        console.log(`[Exist] New Client Message for ${client.clientId}`);
        clients[client.clientId].push({ date, client});
      }
});

Back to question, I have all the events for each client, but how do I aggregate the random times into fixed 15 minute windows?

It is a bit of a vague question. Why not just push the object to an array and use reduce? — Imre_G, Dec 12 '19 at 12:09
what if you have no events in some quarter? Also can't you just implement it yourself? And also, I would assume that you don't send events with a date in the past (like a quarter in the past), is it correct? — grodzi, Dec 12 '19 at 12:09
@Imre_G I have pushed the object into an array for each client Id. How would you use reduce to aggregate it by time? — TheUnreal, Dec 12 '19 at 12:11
@grodzi If I have no events the counter for the clientId should be simply 0 at the time period (or if there are no events at all, I don't know it exists) — TheUnreal, Dec 12 '19 at 12:12

score 0 · Answer 1 · answered Dec 12 '19 at 12:30

0

You will need to store the events as a flat object in an array and use Array.reduce() or lodash's groupBy() to group the array of event objects by any of the desired fields.

Here's an example using your supplied event example

Your incoming event looks like this:

{
  "date": "2019-12-12T09:55:05.679Z",
  "clientId": "Client01",
  "ip": "10.1.1.1",
  "event": "someEvent"
}

Now when this event is received, here's what happens:

const events = []
const onEventArrived = (event) => {
     events = [...events, event ]
});

Next you run a group by as follows:

Array.prototype.groupBy = function(k) {
  return this.reduce((acc, item) => ((acc[item[k]] = [...(acc[item[k]] || []), item]), acc),{});
};

events.groupBy("date")
// OR
events.groupBy("clientId")

You can extend this to fit your use case

Reference: How to group an array of objects by key

answered Dec 12 '19 at 12:30

Samuel_NET

345
3
9

It's not my case, your solution will create a different group for each date, and I can get a new event every second. I want to aggregate all the events within a specific time window, in my case, 15 minute groups. Aggregating all events in the same second will not work well. – TheUnreal Dec 12 '19 at 13:03
1

@TheUnreal this works. Just add a key to every event object caled `dateSlot` or something and round the timestamp to quarters. Then, you can use groupBy or reduce as described here. – Imre_G Dec 12 '19 at 13:45
@TheUnreal oh yes it works for your use case. You can use this utility tool to do your checks. https://date-fns.org/v2.8.1/docs/isWithinInterval This allows you evaluate your supplied interval. However, please note that you may need to extracts the events within a regular interval using some scheduled function and then update the groupBy function to check if `k === "date"` to do a preprocessing to establish that you only need the events in a predefined interval or just follow @Imre_G's advice – Samuel_NET Dec 12 '19 at 14:37

score 0 · Answer 2 · answered Dec 12 '19 at 12:48

An idea can be to:

associate an id to each date (a quarterId)
get the minQuarterId, the maxQuarterId and fill those in between with 0

const bulks = {}
const clientIds = new Set
function onev(ev) {
  clientIds.add(ev.clientId)
  const quarters = ev.date.getTime() / (15 * 60 * 1000)
  const quarterId = Math.floor(quarters)
  bulks[quarterId] = (bulks[quarterId] || 0) + 1
}

function restitute() {
  if(!Object.keys(bulks).length){ return { ts: [], nevs: [] } }
  const ts = []
  const nevs = []
  const {min, max} = Object.keys(bulks).reduce((acc, k) => {
    const pk = parseInt(k)
    if(pk < acc.min){
      acc.min = pk
    }
    if (pk > acc.max) {
      acc.max = pk
    }
    return acc
  }, {min:1e15, max:0})
  for(let i = min; i <= max; ++i){
    ts.push(new Date(i * 15 * 60 * 1000))
    nevs.push(bulks[i] || 0) // fill with 0 if no quarterId "i" exists
  }
  return {ts, nevs}
}


onev({clientId: 1, date:new Date(2019, 11, 12, 13, 24)})
onev({clientId: 2, date:new Date(2019, 11, 12, 13, 28)})
onev({clientId: 1, date:new Date(2019, 11, 12, 13, 29, 30)})
onev({clientId: 1, date:new Date(2019, 11, 12, 13, 31, 30), newquarter:1})
onev({clientId: 4, date:new Date(2019, 11, 12, 13, 29, 30)})
onev({clientId: 4, date:new Date(2019, 11, 12, 12, 29, 30), emptyquarters:1})
onev({clientId: 4, date:new Date(2019, 11, 12, 12, 45, 30), indapast:1})
console.log(restitute())

data aggregation by time window with JS

2 Answers2