Fast and performance based way to merge objects from three arrays onto one object in javascript

Question

I have an array of objects that has a few properties in javascript that has a basic structure like so:

{
  id: 123-456
  username: sdfjweihg
}

then I have three, very large arrays that are unsorted, all of them are arrays of objects, but the objects all have all sorts of different matching fields, however they all have an acccount_id that matches the id from the above object.

The above objects also come in an array, so I was doing something like:

let fullUsers = [];
for(let i = 0; i < users.length; i++) {
   fullUser.push({
      user: users[i],
      favorite_movie: moviesOfusers.filter((movie) => movie.account_id === users[i].id),
      favorite_toys: toysOfusers.filter((toy) => toy.account_id === users[i].id),
      favorite_colors: colorsOfUsers.filter((color) => color.account_id === users[i].id)
   )};
}

Where moviesOfusers, toysOfUsers and colorsOfUsers are the arrays that have all sorts of other properties on them, but that matching account_id. It works, and does what I want, but I need something less CPU intensive and faster. If anyone has any suggestions to achieve the same search and attaching that's better I'd love some new ideas. Thank you!

Where do the *very large* arrays come from? The thing about arrays is that you can’t do much to get away from iterating over them, aside from reducing the big ones just one time each using all the known account ids. Honestly, though, it sounds like whatever you’re doing would be better off done in SQL. — Adam Jenkins, Jan 05 '23 at 22:46
Give the answer on [group by](https://stackoverflow.com/questions/14446511/most-efficient-method-to-groupby-on-an-array-of-objects) a read first. But then I suppose you could go the other way: iterate over all the moviesOfUsers, and locate the user with the corresponding [id] and push it into the collection for that user. Locating the user quickly could be done with an index. for each user, `user_index[user.id] = user` — Wyck, Jan 05 '23 at 22:57
I would get the data as i want from api. Or i would get the extra data as object not array. That will have O(1). — atilkan, Jan 05 '23 at 23:19
@Adam Yeah, you hit the nail on the head, but there's a constraint there unfortunately blocking me from using that solution too. I might just be hitting a hardware limit and need to up the resources at this point. I was just burnt and spinning my tires earlier and didn't think to reduce them. Might be enough, but either way it'll be a step in the right direction. — williambutcherthe99th, Jan 06 '23 at 03:58

score 1 · Accepted Answer · answered Jan 05 '23 at 22:56

Best you can do is reduce each of the large arrays in a single iteration each to a map...

const colorsByAccountId = colorsOfUsers.reduce((acc,color) => {
   const { account_id } = color
   if(acc[account_id]) {
     acc[account_id].push(color)
   } else {
     acc[account_id] = [color]
   }
   return acc
},{})

and then...

const fullUsers = users.map((user) => ({
   user,
   colors: colorsByAccountId[user.id] || [],
   ...other stuff
})

Not sure if that'll get you enough of a gain or not.

I'll report back tomorrow when I give it a test, really appreciate your suggestion Adam thank you! — williambutcherthe99th, Jan 06 '23 at 03:51

score 1 · Answer 2 · answered Jan 05 '23 at 22:56

You can precompute bookkeeping arrays of account IDs to movies, toys, and cars:

const userToMovie = {}
const userToToy = {}
const userToColor = {}
moviesOfusers.forEach(movie => { userToMovie[movie.account_id] = movie })
toysOfusers.forEach(toy => { userToToy[toy.account_id] = toy })
colorsOfUsers.forEach(color => { userToColor[color.account_id = color })

let fullUsers = [];
for(let i = 0; i < users.length; i++) {
   fullUsers.push({
      user: users[i],
      favorite_movie: userToMovie[users[i].id],
      favorite_toys: userToToy[users[i].id],
      favorite_colors: userToColor[users[i].id]
   )};
}

cool, I'll give this one a go as well too thank you! – williambutcherthe99th Jan 06 '23 at 03:52 — williambutcherthe99th, Jan 06 '23 at 03:52

epascarello · Answer 3 · 2023-01-05T23:21:07.430

Personally I would only map the users into a keyed set and then use that to loop over the other items. This way

You loop over the users once (reduce)
You loop over each data set once (forEach X each dataset)
You look over the values of the object once. (Object.keys)

/* Data */

const users = [
  {id: 1, name: 'Bob'}, 
  { id: 2, name: 'Al' }
];

const toys = [
  { id:1, name: 'foo'},
  { id:1, name: 'bar'},
  { id:2, name: 'baz'}
];

const books = [
  {id: 1, name: 'book1'}
]

/* First step, make the easy loop up by id for the users */
const mappedById = users.reduce((acc, user) => {
  acc[user.id] = {
    user,
    toys: [],
    books: []
  };
  return acc;
}, {});

/* Simple function what will append the item to the array */
const setUserData = (mappedById, items, key) => {
  items.forEach(item => mappedById[item.id] ? .[key] ? .push(item))
}

/* Look over the sub data sets */
setUserData(mappedById, toys, 'toys');
setUserData(mappedById, books, 'books');

/* Get the final array out of the users */
const result = Object.values(mappedById);

console.log(result);

Other way is to map all of the data sources and combine them when you loop over the users.

Loop over each dataset (reduce X each dataset)
Loop over the user dataset (forEach)

/* Data */
const users = [
  {id: 1, name: 'Bob'}, 
  { id: 2, name: 'Al' }
];

const toys = [
  { id:1, name: 'foo'},
  { id:1, name: 'bar'},
  { id:2, name: 'baz'}
];

const books = [
  {id: 1, name: 'book1'}
]


// Function to build array by user id for the sub data
const setUserData = (acc, item) => {
  acc[item.id] = acc[item.id] || [];
  acc[item.id].push(item);
  return acc;
}

/* Look over the sub data sets */
const toysMapped = toys.reduce(setUserData, {});
const booksMapped = books.reduce(setUserData, {});


/* Build the user array with the sub data */
const result = users.map((user) => ({
  user,
  toys: toysMapped[user.id] || [],
  books: booksMapped[user.id] || []
}));

console.log(result);

Fast and performance based way to merge objects from three arrays onto one object in javascript

3 Answers3