0

I'm upserting Documents into a Collection using mongoimport. I'm using shelljs to just run the actual mongoimport command the same way it'd be run through a CLI.

E.g. mongoimport docs.json -d DB -c Collection --jsonArray --upsertFields=foo,bar This just outputs the # of Documents updated - but doesn't give any further details on what exactly was changed.

However, I need to be able to see which documents were actually updated/inserted and then do something with these results. I would've thought there'd be some way to get the resulting Object IDs of these documents (perhaps through some -v verbose flag) - but upon doing research, it surprisingly seems that there is no way.

Does anybody know how to get the updated Documents using mongoimport? If it's truly not possible, is there some way to do this using Mongoose? Also, is there any way to check a diff of what exactly was updated on a Document i.e. what fields changed on an upsert? Thanks!

user2402616
  • 1,434
  • 4
  • 22
  • 38

1 Answers1

1

I don't think it's possible to get such information from a bulk operation.

Using Mongoose, you'll have to iterate through every document and compare the current document and the new document to get the fields that are different. You can either:

  1. Make use of document.modifiedPaths() or document.directModifiedPaths() by getting the diff before you update each document
const doc = await MyModel.findById(newDoc._id) // or .findOne for other upsert query conditions
if (doc) {
  const { foo, bar } = newDoc // assuming foo, are are top level fields
  doc.set({ foo, bar })
  const modifiedPaths = doc.directModifiedPaths() // should work if foo, bar are not objects
} else {
  // no previous doc, insert...
  await MyModel.create(newDoc)
}
  1. Compare by yourself
const { foo, bar } = newDoc
const doc = await MyModel.findByIdAndUpdate(newDoc._id, { $set: { foo, bar } }, { new: false, upsert: true })
// or .findOneAndUpdate for other upsert query conditions
/* since we passed new: false, the operation will return the original document or null if not exists */

const { foo: newFoo, bar: newBar } = doc
// compare foo & newFoo, bar & newBar
thammada.ts
  • 5,065
  • 2
  • 22
  • 33
  • How efficient would this be when trying to upsert ~5k records into a 1/4M Document Collection though? The actual `mongoimport` version of this is instant with an index. There's only half a dozen fields per document. – user2402616 May 21 '20 at 19:42
  • 1
    You can execute them in parallel, It wouldn't be an instant but it shouldn't take more 30 seconds. probably just 5-10 seconds. – thammada.ts May 21 '20 at 20:01
  • Any suggestions on how to execute that in parallel. Only other parallel programming I've doing was in C using `threads` and in OpenCL – user2402616 May 21 '20 at 20:07
  • 1
    @user2402616 check out this [answer](https://stackoverflow.com/a/11488129) and [bluebird.js](http://bluebirdjs.com/docs/why-promises.html) – thammada.ts May 24 '20 at 21:06