Trying to do a bulk upsert with Mongoose. What's the cleanest way to do this?

Question

I have a collection that holds documents that contains three fields: first_name, last_name, and age. I'm trying to figure out what query in Mongoose I can use to do a bulk upsert. My app is occasionally receiving a new array of objects with those same three fields. I want the query to check if the first AND last name already exist within a document, and if they do - update the age if it's different. Otherwise, if the first and last name don't exist, insert a new document.

Currently, I'm only doing the import - and haven't yet built out the logic for this upsert piece.

app.post('/users/import', function(req, res) {
  let data = req.body;
  let dataArray = [];
  data.forEach(datum => {
    dataArray.push({
        first: datum.first,
        last: datum.last,
        age: datum.age
    })
})

User.insertMany(dataArray, answer => {
    console.log(`Data Inserted:`,answer)
})

`

And my User model looks like this:

const mongoose = require('mongoose');

const Schema = mongoose.Schema;

const userSchema = new Schema({
  first: String,
  last: String,
  age: Number,
  created_at: { type: Date, default: Date.now }
});

var User = mongoose.model('User', userSchema);
module.exports = User;

Answer 1

(mongoose@4.9.1, mongodb@3.4.2)

TL;DR

await GasStation.collection.bulkWrite([ // <<==== use the model name
  {
    'updateOne': {
      'filter': { 'id': '<some id>' },
      'update': { '$set': { /* properties to update */ } },
      'upsert': true,  // <<==== upsert in every document
    }
  },
  /* other operations here... */
]);

Long story:

After struggling with Mongoose API poor documentation, I solved the bulk upsert tweaking updateOne:{} operation in the bulkWrite() method.

A couple of undocumented things to consider:

// suppose:
var GasStation = mongoose.model('gasstation', gasStationsSchema);
var bulkOps = [ ];

// for ( ... each gasStation to upsert ...) {
  let gasStation = { country:'a', localId:'b', xyz:'c' };
  // [populate gasStation as needed]
  // Each document should look like this: (note the 'upsert': true)
  let upsertDoc = {
    'updateOne': {
      'filter': { 'country': gasStation.country, 'localId': gasStation.localId },
      'update': gasStation,
      'upsert': true
  }};
  bulkOps.push(upsertDoc);
// end for loop

// now bulkWrite (note the use of 'Model.collection')
GasStation.collection.bulkWrite(bulkOps)
  .then( bulkWriteOpResult => {
    console.log('BULK update OK');
    console.log(JSON.stringify(bulkWriteOpResult, null, 2));
  })
  .catch( err => {
    console.log('BULK update error');
    console.log(JSON.stringify(err, null, 2));
  });

The two key things here are incomplete API documentation issues (at the time of writing, at least):

• 'upsert': true in each document. This is not documented in Mongoose API (), which often refers to node-mongodb-native driver. Looking at updateOne in this driver, you could think to add 'options':{'upsert': true}, but, no... that won't do. I also tried to add both cases to the bulkWrite(,[options],) argument, with no effect either.
• GasStation.collection.bulkWrite(). Although Mongoose bulkWrite() method claims it should be called Model.bulkWrite() (in this case, GasStation.bulkWrite()), that will trigger MongoError: Unknown modifier: $__. So, Model.collection.bulkWrite() must be used.

Additionally, note:

• You don't need to use the $set mongo operator in the updateOne.update field, since mongoose handles it in case of upsert (see bulkWrite() comments in example).
• Note that my unique index in the schema (needed for upsert to work properly) is defined as:

gasStationsSchema.index({ country: 1, localId: 1 }, { unique: true });

Hope it helps.

==> EDIT: (Mongoose 5?)

As noticed by @JustinSmith, the $set operator added by Mongoose doesn't seem to be working anymore. Maybe it's because of Mongoose 5?

In any case, using $set explicitly should do:

'update': { '$set': gasStation },

Answer 2

Thank @maganap. I used his/her answer and reached below concise approach:

await Model.bulkWrite(docs.map(doc => ({
    updateOne: {
        filter: {id: doc.id},
        update: doc,
        upsert: true,
    }
})))

---

Or more verbose:

const bulkOps = docs.map(doc => ({
    updateOne: {
        filter: {id: doc.id},
        update: doc,
        upsert: true,
    }
}))

Model.bulkWrite(bulkOps)
        .then(console.log.bind(console, 'BULK update OK:', bulkWriteOpResult))
        .catch(console.error.bind(console, 'BULK update error:'))

Answer 3

I have released a small plugin for Mongoose that exposes a static upsertMany method to perform bulk upsert operations with a promise interface. This should provide a very clean way of doing bulk upserts with Mongoose, while retaining schema validation etc:

MyModel.upsertMany(items, ['matchField', 'other.nestedMatchField']);

You can find this plugin on npm or Github:

https://github.com/meanie/mongoose-upsert-many https://www.npmjs.com/package/@meanie/mongoose-upsert-many

Answer 4

I tried @magnap's solution above and found that it was overwriting the currently existing documents that I simply wanted to update. Rather than updating the fields I set in updates.updateOne, it was selecting the document and replacing all of its fields with the ones specified in .update.

I eventually had to use $set in my update method to resolve this. Here's what my controller ended up looking like:

const { ObjectId } = require('mongodb');

exports.bulkUpsert = (req, res, next) => {
     const { updates } = req.body;
     const bulkOps = updates.map(update => ({
         updateOne: {
             filter: { _id: ObjectId(update.id) },
             // Where field is the field you want to update
             update: { $set: { field: update.field } },
             upsert: true
          }
      }));
    // where Model is the name of your model
    return Model.collection
        .bulkWrite(bulkOps)
        .then(results => res.json(results))
        .catch(err => next(err));
};

This works with Mongoose 5.1.2.

Answer 5

you can use array.map instead use for

 const result = await Model.bulkWrite(
    documents.map(document => {

        document = {
          ...document, 
          ...{
            last_update: Date.now(),
            foo: 'bar'
          }
        }

        return {
          updateOne: {
            filter: { document_id: document.document_id }, //filter for each item
            update: {
              $set: document,//update whole document
              $inc: { version: 1 }//increase version + 1
            },
            upsert: true //upsert document
          }
        }

      }
    ));

Answer 6

Hope my answer HERE helps you out. It deals with bulk upsert for an ecommerce domain asynchronously

Answer 7

Found the official solution on: https://docs.mongodb.com/manual/reference/method/Bulk.find.upsert/

And Mongoose also supports same chain.

Bulk.find(<query>).upsert().update(<update>);
Bulk.find(<query>).upsert().updateOne(<update>);
Bulk.find(<query>).upsert().replaceOne(<replacement>);

Tested it works:

BulkWriteResult {
  result:
   { ok: 1,
     writeErrors: [],
     writeConcernErrors: [],
     insertedIds: [],
     nInserted: 0,
     nUpserted: 1,
     nMatched: 4186,
     nModified: 0,
     nRemoved: 0,
     upserted: [ [Object] ] } }

Answer 8

check this i hope this will helpfull for you link

link2

I think you are looking for the

Bulk.find().upsert().update()

yo can use this

bulk = db.yourCollection.initializeUnorderedBulkOp();
for (<your for statement>) {
    bulk.find({ID: <your id>, HASH: <your hash>}).upsert().update({<your update fields>});
}
bulk.execute(<your callback>)

• If it finds one, it will update that document using {}
• Otherwise, it will create a new document