14

My Code

I have a MongoDB with two collections, Items and Calculations. 

Items
  value:  Number
  date:   Date

Calculations
  calculation:  Number
  start_date:   Date
  end_date:     Date

A Calculation is a stored calcluation based off of Item values for all Items in the DB which have dates in between the Calculation's start date and end date.

Mongo Change Streams

I figure a good way to create / update Calculations is to create a Mongo Change Stream on the Items collection which listens for changes to the Items collection to then recalculate relevant Calculations.

The issue is that according to the Mongo Change Event docs, when a document is deleted, the fullDocument field is omitted which would prevent me from accessing the deleted Item's date which would inform which Calculations should be updated.

Question

Is there any way to access the fullDocument of a Mongo Change Event that was fired due to a document deletion?

Alex Crist
  • 1,059
  • 2
  • 12
  • 22
  • if Items/date combination is unique you could make that your primary key (_id) and then it's available no matter what operation is performed on it. – Asya Kamsky Jul 31 '20 at 20:35

3 Answers3

5

No I don't believe there is a way. From https://docs.mongodb.com/manual/changeStreams/#event-notification:

Change streams only notify on data changes that have persisted to a majority of data-bearing members in the replica set.

When the document was deleted and the deletion was persisted across the majority of the nodes in a replica set, the document has ceased to exist in the replica set. Thus the changestream cannot return something that doesn't exist anymore.

The solution to your question would be transactions in MongoDB 4.0. That is, you can adjust the Calculations and delete the corresponding Items in a single atomic operation.

kevinadi
  • 13,365
  • 3
  • 33
  • 49
  • 2
    actually the solution would be to include immutable fields as part of the document key (i.e. the _id). – Asya Kamsky Jul 31 '20 at 20:34
  • @AsyaKamsky Can you clarify what you mean with a link to an example (or by adding your own answer)? Are you suggesting using [`createIndex`](https://docs.mongodb.com/manual/reference/method/db.collection.createIndex/), or manually setting the `_id` field on every `insert`, or something else? – Altay_H Feb 25 '21 at 23:16
  • I mean that if the _id field includes the date as well as whatever item id is, then you'd know which date's calculation needs to be updated. – Asya Kamsky Mar 10 '21 at 21:52
2

fullDocument is not returned when a document is deleted.

But there is a workaround.

Right before you delete the document, set a hint field. (Obviously use a name that does not collide with your current properties.)

await myCollection.updateOne({_id:theId}, {_deleting: true})
await myCollection.deleteOne({_id:theId})

This will trigger one final change event in the stream, with a hint that the document is getting deleted. Then in your stream watcher, you simple check for this value.

stream.on('change', event => {
  if (!event.fullDocument) {
    // The document was deleted
  }
  else if (event.fullDocument._deleting) {
    // The document is going to be deleted
  }
  else {
   // Document created or updated
  }

})

My oplog was not getting updated fast enough, and the update was looking up a document that was already removed, so I needed to add a small delay to get this working.

myCollection.updateOne({_id:theId}, {_deleting: true})
setTimeout( ()=> {
  myCollection.deleteOne({_id:theId})
}, 100)

If the document did not exist in the first place, it won't be updated or deleted, so nothing gets triggered.

Using TTL Indexes

Another way to make this work is to add a ttl index, and then set that field to the current time. This will trigger an update first, and then delete the document automatically.

myCollection.setIndex({_deleting:1}, {expireAfterSeconds:0})

myCollection.updateOne({_id:theId}, {$set:{_deleting:new Date()}})

The problem with this approach is that mongo prunes TTL documents only during certain intervals, 60s or more as stated in the docs, so I prefer to use the first approach.

Steven Spungin
  • 27,002
  • 5
  • 88
  • 78
0

In Mongo v6.0, you are now able to access pre- and post-operation images of documents in change events. You need to enable this on your watched collection before you can access the pre-document. Once you do that, you can include it in your options object that you pass to the change stream. For deleted documents, fullDocumentBeforeChange is what you're looking for - you access it the same way you access fullDocument in the change event currently.

abischolz
  • 11
  • 3