1

I have two queues that both have distinct data types that affect one another as they're being processed by my application, therefore processing messages from the two queues asynchronously would cause a data integrity issue.

I'm curious as to the best practice for making sure only one consumer is consuming at any given time. Here is a summary of what I have so far:

EventMessages receive information about external events that may or may not have an impact on the enqueued/existing PurchaseOrderMessages.

Since we anticipate we'll be consuming more PurchaseOrderMessage than EventMessage, maybe we should just ensure the EventMessage Queue is empty (via the API) before we process anything in PurchaseOrderMessage Queue - but that gets into the question of wait times, etc. and this all needs to happen as close to real time as possible.

If there's a way to simply pause a Consumer A until Consumer B is at rest that might be the simplest solution, I'm just not quite sure which direction I need to go in.

UPDATE

To provide some additional context, a PurchaseOrderMessage will contain a Origin and Destination.

A EventMessage also contains location data.

Each time a PurchaseOrderMessage is processed, it will query the current EventMessage records for any Event locations that match the Origin and Destination of that PurchaseOrder and create an association.

Each time an EventMessage is processed, it will query the current PurchaseOrderMessage records for any Origins of Destinations that match that Event and create an association.

If synchronous queues aren't a good solution, what's an alternative that would insure none of the associations are missed when EventMessages and PurchaseOrderMessages are getting published to the app at the same time?

UPDATE 2

Ultimately this data will serve a UI which will have a list of PurchaseOrders and the events that might be affecting their delivery dates. It would be too slow to do the "Event Check" as the PurchaseOrder data was being rendered/retrieved by the end user which is why we're wanting to do it as they're processed/consumed.

daniel9x
  • 755
  • 2
  • 11
  • 28
  • It's not exactly clear what portion of your messages is requiring synchronization. My guess is that you'll need to re-tool the design of the services a little. – theMayer Nov 30 '17 at 03:11
  • What is the purpose of storing the associations? – theMayer Nov 30 '17 at 15:44
  • Another great question! I have updated the question with that info. – daniel9x Nov 30 '17 at 15:52
  • That's fine, but I still don't understand. What does the Purchase Order queue accomplish, and why would an event affect something in the queue? – theMayer Nov 30 '17 at 16:34
  • To continue with the DisneyExample theme, I'm ordering some Mickey Mouse slippers (PurchaseOrder) from the Online Disney Store that's shipping from LA to NY but it turns out there's a UPS Strike (Event) in the NY distribution center which might affect my delivery estimate. I want to be able to look at all my orders and see the events that might impact them getting to their destination on time – daniel9x Dec 01 '17 at 14:58
  • Is your queue for purchase orders a queue to create them? If so, how could there be events affecting a purchase order that has not technically been created yet? – theMayer Dec 01 '17 at 16:15
  • No, the purchase orders exist in a separate system. This service simply retains a snapshot of the transaction with the related (event) associations. – daniel9x Dec 01 '17 at 16:22
  • Then what operation is your queue for? You still haven't clarified that aspect of it. – theMayer Dec 01 '17 at 16:45
  • Maybe it will help by prefacing that we have a MicroServices structure and this queue is specifically for a service whose primary purpose is to save a snapshot/summarized version of the purchase order for purposes of a UI that will allow and end user to see the basic attributes of that order and the related events that might be affecting it's estimated delivery date. – daniel9x Dec 01 '17 at 19:53
  • OK, so if I understand correctly, this service is building your "associations" -> and your concern is that there might be an association out there that it doesn't know about when it builds these? – theMayer Dec 01 '17 at 20:06
  • Exactly. If an event is being consumed at the same time a purchase order is being consumed, they might not come up in their respective “searches” – daniel9x Dec 01 '17 at 20:08
  • How is this any different from the case where an event happens after this "summary" has been done? I am assuming you know that you can't have a circular dependency (i.e. a purchase order object cannot depend on an event object if the event object also depends on the purchase order object). – theMayer Dec 01 '17 at 20:10
  • They’re not dependencies. Events will get recorded and purchase orders will get recorded even if there are no associations to create. The association table will just have a list of where events and orders that relate to each other. A many to many relationship. – daniel9x Dec 01 '17 at 20:12
  • We are going around in circles here. I am really struggling to understand what it is that is going on, such that you need queues to be synchronized. Such a need implies a serious defect in implementation, design, or domain model. – theMayer Dec 01 '17 at 20:18
  • Okay, I'll try to summarize it one last time. We have a service whose main object is to consume Event Data and Purchase Order Data and create associations where they intersect. Users can see "All Events", "All Orders" and see what Orders are impacted by a particular event, and what events might be impacted a specific order. The events are coming from an external 3rd party system so this is how we need to ingest them. We are receiving the PurchaseOrders through another system within our architecture but we use the message queue flow throughout our infrastructure and wanted to keep doing that. – daniel9x Dec 01 '17 at 20:27
  • I recommend the use of a configurable big data store. [Couchbase](https://www.couchbase.com/products/server) has a facility called "Views" which would achieve this automatically for you. All you need to do is store the objects with enough information that they can be readily queried by a view. I don't think a message queue is the tool for this job. – theMayer Dec 02 '17 at 02:56
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/160323/discussion-between-themayer-and-daniel9x). – theMayer Dec 02 '17 at 02:56

1 Answers1

4

Let me begin with the bottom line up front - on the face of it, what you are asking doesn't make sense.

Queues should never require synchronization. The very thought of doing so entirely defeats the purpose of having a queue. For some background, visit this answer.

Let's consider some common places from real life where we encounter multiple queues:

  1. Movie theaters (box office, concession counter, usher)
  2. Theme parks (snack bars, major attractions)
  3. Manufacturing floors (each station may have a queue waiting to process)

In each of these examples, from the point of view of the object in the queue, it can only wait in one at a time. It cannot wait in one line while it is waiting in another- such a thing is physically impossible.

Your example seems to take two completely unrelated things and merge them together. You have a queue for PurchaseOrder objects - but what is the queue for? That's the equivalent of going to Disney World and waiting in the Customer queue - what is the purpose of such a queue? If the purpose is not clear, it's not a real queue.

Addressing your issue

This particular issue needs to be addressed first by clearly defining the various operations that are being done to a PurchaseOrder, then creating queues for each of those operations. If these operations are truly synchronous, then your business logic should be coded to wait for one operation to complete before starting another. In this circumstance, it would be considered an exception if a PurchaseOrder got to the head of one queue without fulfilling a pre-requisite.

Please remember that a message queue typically serves a stateless operation. Good design dictates that messages in the queue contain all the information needed for the processor to process the message. If you don't adhere to this, then your database becomes a single point of contention for your system - and while this is not an insurmountable problem, it does make the design more complex.

Waiting in Multiple Queues

Now, if you've ever been to Disney World, you'll also know that they have something called a FastPass+ (FP+), which allows the holder to skip the line at the designated attraction. Disney allocates a certain number of slots per hour for each major attraction at the park, and guests are able to request up to three FP+s during each day. FP+ times are allocated for one hour blocks, and guests cannot have two overlapping FP+ time blocks. Once all FP+ slots have been issued for the ride, no more are made available. The FP+ system ensures these rules are enforced, independently of the standby queues for each ride. Essentially, by using FastPass+, guests can wait in multiple lines virtually and experience more attractions during their visit.

If you are unable to analyze your design and come up with an alternative, perhaps the FastPass+ approach could help alleviate some of the bottlenecks.

Disclaimer: I don't work for Disney, but I do go multiple times per month, always getting my FastPass first

theMayer
  • 15,456
  • 7
  • 58
  • 90
  • Thanks for the awesome response. You're right, I need to explain the purpose of the queues better and the processes that follow afterwards. I've updated the original question with that info. – daniel9x Nov 30 '17 at 13:42