1

This is the scenario: Using AWS Kinesis > To get records from Kinesis I need a shard iterator > Kinesis won't return a new shard iterator until a request is complete > The call to "getRecords" is asynchronous > Attempt to iterate this process fails because the request hasn't resolved before the shard iterator is needed

const getRecordsInShard = (shard, batchSize, streamName) => {
  const records = [];
  const loop = (response) => {
    if (_.isEmpty(response.NextShardIterator)) {
      return Promise.resolve(records);
    }
    records.push(response.Records);
    return getRecordsByShardIterator(response.NextShardIterator, batchSize).then(loop);
  };

  return getStreamIterator(shard.ShardId, shard.SequenceNumberRange.StartingSequenceNumber, (streamName || process.env.KINESIS_STREAM))
  .then(response => getRecordsByShardIterator(response.ShardIterator, batchSize))
  .then(loop);
};

The above code fails because the promise returned from loop doesn't resolve before the return of the super function. How can I iterate a promise function sequentially, using the return value from the previous iteration as the next's input?

Caveats: Each iteration relies on the information returned by the previous, the result needs to collect records from each iteration altogether

Drakken Saer
  • 859
  • 2
  • 10
  • 29
  • 1
    [This thread](https://stackoverflow.com/questions/24586110/resolve-promises-one-after-another-i-e-in-sequence) might be what you need. – saille Sep 28 '17 at 01:11
  • What do you mean by "*doesn't resolve before the return of the super function*"? How does it fail exactly? – Bergi Sep 28 '17 at 01:16
  • Possible duplicate of [Resolve promises one after another (i.e. in sequence)?](https://stackoverflow.com/questions/24586110/resolve-promises-one-after-another-i-e-in-sequence) – alexmac Sep 28 '17 at 06:13
  • @Bergi It returns a promise which falls into the single threaded event loop. Because JavaScript is single threaded, execution continues without processing the event loop first (the whole point of promises). So what I'm saying is, the function resolves to an event loop promise, but not the result of the promise itself (that executes long after I need it). – Drakken Saer Sep 28 '17 at 17:41
  • 1
    @DrakkenSaer A function doesn't "resolve", it returns. In your case, it returns a promise. A promise is not a magical value that transforms into the result it is resolved with, or blocks the event loop upon being returned from a function. If you are doing something asynchronous, it means that you *cannot* get the result immediately. If you "need" it, you have to invent time travel first. – Bergi Sep 28 '17 at 17:44
  • @saille Thank you for the link, however I don't just need to loop through promises sequentially, I also need to resolve them before the next promise (basically, I ALSO need their respective return values, instead of just the promise's side effects). In addition to that, I am also not working with an array of promises (as I can't get the input for the next promise until the previous one has finished). – Drakken Saer Sep 28 '17 at 17:46
  • @Bergi Yes, I know this. That is the problem I am having. Unfortunately I do not have control of the API – Drakken Saer Sep 28 '17 at 17:47
  • @DrakkenSaer If you know this, you should also know that there is no solution. Whatever calls `getRecordsInShard`, it needs to consume the result promise asynchronously itself. – Bergi Sep 28 '17 at 17:48
  • @Bergi I can't form another call without the information from the previous call So it cannot asynchronously consume, that is the problem. – Drakken Saer Sep 28 '17 at 17:51
  • @DrakkenSaer But you *are* consuming it asynchronously, by placing the `loop` call in the `.then(…)`, getting the resolution of `getRecordsByShardIterator` as the argument to `loop`! The code works, does it not? Are you getting any error? – Bergi Sep 28 '17 at 18:43

1 Answers1

-1

I don't know why it went so complicated when the library provider give perfect examples to consume kinesis stream.

https://github.com/awslabs/amazon-kinesis-client-nodejs/tree/master/samples

It has basic samples and clickstream sample.

Hope it helps.

Kannaiyan
  • 12,554
  • 3
  • 44
  • 83