How can I make a callback that requires info of its child function

Question

I am working in a project that uses Node.js for a Haraka (an smtp server) plugin.

This is Node.JS and I have a little problem whith callbacks. I haven't been able to convert this particular code to use a callback.

So, this is the code that I have:

exports.hook_data = function (next, connection) {
    connection.transaction.add_body_filter('', function (content_type, encoding, body_buffer) {
        var header = connection.transaction.header.get("header");
        if (header == null || header == undefined || header == '') return body_buffer;

        var url = 'https://server.com/api?header=' + header ;
        var request = require('request');
        request.get({ uri: url },
          function (err, resp, body) {
              var resultFromServer = JSON.parse(body);
              return ChangeBuffer(content_type, encoding, body_buffer, resultFromServer);
          }
        );
    });
    return next();
}

This code does not work because It doesn't wait the callback of the Request to continue. I need to finish the request before next();

And these are the requirements:

At the end of exports.hook_data is mandatory to return next(). But it only have to return it after the request.
I need to return a Buffer in add_body_filter but I need to create the buffer with information getted from a server.
To make the Get Request I need to use a parameter (header) that I only have inside add_body_filter.

So the issue is that I can not make the request before add_body_filter and put the result inside a callback because the parameter that I need to make the request are only inside add_body_filter.

Any advice please?

You can't use an async function in Javascript, but demand that it behave synchronously. That is simply NOT possible. If you have an async operation, then you MUST code to use it asynchronously. That means returning results in a promise or via a callback, NOT via the return value from the function because the function will return BEFORE the async operation is even complete. You would probably benefit from reading this [How to return response from asynchronous operation](http://stackoverflow.com/questions/14220321/how-do-i-return-the-response-from-an-asynchronous-call/14220323#14220323). — jfriend00, Dec 15 '15 at 22:30
And where do you use `body_buffer` (see the line #4) and the result of `ChangeBuffer(content_type, encoding, body_buffer, resultFromServer)` (see the line #11) which you return? I guess nowhere, right? — Alexander Elgin, Dec 18 '15 at 07:22
@AlexanderElgin the return of `add_body_filter` is a buffer, this buffer contains the body of an email. I dont use it directly, the server uses it when processing an email. I write ChangeBuffer so I can pass a callback If needed. — Ricardo Polo Jaramillo, Dec 18 '15 at 12:51
Can you use Async.js or Promises to control the flow? Otherwise, you'll just end up with a really hacky solution to be honest. — Quy, Dec 20 '15 at 03:23
@Quy Are there any counterproductive things of using something like That or some kind of syncronouz request? It will impact in the scalability of the solution in some way? I have heard that you should always be asynchronous in Node.js — Ricardo Polo Jaramillo, Dec 20 '15 at 03:27
@RicardoPolo A lot of things are async, but in order for this to work you need to call the async functions in order to get your desired behavior. All the async calls you make from this function will still get queued in the event loop and get called eventually with all the other IO tasks. Node will take care of that for you. Your current thread, I can't speak much about since I don't know the architecture of your app or what you're doing. For an HTTP server using express, it'll handle the threading on all the requests so you never really have to worry about it. Hope this helps. — Quy, Dec 20 '15 at 03:36
Can you tell us how `hook_data` function is called? How the caller attempts to use its results? — xersiee, Dec 22 '15 at 21:46
@RicardoPolo please, add an example in your question of how you suppose to use `hook_data` function. Is it intended to create a queue of data readers that is filled with loop? — Eugene Tiurin, Dec 23 '15 at 18:17
better look at some libs like https://github.com/caolan/async — Marcel Djaman, Dec 24 '15 at 16:57
add_body_filter is written by you or its library function? I think you need to modify add_body_filter function. — rishabh dev, Dec 24 '15 at 18:46
var header = connection.transaction.header.get("header"); Why this code would work only inside add_body_filter. — rishabh dev, Dec 24 '15 at 18:59
@rishabhdev is of the add_body_filter. Only works inside because the body filters runs at the end of the SMTP transmision, if you run outside it you get a empty header because it has not been transmited — Ricardo Polo Jaramillo, Dec 27 '15 at 17:43

score 4 · Accepted Answer · answered Dec 15 '15 at 20:31

Unless you are willing to use synchronous, blocking functions, it is impossible to satisfy the requirements you have numbered.

I would examine reasons underlying each requirement and see if you accomplish your goals in a different way.

On face value, looking at just the code you have there, I would look for a way to alter the contract of exports.hook_data so that you're able to call next() from inside the request.get callback.

score 1 · Answer 2 · answered Dec 20 '15 at 03:43

Use Async.js or Promises.

Async implementation:

exports.hook_data = function (next, connection) {
  async.waterfall([
    function( done ) {
      connection.transaction.add_body_filter('', function( content_type, encoding, body_buffer ) {
        var header = connection.transaction.header.get("header");
        if ( header == null || header == undefined || header == '' ) {
          done(null, body_buffer);
        }
        done(null, header);
      });
    },
    function( header, done ) {
      var url = 'https://server.com/api?header=' + header;
      var request = require('request');
      request.get({ uri: url },
        function( err, resp, body ) {
          // do something if there's an error
          var resultFromServer = JSON.parse(body);
          done(null, ChangeBuffer(content_type, encoding, body_buffer, resultFromServer));
        }
      );
    }
  ], function( error, buffer ) {
    // do something if there's error
    next(buffer);
  });
}

There's a lot here and I recommend you reading the docs on async.js#waterfall. Essentially it breaks up each async call into it's on function block and waits for it to return before it continues onto the next block.

Again, since these are all async, Node submits these tasks to the IO event loop and it's taken care of there (non-blocking). When this thread is waiting, it'll probably move on to the next request while this request is waiting.

score 0 · Answer 3 · edited May 23 '17 at 10:29

Looking at the Haraka manual for Transaction object & Header object I don't see there is any dependency of Header on add_body_filter. Also your code doesn't show any dependency on add_body_filter. So your 3rd requirement looks invalid.

Considering that, I think following pseudocode(as I can't test it) should work for you.

exports.hook_data = function (next, connection) {
    var header = connection.transaction.header.get("header");
    if (header == null || header == undefined || header == '') {
        return next();

    var url = 'https://server.com/api?header=' + header ;
    var request = require('request');
    request.get({ uri: url },
        function (err, resp, body) {
            var resultFromServer = JSON.parse(body);
            connection.transaction.add_body_filter('', function (content_type, encoding, body_buffer) {
                return ChangeBuffer(content_type, encoding, body_buffer, resultFromServer);
            });
            next();
        }
    );
}

If Haraka manual has failed to highlight the dependency of Header on add_body_filter and / or you have discovered it based on your practical experience, then Quy's approach seems to be the way to go.

If you are wondering when to use next() v/s return next()

sadly you cant use connection.transaction.header.get("header") outside add_body_filter. You can use it in another events but in the Data event it will always return empty values :( — Ricardo Polo Jaramillo, Dec 24 '15 at 03:47

score 0 · Answer 4 · answered Dec 24 '15 at 16:58

For handling priority of each part of your code I use Q, You can use it in few steps:

npm install q.

var request = require('request');
var Q = require('q');

exports.hook_data = function () {
   var url, header;
   Q()
   .then(function(){
     connection.transaction.add_body_filter('', function(content_type, encoding, body_buffer) {

      header = connection.transaction.header.get("header");

      if (header == null || header == undefined || header == ''){return body_buffer;}
      url = 'https://server.com/api?header=' + header ;

   })
   .then(function(){

       request.get({ uri: url }, function (err, resp, body) {

           var resultFromServer = JSON.parse(body);

           return ChangeBuffer(content_type,encoding,body_buffer,resultFromServer);
       });

   })

}

In async scenarios, using q for callbacks is more better than other ways I think.

are you sure from script above it's proper usage of Q. Doubt about it! — Marcel Djaman, Dec 24 '15 at 17:12
Yes I totally sure about it. but not best way. Maybe best way is using `Q.all` to be sure about needed data and with `.spread` continue the steps. — , Dec 24 '15 at 17:24

score 0 · Answer 5 · answered Dec 24 '15 at 17:09

JavaScript is asynchronous in nature. Asynchronous is a programming pattern which provides the feature of non-blocking code i.e do not stop or do not depend on another function / process to execute a particular line of code. Ref: Codementor Article

From Wikipedia

Node.js provides an event-driven architecture and a non-blocking I/O API designed to optimize an application's throughput and scalability for real-time web applications

What can you do?

Use a sleep/wait ie setTimeout(Not recommended)
Use some async lib like https://github.com/caolan/async (Recommended)
Use some promise lib like Q

How can I make a callback that requires info of its child function

5 Answers5