Parallel cURL Request with WRITEFUNCTION Callback

Question

I'm trying to limit my cURL responses as suggested in these posts:Retrieve partial web page and PHP CURLOPT_WRITEFUNCTION doesn't appear to be working. The idea is to limit the response to 4000 characters as specified in the callback function.

I wrote the following function, but I already know that it doesn't make sense, because a parameter in the callback function definition doesn't vary within a loop as it would within a function call. By the time the functions are actually called, the value for $key is fixed, so my references to that index won't vary.

It seems that I need a new closure function for each of the loops, and each one needs to reference its own $full_length variable. However, I don't see how that's possible. In order to do that, it seems I would have to somehow make a reference the closure object in order to specify the specific $full_length variable.

Any help would be appreciated. Thanks.

function get_headers($urls){
    $curly = array();
    $result = array();
    $mh = curl_multi_init();
    $obj = $this;
    foreach ($urls as $key => $url) {
        $this->full_length[$key] = 0;
        $callback = function ($ch, $string) use ($obj, $key){
                    $length = strlen($string);
                    $obj->full_length[$key] += $length;
                    if($obj->full_length[$key] >= 4000){
                        return -1;
                    }
                    return $length;
                };
        $curly[$key] = curl_init
        curl_setopt($curly[$key], CURLOPT_URL,            $url);
        curl_setopt($curly[$key], CURLOPT_HEADER,         0);
        curl_setopt($curly[$key], CURLOPT_WRITEFUNCTION, $callback);
        curl_setopt($curly[$key], CURLOPT_RETURNTRANSFER, 1);
        curl_multi_add_handle($mh, $curly[$key]);
    }
    $running = null;
    do {
        curl_multi_exec($mh, $running);
    } while($running > 0);
    foreach($curly as $key => $cnt) {
        $content = curl_multi_getcontent($cnt);
        curl_multi_remove_handle($mh, $cnt);
        if (strlen($content) > 0){
            $result[$key] = $content;
        } else {
            curl_multi_close($mh);
            return FALSE;
        }
    }
    curl_multi_close($mh);
    return $result;
}

EDIT:

I found a post that does exactly what I'm trying to do, but it's in javascript: closure inside a for loop - callback with loop variable as parameter . I wrote the following function to try to do the same thing in PHP:

function get_write_function($key){
    $this->full_length[$key] = 0;
    $obj = $this;
    $funky = function ($ch, $str) use ($obj, $key){
        $length = strlen($str);
        $obj->full_length[$key] += $length;
        if($obj->full_length[$key] >= 4000){
            return -1;
        }
        return $length;
    };
    return $funky;
}

The code ran without errors, but it still didn't do what I wanted. After closing my cURL handles I dumped the $full_length array, and it only showed:

array([0] => 0, [1] => 0)

That indicates that they were initialized by the get_write_function (since I didn't initialize anything in the class declaration), but that the values were never updated afterwards.

score 1 · Accepted Answer · edited May 23 '17 at 10:29

I finally got it figured out. The biggest problem was the fact that cURL was ignoring the WRITEFUNCTION until I placed it as the very last option specified, as I posted here: cURL WRITEFUNCTION not Being Called. I actually didn't need the return transfer, since I wrote the output to a class variable. That was necessary because when the callback returns -1, nothing gets returned. The following code works great:

var $full_length = array();
var $result = array();

function get_headers($urls){
    $curly = array();
    $mh = curl_multi_init();
    foreach ($urls as $key => $url) {
        $callback = $this->get_write_function($key);
        $curly[$key] = curl_init
        curl_setopt($curly[$key], CURLOPT_URL,            $url);
        curl_setopt($curly[$key], CURLOPT_HEADER,         0);
        curl_setopt($curly[$key], CURLOPT_WRITEFUNCTION, $callback);
        curl_multi_add_handle($mh, $curly[$key]);
    }
    $running = null;
    do {
        curl_multi_exec($mh, $running);
    } while($running > 0);
    foreach($curly as $key => $cnt) {
        curl_multi_remove_handle($mh, $cnt);
    }
    curl_multi_close($mh);
    return $this->result;
}

function get_write_function($key){
    $this->full_length[$key] = 0;
    $this->result[$key] = '';
    $obj = $this;
    $funky = function ($ch, $str) use ($obj, $key){
        $obj->result[$key] .= $str;
        $length = strlen($str);
        $obj->full_length[$key] += $length;
        if($obj->full_length[$key] >= 4000){
            return -1;
        }
        return $length;
    };
    return $funky;
}

Parallel cURL Request with WRITEFUNCTION Callback

1 Answers1

Linked