Curl easy perform using multiple threads C++ Programming

Question

So far, I've been successful in pulling information from a service provider. However, I need to invoke this over parallel process with multiple threads for millions of requests.

Following is the piece of code

    size_t WriteCallback(void *contents, size_t size, size_t nmemb, void *userp)
    {
        ((std::string*)userp)->append((char*)contents, size * nmemb);
        return size * nmemb;
    }
    int main()
    {
        CURL *curl = curl_easy_init();
        std::string readBuffer;
        if(curl) {
          CURLcode res;
          curl_easy_setopt(curl, CURLOPT_URL, "service-url");
          curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
          res = curl_easy_perform(curl);
          curl_easy_cleanup(curl);
        }
    }

Here are my two options a) One is thread pool (Visual studio C++ 2010 - Thus no access to C++ 11) b) Using curl_multi_perform

When I use thread pool -> Does invoking curl become a worker thread. How do I make user that the WriteCallback is specific to the thread so that no two threads overwrite the contents.

If I use curl_multi_perform, what do I need to do, to make sure that WriteCallback gives me the output for that particular handle?

Use the `void* userp` to store whatever instance/thread/callback-specific data you need ([random example](https://stackoverflow.com/questions/400257/how-can-i-pass-a-class-member-function-as-a-callback)). And check the documentation for the concurrency limitations of curl. — Max Langhof, Apr 05 '19 at 16:32
I realized that using curl_multi_perform is not feasible. As I need to poll the service and wait for 'N' seconds after each poll. This number of seconds to wait will differ from each request. — nsivakr, Apr 05 '19 at 16:53
In that case [this](https://stackoverflow.com/questions/7974371/libcurl-callbacks-w-c-class-members) (and hundreds of other similar questions) should get you started. — Max Langhof, Apr 05 '19 at 16:55
If you have millions of requests you will run out of threads or network I/O will be a bottleneck. — Raedwald, Apr 05 '19 at 16:58
I am allowed to send only few requests at a time (20 at a time). Thus, I'm looking for the best way to implement it. I've tested sending one by one request and I'm successful so far. Have to find a way to send 20 requests to curl at a time and handle responses simultaneously. — nsivakr, Apr 05 '19 at 17:05
@nsivakr oh only 20? yeah then use 20 worker threads each having a curl_easy handle, you don't need the complexity of curl_multi for 20 simultaneous requests :P — hanshenrik, Apr 07 '19 at 17:07
`Visual studio C++ 2010 - Thus no access to C++ 11` - PS Visual Studio 2019 Community Edition is freeware, you may want to upgrade (for free!): https://visualstudio.microsoft.com/vs/community/ - ... but they dropped support for inline asm tho — hanshenrik, Apr 07 '19 at 23:17

score 0 · Answer 1 · answered Apr 05 '19 at 20:37

0

You can use the multi interface to send X simultaneously requests and handle each request when response given.

Have a look at that C Example.

answered Apr 05 '19 at 20:37

Adir Ratzon

181
3
8

This is an excellent example. The only issue I've with curl_multi_perform is that the server(service provider), will not return the response right away and request us to come back after 'N' seconds. In that case, I need to be able to do another request after 'N' seconds only for that thread. In other words, i need to be able to wait for 'N' seconds in one thread, while other threads are working. – nsivakr Apr 05 '19 at 20:43
In business logic aspects it sounds a little weird. Anyway, you can monitor the res code (or any other idle indicator), and avoid removing the easy handle from the multi object. ``curl_multi_remove_handle(cm, e);`` – Adir Ratzon Apr 05 '19 at 20:52
he can only send 20 requests at once. Normally i'd agree with you, he wants to do millions of requests, he should be using curl_multi - but when you can only do 20 requests at once, it will be easier to just use 20 worker threads. – hanshenrik Apr 07 '19 at 17:08
@hanshenrik That also an option, but then he should be able to build it right with thread locking. – Adir Ratzon Apr 07 '19 at 18:47

Curl easy perform using multiple threads C++ Programming

1 Answers1