Cannot download more than one file with sockets in C

Question

When i try to download a set of files using stream sockets over an HTTP protocol, it only gets data from the first file i try to download.
Assume a loop like the following...

char* file = (char*) malloc(enough_space);
char page[] = {"www.foobar.com"};
for(int n=0 ; n<10 ; n++)
    {
        sprintf(file, "file%i.html", n);
        fopen(file, "wb");
        sprintf(request, "GET %s HTTP/1.1\nHost: %s\n\n", file, page);
        write( socket, request, strlen(request) );
        read_file(output_file);
        fclose(output_file);
    }

Where a connection has been established first.
This code would give me file1.html, including its header from the server.. But only the first file, and this puzzles me.. What will i have to do in order to get them all?
Thanks up front.

You loop starts at 0 so the first file should be file0.html not file1.html. I think there is more going on. — Nostromoo, Oct 27 '14 at 21:21
HTTP doesn't allow that simply for multiple requests given one connection. You want to experiment with http://en.wikipedia.org/wiki/HTTP_persistent_connection - but in general, I suggest using a 3rd-party-lib such as libcurl to deal with all the intricacies of HTTP, including SSL, redirects, encodings and content-types. — deets, Oct 27 '14 at 21:23

score 1 · Accepted Answer · edited May 23 '17 at 12:05

1

HTTP was designed so that just a single file can be downloaded over a TCP connection. To download multiple files over one TCP connection, you could use HTTP Pipelining. You can read more here: HTTP pipelining request text example

Or you could just use one of the many libraries that will handle this, and many other caveats of HTTP for you: libcurl, libsoup...

edited May 23 '17 at 12:05

Community

1
1

answered Oct 27 '14 at 21:25

jmajnert

418
4
8

Thank you very much, sending all the requests at once solved my problem :) – Emíl Rahbek Oct 27 '14 at 22:10
1

HTTP pipelining is not the answer. Not all servers support pipelining. Handling HTTP responses and keep-alives correctly is the answer. The server could be closing the socket after any/each file. You have to analyze each response, determine if the server is closing the socket, and re-connect if needed, before sending the next request. Pipelining requires proper keep-alive handling. The purpose of pipelining is to improve network performance by sending multiple requests using fewer TCP packets. Pipelining does not provide download management, you need proper HTTP processing for that. – Remy Lebeau Oct 28 '14 at 04:04

Cannot download more than one file with sockets in C

1 Answers1