1

I am currently writing a mini project with C to better understand TCP, TLS, HTTP methods, and C itself.

Here is a simplified snippet of the GET portion of my program (no error checking, removed OpenSSL functions):

void htmlGET(char * path, char * address, int sockfd) {

    struct pollfd fds[1];
    fds[0].fd = sockfd;
    fds[0].events = POLLIN | POLLHUP | POLLERR;

    char * header;
    header = malloc(strlen(address)+50);
    sprintf(header, "GET %s HTTP/1.1\r\nHost: %s\r\n\r\n", path, address);
    
    write(sockfd, header, strlen(header));
    
    char buf[BUFSIZE];
    int rcount;
    
    while(1) {
        poll(fds, 1, 0);

        if (fds[0].revents & (POLLHUP | POLLERR)) { break; }

        else if (fds[0].revents & POLLIN){
            rcount = read(sockfd, buf, sizeof(buf));
            write(1, buf, rcount);
        }
    }
}

My program performs a GET request and receives data just fine without polling. However, I've found that some websites will send the header, and then the rest of the HTML in another message, so I decided to implement polling to receive everything. However, whenever I run this code, the program loops indefinitely, and I haven't been able to find the root cause. Any suggestions on what might be wrong?

Update: The program seems to work to an extent. I found that it does finish at some point in time, so I decided to run time(1) on it. Here's a sample result:

3.33s user 23.74s system 20% cpu 2:14.94 total

Any ideas on why it's so slow? Sometimes the HTML loads up instantly and the program polls a very long time, sometimes the program polls a very long time and then the HTML loads up.

Nold
  • 60
  • 1
  • 7
  • 2
    You have multiple problems with the code you show, including (but not limited to): Memory leaks, failure to check for errors, and wrong types for your data. – Some programmer dude Aug 20 '20 at 04:53
  • @Someprogrammerdude I mentioned that I removed error checking for the sake of simplifying the code here, I can add it in if you want (also that call to free). Also can you elaborate on the wrong types? – Nold Aug 20 '20 at 04:57
  • What is the type of `buf`? How are you using it? – Some programmer dude Aug 20 '20 at 05:03
  • @Someprogrammerdude Sorry I must've copied that over wrong. Just fixed it. – Nold Aug 20 '20 at 05:06
  • 2
    The main problem is that TCP is a *streaming* protocol, without "packets" or message boundaries. It's just a stream of bytes. That means you generally need to read in a loop to get a full message. *And* be able to detect higher-level protocol messages received in a single read. For HTTP you need to read in a loop until you have received the full header. Then (depending om the response) use another loop to read the data. – Some programmer dude Aug 20 '20 at 05:32
  • @Someprogrammerdude that's kind of what I'm trying to achieve by polling and looping. Also, I just added an update. It turns out that the program does work, just that it takes a really long time. – Nold Aug 20 '20 at 05:38
  • 1
    You can't really do it with `poll` (or `select`). Your system might have already received *everything*, it's just waiti g in buffers for you to fetch. You need to call *`read`* in a loop, not `poll`. – Some programmer dude Aug 20 '20 at 05:41
  • @Someprogrammerdude I thought the same thing. But when I used only read(2), my program ended up blocking on the read(2) call. – Nold Aug 20 '20 at 05:56

2 Answers2

0

regarding:

poll(fds, 1, 0);

from the MAN page:

Note that the timeout interval will be rounded up to the system clock granularity, and kernel scheduling delays mean that the blocking interval may overrun by a small amount. Specifying a negative value in timeout means an infinite timeout. Specifying a timeout of zero causes poll() to return immediately, even if no file descriptors are ready. (Emphasis mine)

suggest a negative value for the timeout, rather than 0 For instance: -1

user3629249
  • 16,402
  • 1
  • 16
  • 17
  • I could have used -1, but it doesn't really matter in this case. Either poll(2) blocks continuously with -1 or poll(2) is called continuously with 0 until the state changes. In any case, I tried a request on example.com with -1 and here was the `time` result: `0.00s user 0.00s system 0% cpu 2:15.00 total`. – Nold Aug 20 '20 at 05:52
  • well, your question was: *However, whenever I run this code, the program loops indefinitely, and I haven't been able to find the root cause. Any suggestions on what might be wrong?* I answered that question – user3629249 Aug 20 '20 at 06:23
  • That still doesn't answer the question. I wrote the program fully understanding that there would be some form of looping. Someprogrammerdude provided the best understanding of the root cause as far as I can tell. – Nold Aug 20 '20 at 18:54
0

From this post, it's clear that you can't expect the web server to close the connection with the client, and thus using polling to determine when to stop reading from the socket isn't the complete solution.

I decided to incorporate parsing of the message header for either Transfer-Encoding: Chunked or Content-Length: #. This gives me the necessary information to determine if all the data promised has been received.

For example, using Content-Length, I used the fact that the message header ends with a blank link (\r\n\r\n) to determine where the message body starts, and then counted the number of bytes I received in the buffer to see if I needed to loop back and do another read for the rest of the body.

For Transfer-Encoding: Chunked I used the suggestion on the same post and this guide, and found that simply checking for the end chunk with 0\r\n\r\n did the trick.

Nold
  • 60
  • 1
  • 7