Socket programming issue with recv() receiving partial messages

Question

I have a socket that is receiving streaming stock tick data. However, I seem to get a lot of truncated messages, or what appears to be truncated messages. Here is how I am receiving data:

if((numbytes = recv(sockfd, buf, MAXDATASIZE-1, 0)) == -1) {
    perror("recv()");
    exit(1);
}
else {
    buf[numbytes] = '\0';
    // Process data
}

Can recv() receive just a partial message of what was sent?

My feeling is I might need another loop around the recv() call that receives until a complete message is sent. I know that a libcurl implementation I have (not possible to use libcurl here I would think) has an outer loop:

// Read the response (sum total bytes read in tot_bytes)
for(tot_bytes=0; ; tot_bytes += iolen)
{  
    wait_on_socket(sockfd, 1, 60000L);
    res = curl_easy_recv(curl, buf + tot_bytes, sizeof_buf - tot_bytes, &iolen);

    if(CURLE_OK != res) {
        // printf( "## %d", res );
        break;
    }
}

Do I need an recv() loop similar to the libcurl example (that verifiably works)?

Not entirely related, but why are you sending tick data over TCP? That stuff is normally multicast over UDP. — chrisaycock, Apr 20 '12 at 17:26
I am receiving, not sending. they just emailed me a hostname and a port number, I didn't get any other technical instructions. I just sent an email asking if I should use UDP. but if I am using TCP and it works (aside from partial message issue I opened topic with) would it also work with UDP? ie - they send UDP, I can recv() with UDP or TCP? Also, they do send headers so I think I need to use those to detect a partial message and then when I recv() the next time thru, offset into the partial message in order to try and complete it .. thx. — Don Wool, Apr 20 '12 at 18:08
the UDP comment I don't get - you can lose data if you use UDP, right? — Don Wool, Apr 23 '12 at 22:21
Yes, UDP can lose data. That's why it's used for ticks; if you miss a quote update, the exchange isn't going to wait and resend. The exchange just multicasts to everyone and hopefully you'll get most of it. I'm confused as to why your market-data system would be set-up differently. Usually the host and port refer to a multicast *group* that you have the *join*. — chrisaycock, Apr 23 '12 at 22:30

score 8 · Answer 1 · edited Dec 22 '14 at 21:10

8

We can also pass the flag to recv to wait until all the message has arrived. It works when you know the number of bytes to receive. You can pass the command like this.

numbytes =  recv(sockfd, buf, MAXDATASIZE-1, MSG_WAITALL);

edited Dec 22 '14 at 21:10

senshin

10,022
7
46
59

answered Dec 22 '14 at 20:35

Anshuman

421
4
11

score 7 · Accepted Answer · answered Apr 20 '12 at 17:20

You're right, you need a loop. recv only retrieves the data that's currently available; once any data has been read, it doesn't wait for more to appear before it returns.

The manual page says "The receive calls normally return any data available, up to the requested amount, rather than waiting for receipt of the full amount requested."

chrisaycock · Answer 3 · 2012-08-14T03:29:17.203

3

TCP does not respect message boundaries. That means that recv() is not guaranteed to get the entire message, exactly as you hypothesize. And that is indeed why you need a loop around your recv(). (That's also why upper-layer protocols like HTTP either close the socket, or prepend a length indicator, so the recipient knows exactly when to stop reading from the socket.)

edited Aug 14 '12 at 03:29

answered Apr 20 '12 at 17:20

chrisaycock

36,470
14
88
125

score 1 · Answer 4 · edited May 23 '17 at 11:47

1

can recv() receive just a partial message of what was sent?

Yes, indeed, if you use TCP. I think this can help you. Handling partial return from recv() TCP in C

edited May 23 '17 at 11:47

Community

1
1

answered Apr 20 '12 at 17:20

young

2,163
12
19

thx for link. from reading that, actually the msgs I receive have headers. eg 2,, . My recv typically returns 100's and last is truncated. So, I should probably "process what I get" and set the next recv() to use an offset such that the last partial msg is completed. don't think I can get these one by one, too much data. this is RT feed for BATS. – Don Wool Apr 20 '12 at 17:42

Socket programming issue with recv() receiving partial messages

4 Answers4

Linked