Why is Socket.Receive timing out on a half-closed connection when timeout is set to infinite?

Question

Below is a C# program demonstrating the problem.

The server starts listening on a socket. The client connects to the server, sends a message, uses Shutdown(SocketShutdown.Send) to close its send half of the connection to let the server know where the end of the message is, and waits for a response from the server. The server reads the message, does some lengthy computation (simulated here with a sleep call), sends a message to the client, and closes the connection.

On Windows, the client's Receive call always fails after exactly 2 minutes with "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond" even though the timeout is set to infinite.

If I run the program in Linux with Mono, the timeout does not occur even if I set the "lengthy operation" to 10 minutes, but it happens in Windows whether I run it with Mono or .NET. If I set the timeout to 1 second, it times out after 1 second. In other words, it times out in the timeout I set or 2 minutes, whichever is less.

A similar sample program in which the server sends a message to the client, with no message from client to server and no half-close, works as expected with no timeout.

I can get around this by modifying my protocol to use some other method of indicating to the server when a message is complete (perhaps prefixing the message with the length of the message). But I want to know what's going on here. Why does Socket.Receive time out on a half-closed connection when the timeout is set to infinite?

From what I understand, a connection with only its send half closed should be able to continue receiving data indefinitely. It seems unlikely that there would be a bug in such a fundamental part of Windows. Am I doing something wrong?

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Net.Sockets;
using System.Net;
using System.Threading.Tasks;
using System.Diagnostics;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            // Start server thread
            Thread serverThread = new Thread(ServerStart);
            serverThread.IsBackground = true;
            serverThread.Start();

            // Give the server some time to start listening
            Thread.Sleep(2000);

            ClientStart();
        }

        static int PortNumber = 8181;

        static void ServerStart()
        {
            TcpListener listener = new TcpListener(new IPEndPoint(IPAddress.Any, PortNumber));
            listener.Start();
            while (true)
            {
                TcpClient client = listener.AcceptTcpClient();
                Task connectionHandlerTask = new Task(ConnectionEntryPoint, client);
                connectionHandlerTask.Start();
            }
            listener.Stop();
        }

        static void ConnectionEntryPoint(object clientObj)
        {
            using (TcpClient client = (TcpClient)clientObj)
            using (NetworkStream stream = client.GetStream())
            {
                // Read from client until client closes its send half.
                byte[] requestBytes = new byte[65536];
                int bufferPos = 0;
                int lastReadSize = -1;
                while (lastReadSize != 0)
                {
                    lastReadSize = stream.Read(requestBytes, bufferPos, 65536 - bufferPos);
                    bufferPos += lastReadSize; 
                }
                client.Client.Shutdown(SocketShutdown.Receive);
                string message = Encoding.UTF8.GetString(requestBytes, 0, bufferPos);

                // Sleep for 2 minutes, 30 seconds to simulate a long-running calculation, then echo the client's message back
                byte[] responseBytes = Encoding.UTF8.GetBytes(message);
                Console.WriteLine("Waiting 2 minutes 30 seconds.");
                Thread.Sleep(150000);

                try
                {
                    stream.Write(responseBytes, 0, responseBytes.Length);
                }
                catch (SocketException ex)
                {
                    Console.WriteLine("Socket exception in server: {0}", ex.Message);
                }
            }
        }

        static void ClientStart()
        {
            using (Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
            {
                // Set receive timeout to infinite.
                socket.ReceiveTimeout = -1;

                // Connect to server
                socket.Connect(IPAddress.Loopback, PortNumber);

                // Send a message to the server, then close the send half of the client's connection
                // to let the server know it has the entire message.
                string requestMessage = "Hello";
                byte[] requestBytes = Encoding.UTF8.GetBytes(requestMessage);
                socket.Send(requestBytes);
                socket.Shutdown(SocketShutdown.Send);

                // Read the server's response. The response is done when the server closes the connection.
                byte[] responseBytes = new byte[65536];
                int bufferPos = 0;
                int lastReadSize = -1;

                Stopwatch timer = Stopwatch.StartNew();
                try
                {
                    while (lastReadSize != 0)
                    {
                        lastReadSize = socket.Receive(responseBytes, bufferPos, 65536 - bufferPos, SocketFlags.None);
                        bufferPos += lastReadSize;
                    }

                    string responseMessage = Encoding.UTF8.GetString(responseBytes, 0, bufferPos);
                    Console.WriteLine(responseMessage);
                }
                catch (SocketException ex)
                {
                    // Timeout always occurs after 2 minutes. Why?
                    timer.Stop();
                    Console.WriteLine("Socket exception in client after {0}: {1}", timer.Elapsed, ex.Message);
                }
            }
        }
    }
}

The following program prefixes messages with a 4-byte message length rather than using socket.Shutdown(SocketShutdown.Send) to signal end of message. The timeout does not occur in this program.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net.Sockets;
using System.Net;
using System.Threading.Tasks;
using System.Diagnostics;
using System.Threading;

namespace WithoutShutdown
{
    class Program
    {
        static void Main(string[] args)
        {
            // Start server thread
            Thread serverThread = new Thread(ServerStart);
            serverThread.IsBackground = true;
            serverThread.Start();

            // Give the server some time to start listening
            Thread.Sleep(2000);

            ClientStart();
        }

        static int PortNumber = 8181;

        static void ServerStart()
        {
            TcpListener listener = new TcpListener(new IPEndPoint(IPAddress.Any, PortNumber));
            listener.Start();
            while (true)
            {
                TcpClient client = listener.AcceptTcpClient();
                Task connectionHandlerTask = new Task(ConnectionEntryPoint, client);
                connectionHandlerTask.Start();
            }
            listener.Stop();
        }

        static void SendMessage(Socket socket, byte[] message)
        {
            // Send a 4-byte message length followed by the message itself
            int messageLength = message.Length;
            byte[] messageLengthBytes = BitConverter.GetBytes(messageLength);
            socket.Send(messageLengthBytes);
            socket.Send(message);
        }

        static byte[] ReceiveMessage(Socket socket)
        {
            // Read 4-byte message length from the client
            byte[] messageLengthBytes = new byte[4];
            int bufferPos = 0;
            int lastReadSize = -1;
            while (bufferPos < 4)
            {
                lastReadSize = socket.Receive(messageLengthBytes, bufferPos, 4 - bufferPos, SocketFlags.None);
                bufferPos += lastReadSize;
            }
            int messageLength = BitConverter.ToInt32(messageLengthBytes, 0);

            // Read the message
            byte[] messageBytes = new byte[messageLength];
            bufferPos = 0;
            lastReadSize = -1;
            while (bufferPos < messageLength)
            {
                lastReadSize = socket.Receive(messageBytes, bufferPos, messageLength - bufferPos, SocketFlags.None);
                bufferPos += lastReadSize;
            }

            return messageBytes;
        }

        static void ConnectionEntryPoint(object clientObj)
        {
            using (TcpClient client = (TcpClient)clientObj)
            {
                byte[] requestBytes = ReceiveMessage(client.Client);
                string message = Encoding.UTF8.GetString(requestBytes);

                // Sleep for 2 minutes, 30 seconds to simulate a long-running calculation, then echo the client's message back
                byte[] responseBytes = Encoding.UTF8.GetBytes(message);
                Console.WriteLine("Waiting 2 minutes 30 seconds.");
                Thread.Sleep(150000);

                try
                {
                    SendMessage(client.Client, responseBytes);
                }
                catch (SocketException ex)
                {
                    Console.WriteLine("Socket exception in server: {0}", ex.Message);
                }
            }
        }

        static void ClientStart()
        {
            using (Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
            {
                // Set receive timeout to infinite.
                socket.ReceiveTimeout = -1;

                // Connect to server
                socket.Connect(IPAddress.Loopback, PortNumber);

                // Send a message to the server
                string requestMessage = "Hello";
                byte[] requestBytes = Encoding.UTF8.GetBytes(requestMessage);
                SendMessage(socket, requestBytes);

                // Read the server's response.
                Stopwatch timer = Stopwatch.StartNew();
                try
                {
                    byte[] responseBytes = ReceiveMessage(socket);
                    string responseMessage = Encoding.UTF8.GetString(responseBytes);
                    Console.WriteLine(responseMessage);
                }
                catch (SocketException ex)
                {
                    // Timeout does not occur in this program because it does not call socket.Shutdown(SocketShutdown.Send)
                    timer.Stop();
                    Console.WriteLine("Socket exception in client after {0}: {1}", timer.Elapsed, ex.Message);
                }
            }
        }
    }
}

Answer 1

This behavior is by design. When the client has closed its half on the connection and the server acknowledges the close, the client is in the FIN_WAIT_2 state, waiting for the server to close the connection. http://support.microsoft.com/kb/923200 states that there is a FIN_WAIT_2 timeout of 2 minutes. If no data is received in a 2 minute period when a connection is in the FIN_WAIT_2 state, the client forcibly closes the connection (with a RST).

By default in Windows Server 2003, TCP connections must close after the TCP connection state has been set to FIN_WAIT_2 for two minutes.

This old Apache article suggests the reason for the timeout: malicious or misbehaving applications could keep the other end of the connection in FIN_WAIT_2 indefinitely by never closing their end of the connection and thus tie up operating system resources.

Linux apparently has a timeout as well You can check the value with

$ cat /proc/sys/net/ipv4/tcp_fin_timeout

I'm not sure why the timeout wasn't occurring for me on Linux. Perhaps because it was a loopback connection and therefore DoS attacks are not a concern or loopback connections use different code that does not use the tcp_fin_timeout setting?

Bottom line: The operating system has a good reason for making the connection time out. Avoid using Shutdown as an application-layer signalling mechanism and use an actual application-layer method instead.

Answer 2

Socket.Receive seems to have an upper limit on the ReceiveTimeout of two minutes. This is something that is supposedly specified in the registry, though I could find no concrete evidence of either the truth of this or the keys to modify. This might explain the different behaviour on Linux vs Windows.

I looked into various solutions for this, but the simplest (and only one that worked) is probably to have the server end send a heartbeat every few seconds. Essentially, this just makes sure that you never hit the two minute timeout.

string boundary = string.Format("--{0}--", Guid.NewGuid());
byte[] boundaryBytes = Encoding.ASCII.GetBytes(boundary);

//Every 15 seconds write a byte to the stream.
for (int i = 0; i < 10; i++)
{
    stream.WriteByte(0);
    Thread.Sleep(15000);
}

//Indicate where the end of the heartbeat bytes is.
stream.Write(boundaryBytes, 0, boundaryBytes.Length);

//Same code as before.
try
{
    stream.Write(responseBytes, 0, responseBytes.Length);
}
catch (SocketException ex)
{
    Console.WriteLine("Socket exception in server: {0}", ex.Message);
}

What I've done here is simulate a long running task (it will sleep for 2.5 minutes in total), but every 15 seconds it writes a single byte to the stream to prevent the timeout.

The problem with doing this is that you wind up with a bunch of unwanted garbage at the beginning of the response. This is where the boundaryBytes come in: with these you can clearly separate the unwanted bits from the actual result. The important thing is that the client must be aware of what the boundary is up front.

Edit:

I see from your comment below that removing the socket.Shutdown(SocketShutdown.Send) seemed to do the trick. I had wondered about this myself, but didn't really investigate it.

What I don't understand is why calling this method has the effect it does. Doing some decompilation, that Shutdown method basically calls the shutdown method in the underlying WinSock library (ws2_32.dll) via pinvoke, does some error handling, then sets the socket to be disconnected. In the absence of any other information, this suggests that the 2 minute problem is created in that WinSock call.

I tried to diagnose this by enabling WinSock logging in the Event Viewer, but there didn't seem to be anything obvious that would point to why this was happening.

Doing some more research down at the WinSock level turned up these questions:

Winsock recv not working after shutdown

Multipe Send()'s and Recv()'s using Winsock2

Why HTTP server close the connection when client close only the send half of the connection?

Why does .Net Socket.Disconnect take two minutes?

The common theme seems to be that socket.Shutdown(SocketShutdown.Send) isn't such a good idea if you intend on using the socket afterward for receiving. The fact that this method sets the socket.Connected property to false could be quite telling.

The last link in the above list seemed rather close to the mark with the 2 minute situation, though the OP refers to registry settings but does not say what they are.