Why does connection pool size keep increasing with Golang HTTP client?

Question

I am basically making a health check crawler to a huge list of domains. I have a Golang script that creates ~256 routines that make requests to the list of domains. I am using the same client with the following transport configuration:

# init func
this.client = &http.Client{
        Transport: &http.Transport{
            ForceAttemptHTTP2:   true,
            TLSHandshakeTimeout: TLSHandShakeTimeout,
            TLSClientConfig:     &tls.Config{InsecureSkipVerify: true},
            MaxConnsPerHost:     -1,
            DisableKeepAlives:   true,
        },
        Timeout: RequestTimeout,
    }
... 
# crawler func
req, err := http.NewRequestWithContext(this.ctx, "GET", opts.Url, nil)
if err != nil {
    return nil, errors.Wrap(err, "failed to create request")
}

res, err := this.client.Do(req)
if err != nil {
    return nil, err
}
defer res.Body.Close()
...

I ran netstat -anp | wc -l and can see over 2000+ connections with TIME_WAIT.

I think MaxConnsPerHost means something other than what you expect it to mean. — Ярослав Рахматуллин, Oct 10 '22 at 03:11
consider what MaxIdleConns does. https://www.loginradius.com/blog/engineering/tune-the-go-http-client-for-high-performance/ you want something like MaxIdleConns 255 and MaxConnsPerHost 1 /// not sure... just reading the docs. — Ярослав Рахматуллин, Oct 10 '22 at 03:13
Even with MaxIdleConns=256, MaxConnsPerHost=1, I can see connections rising over 1000+ — Abe, Oct 10 '22 at 03:32
https://stackoverflow.com/questions/47385692/limited-concurrent-connections-in-go — Ярослав Рахматуллин, Oct 10 '22 at 03:46
https://stackoverflow.com/questions/37774624/go-http-get-concurrency-and-connection-reset-by-peer/37813844#37813844 specifically https://stackoverflow.com/a/45362012/657703 — Ярослав Рахматуллин, Oct 10 '22 at 03:47
Yes, TIME_WAIT, but is set MaxIdleConnsPerHost=-1 and I close the body — Abe, Oct 11 '22 at 18:10
@Abe, the incorrect `MaxConnsPerHost` and `DisableKeepAlives` is used in your case. For more details, please refer to my answer. Hope it will solve your issue. — zangw, Oct 12 '22 at 05:23
@zangw please make sure to read my question carefully. This would make sense if you are making requests to the same host, HOWEVER, I am making requests to thousands of different hots. Which is why we do want to disable keep alives. — Abe, Oct 12 '22 at 15:58
@Abe, Sorry for misunderstanding the question before. For your issue, my answer is updated, hope it could help you. — zangw, Oct 13 '22 at 04:46

zangw · Answer 1 · 2022-10-13T04:45:40.913

The default number of goroutines per host for http.Client is 2. One is for the receiver and the other for the sender. So for thousands of domains, there could be thousands of goroutines here.

As the DisableKeepAlives is set to true, so the connection will be closed when the response of HTTP is done. The TIME_WAIT is the normal TCP state after closing a connection.

However, the default timeout of TIME_WAIT state on Linux is 60 seconds. The huge number of TIME_WAIT states could cause the server (such as probe/crawler) connection issue.

In order to solve the TIME_WAIT issue. The SO_LINGER option could help. It disables the default TCP delayed-close behavior, which sends the RST to the peer when the connection is closed. And it would remove the TIME_wAIT state of the TCP connection.

More discussion could be found here When is TCP option SO_LINGER (0) required?

Sample

    dialer := &net.Dialer{
        Control: func(network, address string, conn syscall.RawConn) error {
            var opterr error
            if err := conn.Control(func(fd uintptr) {
                l := &syscall.Linger{}
                opterr = syscall.SetsockoptLinger(int(fd), unix.SOL_SOCKET, unix.SO_LINGER, l)
            }); err != nil {
                return err
            }
            return opterr
        },
    }
    client := &http.Client{
        Transport: &http.Transport{
            DialContext: dialer.DialContext,
        },
    }

Moreover, here is another SO_LINGER use case in EaseProbe. It is a simple, standalone, and lightweight tool that can do health/status checking.

@Adrian, my bad, just copy from my sample codes. Thank you very much. — zangw, Oct 11 '22 at 07:07
@Abe, I reproduce your issue. and fix it through correct the parameter of `Transport` — zangw, Oct 12 '22 at 05:23
This is incorrect for crawling different hosts which does require to disable keep alives. Your example only tests a single host. — Abe, Oct 12 '22 at 15:58
Just post anther option with `SO_LINGER`, hope it could solve your issue @Abe — zangw, Oct 13 '22 at 04:49

Why does connection pool size keep increasing with Golang HTTP client?

1 Answers1