0

In a nutshell, SO_REUSEPORT socket option allow to create multiple sockets on ip:port pair. For example, program1 and program2 both can call functions chain socket()->bind()->listen()->accept() for the same port and IP, and kernel scheduler will evenly distribute incoming connections between this two programs.

I assumed that with this option, you can get rid of the use of fork() for spawning additional workers and can simply run new program instance.

I wrote a simple epoll socket server, based on this logic, and test it with weighttp:

weighttp -n 1000000 -c 1000 -t 4 http://127.0.0.1:8080/

For two running instances the results is ~44000 RPS, for one running instance - near ~51000 RPS. I am very surprised with 7000 RPS differece.

After this test I add fork() before listen() and run one instance of server, so now it has the same logic that previous implementation - two process with epoll loop listening socket, but socket()->bind() called only once, before fork(), and second process receive it FD copy for listen() call.

I run tests again and it shows ~50000 RPS!

So, my question is very simple: what magic do fork() in this case and why it works faster than two independent process with it own socket()? Kernel do the same job for scheduling, I dont see any important difference.

Alex
  • 571
  • 1
  • 8
  • 26
  • Related answer: http://stackoverflow.com/a/14388707/694576 – alk Mar 11 '15 at 17:49
  • Iam read this answer, its not cover my question. – Alex Mar 11 '15 at 18:34
  • Did you enable `SO_REUSEPORT` before `bind()`ing the address in each of the processes? From some quick reading I don't think `SO_REUSEPORT` would make any difference if you do the `bind()` before forking off the threads. Each thread should `bind()` separately instead. – Ulfalizer Mar 11 '15 at 21:27
  • `setsockopt()` with `SO_REUSEPORT` and `bind()` are called only in parent, before `fork()`. Child do ony `listen()` after fork. – Alex Mar 11 '15 at 21:44
  • @Alex: In that case `SO_REUSEPORT` might not make any difference. My understanding is that you should create a separate `socket()` in each of the threads and bind them to the same port. – Ulfalizer Mar 11 '15 at 22:11
  • Also, you should post your app, yeah. :) – Ulfalizer Mar 11 '15 at 22:11
  • And what exactly is "*RPS*", "*reads-per-.second*"? – alk Mar 12 '15 at 05:49

1 Answers1

0

Make this comment to flag question as answered: due to my research, internal OS scheduler is far from optimal and depends on hardware. Better is to use user-land scheduler.

Alex
  • 571
  • 1
  • 8
  • 26
  • I think you create epoll descriptor before calling fork() which means you actually have a single epoll() list and a single socket kernel buffer. And everything else is consistent with contention overhead. – wick May 14 '21 at 18:49