Perf Profiler Reporting Excess Use of "do_syscall_64"

Question

I'm attempting to optimize a program I wrote which aims to replicate network flows by sending packets to a specified MAC address.

The main loop of my program that is responsible for the sending and removal of flows is as follows:

while (size != 0 || response) {
    for (i = 0; size != 0 && i < size; ++i) {
       curFlow = *pCurFlow;
       while (curFlow.cur_time < now) {
           // Sending Packet
           sendto(sockfd, curFlow.buff, curFlow.length, 0, \
                 memAddr, sAddrSize);

           // Adjusting Packet Attributes
           curFlow.packets_left -= 1;
           curFlow.cur_time += curFlow.d_time;

           // If the packet has no packets left, delete it
           if (!curFlow.packets_left) {
                pCurFlow -> last -> next = pCurFlow -> next;
                pCurFlow -> next -> last = pCurFlow -> last;
                size -= 1;
                break;
            }
        }
        *pCurFlow = curFlow;
        pCurFlow = pCurFlow -> next;
    }
}

I've begun using the perf profiler to record what sort of function calls I'm making and how expensive each overhead is. However, every time I ask perf to give me a report, the outcome looks like:

Overhead  Command    Shared Object  Symbol
15.34%   packetize   /proc/kcore    0x7fff9c805b73   k [k] do_syscall_64
6.19%    packetize   /proc/kcore    0x7fff9d20214f   k [k] syscall_return_via_sysret
5.98%    packetize   /proc/kcore    0x7fff9d1a3de6   k [k] _raw_spin_lock      
5.29%    packetize   /proc/kcore    0x7fffc0512e9f   k [k] mlx4_en_xmit
5.26%    packetize   /proc/kcore    0x7fff9d16784d   k [k] packet_sendmsg

(Note: "packetize" is the name of my program)

My question is, what the heck is "do_syscall_64"?? After conducting some research, it seems like this particular function is a kernel tool used as an interrupt request.

Furthermore, I've found that the directory /proc/kcore is responsible for some components of memory management, although upon purposefully ransacking my program with memory references, the dynamic library I use with my program was the only overhead that increased from perf report.

Please let me know if you have any advice for me. Thank you!

syscall is short for system call, i.e. a call to a system/kernel function. — Mat, Aug 10 '19 at 05:31
Thank you for the comment! That makes sense... do you have any advice on how to find the particular system call I am making in order to reduce its overhead? — NolanRudolph, Aug 10 '19 at 05:34

score 3 · Accepted Answer · answered Aug 10 '19 at 05:41

It's not an interrupt request; it's the C function called from the syscall entry point that dispatches to the appropriate C function that implements the system call selected by a register passed by user-space.

Presumably sys_sendto in this case.

In older versions of Linux, the x86-64 syscall entry point used the system-call table of function pointers directly (e.g. as shown in this Q&A where only the 32-bit entry points like for int 0x80 used a C wrapper function).

But with the changes for Spectre and Meltdown mitigation, the native 64-bit system call entry point (into a 64-bit kernel from 64-bit user-space) also uses a C wrapper around system call dispatching. This allows using C macros and gcc hints to control speculation barriers before the indirect branch. The current Linux version of do_syscall_64 on github is a pretty simple function; it's somewhat surprising it's getting so many cycles itself unless nr = array_index_nospec(nr, NR_syscalls); is a lot more expensive than I'd expect on your CPU.

There's definitely expensive stuff that happens in the hand-written-asm syscall entry point, e.g. writing the MSR that flushes the branch-prediction cache. Oh, maybe lack of good branch prediction is costing extra cycles in the first C function called after that.

System-call intensive workloads suffer a lot from Spectre / Meltdown mitigations. It might be interesting to try booting with some of them disabled, and/or with an older kernel that doesn't have that code at all.

Meltdown / L1TF / etc. are completely fixed in the newest Intel CPUs with no performance cost, so disabling workarounds for that might give you some clue how much benefit you'd get from a brand new CPU.

(Spectre is still a very hard problem and can't be easily fixed with a local change to the load ports. IDK what the details are of how efficient various mitigation microcode-assisted or not strategies for mitigating it are on various CPUs.)

Thank you so much for the informative answer! This is exactly what I needed, and perhaps you're correct about branch prediction as I am currently at a 2% misprediction (which with the number of times this loop runs is actually 95 million misses for a 10 Gb/s flow, yikes!). I was wondering if you had any suggestions on a profiler that could be used to find what sort of kernel calls are being utilized, although sys_sendto is an intelligent guess. — NolanRudolph, Aug 10 '19 at 05:57
@NolanRudolph: you can use `strace` to log what system calls your code makes. Or probably you get some hits further down in the profile for the actual system-call implementations. There might be tracepoints or whatever that let you get `perf` itself to record which system calls are being invoked, IDK. I usually tune / SIMD optimize computationally intensive code that makes only a few system calls for I/O so it's not something I know a lot about digging into. — Peter Cordes, Aug 10 '19 at 06:00
That's quite alright, and I'll definitely make sure to look into strace! I appreciate all the help, you've aided my project greatly :) — NolanRudolph, Aug 10 '19 at 06:04

Perf Profiler Reporting Excess Use of "do_syscall_64"

1 Answers1