1

we want to understand what are the aspects when net.core.netdev_max_backlog kernel value is very low and not as recommended

on our Linux RHEL machines , the value for this parameter is 1000

since our machines are HADOOP machines ( BIGDATA cluster )

we saw that best practice is to increase the value to 65536

as describe on:

https://datasayans.wordpress.com/2015/11/04/performance-kernel-tuning-for-hadoop-environment/

background:

The kernel parameter “netdev_max_backlog” is the maximum size of the receive queue. The received frames will be stored in this queue after taking them from the ring buffer on the NIC. Use high value for high speed cards to prevent loosing packets. In real time application like SIP router, long queue must be assigned with high speed CPU otherwise the data in the queue will be out of date (old).

so - what could be the aspects when this kernel parameter is with un-Insufficient value ?

other - reference - https://gist.github.com/leosouzadias/e37cd189794bb78de502ac25cb605576

https://community.cloudera.com/t5/Community-Articles/OS-Configurations-for-Better-Hadoop-Performance/ta-p/247300

https://www.senia.org/2016/02/28/hadoop-and-redhat-system-tuning-etcsysctl-conf/

https://mapredit.blogspot.com/2014/11/hadoop-server-performance-tuning.html

https://gist.github.com/phaneesh/38b3d80b38cc76abb1d010f598fbc90a

https://docs.datastax.com/en/dse/5.1/dse-dev/datastax_enterprise/config/configRecommendedSettings.html

PDF - https://www.cisco.com/c/dam/en/us/solutions/collateral/data-center-virtualization/big-data/cloudera-intel-cisco-hadoop-benchmark.pdf

King David
  • 1,001

1 Answers1

4

What could the effects be when this kernel parameter value is insufficient

Run the following command:

for i in {1..30}; do date; cat /proc/net/softnet_stat; sleep 1; done

The following will be the output:

001c1216 000000b0 00000040 00000000 00000000 00000000 00000000 00000000 00000000 00000000
02fdffa5 000000aa 00000032 00000000 00000000 00000000 00000000 00000000 00000000 00000000

02fdffa5 is the packet count 000000aa is the number of packets that have been dropped

Run the command and see if the amount of packets that have been dropped has increased by converting the value to decimal. If the number of packets has increased then increase net.core.netdev_max_backlog to an appropriate value.

You can temporarily increase the value by running the following command:

sysctl -w net.core.netdev_max_backlog=2000

You can make the value permanent by running the following command:

echo "net.core.netdev_max_backlog = X" >> /etc/sysctl.conf

Additionally, a value of 65536 is excessive and unnecessary for proper system performance. You should obviously change X to the value you wish to use.

If the second and third column of softnet_stat is not growing then it's necessary to increase net.core.netdev_max_backlog

Source: How to tune net.core.netdev_max_backlog and net.core.netdev_budget sysctl kernel tunables?

Ramhound
  • 44,080