4

I am using IPSet to managed tens of thousands of IPv4 CIDR/netmask ranges that then get linked to IPTables rules. This setup is working great, but I would like to get a good, high-level overview count of the IP host addresses IPSet acts on for client reporting purposes.

The IPSet entry formatting is consistently like this:

123.456.0.0/16 timeout 86400

So I can grep on the lines that have timeout to get values to act on the CIDR/netmask ranges that the entry contains.

For example, if I save the IPSet output (via ipset -L -n > ipset-20181228.txt) to a text file named ipset-20181228.txt and then run a combination of grep and wc -l like this:

grep  "timeout" ipset-20181228.txt  | wc -l

I get a count of 39,000+ items which equate to 39,000+ CIDR/netmask ranges. But that is (of course) only counting the CIDR/netmask ranges and not full counts of IP host addresses in that range.

I attempted to use prips (which expands CIDR/netmask values to actual IP addresses in Bash) with grep to cull out only items with CIDR/netmask ranges like this:

grep -oE '(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\/([0-9]{1,2})' ipset-20181228.txt | awk 'NF { system( "prips " $0)  }' | wc -l

And after a whopping 20 to 30 minutes (!!!) on my 2018 MacBook Air (with the fans kicking in), the count I got was 736,000,000+ which is what I am going for… But 20 to 30 minutes is way too long. I want this to be as scriptable and non-intrusive as possible, and can’t trust a command like that to run on a production server without eating up resources; I mean look at how it behaves on my local 2018 MacBook Air development setup.

Is there any way to just calculate the CIDR/netmask range count based on simply the CIDR/netmask value? I am hoping there is just some command line tool—or option in existing tools I am using—that I am unaware of that can help.

Giacomo1968
  • 58,727

2 Answers2

2

If your grep command outputs lines like 123.456.0.0/16, then you need to pipe them to

awk -F / '{ count[$2]++ } END { for (mask in count) total+=count[mask]*2^(32-mask); print total }'

The command only extracts masks (i.e. what is after /) and counts occurrences of each mask. At the end the number of hosts is calculated for each encountered mask (2^(32-mask)), multiplied by the number of occurrences and summed up.

Notes:

  • No sanity check is performed. E.g. input like 1.2.3.4/40 will be accepted, non-integer output will be calculated. Improve your preliminary grep filter if needed.
  • Each range independently contributes to the total number. If your ranges overlap then you will get an inflated result (I think your try with prips was no better in this).
Giacomo1968
  • 58,727
Kamil Maciorowski
  • 81,893
0

I thought something like this would work since the original poster is running grep to get the CIDR from lines with timeout:

awk -F'[ /]' '/timeout/ {hosts+=2^(32-$2)};ENDFILE{print "Hosts number in "FILENAME": "hosts;total+=hosts;hosts=0};END {print "Total: "total}' ipset*.txt

EDIT- The awk program above runs ok only with GNU awk. ENDFILE is a GNU extension.

I guess BSD awk ignores ENDFILE and runs that section if it was part of the main program section.

This is compatible with GNU and BSD awk.

awk -F'[ /]' '{if (filename != FILENAME) hosts=0};/timeout/ {hosts+=2^(32-$2)};{filename=FILENAME;file_total[filename]=hosts};END{for (i in file_total) {print "Hosts number in "i": "file_total[i];total+=file_total[i]};{print "Total: "total};}' ipset*.txt
Paulo
  • 656