On macOS Sonoma 14.6.1 (the kernel is Darwin 23.6.0) I start five nodes of Hazelcast 3.12 on localhost to emulate a split–brain. Each of those nodes listens on the port for communications from other nodes, and communicates to other nodes from the configured ports:
| listens on | 5701 | 5702 | 5703 | 5704 | 5705 |
|------------|-------|-------|-------|-------|-------|
| sends from | 33712 | 33721 | 33731 | 33741 | 33751 |
| | 33713 | 33723 | 33732 | 33742 | 33752 |
| | 33714 | 33724 | 33734 | 33743 | 33753 |
| | 33715 | 33725 | 33735 | 33745 | 33754 |
Now I want to emulate a split–brain scenario by dropping all tcp packets between nodes–1,2,3 and nodes–4,5. For that purpose I created a pf (Packet Filter) rules in /etc/pf.anchors/hazelcast:
block out quick on lo0 proto tcp from 127.0.0.1 port 33711:33715 to 127.0.0.1 port 5704
block out quick on lo0 proto tcp from 127.0.0.1 port 33721:33725 to 127.0.0.1 port 5704
block out quick on lo0 proto tcp from 127.0.0.1 port 33731:33735 to 127.0.0.1 port 5704
block in quick on lo0 proto tcp from 127.0.0.1 port 33711:33715 to 127.0.0.1 port 5704
block in quick on lo0 proto tcp from 127.0.0.1 port 33721:33725 to 127.0.0.1 port 5704
block in quick on lo0 proto tcp from 127.0.0.1 port 33731:33735 to 127.0.0.1 port 5704
block out quick on lo0 proto tcp from 127.0.0.1 port 33711:33715 to 127.0.0.1 port 5705
block out quick on lo0 proto tcp from 127.0.0.1 port 33721:33725 to 127.0.0.1 port 5705
block out quick on lo0 proto tcp from 127.0.0.1 port 33731:33735 to 127.0.0.1 port 5705
block in quick on lo0 proto tcp from 127.0.0.1 port 33711:33715 to 127.0.0.1 port 5705
block in quick on lo0 proto tcp from 127.0.0.1 port 33721:33725 to 127.0.0.1 port 5705
block in quick on lo0 proto tcp from 127.0.0.1 port 33731:33735 to 127.0.0.1 port 5705
block out quick on lo0 proto tcp from 127.0.0.1 port 33741:33745 to 127.0.0.1 port 5701
block out quick on lo0 proto tcp from 127.0.0.1 port 33741:33745 to 127.0.0.1 port 5702
block out quick on lo0 proto tcp from 127.0.0.1 port 33741:33745 to 127.0.0.1 port 5703
block in quick on lo0 proto tcp from 127.0.0.1 port 33741:33745 to 127.0.0.1 port 5701
block in quick on lo0 proto tcp from 127.0.0.1 port 33741:33745 to 127.0.0.1 port 5702
block in quick on lo0 proto tcp from 127.0.0.1 port 33741:33745 to 127.0.0.1 port 5703
block out quick on lo0 proto tcp from 127.0.0.1 port 33751:33755 to 127.0.0.1 port 5701
block out quick on lo0 proto tcp from 127.0.0.1 port 33751:33755 to 127.0.0.1 port 5702
block out quick on lo0 proto tcp from 127.0.0.1 port 33751:33755 to 127.0.0.1 port 5703
block in quick on lo0 proto tcp from 127.0.0.1 port 33751:33755 to 127.0.0.1 port 5701
block in quick on lo0 proto tcp from 127.0.0.1 port 33751:33755 to 127.0.0.1 port 5702
block in quick on lo0 proto tcp from 127.0.0.1 port 33751:33755 to 127.0.0.1 port 5703
At the end of /etc/pf.conf, I added:
anchor "hazelcast/*"
load anchor "hazelcast" from "/etc/pf.anchors/hazelcast"
Then I ran the command:
sudo pfctl -Evf /etc/pf.conf
It printed:
...
Loading anchor hazelcast from /etc/pf.anchors/hazelcast
block drop out quick on lo0 inet proto tcp from 127.0.0.1 port 33741:33745 to 127.0.0.1 port = 5701
block drop out quick on lo0 inet proto tcp from 127.0.0.1 port 33741:33745 to 127.0.0.1 port = 5702
block drop out quick on lo0 inet proto tcp from 127.0.0.1 port 33741:33745 to 127.0.0.1 port = 5703
block drop out quick on lo0 inet proto tcp from 127.0.0.1 port 33751:33755 to 127.0.0.1 port = 5701
block drop out quick on lo0 inet proto tcp from 127.0.0.1 port 33751:33755 to 127.0.0.1 port = 5702
block drop out quick on lo0 inet proto tcp from 127.0.0.1 port 33751:33755 to 127.0.0.1 port = 5703
block drop out quick on lo0 inet proto tcp from 127.0.0.1 port 33711:33715 to 127.0.0.1 port = 5704
block drop out quick on lo0 inet proto tcp from 127.0.0.1 port 33721:33725 to 127.0.0.1 port = 5704
block drop out quick on lo0 inet proto tcp from 127.0.0.1 port 33731:33735 to 127.0.0.1 port = 5704
block drop out quick on lo0 inet proto tcp from 127.0.0.1 port 33711:33715 to 127.0.0.1 port = 5705
block drop out quick on lo0 inet proto tcp from 127.0.0.1 port 33721:33725 to 127.0.0.1 port = 5705
block drop out quick on lo0 inet proto tcp from 127.0.0.1 port 33731:33735 to 127.0.0.1 port = 5705
block drop in quick on lo0 inet proto tcp from 127.0.0.1 port 33741:33745 to 127.0.0.1 port = 5701
block drop in quick on lo0 inet proto tcp from 127.0.0.1 port 33741:33745 to 127.0.0.1 port = 5702
block drop in quick on lo0 inet proto tcp from 127.0.0.1 port 33741:33745 to 127.0.0.1 port = 5703
block drop in quick on lo0 inet proto tcp from 127.0.0.1 port 33751:33755 to 127.0.0.1 port = 5701
block drop in quick on lo0 inet proto tcp from 127.0.0.1 port 33751:33755 to 127.0.0.1 port = 5702
block drop in quick on lo0 inet proto tcp from 127.0.0.1 port 33751:33755 to 127.0.0.1 port = 5703
block drop in quick on lo0 inet proto tcp from 127.0.0.1 port 33711:33715 to 127.0.0.1 port = 5704
block drop in quick on lo0 inet proto tcp from 127.0.0.1 port 33721:33725 to 127.0.0.1 port = 5704
block drop in quick on lo0 inet proto tcp from 127.0.0.1 port 33731:33735 to 127.0.0.1 port = 5704
block drop in quick on lo0 inet proto tcp from 127.0.0.1 port 33711:33715 to 127.0.0.1 port = 5705
block drop in quick on lo0 inet proto tcp from 127.0.0.1 port 33721:33725 to 127.0.0.1 port = 5705
block drop in quick on lo0 inet proto tcp from 127.0.0.1 port 33731:33735 to 127.0.0.1 port = 5705
pf enabled
Token : 14399845021355597821
Then I started the nodes 1 and 4. In the log of the node-1, I see:
Initialized new cluster connection between /127.0.0.1:33715 and /127.0.0.1:5704
What am I doing wrong?
