I'm fairly sure the accepted answer is incorrect. (See tested versions listed at the end.) What Minecraft actually sends with "Open to LAN," and listens for when saying "Scanning for games on your local network," is a UDP multicast packet sent about once a second to 224.0.2.60:4445 with a message such as [MOTD]HostingUserName - WorldName[/MOTD][AD]12435[/AD] where 12345 is the hosting port number.
Methodology: SysInternals Process Monitor filtering to process name javaw.exe and using the toolbar buttons to filter to network activity only, plus Wireshark to examine the packets. No IGMP packets seem to be sent by Minecraft, but this UDP packet is sent. I also sent a UDP packet manually as a test, and it was autodiscovered by Minecraft.
Minecraft versions checked: 1.21.4, 1.9, 1.7.10, 1.4.7, 1.3.1 (the earliest version with the Open to LAN option).
To answer the question "why isn't it working," Minecraft seems to send to the first active network interface only. This is the behavior you get when you don't bind the socket to a specific network interface. In my case, my first active network interface listed in ipconfig was the Hyper-V vEthernet interface, and my LAN was second. I have reported this at https://report.bugs.mojang.com/servicedesk/customer/portal/2/MC-289175 and proposed the proof of concept shown below.
This likely also affects the receiving side. Minecraft is likely only listening on the default network interface on the receiving PC. If that's not its LAN interface, then it won't receive the message even if it's properly sent.
PowerShell to send an autodiscovery message to all network interfaces
$payload = '[MOTD]Your message here[/MOTD][AD]12345[/AD]'
$bytes = [System.Text.Encoding]::UTF8.GetBytes($payload)
foreach ($address in [System.Net.Dns]::GetHostEntry([System.Net.Dns]::GetHostName()).AddressList) {
if ($address.AddressFamily -ne [System.Net.Sockets.AddressFamily]::InterNetwork) {
continue
}
$udpClient = [System.Net.Sockets.UdpClient]::new()
$udpClient.Client.Bind(([IPEndPoint]::new($address, 0)))
[void] $udpClient.Send($bytes, $bytes.Length, '224.0.2.60', 4445)
$udpClient.Dispose()
}
