The theory of the subnet mask is that it defines what part of the IP address is the network address and what part of the IP address is the host address:
10.100.0.1 - IP address;
255.0.0.0 - Subnet mask;
10 - network address, 100.0.1 - host address.
Hosts within the same subnet can talk directly to each other. That means if host A and B are located within the same subnet and A wants to talk to B then A will send its traffic directly to B. If host A wants to talk to host C which is located in a different subnet then A will have to route this traffic to a gateway which knows (hopefully) how to reach the destination network. So, it's up to the host to decide where to send traffic:
- Directly to the host (second host is within the same subnet)
- To a gateway (second host belongs to a different subnet)
What happens in your case is that your "Authorized" clients have IP addresses 10.100.0.10 - 10.100.0.250 (I assume the subnet mask is 255.0.0.0). The server has IP address 10.100.0.1. To a host from the "Authorized" range this server is located in the same subnet.
If host 10.100.0.10 from the "Authorized" range wants to talk to the server - it first checks if this server is located within the same subnet or not. For the host 10.100.0.10 with subnet mask 255.0.0.0 same subnet would be all hosts within the range 10.0.0.1 - 10.255.255.254. Server's IP address happens to be in this range. For this reason a host from "Authorized" range makes an attempt to reach the server directly and (assuming they are located on the same Layer 2 network) this attempt succeeds.
In this case even though the server has different subnet mask - it happens to be located in the bigger subnet (which is also a subnet for the "Authorized" clients). If your server will have different second byte in the IP address (10.150.0.1 for example) it will be unable to reply to the host from the "Authorized" range, because from the server's perspective, the "Authorized" range would look like a different subnet and the server would need to send traffic to a router. If there would be no router - then there would be no communication.
If you want to separate your network to the "Guests" and "Authorized" parts then you need to make them to be located in different subnets that do not overlap.
For example:
- "Guests" -
10.10.0.1, subnet mask 255.255.0.0
- "Authorized" -
10.20.0.1, subnet mask 255.255.0.0
The Server would be located within "Authorized" part of the network having IP address 10.20.0.100, subnet mask 255.255.0.0.
With this setup these subnets will be effectively separated from each other, since parts of IP addresses representing their subnet will differ:
10.10 for Guests
10.20 for Authorized
At this point communication between these subnets will be possible only via a router that has interfaces in both subnets.
Also, it is worth mentioning, that while all your computers are on the same layer 2 network, nothing prevents hosts from the "Guests" range from manually assigning themselves IP addresses from the "Authorized" range. This will effectively make them to be part of the Authorized network.