How to bind to all addresses of only one network interface (Linux)?

Question

What I am trying to achieve is binding an IPv6 socket to any address of just one particular device, not system-wide. My intuition is that I could setsockopt() with SO_BINDTODEVICE followed by a bind to ::. It mostly does what I expect it to do. The behaviour is the same in v4.

The sockets bound to an interface with SO_BINDTODEVICE will only accept connections made to addresses on that interface. That much is expected.

However, I run into errno "Address already in use", if I'm trying to bind to a source port on interface B when there is a socket using the same port but on interface A.

Ex:

• nic A has IPv6 fd00:aaaa::a/64
• nic B has IPv6 fd00:bbbb::b/64
• they do not share networks.

Put shortly (pseudocode):

• process 1 calls socket(...) and binds bind(fd00:aaaa::a/64, 9000).
• process 2 calls socket(...) and setsockopt(SO_BINDTODEVICE, "B")
• process 2 (continued) calls bind(::, 9000) and gets EADDRINUSE. Why?

How does SO_BINDTODEVICE really work? Does the determination for "addresses in use" ignore, conservatively, the interface sockets are bound to? Is it a networking stack layering issue?

Example traces:

1. I start a listening socket (server) on a specific address: nc -l fd00:aaaa::a 9000. Its trace is as follows:

socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
bind(3, {
    sa_family=AF_INET6,
    sin6_port=htons(9000),
    inet_pton(AF_INET6, "fd00:aaaa::a", &sin6_addr),
    sin6_flowinfo=0, sin6_scope_id=0
}, 28) = 0
listen(3, 1)                            = 0
accept(3, ...

2. Connecting to it (client) fails if I bind to the port in use by the other interface, even though I've already bound to a different interface:

socket(PF_INET6, SOCK_STREAM, IPPROTO_IP) = 3
setsockopt(3, SOL_SOCKET, SO_BINDTODEVICE, "nicB\0", 5) = 0
bind(3, {sa_family=AF_INET6,
         sin6_port=htons(9000),
         inet_pton(AF_INET6, "::", &sin6_addr), 
         sin6_flowinfo=0,
         sin6_scope_id=0
        }, 28) = -1 //EADDRINUSE (Address already in use)

3. However, if I don't specify the port, then all is good when binding to :: (while the listener still runs):

socket(PF_INET6, SOCK_STREAM, IPPROTO_IP) = 3
setsockopt(3, SOL_SOCKET, SO_BINDTODEVICE, "nicB\0", 5) = 0
bind(3, {
    sa_family=AF_INET6,
    sin6_port=htons(0),
    inet_pton(AF_INET6, "::", &sin6_addr),
    sin6_flowinfo=0, sin6_scope_id=0
}, 28) = 0
connect(3, {
    sa_family=AF_INET6,
    sin6_port=htons(9000),
    inet_pton(AF_INET6, "fd00:aaaa::a", &sin6_addr),
    sin6_flowinfo=0, sin6_scope_id=0
}, 28) = ...

Note: This is on 3.19.0-68-generic x86_64 . Ubuntu 14.04. In case it makes a difference, for my tests, nicB is a macvlan in bridge mode whose parent is nicA.

Answer 1

I've found a satisfying explanation for this problem.

The observation is that even though only interface "A" has IP fd00:aaaa::a/64 when the program is started, the listening socket could accept connections coming in over different interfaces if they were to receive that IP in the future. IPs can be added and removed -- and server processes listening on :: or (0.0.0.0 in v4) need not be restarted when interfaces receive new IPs.

So, in a way, process 1's bind("fd00:aaaa::a/64", 9000) binds implicitly to ALL interfaces. Even though process 2 only needs to use interface B, process 1's already got first dibs, because it uses port 9000 on both interfaces, so process 2 gets denied.

If I change program 1 so that it too uses SO_BINDTODEVICE (to interface "A"), then both processes can bind(::, 9000) without issues.

experiment

I've tested this out with a little LD_PRELOAD goop, which precedes calls to bind() with setsockopt(...SO_BINDTODEVICE...). The two following TCP listeners can both bind to port 9000 simulateneously if they are each bound to a different interface.

# LD_PRELOAD=./bind_hook.so _BINDTODEVICE=eth0 nc -l 0.0.0.0 9000

# LD_PRELOAD=./bind_hook.so _BINDTODEVICE=eth1 nc -l 0.0.0.0 9000

If only one of the two uses SO_BINDTODEVICE, then the last process gets EADDRINUSE. Which is the situation put forward in the question.

I'm including the C code (GNU/Linux) for my tool in case someone needs something similar:

/**                                                                                                                                                              
 * bind_hook.c                                                                                                                                                   
 *                                                                                                                                                               
 * Calls setsockopt() with #SO_BINDTODEVICE before _any_ bind().                                                                                                       
 * The name of the interface to bind to is obtained from                                                                                                         
 * environment variable `_BINDTODEVICE`.
 *
 * Needs root perms. errors are not signalled out.
 *                                                                                                                                                               
 * Compile with:
 *   gcc -Wall -Werror -shared -fPIC -o bind_hook.so -D_GNU_SOURCE bind_hook.c -ldl
 * Example usage:
 *   LD_PRELOAD=./bind_hook.so _BINDTODEVICE=eth0 nc -l 0.0.0.0 9500
 *                                                                                                                                                               
 * @author: init-js
 **/
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <net/if.h>
#include <dlfcn.h>
#include <errno.h>


static char iface[IF_NAMESIZE];
static int (*bind_original)(int, const struct sockaddr*, socklen_t addrlen);

int bind(int sockfd, const struct sockaddr *addr,
         socklen_t addrlen);

__attribute__((constructor))
void ctor() {
        bind_original = dlsym(RTLD_NEXT, "bind");

        char *env_iface = getenv("_BINDTODEVICE");
        if (env_iface) {
                strncpy(iface, env_iface, IF_NAMESIZE - 1);
        }
}

/* modified bind() -- call setsockopt first */
int bind(int sockfd, const struct sockaddr *addr, socklen_t addrlen) {
        int _errno;

        if (iface[0]) {
                /* preserve errno */
                _errno = errno;
                setsockopt(sockfd, SOL_SOCKET, SO_BINDTODEVICE,
                           (void*)iface, IF_NAMESIZE);
                errno = _errno;
        }
        return bind_original(sockfd, addr, addrlen);
}

Answer 2

If there is a socket already bound to a specific IP address and port, you can only bind to that port again if you provide another specific IP address. You cannot use INADDR_ANY in this circumstance.