0

Given an extra loopback interface in the 127.0.0.X (X>1) range on BoxA (which can be running either OSX or Linux), I want to bind port 22 of this extra loopback interface to a forward SSH tunnel (ie. local port forward) that is pointed at BoxB.

On OSX this works fine (strangely, in retrospect). [Taking X=2] after bringing up the loopback alias with ifconfig lo0 alias 127.0.0.2 up, SSH can establish a tunnel with ssh -NfL 127.0.0.2:22:localhost:22 BoxB. Then in a new shell on BoxA, ssh 127.0.0.2 logs me into BoxB.

On Ubuntu, I can bring up the loopback alias on BoxA, but when trying to establish the SSH tunnel, ssh complains about not being able to bind (and hence forward) BoxA's port 22. The subsequent ssh 127.0.0.2 (in a new shell on BoxA) gives a fingerprint warning, which if bypassed, logs me back into BoxA. Makes sense - sshd on BoxA is listening to all interfaces.

Looking at the sshd_config in each, both are configured to listen on 0.0.0.0 (and :: for IPv6).

lsof for OSX gives:

launchd       1           root   40u  IPv6 0xddfcabed61001f0d      0t0  TCP *:ssh (LISTEN)
launchd       1           root   41u  IPv4 0xddfcabed6100413d      0t0  TCP *:ssh (LISTEN)
launchd       1           root   43u  IPv6 0xddfcabed61001f0d      0t0  TCP *:ssh (LISTEN)
launchd       1           root   44u  IPv4 0xddfcabed6100413d      0t0  TCP *:ssh (LISTEN)

and for Ubuntu:

sshd       1287              0    3u     IPv4           21903340        0t0        TCP *:ssh (LISTEN)

So both are listening on all interfaces, though I'm not sure why OSX uses 4 processes. In any case, Ubuntu gives the expected behaviour. Why does OSX behave differently?

The follow up question of course, is how to make Ubuntu behave like OSX in this regard.

While I wish for the sshd_config to have state, wildcards and/or logical operators (e.g. "do not listen on 127.0.0.*; listen on 127.0.0.1") like iptables, it doesn't seem to be the case...

Kamil Maciorowski
  • 81,893
Congee
  • 1

1 Answers1

1

That's reproducible using socat (1.7.3.2-2, not 1.7.3.1-2+deb9u1 which ignores reuseport) on Debian (so similar for Ubuntu):

term1$ socat -d -d  TCP4-LISTEN:5555,reuseaddr,fork -
2018/08/03 08:20:43 socat[25084] N listening on AF=2 0.0.0.0:5555

term2$ socat -d -d TCP4-LISTEN:5555,bind=127.0.0.2,reuseaddr,fork -
2018/08/03 08:21:00 socat[25085] E bind(5, {AF=2 127.0.0.2:5555}, 16): Address already in use
2018/08/03 08:21:00 socat[25085] N exit(1)

Now if you add reuseport on both:

term1$ socat -d -d  TCP4-LISTEN:5555,reuseaddr,reuseport,fork -
2018/08/03 08:21:28 socat[25086] N listening on AF=2 0.0.0.0:5555

term2$ socat -d -d TCP4-LISTEN:5555,bind=127.0.0.2,reuseaddr,reuseport,fork -
2018/08/03 08:21:56 socat[25092] N listening on AF=2 127.0.0.2:5555


otherterm$ netstat -tnlp 2>/dev/null|grep :5555 
tcp        0      0 127.0.0.2:5555          0.0.0.0:*               LISTEN      25092/socat         
tcp        0      0 0.0.0.0:5555            0.0.0.0:*               LISTEN      25086/socat

Using an other socat or netcat to connect will show it's properly routed to the correct listening socat depending on the IP address.

So the behaviour difference is most certainly on how the bind and setsockopt calls are done on MacOS and Ubuntu for ssh (maybe launchd has a role to play too?). Using strace, this is added in the working socat case:

 setsockopt(5, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0

UPDATE:
While I will address the letter of the OP's question: make it work, I also think the question doesn't tell about the broader goal and this goal might be achieved with an entirely different solution.

So to alter sshd and ssh behaviour when they bind a socket, it's possible to use a dlsym() wrapper that will alter bind(3)'s function behaviour before the real bind(2) syscall.

compile the file reuseport-wrapper.c below with:

gcc -shared -fPIC -o reuseport-wrapper.so reuseport-wrapper.c -ldl

#define _GNU_SOURCE
#include <dlfcn.h>

#include <sys/socket.h>
#include <stddef.h> /* NULL */

int bind(int socket, const struct sockaddr *address, socklen_t address_len) {
    static const int optval=1;
    static int (*orig_bind)(int, const struct sockaddr *,socklen_t)=NULL;

    if (orig_bind == NULL && (orig_bind=dlsym(RTLD_NEXT,"bind")) == NULL)
        return -1;
    if (address != NULL && (address->sa_family == AF_INET || address->sa_family == AF_INET6)) {
        if (setsockopt(socket, SOL_SOCKET, SO_REUSEPORT, &optval, sizeof optval) != 0) {
            return -1;
        }
    }
    return orig_bind(socket, address, address_len);
}

This will put a wrapper on bind(3) to always set SO_REUSEPORT before binding an inet or inet6 socket.

Place it somewhere (eg as /usr/local/lib/reuseport-wrapper.so but it really doesn't matter since it's explicitely loaded like would a plugin).

Edit /etc/default/ssh and add:

LD_PRELOAD=/usr/local/lib/reuseport-wrapper.so

restart the ssh server (eg service ssh restart)

And run any other tool (including ssh) as root (see below why) with LD_PRELOAD=/usr/local/lib/reuseport-wrapper.so exported. For example:

$ sudo su -
[...]
# LD_PRELOAD=/usr/local/lib/reuseport-wrapper.so ssh -NfL 127.0.0.2:22:localhost:22 userB@BoxB

This command has to be started as root for two reasons: port 22 requires root to bind to it, and even with some additional capability, there's an additional restriction described in socket(7):

SO_REUSEPORT (since Linux 3.9)
Permits multiple AF_INET or AF_INET6 sockets to be bound to an identical socket address. This option must be set on each socket (including the first socket) prior to calling bind(2) on the socket. To prevent port hijacking, all of the processes binding to the same address must have the same effective UID. This option can be employed with both TCP and UDP sockets.

Once this is done:

# netstat -tnlp |grep -w 22
tcp        0      0 127.0.0.2:22            0.0.0.0:*               LISTEN      157/ssh             
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      143/sshd            
tcp6       0      0 :::22                   :::*                    LISTEN      143/sshd

And the tunnel will work as expected.

A.B
  • 6,306