I've got a raspi (in my robot boat project) connected to the internet over a 3G connection using a dongle with a simcard. I then use ngrok to expose ssh to a static address so that I can always ssh into it using:
ssh -p 29xxx pi@1.tcp.ngrok.io
That has always worked great for me. Today I wanted to do another test with my robot, but I can't ssh into the machine through ngrok anymore. It just gives me kex_exchange_identification: Connection closed by remote host
So this is what I did to debug it:
- I can ssh into it over the local network fine, so I guess sshd on the pi itself is not the problem.
- I checked the ngrok website to see if the pi connected to the ngrok network. It lists the connection saying it was established a couple minutes ago.
- I ssh'ed into the pi over the local network, stopped the tunnel and then manually started it to see the logs. It shows me this:
pi@myrobot:~$ ngrok tcp -remote-addr=1.tcp.ngrok.io:29xxx --log=stdout 22
INFO[09-11|09:39:33] no configuration paths supplied
INFO[09-11|09:39:33] using configuration at default config path path=/home/pi/.ngrok2/ngrok.yml
INFO[09-11|09:39:33] open config file path=/home/pi/.ngrok2/ngrok.yml err=nil
t=2020-09-11T09:39:33+0000 lvl=info msg="starting web service" obj=web addr=127.0.0.1:4040
t=2020-09-11T09:39:34+0000 lvl=info msg="tunnel session started" obj=tunnels.session
t=2020-09-11T09:39:34+0000 lvl=info msg="client session established" obj=csess id=1b6463ec0724
t=2020-09-11T09:39:34+0000 lvl=info msg="started tunnel" obj=tunnels name=command_line addr=//localhost:22 url=tcp://1.tcp.ngrok.io:29xxx
t=2020-09-11T09:39:38+0000 lvl=warn msg="failed to check for update" obj=updater err="Post https://update.equinox.io/check: context deadline exceeded"
- I then tried ssh'ing into the pi over the internet again using verbose output (
ssh -v -p 29xxx pi@1.tcp.ngrok.io), which gives me the following output:
$ ssh -v -p 29xxx pi@1.tcp.ngrok.io
OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 1.tcp.ngrok.io [3.13.191.xxx] port 29xxx.
debug1: Connection established.
debug1: identity file /home/kramer65/.ssh/id_rsa type 0
debug1: identity file /home/kramer65/.ssh/id_rsa-cert type -1
debug1: identity file /home/kramer65/.ssh/id_dsa type -1
debug1: identity file /home/kramer65/.ssh/id_dsa-cert type -1
debug1: identity file /home/kramer65/.ssh/id_ecdsa type -1
debug1: identity file /home/kramer65/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/kramer65/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/kramer65/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/kramer65/.ssh/id_ed25519 type -1
debug1: identity file /home/kramer65/.ssh/id_ed25519-cert type -1
debug1: identity file /home/kramer65/.ssh/id_ed25519_sk type -1
debug1: identity file /home/kramer65/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/kramer65/.ssh/id_xmss type -1
debug1: identity file /home/kramer65/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
kex_exchange_identification: Connection closed by remote host
In the terminal on the pi I see nothing happening. But the output above also suggests that it never actually reaches the pi.
Could it be that ngrok doesn't pass on the connection properly? Is the problem in the pi? or on my laptop locally? All tips are welcome!
[EDIT]
After some more debugging I found the problem has to be with the mobile connection. When I remove the 3G-dongle and connect the pi to the internet over wifi I can perfectly ssh into it using the ngrok address. But when I connect over 3G I cannot. I checked whether the internet over 3G works by ssh'ing into the pi over the wifi network and using curl ip.me to check whether the public ip changes when I connect over 3G (plus, a ping to 8.8.8.8 increases from 10ms to about 40ms).
I also checked the syslog and that doesn't say the anything about the incoming message (I would expect a Started Session c7 of user pi). Furthermore, when I start the ngrok tunnel over 3G it adds the line below to the output. The rest (including "client session established") is the same though
lvl=warn msg="failed to check for update" obj=updater err="Post https://update.equinox.io/check: context deadline exceeded"
So why would the tunnel fail over 3G? Could it be that my telco closes all kinds of ports or blocks traffic? Any way of debugging this further?
[EDIT]
Ok, this is weird, but the problem somehow solved itself. Today I tried it again, in various combinations. They all failed. I then went sway for some time. After I came back and out of sheer hopelessness tried again, it suddenly worked. I'm just stumped. I rebooted, disconnected from the 3G network again and now it works every time again. This kind of frightens me. I had it fail so many times over a week or so, with a million combinations, but nothing worked. I had it up and it failed, and without changing anything it suddenly works again.
This does frighten me a bit. I don't understand why it fails, which means that when my robot boat is on the North sea it will suddenly fail again and it will become unreachable.
But anyway. It works, so for now I won't bother about it anymore.
