4

After using Mosh, I noticed that UDP ports held by mosh-server were still in use after all processes exited.

Running netstat -ln, it shows that these ports are in use:

Active Internet connections
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)    

udp4       0      0  10.0.106.61.60002      *.*                               
udp4       0      0  10.0.106.61.61006      *.*

Since this is OS X, the netstat -p flag is not supported, so I cannot find the PID of the process, as https://stackoverflow.com/a/3855359 points out. Instead, I run sudo lsof -i :61006, which returns nothing.

Okay... perhaps I can list all open files with numeric ports and hostnames and grep through it? sudo lsof -i -n -P | grep 61006 ...Nope, nothing again.

Obviously these ports do get cleaned up on reboot... But the challenge here is to diagnose and free them up without a reboot.

Any ideas?

mxxk
  • 411

1 Answers1

2

A little over a year later, I ran into this issue again, but this time diagnosed it deeper. This time, the phantom port-bound process was Java and not Mosh and it was using TCP instead of UDP. In this case, the process turned out to be in an "exiting" state and could not be killed at all, save for a system reboot.

While diagnosing, I discovered some interesting points along the way:

  1. It is possible to see the PID in netstat using the -v flag:

    $ netstat -avn
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)     rhiwat shiwat    pid   epid
...
tcp4       0      0  *.6000                 *.*                    LISTEN      131072 131072  50207      0

  2. Process 50207, which shows up in netstat, does not appear in any variants of lsof, with or without sudo. This is the same behavior documented in the question.

  3. The process couldn't be killed even with sudo kill -9! Inspecting its entry in ps, it looks to be in state an interesting "exiting" state corresponding to "?E":

    $ ps aux
USER               PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
...
mxxk             50207   0.0  0.0        0      0   ??  ?E    5Aug19   0:00.00 (java)

  4. As documented in https://serverfault.com/questions/85799/how-to-kill-an-exiting-process-on-os-x-state-e, it seems that the only way to terminate an exiting process in MacOS is to reboot. (Not sure if Linux behaves differently.)

All of this was done on MacOS High Sierra (10.13.6).

mxxk
  • 411