That simply says:
- If the first numbers are equal return the difference between the second.
- Otherwise, return the diff between the first.
Which then boils down your question to:
"Why does a-b form a valid comparator"?
The answer is: It does not, actually.
A comparator must return a negative number if 'a' comes before 'b', a positive number if 'a' comes after 'b', and 0 if 'a' and 'b' are, in terms of comparison, equal.
a-b actually sort of does that. For example, 3-5 is -2. -2 is negative, which means '3' comes before '5'. Indeed it does.
It's not a good idea because there's such a thing as overflow. -2_000_000_000 surely comes before 1_000_000_000. Nevertheless, -2_000_000_000 - 1_000_000_000 is nevertheless positive (try it!) because of overflow.
The proper way to write that code as above would instead be:
Arrays.sort(nx2Matrix, (a, b) -> {
    if (a[0] == b[0]) return Integer.comparing(a[1], b[1]);
    return Integer.comparing(a[0], b[0]);
});
or perhaps even:
Arrays.sort(nx2Matrix, Comparator
  .comparingInt(a -> a[0])
  .thenComparingInt(a -> a[1]));
Which is a lot more readable, and doesn't suffer from the overflow issue.
The above means:
- For any element, 'unpack it' by taking that element and running the function a[0]on it; this gives you anint, then just compare these ints to know which element is first.
- If that results in 2 elements being considered equal, instead compare the ints you get when running a[1]on each element.