Since I found out that it's impossible to have unsigned bytes in java
This is incorrect. There are lots of ways. You can even use byte to represent an unsigned byte. You just need to perform a mapping in places that require it; e.g.
byte b = ...
int unsigned = (b >= 0) ? b : (b + 256);
... and that essentially they take up the same memory as an int.
That is also incorrect. It is true for a byte variable or field. However, the bytes in a byte array occupy 1/4 of the space of integers in an int array.
... is there really any difference when you send them over a packet?
Well yes. A byte sent over the network (in the natural fashion) takes 1/4 of the number of bits as an int sent over the network. If you are sending an "8 bit quantity" as 32 bits, then you are wasting network bandwidth.