Sending the length of jSON array though sockets (java to python)

Question

I am trying to connect with a python server (from my colleague), with java. The aim (for now) is to send a json array. We start by sending the length first. It works with an equivalent python client, which I am trying to translate into python.

This is an excerpt from my java code

  PrintWriter out = new PrintWriter(clientSocket.getOutputStream(), true);
   long length = (long) arraytosend.length();
   out.print(length);

   String arraytosend = new JSONArray(test2).toString(); 
   out.println(new JSONArray(test2).toString());

The python server first reads the length like this (I just copied the relevant commands de):

  N_BYTES_MSG_LEN = 8
    raw_len = connection.recv(N_BYTES_MSG_LEN)
    # here it output 51 as raw_len

  try:
        msg_len = struct.unpack(MSG_LEN_TYPE, raw_len)[0]
        print msg_len
        logger.debug('announced message length: {}'.format(msg_len))

    except:
        logger.warning('could not interpret message length')
        return None

    # read the incoming message
    raw_data = connection.recv(msg_len)

    if len(raw_data) < msg_len:
        logger.info('lost connection')
        return None

After the "51" it immediately goes to lost connection.

The python client code (which I am trying to translate into java), works like this:

try:
        raw_data = json.dumps(dict(data=data))
    except:
        logger.warning('Failed to create a json representation of the data')
        return False

    # TODO: this could fail for *very* large objects
    raw_len = struct.pack('Q', len(raw_data))


    try:
        connection.sendall(raw_len)
        connection.sendall(raw_data)
    except Exception:
        logger.warning('lost connection while sending data')
        raise

I think it may have to do with me having to send the length in bytes? — dorien, Jul 02 '16 at 15:43

score 0 · Answer 1 · answered Jul 02 '16 at 17:02

0

Your receiver is assuming the length is expressed in 8 bytes (N_BYTES_MSG_LEN). But you send the long as string. PrintWriter.write(long) is the same as PrintWriter.write(Long.valueof(long).toString). For example if the length is 356 it sends "356". You should lef pad your length first: "00000356".

answered Jul 02 '16 at 17:02

Heri

4,368
1
31
51

I added zeros in from of the lengths, but now python reads it all wrong: and says received length: 3688782553318174768. From what I've read there is a problem translating struct.unpack('Q',...) to java. – dorien Jul 02 '16 at 17:18
I don't know python. But it seems that python reads the 8 bytes as the binary representation of a number which is completely something else than a string. Translating to a such a big number let's me assume that python interpretes the bytes in reverse order. – Heri Jul 02 '16 at 17:25
Mmmm I don't know. It seems like I am sending a string with 8 characters, that's not necessarily 8 bytes is it? – dorien Jul 02 '16 at 17:35
As long all characters are digits (0x30 .. 0x39) each character is one byte. As long as you do not construct your PrintWriter with a defined charset it will use the platform default. This is different on windows, mac an linux! Thus it's good practice always indicate the charset if it comes to byte/String conversion (or vice versa). – Heri Jul 02 '16 at 17:43
Doesn't seem to work. I think it's really expecting bytes or so in machine code. – dorien Jul 02 '16 at 20:49
Yes. The big number you posted, converted to hex dump of the binary long format, gives as Hex 3331303030303030, which is, considering the reverse byte order, the digit string of "00000013". So you need to convert your long into the corresponding binary byte representation as an array. Maybe this link helps: http://stackoverflow.com/questions/4485128/how-do-i-convert-long-to-byte-and-back-in-java – Heri Jul 03 '16 at 08:45

score 0 · Accepted Answer · answered Jul 05 '16 at 18:23

I found the solution, you have to take into account that java uses. You can do this by changing the server (python) code to:

   raw_len = struct.pack('!Q', len(raw_data))

And you can then send it with:

    JSONObject request = new JSONObject();
    request.append("data", array);
    byte[] outBytes = jsonObject.toString().getBytes("UTF-8");
    out.writeLong(outBytes.length);

Sending the length of jSON array though sockets (java to python)

2 Answers2