"struct.error: unpack requires a string argument of length 4" when slicing [:3]

Question

I have a problem with a little server-client assignment in python 2.7.

The client can send 5 types of requests to the server:

1. get the server's IP
2. get contents of a directory on the server
3. run cmd command on the server and get the output
4. open a calculator on the server
5. disconnect

This is the error I get:

error:
msg_type, data_len = unpack("BH", client_structs[:3])
struct.error: unpack requires a string argument of length 4

Code:

client_structs = client_soc.recv(1024)
msg_type, data_len = unpack("BH", client_structs[:3])

Doesn't the substring contain 4 chars including the null?

Would appreciate explanation about this error + how to solve it.

Entire server code:

__author__ = 'eyal'

from struct import pack, unpack, calcsize
import socket
from os import listdir
from subprocess import check_output, call


def server():
    ser_soc = socket.socket()
    ser_soc.bind(("0.0.0.0", 8080))
    ser_soc.listen(1)
    while True:
        accept_flag = raw_input("Would you like to wait for a client? (y/n) ")
        if accept_flag == "y":
            client_soc, client_address = ser_soc.accept()
            while True:
                client_structs = client_soc.recv(1024)
                data_size = calcsize(client_structs) - 3
                data_str = 'c' * data_size
                unpacked_data = unpack("BH" + data_str, client_structs)
                if unpacked_data[0] == 1:
                    ip = socket.gethostbyname(socket.gethostname())
                    ip_data = 'c' * len(ip)
                    to_send = pack("BH" + str(len(ip)) + ip_data, unpacked_data[0], len(ip), ip)
                elif unpacked_data[0] == 2:
                    content = listdir(str(unpacked_data[2]))
                    content_str = "\r\n".join(content)
                    content_data = 'c' * len(content_str)
                    to_send = pack("BH" + str(len(content_str)) + content_data, unpacked_data[0],
                                   len(content_str), content_str)
                elif unpacked_data[0] == 3:
                    command = str(unpacked_data[2:]).split()
                    output = check_output(command)
                    message_data = 'c' * len(output)
                    to_send = pack("BH" + message_data, unpacked_data[0], len(output), output)
                elif unpacked_data[0] == 4:
                    call("gnome-calculator")
                    msg_data = 'c' * len("The calculator is open.")
                    to_send = pack("BH" + msg_data, unpacked_data[0], len("The calculator is open."),
                                   "The calculator is open.")
                elif unpacked_data[0] == 5:
                    client_soc.close()
                    break
                else:
                    to_send = pack("BH" + 'c' * len("invalid message type, try again"),
                                   unpacked_data[0], len("invalid message type, try again"),
                                   "invalid message type, try again")
                if unpacked_data[0] != 5:
                    client_soc.send(to_send)
        else:
            break
    ser_soc.close()


def main():
    server()


if __name__ == "__main__":
    main()

Entire client code:

__author__ = 'eyal'


from struct import pack, unpack, calcsize
import socket


def client():
    my_soc = socket.socket()
    my_soc.connect(("127.0.0.1", 8080))
    while True:
        send_flag = raw_input("Would you like to send the server a request? (y/n) ")
        if send_flag == "y":
            msg_code = input("What type of request would you like to send?\n"
                             "1. Get the server's IP address.\n"
                             "2. Get content of a directory on the server.\n"
                             "3. Run a terminal command on the server and get the output.\n"
                             "4. Open a calculator on the server.\n"
                             "5. Disconnect from the server.\n"
                             "Your choice: ")
            if msg_code == 1 or msg_code == 4 or msg_code == 5:
                to_send = pack("BH", msg_code, 0)
            elif msg_code == 2:
                path = raw_input("Enter path of wanted directory to get content of: ")
                to_send = pack("BH" + 'c' * len(path), msg_code, len(path), path)
            elif msg_code == 3:
                command = raw_input("Enter the wanted terminal command, including arguments: ")
                to_send = pack("BH" + 'c' * len(command), msg_code, len(command), command)
            else:
                print "Invalid message code, try again\n"

            if 1 <= msg_code <= 5:
                my_soc.send(to_send)
        else:
            break
    data = my_soc.recv(1024)
    unpacked_data = unpack("BH" + 'c' * (calcsize(data) - 3), data)
    print "The server's response to your type-" + str(msg_code) + " request:"
    print unpacked_data[2]
    my_soc.close()


def main():
    client()


if __name__ == "__main__":
    main()

Answer 1

Why would there be any null included? The slice includes 3 characters, which is exactly how many you specified — indexed 0 to 2.

Instead, slice it with client_structs[:4]. (Or, as abarnert points out, slice [:3] or [:struct.calcsize('>BH')] and pack/unpack with ">BH" to avoid endianness problems.)

Python is not as tricky about most fencepost errors as most C-ish languages, so you may have inadvertently gotten too clever for yourself.

Answer 2

This is explained in a big box at the very top of the docs:

Note By default, the result of packing a given C struct includes pad bytes in order to maintain proper alignment for the C types involved; similarly, alignment is taken into account when unpacking. This behavior is chosen so that the bytes of a packed struct correspond exactly to the layout in memory of the corresponding C struct. To handle platform-independent data formats or omit implicit pad bytes, use standard size and alignment instead of native size and alignment: see Byte Order, Size, and Alignment for details.

So, on most platforms, 'BH' is 4 bytes—1 byte for the B, 1 byte of padding so the H ends up aligned, and 2 bytes for the H. But on some platforms it might be 3 bytes, or 6. For that matter, B and H aren't even guaranteed to be 1 and 2 bytes respectively in native format, so it could be 17 bytes.

If you don't want that, because you're using this for a network protocol rather than for accessing C structs in memory, don't use the default native order, padding, and alignment. On all platforms, '>BH' is 3 bytes—1 byte for the B, and 2 for the H. (And the H is always in network order, even if one of the two machines communicating is little-endian.)

---

If you do want that, then use [:struct.calcsize('BH')] instead of [:3], to make sure you read the right number of bytes for your platform. But honestly, if you do want that, you're shouldn't.