What's the recommended "pythonic way" for unpacking a byte encoded message with arbitrary bit length fields?

Question

Suppose I have a byte-packed message with the following schema:

Message Name: Ball
    Field #1: 
        Name: Color
        Type: unsigned int
        Start Bit: 0
        Bit Length: 4
    Field #2: 
        Name: Radius
        Type: single
        Start Bit: 4
        Bit Length: 32
    Field #3: 
        Name: New
        Type: bool
        Start Bit: 36
        Bit Length: 1

What is the recommended pythonic way to convert the byte sequence into a python variables? Would the "struct" module work well for unpacking byte arrays that have arbitrary bit length fields?

I suspect this question will end up closed for one reason or another, but in the interest of helping anyway, [`construct`](http://construct.readthedocs.io) would probably be better-suited than `struct`. — jwodder, Apr 27 '17 at 21:28
Hmm, check out [this](http://varx.org/wordpress/2016/02/03/bit-fields-in-python/). — juanpa.arrivillaga, Apr 27 '17 at 21:32
@juanpa.arrivillaga. I think that is a perfect example. I was very familiar with how to do this in C, but I was curious the proper python method. I like the references in that article for using native python libraries. — Pat, Apr 27 '17 at 21:36
@jwodder: Thank you. I'll check that out, but was hoping for a native python approach. — Pat, Apr 27 '17 at 21:36
@Pat yeah, if you want to do something like you would do it in C, generally `ctypes` and `struct` with maybe `array` modules is what you need to look at. — juanpa.arrivillaga, Apr 27 '17 at 21:36
@Pat this even lets you do all sorts of [unsafe, dark magic](http://stackoverflow.com/a/15702647/5014455) that you might not think is possible in Python. Although, I wouldn't recommend the above. — juanpa.arrivillaga, Apr 27 '17 at 21:41
Shouldn't the title of your question be about the unpacking a **bit** encoded message? — martineau, Apr 27 '17 at 21:56

score 3 · Accepted Answer · edited Sep 02 '22 at 14:33

What is the recommended pythonic way to convert the byte sequence into a python variables?

What you want is to read this python doc page about struct and bytearrays.

It will show you how to pack and unpack data using format.

Would the "struct" module work well for unpacking byte arrays that have arbitrary bit length fields?

It will show you how to pack and unpack data using format.

Yup, as follows:

import struct
color, radius, new = struct.unpack("If?", incoming_bytes)

Check the format characters to define the format string and you're done.

You could use more bloated libraries like construct, but TBH, this format is super simple and if you unpack early and pack late, you're free to organize your data as you want in your code.

e.g.:

class Ball:
    def __init__(self, color, radius, new):
        self.color = color
        self.radius = radius
        self.new = new

Ball(*unpack("If?", incoming_bytes))

What's the recommended "pythonic way" for unpacking a byte encoded message with arbitrary bit length fields?

1 Answers1