Given bytes that map to a C struct, how to access the bytes via the C struct's variable names

Question

(Bear with me, I think in C and not Python, so you're probably about to see some real dumb stuff...)

I have many (100+) different C structs, pulled into Python (version 3.5.1) as bytes, that I want to be able to access using the original C struct's variable names. Here's a simple example. In Python I have received these bytes:

# In Python:
example1_bytes = b'\x08\x09\x0a\x0b'

Assume these bytes were provided by something running C, using a struct of the following format:

// In C:
struct example1 {
  uint8_t val1;
  uint8_t val2;
  uint8_t val3;
  uint8_t val4; };

How can I process example1_bytes so that I can access them like this:

# In Python:
result = process_example1_bytes(example1_bytes)
print(result.val1)
# Prints "8"
print(result.val2)
# Prints "9"
# Et cetera

Taking it a step further, what if the C struct is more complex and contains arrays and/or sub-structs? For example, something like this:

// In C:
struct substruct {
  uint8_t ss_val1;
  uint8_t ss_val2; };

struct example2 {
  uint8_t val1;
  uint8_t val2;
  struct substruct ss[8]; };

How can I process example2_bytes so that I can access them like this:

# In Python:
result = process_example2_bytes(example2_bytes)
print(result.val1)
print(result.ss[3].ss_val2)

I've experimented a bit using Python's struct unpack, which returns tuples and I think is a step in the right direction, but it hasn't quite gotten me to the usable solution I want. I'm not sure if I need to go down the namedtuple path or take some other direction.

Answer 1

You are looking for the ctypes library, which allows you to define Python wrappers around for complex, underlying C-struct. For a simple type:

import ctypes

example1_bytes = b'\x08\x09\x0a\x0b'

class Example1(ctypes.Structure):
    _fields_ = (
        ('val1', ctypes.c_uint8),
        ('val2', ctypes.c_uint8),
        ('val3', ctypes.c_uint8),
        ('val4', ctypes.c_uint8)
    )

ex1 = Example1.from_buffer_copy(example1_bytes)

print(ex1.val1, ex1.val2, ex1.val3, ex1.val4, sep='|')
# 8|9|10|11

More complex structures:

class substruct(ctypes.Structure):
    _fields_ = (
        ('ss_val1', ctypes.c_uint8),
        ('ss_val2', ctypes.c_uint8),
    )

class Example2(ctypes.Structure):
    _fields_ = (
        ('val1', ctypes.c_uint8),
        ('val2', ctypes.c_uint8),
        ('ss', substruct*8), #array type!
    )

Note, you define an array of type T of size n using the multiplication operator!: T*n

So, along with Structures and Arrays, it supports Unions and pointers, as well as containing all sorts of goodies for C-programmers.

Note, you are using bytes objects, which are immutable and will require a copy when creating a structure. however, if you use a bytearray, you won't require a copy of the underlying buffer!:

In [4]: example1_bytes
Out[4]: b'\x08\t\n\x0b'

In [5]: ex1.val2 = 99

In [6]: example1_bytes
Out[6]: b'\x08\t\n\x0b'

However, using a bytearray:

In [16]: buffer = bytearray(example1_bytes)

In [17]: ex2 = Example1.from_buffer(buffer)

In [18]: ex2
Out[18]: <__main__.Example1 at 0x10a5b5598>

In [19]: buffer
Out[19]: bytearray(b'\x08\t\n\x0b')

In [20]: ex2.val2 = 99

In [21]: buffer
Out[21]: bytearray(b'\x08c\n\x0b')