How to pack and unpack using ctypes (Structure <-> str)

Question

This might be a silly question but I couldn't find a good answer in the docs or anywhere.

If I use struct to define a binary structure, the struct has 2 symmetrical methods for serialization and deserialization (pack and unpack) but it seems ctypes doesn't have a straightforward way to do this. Here's my solution, which feels wrong:

from ctypes import *

class Example(Structure):
    _fields_ = [
        ("index", c_int),
        ("counter", c_int),
        ]

def Pack(ctype_instance):
    buf = string_at(byref(ctype_instance), sizeof(ctype_instance))
    return buf

def Unpack(ctype, buf):
    cstring = create_string_buffer(buf)
    ctype_instance = cast(pointer(cstring), POINTER(ctype)).contents
    return ctype_instance

if __name__ == "__main__":
    e = Example(12, 13)
    buf = Pack(e)
    e2 = Unpack(Example, buf)
    assert(e.index == e2.index)
    assert(e.counter == e2.counter)
    # note: for some reason e == e2 is False...

That looks about right to me. ctypes is not for serialization, so the fact that you can do it in 7 lines of code seems pretty good actually. — Jason Orendorff, Dec 01 '09 at 12:17

score 36 · Accepted Answer · edited Jun 20 '20 at 09:12

36

The PythonInfo wiki has a solution for this.

FAQ: How do I copy bytes to Python from a ctypes.Structure?
def send(self):
    return buffer(self)[:]
FAQ: How do I copy bytes to a ctypes.Structure from Python?
def receiveSome(self, bytes):
    fit = min(len(bytes), ctypes.sizeof(self))
    ctypes.memmove(ctypes.addressof(self), bytes, fit)

Their send is the (more-or-less) equivalent of pack, and receiveSome is sort of a pack_into. If you have a "safe" situation where you're unpacking into a struct of the same type as the original, you can one-line it like memmove(addressof(y), buffer(x)[:], sizeof(y)) to copy x into y. Of course, you'll probably have a variable as the second argument, rather than a literal packing of x.

edited Jun 20 '20 at 09:12

Community

1
1

answered Dec 01 '09 at 17:24

Mark Rushakoff

249,864
45
407
398

1

I tested this solution and it works as well. What was more important to me was that you found an official python.org entity (the FAQ in the wiki is good enough) state that hacking it is the way to go. I just felt as though these 2 functions/methods had to be somewhere in ctypes.py already so hacking it using pointers seemed very unpythonic. I know some people say ctypes isn't built for serializing etc, but I like the ctypes OOP-ness a lot more than the perl-ish struct module. – Mr Temp Dec 01 '09 at 18:06
1

What about Python3 which doesn't have `buffer`? If you try to simply replace it with `memoryview`, you'll get `TypeError: invalid indexing of 0-dim memory` – Jonathon Reinhart Feb 28 '17 at 15:39
11

In Python3 you can use `bytes(self)` to extract the structure bytes. – Zanapher May 12 '17 at 14:19

score 19 · Answer 2 · edited May 25 '20 at 00:35

19

Have a look at this link on binary i/o in python:

http://www.dabeaz.com/blog/2009/08/python-binary-io-handling.html

Based on this you can simply write the following to read from a buffer (not just files):

g = open("foo","rb")
q = Example()
g.readinto(q)

To write is simply:

g.write(q)

The same for using sockets:

s.send(q)

and

s.recv_into(q)

I did some testing with pack/unpack and ctypes and this approach is the fastest except for writing straight in C

edited May 25 '20 at 00:35

sigjuice

28,661
12
68
93

answered Dec 29 '09 at 00:40

Ralph Paul

191
2

8

In 2.6+, a more generic `pack` is just `bytearray(q)`, which also uses the buffer protocol. For generic unpacking 2.6 also added, for example, `Example.from_buffer(buf)` if `buf` is mutable, else `Example.from_buffer_copy(buf)`. – Eryk Sun Jul 16 '13 at 14:24

score 13 · Answer 3 · answered May 24 '18 at 18:41

13

Tested on Python3

e = Example(12, 13)
serialized = bytes(e)
deserialized = Example.from_buffer_copy(serialized)

answered May 24 '18 at 18:41

TomDotTom

6,238
3
41
39

milahu · Answer 4 · 2023-03-31T19:12:28.610

see also the kaitai binary parsing framework

Kaitai Struct is a declarative language used to describe various binary data structures

example: sqlite3.py generated from sqlite3.ksy

# sqlite3.py was generated by
# kaitai-struct-compiler -t python --python-package . sqlite3.ksy
from . import sqlite3

db = sqlite3.Sqlite3.from_file("some_database.db")

print(f"page size: {db.len_page} bytes")
print(f"db size: {db.num_pages} pages", )

How to pack and unpack using ctypes (Structure <-> str)

4 Answers4

Linked

Related