Encode json files into bytes

Question

I want to encode my json files into bytes.

I'm working with sensors that offer a lot of data. Now what I'm doing is obtain each value, convert it to byte using struct.pack(), stack the data and finally send it.

For example, if I have:

{
    'Temperature':9999,
    'Ph': 8888,
    'WindDir': 7777,
    ...
}

I was thinking, maybe exists some function, algorithm, NN, etc that can directly convert this dict to binary, and obviously a decoder.

The idea is doing something like, struct.pack(myFile.json).

I think it's very interesting, because if this is possible I can work not only with numbers, with strings too and send the data as a typical API response.

Note: It's mandatory work in bytes.

So you have JSON which is `str` and want to get `bytes` which could be then converted back to original `str`? — Daweo, May 31 '21 at 08:25
I want to convert the full json to byte, and not work data by data. — Lleims, May 31 '21 at 08:26
How about you dump the JSON into a `str` using `json.dumps`, then encode it? — Nguyen Thai Binh, May 31 '21 at 08:35
Does this answer your question? [Best way to convert string to bytes in Python 3?](https://stackoverflow.com/questions/7585435/best-way-to-convert-string-to-bytes-in-python-3) — Thomas Weller, May 31 '21 at 08:36
NN = neural network? Encoding is about rules, not about guessing and predicting. A neural network is definitely the wrong choice. — Thomas Weller, May 31 '21 at 08:37
How much memory do your sensors have that they can process JSON? — Thomas Weller, May 31 '21 at 08:39
A file is already in bytes. Why are bytes that represent JSON not good bytes? That is your problem. So tell us, what do you want to achieve? Less bytes? Use compression. Faster reading? Just save each float as 4/8 bytes in a predetermined order and read them out like that. — Robin De Schepper, May 31 '21 at 08:40
The main goal is send all my data, the second one is send it as compress as possible. I know how to do it using `struct`, I was thinking if exists other ways. — Lleims, May 31 '21 at 08:55
Typically, the layout of the structs are given to you by the embedded device (sensor) developers. You must adhere to these definitions and provide the exact amount of bytes in the correct order (big endian, little endian). You write the encoder and decoder, e.g. from and to JSON or XML or something else. Using `struct.pack()` is exactly right, but not `struct.pack(json)`. Rather use `struct.pack(myobject.temperature, myobject.ph, myobject.winddir)`. — Thomas Weller, May 31 '21 at 08:55
`struct.pack(json)` was just to explain the concept of my goal, I understand it's not correctly — Lleims, May 31 '21 at 08:56
IMHO you can't compress the data more, because the device will not understand compression. Using compression needs more memory on the device: the device needs to hold the compressed data + a buffer for uncompressed data. — Thomas Weller, May 31 '21 at 08:56
I think the same, but I wanted to know your opinion to know other points of view. — Lleims, May 31 '21 at 08:58

score -1 · Answer 1 · answered May 31 '21 at 08:36

As long as content of your dict is limited to built-in python types, you might use pickle built-in modules effortlessly.

Creating file is as follows:

import pickle
d = {"a":1,"b":"two","c":3.5}
with open("data.p","wb") as f:
    pickle.dump(d, f)

Reading from that file is as follows:

import pickle
with open("data.p","rb") as f:
    d = pickle.load(f)
print(d)  # {'a': 1, 'b': 'two', 'c': 3.5}

Creating bytes is as follows:

import pickle
d = {"a":1,"b":"two","c":3.5}
databytes = pickle.dumps(d)
print(databytes)  # b'\x80\x03}q\x00(X\x01\x00\x00\x00aq\x01K\x01X\x01\x00\x00\x00bq\x02X\x03\x00\x00\x00twoq\x03X\x01\x00\x00\x00cq\x04G@\x0c\x00\x00\x00\x00\x00\x00u.'

Reading bytes is as follows:

import pickle
databytes = b'\x80\x03}q\x00(X\x01\x00\x00\x00aq\x01K\x01X\x01\x00\x00\x00bq\x02X\x03\x00\x00\x00twoq\x03X\x01\x00\x00\x00cq\x04G@\x0c\x00\x00\x00\x00\x00\x00u.'
d = pickle.loads(databytes)
print(d)  # {'a': 1, 'b': 'two', 'c': 3.5}

Pickle will use a binary representation of the Python object, not a JSON string. I doubt that the sensor will be able to process Python objects (except it runs MicroPython) — Thomas Weller, May 31 '21 at 08:41

Encode json files into bytes

1 Answers1