Trouble saving repeated protobuf object to file (Python)

Question

I'm new to protobuf, so I don't know how to frame the question correctly.

Anyways, I'm using this Model Config proto file. I converted it into python using this command protoc -I=. --python_out=. ./model_server_config.proto from Protocol Buffer page. Now I have some python files which I can import and work on. My objective is to create a file (for running the TensorFlow model server with multiple models) which should look like the following:

model_config_list: {
 config: {
    name: "name1",
    base_path: "path1",
    model_platform: "tensorflow"
  },
  config: {
    name: "name2",
    base_path: "path2",
    model_platform: "tensorflow"
  },
  config: {
    name: "name3",
    base_path: "path3",
    model_platform: "tensorflow"
  },
}

Now using the python package compiled, I made a protobuf object which looks like this when I print it out:

model_config_list {
  config {
    name: "name1"
    base_path: "path1"
    model_platform: "tensorflow"
  }
  config {
    name: "name2"
    base_path: "path2"
    model_platform: "tensorflow"
  }
  config {
    name: "name3"
    base_path: "path3"
    model_platform: "tensorflow"
  }
}

But while serializing the object using objectname.SerializeToString(), I get a weird output as :

b'\n\x94\x01\n \n\x04name1\x12\x0cpath1"\ntensorflow\n7\n\x08name2\x12\x1fpath2"\ntensorflow\n7\n\x08name3\x12\x1fpath3"\ntensorflow'

I tried converting it into Json also using the protobuf for python like this:

from google.protobuf.json_format import MessageToJson
MessageToJson(objectname)

which gave me a result like:

{
  "modelConfigList": {
    "config": [
      {
        "name": "name1",
        "basePath": "path1",
        "modelPlatform": "tensorflow"
      },
      {
        "name": "name2",
        "basePath": "path2",
        "modelPlatform": "tensorflow"
      },
      {
        "name": "name3",
        "basePath": "path3",
        "modelPlatform": "tensorflow"
      }
    ]
  }
}

with all the objects in a list and each objects as string, which is not acceptable for TensorFlow model server config.

Any ideas on how to write it into a file correctly? Or am I creating the whole objects incorrectly? Any help is welcome, Thanks in advance.

Your very first example (the desired output): Where does this come from? — Omni, Jul 24 '20 at 21:20
@Omni https://www.tensorflow.org/tfx/serving/serving_config#model_server_config_details — krxat, Jul 25 '20 at 07:27

Rafael Lerm · Accepted Answer · 2020-07-28T19:35:30.710

I don't know anything about what system will be reading your file, so I can't say anything about how you should write it to a file. It really depends on how the Model Server expects to read it.

That said, I don't see anything wrong with how you're creating the message, or any of the serialization methods you've shown.

The print method shows a "text format" proto, which is good for debugging and is sometimes used for storing configuration files. It's not very compact (field names are present in the file) and doesn't have all the backwards- and forwards-compatible features of the binary representation. It's actually funcionally the same as what you've said it "should look like": the colons and commas are actually optional.
The SerializeToString() method uses the binary serialization format. This is arguably what Protocol Buffers were built to do. It's a compact representation and provides backwards and forwards compatibility, but it's not very human-readable.
As the name suggests, the json_format module provides a JSON representation of the message. That's perfectly good if the system you're interacting with expects a JSON, but it's not exactly common.

Appendix: instead of using print(), the google.protobuf.text_format module has utilities better suited to using the text format programmatically. To write to a file, you could use:

from google.protobuf import text_format
(...)
with open(file_path, 'w') as output:
  text_format.PrintMessage(my_message, output)

Thanks for the reply. I'm very new to this concept, so I'm not sure how much I understand the reply. My objective is to create a config file like this https://www.tensorflow.org/tfx/serving/serving_config#model_server_config_details. But if I serialize it and write it into a file, it doesn't work and throw this error `Error parsing text-format tensorflow.serving.ModelServerConfig: 2:1: Interpreting non ascii codepoint 148.` — krxat, Jul 25 '20 at 07:22
Quoting that page: "The Model Server configuration file provided must be an ASCII ModelServerConfig protocol buffer. Refer to the following to understand what an ASCII protocol buffer looks like." It seems like they expect the text-format above (the one you get from `print`). It is (unfortunately) sometimes referred to as "ASCII format", although that's a misnomer (it actually uses utf-8). — Rafael Lerm, Jul 27 '20 at 14:05
Instead of print, I'd use the [`text_format`](https://googleapis.dev/python/protobuf/latest/google/protobuf/text_format.html) module, which is the same thing as just `print()`, but has more controls for using programmatically. `PrintMessage()` looks particularly useful here. — Rafael Lerm, Jul 28 '20 at 19:32

Trouble saving repeated protobuf object to file (Python)

1 Answers1