Writing Delimited C++ Protobuf Messages and Reading in Java

Question

I am writing multiple protobuf messages to a file in C++.

int fifoPipe = open("/media/my_pipe", O_WRONLY);

MyModel *model = new MyModel();

// Write to fifo pipe.
model->SerializeToFileDescriptor(fifoPipe);

I had read that the message size is written to the file automatically before the message data. Apparently this is not the case:

https://developers.google.com/protocol-buffers/docs/techniques?hl=en

So in Java it should be possible to read the message like so:

MyModel.parseDelimitedFrom(fileInputStream);

However there are questions here:

Cannot deserialize protobuf data from C++ in Java

Are there C++ equivalents for the Protocol Buffers delimited I/O functions in Java?

That say the message size must be written manually.

Since the questions are kind of old this procedure may have changed.

What is the proper way of writing multiple messages in C++ and reading those messages in Java?

Also consider how Java would respond to a half written message. In theory it should wait for data the size of message size to be present before returning.

As I read it, the C++ code in http://stackoverflow.com/questions/2340730/are-there-c-equivalents-for-the-protocol-buffers-delimited-i-o-functions-in-ja will write the lengths expected by the Java method parseDelimitedFrom. — JesperSM, Aug 11 '15 at 23:00
@JesperSM Same interpretation here. tzaman's answer indicates Java implements it yet C++ does not. With all the resources google has one has to wonder why things are like this. — BAR, Aug 11 '15 at 23:08
True, but then again code is written by individuals, not companies, especially open source. At least Kenton Varda was nice enough to post a useful implementation. You won't be the first to copy and paste a few lines of code from Stackoverflow in order to get the job done... ;-) — JesperSM, Aug 11 '15 at 23:25

score 6 · Answer 1 · edited May 23 '17 at 12:24

Unfortunately, the parseDelimitedFrom() and writeDelimitedTo() methods still have not been added to the C++ library.

The code I wrote in the answer to one of the questions you referenced is still the best way to implement this in C++:

https://stackoverflow.com/a/22927149/2686899

The main reason why Google with all its resources hasn't gotten around to adding these in C++ is because Google internally does not use this format at all. For network communications, Google uses its internal RPC protocol (very similar to GRPC, which they open sourced recently), and for storing messages to disk they usually a variety of internal formats that are a bit more featureful than this "delimited" format (you might consider using sqlite, for example).

In fact, when Protobuf was first open sourced, parseDelimitedFrom() didn't exist even in Java. I added it later specifically as a stopgap for users of the open source library -- a lot of people asked us how to write multiple messages to a file, and telling them "you should develop your own framing library" didn't seem very nice.

Google may be a huge company, but at that time I was the only person working full-time on Protocol Buffers. Unfortunately, for reasons that I don't quite remember, I only implemented the functions in Java, and never got around to adding them in C++ as well. In retrospect it seems a bit silly -- writing C++ code wouldn't have been very hard, as you can see at the link above. But I had a lot to do, as you might imagine.

The current Protobuf team took over in 2010, and I moved on to other things (and eventually left Google). I don't know for sure why they haven't added this code to the C++ library, but my guess is that not enough people have asked for it and they are focused on other things. I bet if you file a bug, and link to my code -- or better yet, submit a pull request yourself -- they might just take it. (I would submit a pull request myself, but right now I don't have time to write the necessary unit tests, etc...)

EDIT: OK, against my better judgment I took the time to prepare a pull request: https://github.com/google/protobuf/pull/710

Writing Delimited C++ Protobuf Messages and Reading in Java

1 Answers1