atom feed10 messages in com.googlegroups.protobufRe: [protobuf] Re: Serialize with Len...
FromSent OnAttachments
Jay ThomasAug 24, 2009 3:54 pm 
Peter KeenAug 24, 2009 4:05 pm 
Jay ThomasAug 24, 2009 4:07 pm 
Jay ThomasAug 24, 2009 4:17 pm 
Peter KeenAug 24, 2009 4:24 pm 
Peter KeenAug 24, 2009 4:33 pm 
Jay ThomasAug 24, 2009 4:42 pm 
Marc GravellAug 25, 2009 12:07 am 
Paul ShaferFeb 20, 2013 1:32 pm 
Marc GravellFeb 20, 2013 8:57 pm 
Subject:Re: [protobuf] Re: Serialize with Length Prefix
From:Marc Gravell (marc@gmail.com)
Date:Feb 20, 2013 8:57:47 pm
List:com.googlegroups.protobuf

this seems to be a resurrection of a protobuf-net specific discussion, so let me jump in...

The SerializeWithLengthPrefix method, by default, aims to represent data in a way that is a valid protobuf stream - in particular, as though it were simply a member of a parent object or list. As such, it encodes a [ field-number and "string" marker ], then the length - as two separate varints. So assuming the data was stored in this way (the default), consuming 2 defaults should consume the header information - the second of the 2 values being the length of the subsequent payload. If the data was stored with different header-formats, then you may need to adjust - for example the fixed32 obviously requires 4 bytes to be consumed.

Does that make sense? or have I missed the point of the question?

Marc

On 20 February 2013 13:32, Paul Shafer <prab@gmail.com> wrote:

First time poster and new to protobuf/protobuf-net so forgive me if this has already been addressed. But I've searched quite a bit and cant find an"official" answer.

If you want to use the fieldNumber argument in your SerializeWithLengthPrefix/TryDeserializeWithLengthPrefix for message type identification (or for whatever reason), all works great between two C#/.NET apps. However if you have a c++ app using protobuf talking to a C#/.NET app using protobuf-net, the fieldNumber argument will cause grief. It took wireshark, plus diving into protobuf-net code and protobuf code to find that the fieldNumber is encoded like (I assume) any other field within a message. Whereas the c++ protobuf has no such notion (that I could find) unless you want to use a CodedOutputSTream - which then has both a WriteTag and WriteVarint32 (which WriteTag calls). However WriteTag/WriteVarint32 does NOT encode the same as the fieldNumber with SerializeWithLengthPrefix. So... what do to?

Again, perhaps this has been addressed already but here's my solution (on the c++ side):

#define PROTO_NET_TAG(x) (((x) << 3) | (2 & 7))

... google::protobuf::io::ZeroCopyOutputStream* raw_output = new google::protobuf::io::StringOutputStream(str); google::protobuf::io::CodedOutputStream * coded = new google::protobuf::io::CodedOutputStream(raw_output); coded->WriteTag(PROTO_NET_TAG(msgTag)); coded->WriteVarint32(msg->ByteSize()); msg->SerializeToCodedStream(coded);

...

On Monday, August 24, 2009 5:33:26 PM UTC-6, Peter Keen wrote:

I looked at the protobuf-net source a little bit and it looks like if you write the length with WriteLittleEndian32 instead of WriteVarint32 you should be good to go, as long as you use PrefixStyle.Fixed32 within the C# call to DeserialzeWithLengthPrefix().

On Mon, Aug 24, 2009 at 4:17 PM, Jay Thomas<jaydian...@gmail.**com> wrote:

This code ideally has to operate with C# code on the other side of the socket that uses the DeserializeWithLengthPrefix() method call. The message originates in C++ environment on a linux box and terminates in C# environment on a windows PC. Why doesn't C++ support the same methods as C#? The only document that I've seen that has list of methods available is the message.h header file, installed into my Linux usr directory.

On Aug 24, 4:05 pm, Peter Keen <pete@gmail.com> wrote:

You sort of have to roll your own. In my project I'm doing something like this:

coded_output->WriteVarint32(**message->ByteSize()); message->**SerializeToCodedStream(coded_**output);

And then on the reading side:

uint32_t size; if (! coded_input->ReadVarint32(&**size)) { return NULL; // just an example. probably don't want to do this;

}

Message * m = new Message; CodedInputStream::Limit limit = coded_input->PushLimit(size); message->ParseFromCodedStream(**coded_input); coded_input->PopLimit(limit);

The biggest thing that helped me along was finding the Limit docs. I couldn't figure out how to parse from a ZeroCopyInputStream without consuming the entire stream, and then I ran across that and everything became much more clear.

On Mon, Aug 24, 2009 at 3:54 PM, Jay Thomas<jaydianetho...@gmail.**com> wrote:

Hello

I am looking for a way to serialize/deserialize with length prefix under c++. The serialized bytes will sent to a TCP socket. I am aware that C# has a method SerializeWithLengthPrefix() and DeserializeWithLengthPrefix(). Are there any such analogous methods for C++? Please point out any documentation that I may have missed here.

Thanks for any assistance.

-- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsu@googlegroups.com. To post to this group, send email to prot@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.

Marc