v3.0

v3.0 is now available on NuGet, and brings exciting new improvements - and some necessary changes. For most people 3.0 should be a simple update. A few niche features have changed or been removed - more details below. The library tries to guide you at build time if you are particularly impacted.

New Features

• buffer-focused APIs: like many serializers, protobuf-net has historically been focused on the Stream API, but recent advances in .NET IO have brought more focus on things like ReadOnlyMemory<byte>, ReadOnlySequence<byte>, and IBufferWriter<byte>; protobuf-net is now able to directly utilize the APIs, to allow use with modern .NET IO primitives without intermediate layers. Since there are performance advantages to these APIs, protobuf-net will also try to detect scenarios where it can use these features - for example, if you try to deserialize from a MemoryStream, protobuf-net will use TryGetBuffer to access the underlying memory directly when possible
• new serializer core for AOT: the heart of the serialize and deserialize code has been heavily refactored, as part of in-progress work to enable full AOT usage via the proposed “generators” C# language feature; this work is not yet fully available (as the compiler changes are still in preview), but the building blocks are in place and have been validated; AOT usage will be entirely optional, and targets both cold-start performance for “full” runtimes, and ref-emit avoidance for AOT environments like mobile devices and gaming platforms
• new custom serializer API: there is a new API at the heart of protobuf-net that allows custom serializers to be defined in C# - both for messages and primitive values; this API is not yet intended for “consumer” use - that is a later v3 stage - but: it is already opening up a lot of options for the engine, and offers future extensibility for consumers
• performance: the use of modern APIs, plus additional changes (things like ArrayPool<T> usage) should see performance improvements for most callers, in both serialization and deserialization; note that for 3.0 the only aim was “no performance regression” (with a later 3.something deliverable being an in-depth performance review), so getting a performance increase here was a free win!
• pre-measurement of payloads: in a lot of cases, it is necessary to know the length of a payload before you actually write it - for example, for an HTTP Content-Length header; previously this has required serializing the object fully to a buffer, writing the buffer length, then copying the buffer; a new Measure<T> API now allows the caller to perform a pre-computation of the length in advance, in a way that allows the library to reuse that knowledge in the anticipated serialize that will surely follow, which in turn makes the final serialize a “forwards only” affair
• compatibiliy level API: this extends the DataFormat.WellKnown concept from v2 allowing granular changes to how common data types are serialized; this is discussed in more depth here - thanks here go to ServiceStack, who both inspired and sponsored these improvements
• capability API: since there are some changes in v3.0 that might make it not a perfect update for everyone, libraries that consume protobuf-net might not want to take a hard 3.0 dependency, but might still want to use the 3.0 features when they are available; new capability APIs make this possible - for example, protobuf-net.Grpc uses this, allowing it to use the “contextual serializer” gRPC API when 3.0 APIs are detected, or the legacy byte[] API otherwise); this works via interfaces that are advertised in the later 2.4 releases, but not implemented until 3.0; in particular, you can now test a TypeModel for:
  • input - IProtoInput<T> where T is any of:
    • 2.4 and 3.0: Stream, ArraySegment<byte>, byte[]
    • 3.0 only: ReadOnlyMemory<byte>, ReadOnlySequence<byte>
  • output - IProtoInput<T> where T is any of:
    • 2.4 and 3.0: Stream
    • 3.0 only: IBufferWriter<byte>
  • output with measurement - IMeasuredProtoOutput<T> where T is any of:
    • 3.0 only: Stream, IBufferWriter<byte>
• split libraries: as part of the AOT drive, the library has now been split into protobuf-net and protobuf-net.Core, with the “Core” package having the main serializer API but none of the runtime reflection code; this means that future AOT-only scenarios only require the “Core” package, and can omit the overhead of shipping any of the other code at all; you can see this in action already at protogen.marcgravell.com, which uses the schema parting tools and serializer core in the client browser (not at the web-server) via “Blazor”/WASM, via a dependency on protobuf-net.Core (a huge thanks once again to Remi BOURGAREL here, who leant their Blazor expertise)

Changes To Existing Features

Listed in highest-to-lowest impact; the first 2 here could well be reasons not to choose v3 for now

• reference tracking is deprecated; this means any use of AsReference = true, to build an object graph rather than an object tree; the reality is: protobuf itself is a tree serializer, and this feature was painful to use and support; I’m open to discussions around reinstating it, but it is a lot of work, and is a very niche feature that provides a lot of headaches
• dynamic typing is deprecated; this means any usage of DynamicType = true, where a property is typed as object, and the serializer includes the type metadata in the payload; this feature suffers from a number of the same problems as BinaryFormatter, exacerbated by problems due to types moving around between .NET Framework and .NET Core; this made it both brittle and dangerous, in addition to just being terrible for cross-platform work; the plan here is to implement support for the Any feature of protobuf, which should provide a supportable API, while also allowing compatibility with other protobuf implementations; this work is planned for a future v3 stage, and will be back-ported to v2 to allow a migration route between v2 and v3
• enum maps are deprecated; this is where an enum has a [ProtoEnum(Value=...)] marker that was used to provide a different value for serialization - for example, when the enum value is 42 but you want to serialize it as 3; this feature just confused a lot of people and made code brittle; instead, it is suggested to use a shadow property in your code that provides this map, for example via a switch expression, and serialize the shadow property instead
• ProtoReader/ProtoWriter must now be instantiated via ProtoReader.Create, not new (recent v2 releases have been advising this change for some time); most users will not be using these APIs
• minor: ProtoReader/ProtoWriter should now be used with the ref state APIs when used against Stream, and must now be used with the ref state APIs when used against the new buffer APIs; most users will not be using these APIs
• minor: DataFormat.WellKnown now advises you to use CompatibilityLevel instead; if you’re happy with your code “as is”, you can safely igore this warning (#pragma or similar), and your code will continue to work in Level240 (as discussed here)
• minor: the API for creating runtime type models has moved from TypeModel.Create to RuntimeTypeModel.Create; this is largely a side-effect of the library split, and recent v2 releases have been advising this change for some time; if this causes genuine inconvenience to anyone, we can probably “fix” this - just ping me!
• streamlined framework targets; we used to have targets for (takes a deep breath): .NET Framework 2.0/3.5/4.0, .NET Standard 1.0/1.3/2.0, UAP 10.0, and .NET Core 2.1; carrying around this weight made feature changes hard bordering on impossible, and in reality: I wasn’t happy that we were providing suitable testing for platforms that we haven’t used in years; the TFM list is now .NET Standard 2.0/2.1, and .NET Framework 4.6.1; for more discussion on this theme, see my writing here

Summary

Emphasis: for most people, this should be a simple upgrade that “just works”. It has been in use at Stack Overflow for a very long time, without any problems - providing new performance and features. If you are using the reference tracking or dynamic type features, or .NET Framework 3.0, then I totally get that you might be frustrated here. The good news is that nobody is coming to uninstall v2 from your system - it will continue to work just fine.

As always, this library is provided “as is” etc, but if you have any problems or find any bugs, please let me know and we’ll try to fix it.