MemoryPack: Zero encoding extreme performance binary serializer for C#

Zero encoding extreme performance binary serializer for C# and Unity. Compared with System.Text.Json, protobuf-net, MessagePack for C#, Orleans.Serialization. Measured by .NET 7 / Ryzen 9 5950X machine. These serializers have IBufferWriter method, serialized using ArrayBufferWriter and reused to avoid measure buffer copy. For standard objects, MemoryPack is x10 faster and x2 ~ x5 faster than other binary serializers. For struct array, MemoryPack is even more powerful, with speeds up to x50 ~ x200 greater than other serializers.

MemoryPack is my 4th serializer, previously I've created well known serializers, ZeroFormatter, Utf8Json, MessagePack for C#. The reason for MemoryPack's speed is due to its C#-specific, C#-optimized binary format and a well tuned implementation based on my past experience. It is also a completely new design utilizing .NET 7 and C# 11 and the Incremental Source Generator (.NET Standard 2.1 (.NET 5, 6) and there is also Unity support).

Other serializers perform many encoding operations such as VarInt encoding, tag, string, etc. MemoryPack format uses a zero-encoding design that copies as much C# memory as possible. Zero-encoding is similar to FlatBuffers, but it doesn't need a special type; MemoryPack's serialization target is POCO.

Features

• Support modern I/O APIs (IBufferWriter, ReadOnlySpan, ReadOnlySequence)
• Native AOT friendly Source Generator based code generation, no Dynamic CodeGen (IL.Emit)
• Reflectionless non-generics APIs
• Deserialize into existing instance
• Polymorphism (Union) serialization
• Limited version-tolerant (fast/default) and full version-tolerant support
• Circular reference serialization
• PipeWriter/PipeReader based streaming serialization
• TypeScript code generation and ASP.NET Core Formatter
• Unity (2021.3) IL2CPP Support via .NET Source Generator

This library is distributed via NuGet. For best performance, recommend to use .NET 7. Minimum requirement is .NET Standard 2.1.

PM> Install-Package MemoryPack

And also a code editor requires Roslyn 4.3.1 support, for example Visual Studio 2022 version 17.3, .NET SDK 6.0.401. For details, see the Roslyn Version Support document.

For Unity, the requirements and installation process are completely different. See the Unity section for details.

Basic Usage

Define a struct or class to be serialized and annotate it with the [MemoryPackable] attribute and the partial keyword.

using MemoryPack;

[MemoryPackable]
public partial class Person
{
    public int Age { get; set; }
    public string Name { get; set; }
}

Serialization code is generated by the C# source generator feature which implements the IMemoryPackable interface. In Visual Studio you can check a generated code by using a shortcut Ctrl+K, R on the class name and select *.MemoryPackFormatter.g.cs.

Call MemoryPackSerializer.Serialize / Deserialize to serialize/deserialize an object instance.

var v = new Person { Age = 40, Name = "John" };
var bin = MemoryPackSerializer.Serialize(v);
var val = MemoryPackSerializer.Deserialize<Person>(bin);

Serialize method supports a return type of byte[] as well as it can serialize to IBufferWriter or Stream. Deserialize method supports ReadOnlySpan, ReadOnlySequence and Stream. And there are also non-generics versions.

Supported Types

These types can be serialized by default:

• .NET primitives (byte, int, bool, char, double, etc.)
• Unmanaged types (Any enum, Any user-defined struct which doesn't contain reference types)
• string, decimal, Half, Int128, UInt128, Guid, Rune, BigInteger
• TimeSpan, DateTime, DateTimeOffset, TimeOnly, DateOnly, TimeZoneInfo
• Complex, Plane, Quaternion, Matrix3x2, Matrix4x4, Vector2, Vector3, Vector4
• Uri, Version, StringBuilder, Type, BitArray, CultureInfo
• T[], T[,], T[,,], T[,,,], Memory<>, ReadOnlyMemory<>, ArraySegment<>, ReadOnlySequence<>
• Nullable<>, Lazy<>, KeyValuePair, Tuple, ValueTuple
• List<>, LinkedList<>, Queue<>, Stack<>, HashSet<>, SortedSet<>, PriorityQueue<TElement, TPriority>
• Dictionary<TKey, TValue>, SortedList, SortedDictionary, ReadOnlyDictionary
• Collection<>, ReadOnlyCollection<>, ObservableCollection<>, ReadOnlyObservableCollection<>
• IEnumerable<>, ICollection<>, IList<>, IReadOnlyCollection<>, IReadOnlyList<>, ISet<>
• IDictionary, IReadOnlyDictionary, ILookup, IGrouping, ConcurrentBag<>, ConcurrentQueue<>, ConcurrentStack<>, ConcurrentDictionary, BlockingCollection<>
• Immutable collections (ImmutableList<>, etc.) and interfaces (IImmutableList<>, etc.)

Defining Serializable Types

[MemoryPackable] can annotate to any class, struct, record, record struct and interface. If a type is struct or record struct which contains no reference types (C# Unmanaged types) any additional annotation (ignore, include, constructor, callbacks) is not used; that serialize/deserialize directly from the memory. Otherwise, by default, [MemoryPackable] serializes public instance properties or fields. You can use [MemoryPackIgnore] to remove serialization target, [MemoryPackInclude] promotes a private member to serialization target.

[MemoryPackable]
public partial class Sample
{
    // these types are serialized by default
    public int PublicField;
    public readonly int PublicReadOnlyField;
    public int PublicProperty { get; set; }
    public int PrivateSetPublicProperty { get; private set; }
    public int ReadOnlyPublicProperty { get; }
    public int InitProperty { get; init; }
    public required int RequiredInitProperty { get; init; }

    // these types are not serialized by default
    int privateProperty { get; set; }
    int privateField;
    readonly int privateReadOnlyField;

    // use [MemoryPackIgnore] to remove target of a public member
    [MemoryPackIgnore]
    public int PublicProperty2 => PublicProperty + PublicField;

    // use [MemoryPackInclude] to promote a private member to serialization target
    [MemoryPackInclude]
    int privateField2;
    [MemoryPackInclude]
    int privateProperty2 { get; set; }
}

MemoryPack's code generator adds information about what members are serialized to the section. This can be viewed by hovering over the type with Intellisense. All members must be memorypack-serializable, if not the code generator will emit an error. MemoryPack has 35 diagnostics rules (MEMPACK001 to MEMPACK035) to be defined comfortably. If target type is defined MemoryPack serialization externally and registered, use [MemoryPackAllowSerialize] to silent diagnostics.

[MemoryPackable]
public partial class Sample2
{
    [MemoryPackAllowSerialize]
    public NotSerializableType? NotSerializableProperty { get; set; }
}

Member Order

Member order is important; MemoryPack does not serialize the member-name or other information, instead serializing fields in the order they are declared. If a type is inherited, serialization is performed in the order of parent → child. The order of members can not change for the deserialization. For the schema evolution, see the Version tolerant section.

The default order is sequential, but you can choose the explicit layout with [MemoryPackable(SerializeLayout.Explicit)] and [MemoryPackOrder()].

// serialize Prop0 -> Prop1
[MemoryPackable(SerializeLayout.Explicit)]
public partial class SampleExplicitOrder
{
    [MemoryPackOrder(1)]
    public int Prop1 { get; set; }
    [MemoryPackOrder(0)]
    public int Prop0 { get; set; }
}

Constructors

MemoryPack supports both parameterized and parameterless constructors. The selection of the constructor follows these rules (applies to classes and structs):

• If there is [MemoryPackConstructor], use it.
• If there is no explicit constructor (including private), use a parameterless one.
• If there is one parameterless/parameterized constructor (including private), use it.
• If there are multiple constructors, then the [MemoryPackConstructor] attribute must be applied to the desired constructor (the generator will not automatically choose one), otherwise the generator will emit an error.
• If using a parameterized constructor, all parameter names must match corresponding member names (case-insensitive).

[MemoryPackable]
public partial class Person
{
    public readonly int Age;
    public readonly string Name;

    // You can use a parameterized constructor - parameter names must match corresponding members name (case-insensitive)
    public Person(int age, string name)
    {
        this.Age = age;
        this.Name = name;
    }
}

// also supports record primary constructor
[MemoryPackable]
public partial record Person2(int Age, string Name);

public partial class Person3
{
    public int Age { get; set; }
    public string Name { get; set; }

    public Person3() { }

    // If there are multiple constructors, then [MemoryPackConstructor] should be used
    [MemoryPackConstructor]
    public Person3(int age, string name)
    {
        this.Age = age;
        this.Name = name;
    }
}

Serialization Callbacks

When serializing/deserializing, MemoryPack can invoke a before/after event using the [MemoryPackOnSerializing], [MemoryPackOnSerialized], [MemoryPackOnDeserializing], [MemoryPackOnDeserialized] attributes. It can annotate both static and instance (non-static) methods, and public and private methods.

[MemoryPackable]
public partial class MethodCallSample
{
    // method call order is static -> instance
    [MemoryPackOnSerializing]
    public static void OnSerializing1()
    {
        Console.WriteLine(nameof(OnSerializing1));
    }
    // also allows private method
    [MemoryPackOnSerializing]
    void OnSerializing2()
    {
        Console.WriteLine(nameof(OnSerializing2));
    }

    // serializing -> /* serialize */ -> serialized
    [MemoryPackOnSerialized]
    static void OnSerialized1()
    {
        Console.WriteLine(nameof(OnSerialized1));
    }
    [MemoryPackOnSerialized]
    public void OnSerialized2()
    {
        Console.WriteLine(nameof(OnSerialized2));
    }

    [MemoryPackOnDeserializing]
    public static void OnDeserializing1()
    {
        Console.WriteLine(nameof(OnDeserializing1));
    }
    // Note: instance method with MemoryPackOnDeserializing, that not called if instance is not passed by `ref`
    [MemoryPackOnDeserializing]
    public void OnDeserializing2()
    {
        Console.WriteLine(nameof(OnDeserializing2));
    }

    [MemoryPackOnDeserialized]
    public static void OnDeserialized1()
    {
        Console.WriteLine(nameof(OnDeserialized1));
    }
    [MemoryPackOnDeserialized]
    public void OnDeserialized2()
    {
        Console.WriteLine(nameof(OnDeserialized2));
    }
}

Callbacks allows parameterless method and ref reader/writer, ref T value method. For example, ref callbacks can write/read custom header before serialization process.

[MemoryPackable]
public partial class EmitIdData
{
    public int MyProperty { get; set; }

    [MemoryPackOnSerializing]
    static void WriteId(ref MemoryPackWriter<TBufferWriter> writer, ref EmitIdData? value)
        where TBufferWriter : IBufferWriter // .NET Standard 2.1, use where TBufferWriter : class, IBufferWriter
    {
        writer.WriteUnmanaged(Guid.NewGuid()); // emit GUID in header.
    }

    [MemoryPackOnDeserializing]
    static void ReadId(ref MemoryPackReader reader, ref EmitIdData? value)
    {
        // read custom header before deserialize
        var guid = reader.ReadUnmanaged<Guid>();
        Console.WriteLine(guid);
    }
}

If set a value to ref value, you can change the value used for serialization/deserialization. For example, instantiate from ServiceProvider.

// before using this formatter, set ServiceProvider
// var options = MemoryPackSerializerOptions.Default with { ServiceProvider = provider };
// MemoryPackSerializer.Deserialize(value, options);
[MemoryPackable]
public partial class InstantiateFromServiceProvider
{
    static IServiceProvider serviceProvider = default!;
    public int MyProperty { get; private set; }

    [MemoryPackOnDeserializing]
    static void OnDeserializing(ref MemoryPackReader reader, ref InstantiateFromServiceProvider value)
    {
        if (value != null) return;
        value = reader.Options.ServiceProvider!.GetRequiredService<InstantiateFromServiceProvider>();
    }
}

Collection Types

By default, annotated [MemoryPackObject] type try to serialize its members. However, if a type is a collection (ICollection<>, ISet<>, IDictionary), use GenerateType.Collection to serialize it correctly.

[MemoryPackable(GenerateType.Collection)]
public partial class MyList : List<int> { }

[MemoryPackable(GenerateType.Collection)]
public partial class MyStringDictionary : Dictionary<string, string> { }

MemoryPackable class can not define static constructor because generated partial class uses it. Instead, you can define a static partial void StaticConstructor() to do the same thing.

[MemoryPackable]
public partial class CctorSample
{
    static partial void StaticConstructor() { }
}

Polymorphism (Union)

MemoryPack supports serializing interface and abstract class objects for polymorphism serialization. In MemoryPack this feature is called Union. Only interfaces and abstract classes are allowed to be annotated with [MemoryPackUnion] attributes. Unique union tags are required.

// Annotate [MemoryPackable] and inheritance types with [MemoryPackUnion]
// Union also supports abstract class
[MemoryPackable]
[MemoryPackUnion(0, typeof(FooClass))]
[MemoryPackUnion(1, typeof(BarClass))]
public partial interface IUnionSample { }

[MemoryPackable]
public partial class FooClass : IUnionSample
{
    public int XYZ { get; set; }
}

[MemoryPackable]
public partial class BarClass : IUnionSample
{
    public string? OPQ { get; set; }
}

// ---
IUnionSample data = new FooClass() { XYZ = 999 };

// Serialize as interface type.
var bin = MemoryPackSerializer.Serialize(data);

// Deserialize as interface type.
var reData = MemoryPackSerializer.Deserialize<IUnionSample>(bin);

switch (reData)
{
    case FooClass x:
        Console.WriteLine(x.XYZ);
        break;
    case BarClass x:
        Console.WriteLine(x.OPQ);
        break;
    default:
        break;
}

Tag allows 0 ~ 65535, it is especially efficient for less than 250.

If an interface and derived types are in different assemblies, you can use MemoryPackUnionFormatterAttribute instead. Formatters are generated the way that they are automatically registered via ModuleInitializer in C# 9.0 and above. Note that ModuleInitializer is not supported in Unity, so the formatter must be manually registered. To register your union formatter invoke {name of your union formatter}Initializer.RegisterFormatter() manually in Startup. For example UnionSampleFormatterInitializer.RegisterFormatter().

// AssemblyA
[MemoryPackable(GenerateType.NoGenerate)]
public partial interface IUnionSample { }

// AssemblyB define definition outside of target type