This post is part of the series Microsoft Orleans - Problems & Solutions.
Problem

Invoking a grain call, results in the call succeeding or failing at a random rate.

System.InvalidOperationException: Insufficient data present in buffer.
Configuration
  • Two silos.
  • One client.
  • Grain placement strategy is random placement.
  • The default fallback serializer for unknown types and exceptions is BinaryFormatter (Orleans 3.x)
  • Both silo hosts are ASP.NET Core applications (>=5.0).
Solution

Set EnableUnsafeBinaryFormatterSerialization=true, in each of the ASP.NET Core csproj files.

Explanation

In ASP.NET Core 5.0 and later versions, the BinaryFormatter methods are prohibited and will throw an exception, unless the developer has explicitly re-enabled those.

In this specific case the issue was that one of the silo hosts had BinaryFormatter serialization enabled, and the other did not! This explains why the grain call was sometimes succeeding and sometimes failing, at a random rate.

Because of the random grain placement strategy, the grain was sometimes being activated on the silo, which its host had BinaryFormatter serialization enabled, and was failing when the grain was activated on the silo, which its host didn't have this enabled.

The call succeeds when the client makes a call to silo 1 (where grain is activated) and the host has BinaryFormatter serialization enabled.

The call fails when the client makes a call to silo 2 (where grain is activated) and the host has BinaryFormatter serialization disabled.

If you found this article helpful please give it a share in your favorite forums 😉.