Inconsistent data and Orleans exception in my logs when running in kubernettes cluster

Hosting (OnPremise)
21

Still getting something like this,

{
“logLevel”: “Error”,
“message”: “Caught and ignored exception: Squidex.Infrastructure.States.InconsistentStateException with message: Requested version 453, but found 671. thrown from timer callback GrainTimer. TimerCallbackHandler:Squidex.Domain.Apps.Entities.Rules.UsageTracking.UsageTrackerGrain->System.Threading.Tasks.Task b__5_0(System.Object)”,
“eventId”: {
“id”: 101413
},
“exception”: {
“type”: “Squidex.Infrastructure.States.InconsistentStateException”,
“message”: “Requested version 453, but found 671.”,
“stackTrace”: " at Squidex.Infrastructure.MongoDb.MongoExtensions.UpsertVersionedAsync[T,TKey](IMongoCollection1 collection, TKey key, Int64 oldVersion, Int64 newVersion, Func2 updater) in /src/src/Squidex.Infrastructure.MongoDb/MongoDb/MongoExtensions.cs:line 131\n at Squidex.Infrastructure.States.MongoSnapshotStore2.WriteAsync(TKey key, T value, Int64 oldVersion, Int64 newVersion) in /src/src/Squidex.Infrastructure.MongoDb/States/MongoSnapshotStore.cs:line 60\n at Squidex.Infrastructure.States.Persistence2.WriteSnapshotAsync(TSnapshot state) in /src/src/Squidex.Infrastructure/States/Persistence{TSnapshot,TKey}.cs:line 139\n at Squidex.Domain.Apps.Entities.Rules.UsageTracking.UsageTrackerGrain.CheckUsagesAsync() in /src/src/Squidex.Domain.Apps.Entities/Rules/UsageTracking/UsageTrackerGrain.cs:line 110\n at Orleans.Runtime.GrainTimer.ForwardToAsyncCallback(Object state)"
},
“app”: {
“name”: “Squidex”,
“version”: “3.0.0.0”,
“sessionId”: “14f0c5fa-e33f-4520-ab94-7f7e0a57b815”
},
“timestamp”: “2019-09-13T07:36:49Z”,
“category”: “Orleans.Runtime.GrainTimer”
}

{
“logLevel”: “Error”,
“message”: “Caught and ignored exception: Squidex.Infrastructure.States.InconsistentStateException with message: Requested version 454, but found 671. thrown from timer callback GrainTimer. TimerCallbackHandler:Squidex.Domain.Apps.Entities.Rules.UsageTracking.UsageTrackerGrain->System.Threading.Tasks.Task b__5_0(System.Object)”,
“eventId”: {
“id”: 101413
},
“exception”: {
“type”: “Squidex.Infrastructure.States.InconsistentStateException”,
“message”: “Requested version 454, but found 671.”,
“stackTrace”: " at Squidex.Infrastructure.MongoDb.MongoExtensions.UpsertVersionedAsync[T,TKey](IMongoCollection1 collection, TKey key, Int64 oldVersion, Int64 newVersion, Func2 updater) in /src/src/Squidex.Infrastructure.MongoDb/MongoDb/MongoExtensions.cs:line 131\n at Squidex.Infrastructure.States.MongoSnapshotStore2.WriteAsync(TKey key, T value, Int64 oldVersion, Int64 newVersion) in /src/src/Squidex.Infrastructure.MongoDb/States/MongoSnapshotStore.cs:line 60\n at Squidex.Infrastructure.States.Persistence2.WriteSnapshotAsync(TSnapshot state) in /src/src/Squidex.Infrastructure/States/Persistence{TSnapshot,TKey}.cs:line 139\n at Squidex.Domain.Apps.Entities.Rules.UsageTracking.UsageTrackerGrain.CheckUsagesAsync() in /src/src/Squidex.Domain.Apps.Entities/Rules/UsageTracking/UsageTrackerGrain.cs:line 110\n at Orleans.Runtime.GrainTimer.ForwardToAsyncCallback(Object state)"
},

22

Yes, this is not really a bug, in general all system use the the storage system and consistency checks. This grain is a background system and it will just flood the logs a little bit but does not produce any harm.

The real reason is that your MongoDB replica members are not in sync. In this case it is really hard to provide consistent results.

23

Okay, so this would go away as soon as the replica is in sync, or is it a sign, that it’s in general out of sync?

24

Yes, exactly, the exception just shows the inconsistency and does not hide it.

25

I got similar issues. i currently using 3.2.2 version. this 4 service running for about 4 days. even I clear the Orleans_OrleansMembershipV2. it still not works. do you have any solutions

{
  "logLevel": "Error",
  "action": "GrainInvoked",
  "status": "Failed",
  "grain": "[SystemTarget: S172.30.218.216:11111:307096944*stg/10/0000000a@S0000000a]",
  "grainMethod": "System.Threading.Tasks.Task`1[Orleans.GrainDirectory.AddressesAndTag] LookupAsync(Orleans.Runtime.GrainId, Int32)",
  "exception": {
    "type": "Orleans.Runtime.OrleansMessageRejectionException",
    "message": "Silo S172.30.218.216:11111:307096944 is rejecting message: Request S172.30.218.216:11111:307096944*stg/10/0000000a@S0000000a->S172.30.172.148:11111:307045967*stg/10/0000000a@S0000000a #3802981: . Reason = Recent (00:00:00.6649665 ago, at 2019-09-29 06:35:03.026 GMT) connection failure trying to reach target silo S172.30.172.148:11111:307045967. Going to drop Request msg 3802981 without sending. CONNECTION_RETRY_DELAY = 00:00:01.",
    "stackTrace": "   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)\n   at Orleans.Runtime.OrleansCodeGenRemoteGrainDirectoryMethodInvoker.Invoke(IAddressable grain, InvokeMethodRequest request)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at OrleansDashboard.Metrics.GrainProfilerFilter.Invoke(IIncomingGrainCallContext context)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at Squidex.Infrastructure.Orleans.LoggingFilter.Invoke(IIncomingGrainCallContext context) in /src/src/Squidex.Infrastructure/Orleans/LoggingFilter.cs:line 30"
  },
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:03Z"
}

{
  "logLevel": "Warning",
  "message": "LookupAsync - It seems we are not the owner of grain *grn/68072C12/00000000+172.30.172.164:11111@307096950, trying to forward it to S172.30.140.158:11111:307046046 (hopCount=1)",
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:03Z",
  "category": "Orleans.Runtime.GrainDirectory.LocalGrainDirectory"
}

{
  "logLevel": "Warning",
  "message": "Exception getting a sending socket to endpoint S172.30.140.158:11111:307046046",
  "eventId": {
    "id": 101021
  },
  "exception": {
    "type": "Orleans.Runtime.OrleansException",
    "message": "Could not connect to 172.30.140.158:11111: ConnectionRefused",
    "stackTrace": "   at Orleans.Runtime.SocketManager.Connect(Socket s, IPEndPoint endPoint, TimeSpan connectionTimeout)\n   at Orleans.Runtime.SocketManager.SendingSocketCreator(IPEndPoint target)\n   at Orleans.Runtime.LRU`2.Get(TKey key)\n   at Orleans.Runtime.Messaging.SiloMessageSender.GetSendingSocket(Message msg, Socket& socket, SiloAddress& targetSilo, String& error)"
  },
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:03Z",
  "category": "Runtime.Messaging.SiloMessageSender/SystemSender"
}

{
  "logLevel": "Error",
  "action": "GrainInvoked",
  "status": "Failed",
  "grain": "[SystemTarget: S172.30.218.216:11111:307096944*stg/10/0000000a@S0000000a]",
  "grainMethod": "System.Threading.Tasks.Task`1[Orleans.GrainDirectory.AddressesAndTag] LookupAsync(Orleans.Runtime.GrainId, Int32)",
  "exception": {
    "type": "Orleans.Runtime.OrleansMessageRejectionException",
    "message": "Silo S172.30.218.216:11111:307096944 is rejecting message: Request S172.30.218.216:11111:307096944*stg/10/0000000a@S0000000a->S172.30.140.158:11111:307046046*stg/10/0000000a@S0000000a #3802982: . Reason = Exception getting a sending socket to endpoint S172.30.140.158:11111:307046046",
    "stackTrace": "   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)\n   at Orleans.Runtime.OrleansCodeGenRemoteGrainDirectoryMethodInvoker.Invoke(IAddressable grain, InvokeMethodRequest request)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at OrleansDashboard.Metrics.GrainProfilerFilter.Invoke(IIncomingGrainCallContext context)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at Squidex.Infrastructure.Orleans.LoggingFilter.Invoke(IIncomingGrainCallContext context) in /src/src/Squidex.Infrastructure/Orleans/LoggingFilter.cs:line 30"
  },
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:03Z"
}

{
  "logLevel": "Warning",
  "message": "LookupAsync - It seems we are not the owner of grain *grn/FFE8F7B3/00000000, trying to forward it to S172.30.172.148:11111:307045967 (hopCount=1)",
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:03Z",
  "category": "Orleans.Runtime.GrainDirectory.LocalGrainDirectory"
}

{
  "logLevel": "Warning",
  "message": "LookupAsync - It seems we are not the owner of grain *grn/68072C12/00000000+172.30.199.41:11111@307096966, trying to forward it to S172.30.140.158:11111:307046046 (hopCount=1)",
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:03Z",
  "category": "Orleans.Runtime.GrainDirectory.LocalGrainDirectory"
}

{
  "logLevel": "Error",
  "action": "GrainInvoked",
  "status": "Failed",
  "grain": "[SystemTarget: S172.30.218.216:11111:307096944*stg/10/0000000a@S0000000a]",
  "grainMethod": "System.Threading.Tasks.Task`1[Orleans.GrainDirectory.AddressesAndTag] LookupAsync(Orleans.Runtime.GrainId, Int32)",
  "exception": {
    "type": "Orleans.Runtime.OrleansMessageRejectionException",
    "message": "Silo S172.30.218.216:11111:307096944 is rejecting message: Request S172.30.218.216:11111:307096944*stg/10/0000000a@S0000000a->S172.30.172.148:11111:307045967*stg/10/0000000a@S0000000a #3802983: . Reason = Recent (00:00:00.9222869 ago, at 2019-09-29 06:35:03.026 GMT) connection failure trying to reach target silo S172.30.172.148:11111:307045967. Going to drop Request msg 3802983 without sending. CONNECTION_RETRY_DELAY = 00:00:01.",
    "stackTrace": "   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)\n   at Orleans.Runtime.OrleansCodeGenRemoteGrainDirectoryMethodInvoker.Invoke(IAddressable grain, InvokeMethodRequest request)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at OrleansDashboard.Metrics.GrainProfilerFilter.Invoke(IIncomingGrainCallContext context)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at Squidex.Infrastructure.Orleans.LoggingFilter.Invoke(IIncomingGrainCallContext context) in /src/src/Squidex.Infrastructure/Orleans/LoggingFilter.cs:line 30"
  },
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:03Z"
}

{
  "logLevel": "Error",
  "action": "GrainInvoked",
  "status": "Failed",
  "grain": "[SystemTarget: S172.30.218.216:11111:307096944*stg/10/0000000a@S0000000a]",
  "grainMethod": "System.Threading.Tasks.Task`1[Orleans.GrainDirectory.AddressesAndTag] LookupAsync(Orleans.Runtime.GrainId, Int32)",
  "exception": {
    "type": "Orleans.Runtime.OrleansMessageRejectionException",
    "message": "Silo S172.30.218.216:11111:307096944 is rejecting message: Request S172.30.218.216:11111:307096944*stg/10/0000000a@S0000000a->S172.30.140.158:11111:307046046*stg/10/0000000a@S0000000a #3802984: . Reason = Recent (00:00:00.2420637 ago, at 2019-09-29 06:35:03.708 GMT) connection failure trying to reach target silo S172.30.140.158:11111:307046046. Going to drop Request msg 3802984 without sending. CONNECTION_RETRY_DELAY = 00:00:01.",
    "stackTrace": "   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)\n   at Orleans.Runtime.OrleansCodeGenRemoteGrainDirectoryMethodInvoker.Invoke(IAddressable grain, InvokeMethodRequest request)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at OrleansDashboard.Metrics.GrainProfilerFilter.Invoke(IIncomingGrainCallContext context)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at Squidex.Infrastructure.Orleans.LoggingFilter.Invoke(IIncomingGrainCallContext context) in /src/src/Squidex.Infrastructure/Orleans/LoggingFilter.cs:line 30"
  },
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:03Z"
}

{
  "logLevel": "Warning",
  "message": "LookupAsync - It seems we are not the owner of grain *grn/FFE8F7B3/00000000, trying to forward it to S172.30.172.148:11111:307045967 (hopCount=1)",
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:04Z",
  "category": "Orleans.Runtime.GrainDirectory.LocalGrainDirectory"
}

{
  "logLevel": "Error",
  "action": "GrainInvoked",
  "status": "Failed",
  "grain": "[SystemTarget: S172.30.218.216:11111:307096944*stg/10/0000000a@S0000000a]",
  "grainMethod": "System.Threading.Tasks.Task`1[Orleans.GrainDirectory.AddressesAndTag] LookupAsync(Orleans.Runtime.GrainId, Int32)",
  "exception": {
    "type": "Orleans.Runtime.OrleansMessageRejectionException",
    "message": "Silo S172.30.218.216:11111:307096944 is rejecting message: Request S172.30.218.216:11111:307096944*stg/10/0000000a@S0000000a->S172.30.172.148:11111:307045967*stg/10/0000000a@S0000000a #3802985: . Reason = Recent (00:00:00.9759655 ago, at 2019-09-29 06:35:03.026 GMT) connection failure trying to reach target silo S172.30.172.148:11111:307045967. Going to drop Request msg 3802985 without sending. CONNECTION_RETRY_DELAY = 00:00:01.",
    "stackTrace": "   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)\n   at Orleans.Runtime.OrleansCodeGenRemoteGrainDirectoryMethodInvoker.Invoke(IAddressable grain, InvokeMethodRequest request)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at OrleansDashboard.Metrics.GrainProfilerFilter.Invoke(IIncomingGrainCallContext context)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at Squidex.Infrastructure.Orleans.LoggingFilter.Invoke(IIncomingGrainCallContext context) in /src/src/Squidex.Infrastructure/Orleans/LoggingFilter.cs:line 30"
  },
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:04Z"
}

{
  "logLevel": "Warning",
  "message": "LookupAsync - It seems we are not the owner of grain *grn/FFE8F7B3/00000000, trying to forward it to S172.30.172.148:11111:307045967 (hopCount=1)",
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:04Z",
  "category": "Orleans.Runtime.GrainDirectory.LocalGrainDirectory"
}

{
  "logLevel": "Warning",
  "message": "Exception getting a sending socket to endpoint S172.30.172.148:11111:307045967",
  "eventId": {
    "id": 101021
  },
  "exception": {
    "type": "Orleans.Runtime.OrleansException",
    "message": "Could not connect to 172.30.172.148:11111: ConnectionRefused",
    "stackTrace": "   at Orleans.Runtime.SocketManager.Connect(Socket s, IPEndPoint endPoint, TimeSpan connectionTimeout)\n   at Orleans.Runtime.SocketManager.SendingSocketCreator(IPEndPoint target)\n   at Orleans.Runtime.LRU`2.Get(TKey key)\n   at Orleans.Runtime.Messaging.SiloMessageSender.GetSendingSocket(Message msg, Socket& socket, SiloAddress& targetSilo, String& error)"
  },
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:04Z",
  "category": "Runtime.Messaging.SiloMessageSender/SystemSender"
}

{
  "logLevel": "Error",
  "action": "GrainInvoked",
  "status": "Failed",
  "grain": "[SystemTarget: S172.30.218.216:11111:307096944*stg/10/0000000a@S0000000a]",
  "grainMethod": "System.Threading.Tasks.Task`1[Orleans.GrainDirectory.AddressesAndTag] LookupAsync(Orleans.Runtime.GrainId, Int32)",
  "exception": {
    "type": "Orleans.Runtime.OrleansMessageRejectionException",
    "message": "Silo S172.30.218.216:11111:307096944 is rejecting message: Request S172.30.218.216:11111:307096944*stg/10/0000000a@S0000000a->S172.30.172.148:11111:307045967*stg/10/0000000a@S0000000a #3802986: . Reason = Exception getting a sending socket to endpoint S172.30.172.148:11111:307045967",
    "stackTrace": "   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)\n   at Orleans.Runtime.OrleansCodeGenRemoteGrainDirectoryMethodInvoker.Invoke(IAddressable grain, InvokeMethodRequest request)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at OrleansDashboard.Metrics.GrainProfilerFilter.Invoke(IIncomingGrainCallContext context)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at Squidex.Infrastructure.Orleans.LoggingFilter.Invoke(IIncomingGrainCallContext context) in /src/src/Squidex.Infrastructure/Orleans/LoggingFilter.cs:line 30"
  },
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:04Z"
}

{
  "logLevel": "Error",
  "message": "SelectTarget failed with Silo S172.30.218.216:11111:307096944 is rejecting message: Request S172.30.218.216:11111:307096944*stg/14/0000000e@S0000000e->S172.30.199.36:11111:307045902*stg/10/0000000a@S0000000a #3802988: . Reason = Recent (00:00:00.9510389 ago, at 2019-09-29 06:35:03.350 GMT) connection failure trying to reach target silo S172.30.199.36:11111:307045902. Going to drop Request msg 3802988 without sending. CONNECTION_RETRY_DELAY = 00:00:01.",
  "eventId": {
    "id": 100071
  },
  "exception": {
    "type": "Orleans.Runtime.OrleansMessageRejectionException",
    "message": "Silo S172.30.218.216:11111:307096944 is rejecting message: Request S172.30.218.216:11111:307096944*stg/14/0000000e@S0000000e->S172.30.199.36:11111:307045902*stg/10/0000000a@S0000000a #3802988: . Reason = Recent (00:00:00.9510389 ago, at 2019-09-29 06:35:03.350 GMT) connection failure trying to reach target silo S172.30.199.36:11111:307045902. Going to drop Request msg 3802988 without sending. CONNECTION_RETRY_DELAY = 00:00:01.",
    "stackTrace": "   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)\n   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()\n   at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)\n   at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)\n   at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)\n   at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()"
  },
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:04Z",
  "category": "Orleans.Runtime.Dispatcher"
}

{
  "logLevel": "Warning",
  "message": "Discarding OneWay rejection for message OneWay S172.30.218.216:11111:307096944*cli/39a77480@ebd94791->*grn/68072C12/00000000+172.30.218.216:11111@307096944 #3802987: . Exc = Silo S172.30.218.216:11111:307096944 is rejecting message: Request S172.30.218.216:11111:307096944*stg/14/0000000e@S0000000e->S172.30.199.36:11111:307045902*stg/10/0000000a@S0000000a #3802988: . Reason = Recent (00:00:00.9510389 ago, at 2019-09-29 06:35:03.350 GMT) connection failure trying to reach target silo S172.30.199.36:11111:307045902. Going to drop Request msg 3802988 without sending. CONNECTION_RETRY_DELAY = 00:00:01.",
  "eventId": {
    "id": 101002
  },
  "app": {
    "name": "Squidex",
    "version": "3.2.2.0",
    "sessionId": "6077eb95-bdef-4caf-8fe6-84c46875fe8a"
  },
  "timestamp": "2019-09-29T06:35:04Z",