I found some more errors in the logs… seems to be mostly Orleans connectivity issues
exception.message
Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.53.132.27:11114:389372401. See InnerException
—> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.53.132.27:11114. Error: ConnectionRefused
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in //src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 59
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in //src/Orleans.Core/Networking/ConnectionFactory.cs:line 56
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task1 task, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 410 at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 293 --- End of inner exception stack trace --- at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 293 at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 139 at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 139 at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask
1 c, Message m) in //src/Orleans.Runtime/Messaging/OutboundMessageQueue.cs:line 130
at Orleans.Runtime.Messaging.OutboundMessageQueue.g__SendAsync|10_0(ValueTask`1 c, Message m) in //src/Orleans.Runtime/Messaging/OutboundMessageQueue.cs:line 130
exception.stackTrace
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object ) in //src/Orleans.Runtime/Placement/DeploymentLoadPublisher.cs:line 117
exception.type
Orleans.Runtime.OrleansMessageRejectionException
errormessage
An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
eventId.id
101,802
exception.message
Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.53.132.27:11114:389372401. See InnerException
—> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.53.132.27:11114. Error: ConnectionRefused
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in //src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 59
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in //src/Orleans.Core/Networking/ConnectionFactory.cs:line 56
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task1 task, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 410 at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 293 --- End of inner exception stack trace --- at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 293 at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 139 at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 139 at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask
1 c, Message m) in //src/Orleans.Runtime/Messaging/OutboundMessageQueue.cs:line 130
at Orleans.Runtime.Messaging.OutboundMessageQueue.g__SendAsync|10_0(ValueTask`1 c, Message m) in //src/Orleans.Runtime/Messaging/OutboundMessageQueue.cs:line 130
exception.stackTrace
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object ) in //src/Orleans.Runtime/Placement/DeploymentLoadPublisher.cs:line 117