RuleEnqueuer can not reset

Hi @Sebastian

I was trying to move squidex to our PRD(K8S) env. but I keep getting some strange issue.
we using MongoDB cluster. 1 master, and few nodes.

for the first time. I found rule is not trigging any event. so I checked admin portal. try to restart RuleEnqueuer.
but I got this message.

“Requested version 1 for object ‘RuleEnqueuer’ (type Squidex.Infrastructure.EventSourcing.Grains.EventConsumerGrain), but found 9.”

Do you have a guide for how to deploy squidex in PRD (K8S) ?

{
  "logLevel": "Error",
  "message": "ReminderTable.ReadRows failed. Exception=An exception occurred while receiving a message from the server.",
  "eventId": {
    "id": 900400
  },
  "exception": {
    "type": "MongoDB.Driver.MongoConnectionException",
    "message": "An exception occurred while receiving a message from the server.",
    "stackTrace": "   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync()\n   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync(Int32 responseTo, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessageAsync(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.WireProtocol.CommandUsingQueryMessageWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocolAsync[TResult](IWireProtocol`1 protocol, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocolAsync(IChannelSource channelSource, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.FindCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.FindOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)\n   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)\n   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)\n   at MongoDB.Driver.IAsyncCursorSourceExtensions.ToListAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)\n   at Orleans.Providers.MongoDB.Reminders.Store.MongoReminderCollection.ReadRowsInRange(UInt32 beginHash, UInt32 endHash)\n   at Orleans.Providers.MongoDB.Reminders.MongoReminderTable.DoAndLog[T](String actionName, Func`1 action)"
  },
  "app": {
    "name": "Squidex",
    "version": "1.0.0.0",
    "sessionId": "aa29f95a-46d0-4f9e-a20b-8794c8cb3360"
  },
  "timestamp": "2019-06-02T12:11:28Z",
  "category": "Orleans.Providers.MongoDB.Reminders.MongoReminderTable"
}

{
  "logLevel": "Error",
  "message": "Failed to read rows from table.",
  "eventId": {
    "id": 102920
  },
  "exception": {
    "type": "MongoDB.Driver.MongoConnectionException",
    "message": "An exception occurred while receiving a message from the server.",
    "stackTrace": "   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync()\n   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync(Int32 responseTo, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessageAsync(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.WireProtocol.CommandUsingQueryMessageWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocolAsync[TResult](IWireProtocol`1 protocol, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocolAsync(IChannelSource channelSource, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.FindCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.FindOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)\n   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)\n   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)\n   at MongoDB.Driver.IAsyncCursorSourceExtensions.ToListAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)\n   at Orleans.Providers.MongoDB.Reminders.Store.MongoReminderCollection.ReadRowsInRange(UInt32 beginHash, UInt32 endHash)\n   at Orleans.Providers.MongoDB.Reminders.MongoReminderTable.DoAndLog[T](String actionName, Func`1 action)\n   at Orleans.Runtime.ReminderService.LocalReminderService.ReadTableAndStartTimers(IRingRange range, Int32 rangeSerialNumberCopy)"
  },
  "app": {
    "name": "Squidex",
    "version": "1.0.0.0",
    "sessionId": "aa29f95a-46d0-4f9e-a20b-8794c8cb3360"
  },
  "timestamp": "2019-06-02T12:11:28Z",
  "category": "Orleans.Runtime.ReminderService.LocalReminderService"
}

{
  "logLevel": "Error",
  "message": "Caught and ignored exception: MongoDB.Driver.MongoConnectionException with message: An exception occurred while receiving a message from the server. thrown from timer callback GrainTimer.ReminderService.ReminderListRefresher. TimerCallbackHandler:[LowPrioritySystemTarget: S172.30.218.226:11111:296979917*stg/0/00000000@S00000000]->System.Threading.Tasks.Task <PromoteToStarted>b__26_0(System.Object)",
  "eventId": {
    "id": 101413
  },
  "exception": {
    "type": "MongoDB.Driver.MongoConnectionException",
    "message": "An exception occurred while receiving a message from the server.",
    "stackTrace": "   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync()\n   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync(Int32 responseTo, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessageAsync(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.WireProtocol.CommandUsingQueryMessageWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocolAsync[TResult](IWireProtocol`1 protocol, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocolAsync(IChannelSource channelSource, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.FindCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.FindOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)\n   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)\n   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)\n   at MongoDB.Driver.IAsyncCursorSourceExtensions.ToListAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)\n   at Orleans.Providers.MongoDB.Reminders.Store.MongoReminderCollection.ReadRowsInRange(UInt32 beginHash, UInt32 endHash)\n   at Orleans.Providers.MongoDB.Reminders.MongoReminderTable.DoAndLog[T](String actionName, Func`1 action)\n   at Orleans.Runtime.ReminderService.LocalReminderService.ReadTableAndStartTimers(IRingRange range, Int32 rangeSerialNumberCopy)\n   at Orleans.Runtime.ReminderService.LocalReminderService.ReadAndUpdateReminders()\n   at Orleans.Runtime.GrainTimer.ForwardToAsyncCallback(Object state)"
  },
  "app": {
    "name": "Squidex",
    "version": "1.0.0.0",
    "sessionId": "aa29f95a-46d0-4f9e-a20b-8794c8cb3360"
  },
  "timestamp": "2019-06-02T12:11:28Z",
  "category": "Orleans.Runtime.GrainTimer"
}

{
  "logLevel": "Error",
  "message": "ReminderTable.ReadRows failed. Exception=An exception occurred while receiving a message from the server.",
  "eventId": {
    "id": 900400
  },
  "exception": {
    "type": "MongoDB.Driver.MongoConnectionException",
    "message": "An exception occurred while receiving a message from the server.",
    "stackTrace": "   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync()\n   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync(Int32 responseTo, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessageAsync(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.WireProtocol.CommandUsingQueryMessageWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocolAsync[TResult](IWireProtocol`1 protocol, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocolAsync(IChannelSource channelSource, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.FindCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.FindOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)\n   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)\n   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)\n   at MongoDB.Driver.IAsyncCursorSourceExtensions.ToListAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)\n   at Orleans.Providers.MongoDB.Reminders.Store.MongoReminderCollection.ReadRowsInRange(UInt32 beginHash, UInt32 endHash)\n   at Orleans.Providers.MongoDB.Reminders.MongoReminderTable.DoAndLog[T](String actionName, Func`1 action)"
  },
  "app": {
    "name": "Squidex",
    "version": "1.0.0.0",
    "sessionId": "aa29f95a-46d0-4f9e-a20b-8794c8cb3360"
  },
  "timestamp": "2019-06-02T12:31:29Z",
  "category": "Orleans.Providers.MongoDB.Reminders.MongoReminderTable"
}

{
  "logLevel": "Error",
  "message": "Failed to read rows from table.",
  "eventId": {
    "id": 102920
  },
  "exception": {
    "type": "MongoDB.Driver.MongoConnectionException",
    "message": "An exception occurred while receiving a message from the server.",
    "stackTrace": "   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync()\n   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync(Int32 responseTo, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessageAsync(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.WireProtocol.CommandUsingQueryMessageWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocolAsync[TResult](IWireProtocol`1 protocol, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocolAsync(IChannelSource channelSource, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.FindCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.Core.Operations.FindOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\n   at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)\n   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)\n   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)\n   at MongoDB.Driver.IAsyncCursorSourceExtensions.ToListAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)\n   at Orleans.Providers.MongoDB.Reminders.Store.MongoReminderCollection.ReadRowsInRange(UInt32 beginHash, UInt32 endHash)\n   at Orleans.Providers.MongoDB.Reminders.MongoReminderTable.DoAndLog[T](String actionName, Func`1 action)\n   at Orleans.Runtime.ReminderService.LocalReminderService.ReadTableAndStartTimers(IRingRange range, Int32 rangeSerialNumberCopy)"
  },
  "app": {
    "name": "Squidex",
    "version": "1.0.0.0",
    "sessionId": "aa29f95a-46d0-4f9e-a20b-8794c8cb3360"
  },
  "timestamp": "2019-06-02T12:31:29Z",
  "category": "Orleans.Runtime.ReminderService.LocalReminderService"
}

I would solve the MongoDB connection issue first.

Sure, After I deleted some of pods. the issue is gone.

1 Like