Squidex 3.5.0 to Mongodb 3.4 - Connectivity sporadic issues

I have…

  • [X ] Checked the logs and have provided the logs if I found something suspicious there

I’m submitting a…

Current behavior

On Squidex 3.5.0
Configured to access a Mongo DB Replicaset.
Squidex is experiencing MongoDBConnection Issues; Logs will be attached
The CMS is available, but sluggish due to not always executing or fulfilling it’s request.
Issue seems to be centered often around the Orleans.Providers.MongoDB.Reminders.MongoReminderTable in the logs

Expected behavior

To have Squidex not have any issue with connectivity to mongodo, which will ressult in the CMS to run at full optimal speed.

Minimal reproduction of the problem

N/A

Environment

Squidex 3.50
Installed On IIS on Window 2016 Server 8 GB Ram
Mongo DB Replica et 3 instances Mongo 3.4

  • [X ] Self hosted with IIS
    Version: 3.5.0

Browser:

  • [ X] All

Others:
The system had been running fine for weeks. Blazing speed between Squidex and the Mongodb backend.

Just recently the system started to slow down.
Logs how the mongodb connectivity problem

The networking between these instances are functioning as expected. IT support team as excellent.
I can ping the mongodb instances from the IIS application servers without issue.
I installed Mongodb’s Compass tool on the web server. I can access the same replicaset without issues.
Running Mongo rs.status() indicates the databases is running and replicated between each of the 3 machines.

I will create the topic and upload a large log file.

Log messages look as follow:

{
  "logLevel": "Error",
  "message": "Failed to read rows from table.",
  "eventId": {
    "id": 102920
  },
  "exception": {
    "type": "MongoDB.Driver.MongoConnectionException",
    "message": "An exception occurred while opening a connection to the server.",
    "stackTrace": "   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)\r\n   at MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken)\r\n   at MongoDB.Driver.Core.Operations.RetryableReadContext.InitializeAsync(CancellationToken cancellationToken)\r\n   at MongoDB.Driver.Core.Operations.RetryableReadContext.CreateAsync(IReadBinding binding, Boolean retryRequested, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.Core.Operations.FindOperation\u00601.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation\u00601 operation, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.MongoCollectionImpl\u00601.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation\u00601 operation, ReadPreference readPreference, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.MongoCollectionImpl\u00601.UsingImplicitSessionAsync[TResult](Func\u00602 funcAsync, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.IAsyncCursorSourceExtensions.ToListAsync[TDocument](IAsyncCursorSource\u00601 source, CancellationToken cancellationToken)\r\n   at Orleans.Providers.MongoDB.Reminders.Store.MongoReminderCollection.ReadRowsInRange(UInt32 beginHash, UInt32 endHash)\r\n   at Orleans.Providers.MongoDB.Reminders.MongoReminderTable.DoAndLog[T](String actionName, Func\u00601 action)\r\n   at Orleans.Runtime.ReminderService.LocalReminderService.ReadTableAndStartTimers(IRingRange range, Int32 rangeSerialNumberCopy)"
  },
  "app": {
    "name": "Squidex",
    "version": "1.0.0.0",
    "sessionId": "ed04adc3-4783-448c-a945-ff14d929d4ac"
  },
  "timestamp": "2020-05-13T15:38:47Z",
  "category": "Orleans.Runtime.ReminderService.LocalReminderService"
}

{
  "logLevel": "Error",
  "message": "ReminderTable.ReadRows failed. Exception=An exception occurred while opening a connection to the server.",
  "eventId": {
    "id": 900400
  },
  "exception": {
    "type": "MongoDB.Driver.MongoConnectionException",
    "message": "An exception occurred while opening a connection to the server.",
    "stackTrace": "   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)\r\n   at MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken)\r\n   at MongoDB.Driver.Core.Operations.RetryableReadContext.InitializeAsync(CancellationToken cancellationToken)\r\n   at MongoDB.Driver.Core.Operations.RetryableReadContext.CreateAsync(IReadBinding binding, Boolean retryRequested, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.Core.Operations.FindOperation\u00601.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation\u00601 operation, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.MongoCollectionImpl\u00601.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation\u00601 operation, ReadPreference readPreference, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.MongoCollectionImpl\u00601.UsingImplicitSessionAsync[TResult](Func\u00602 funcAsync, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.IAsyncCursorSourceExtensions.ToListAsync[TDocument](IAsyncCursorSource\u00601 source, CancellationToken cancellationToken)\r\n   at Orleans.Providers.MongoDB.Reminders.Store.MongoReminderCollection.ReadRowsInRange(UInt32 beginHash, UInt32 endHash)\r\n   at Orleans.Providers.MongoDB.Reminders.MongoReminderTable.DoAndLog[T](String actionName, Func\u00601 action)"
  },
  "app": {
    "name": "Squidex",
    "version": "1.0.0.0",
    "sessionId": "ed04adc3-4783-448c-a945-ff14d929d4ac"
  },
  "timestamp": "2020-05-13T15:38:47Z",
  "category": "Orleans.Providers.MongoDB.Reminders.MongoReminderTable"
}

Where can I attached a full log for you to examine?

You should be able to upload a file or you can also attach a link to a dropbox file or so.

Can you please format our logs and sample with code blocks?

Hi Sebastian - Here is a link to Google Drive:

Sorry, I saw your reply. You mentioned sample with code blocks. I don’t understand the second part.
Did the google link work as a first ask?

In markdown you can use code blocks

```
Code
```

1 Like

I have no idea what this is. Does not sound like a Squidex problem…I am also running mongodb 3.4 btw.

Okay, I see. Ah, I am stressing over this. I can’t guestimate why the app seems to repeat struggle on issues in the db.

Are there any steps you’d take to investigate why perfectly good app server would start experiencing these Mongo db connectivity issues all of a sudden?

Any optimization on the db/replicaset side or tuning?

What is the purpose of the Orleans MongoRemindersTable?

Is there any aspect of applicationSetting I could set to turn this feature off?
We have two CMS app servers out load-balanced. Would this cause any related issues? Or setting I didn’t set for a multi-instance CMS cluster?

Should i attempt to rebuild the cms/db from scratch? Restore the cms off fresh back up?

MongoReminderTable is a list of dynamically created timers that are managed by the clustering framework (Microsoft Orleans).

There is nothing on the squidex side. Perhaps just network issue or so? You could create a simple .net core app that constantly queries a test collection to verify if the issue exists for other apps as well. Or some network monitoring?

You should also monitor you cluster health. Perhaps there is some correlation?