Mongo connection is failing

Navya · May 12, 2021, 10:10am

I have…

[x] Checked the logs and have uploaded a log file and provided a link because I found something suspicious there. Please do not post the log file in the topic because very often something important is missing.

I’m submitting a…

[ ] Regression (a behavior that stopped working in a new release)
[x] Bug report
[ ] Performance issue
[ ] Documentation issue or request

Current behavior

The Mongodb connection is failing. I tried to restart mongodb and webapp from Azure.

Expected behavior

Connect to Squidex normally

Minimal reproduction of the problem

Environment

[ ] Self hosted with docker
[x] Self hosted with IIS
[ ] Self hosted with other version
[ ] Cloud version

Version: [VERSION]

Browser:

[x] Chrome (desktop)
[ ] Chrome (Android)
[ ] Chrome (iOS)
[ ] Firefox
[ ] Safari (desktop)
[ ] Safari (iOS)
[ ] IE
[ ] Edge

Others:
Getting this error in log files:
{
2021-05-12T09:23:37.371129762Z “logLevel”: “Error”,
2021-05-12T09:23:37.371137263Z “message”: “QueueWorkItem was called on a non-null context [SystemTarget: S127.0.0.1:11111:358507384*stg/13/0000000d@S0000000d] but there is no valid WorkItemGroup for it.”,
2021-05-12T09:23:37.371142763Z “eventId”: {
2021-05-12T09:23:37.371147363Z “id”: 101231
2021-05-12T09:23:37.371152263Z }

Log file:
2021-05-12T09:23:06.383222479Z 2021-05-12T09:23:06.383225679Z 2021-05-12T09:23:06.383228879Z 2021-05-12T09:23:06.383232079Z 2021-05-12T09:23:06.383235379Z },
2021-05-12T09:23:06.383238479Z 2021-05-12T09:23:06.383241779Z }
2021-05-12T09:23:06.383244979Z
2021-05-12T09:23:07.260508166Z {
2021-05-12T09:23:07.260574867Z 2021-05-12T09:23:07.260581367Z 2021-05-12T09:23:07.260585867Z 2021-05-12T09:23:07.260589967Z 2021-05-12T09:23:07.260606167Z 2021-05-12T09:23:07.260617968Z 2021-05-12T09:23:07.260622768Z },
2021-05-12T09:23:07.260626868Z 2021-05-12T09:23:07.260631768Z }
2021-05-12T09:23:07.260635768Z
2021-05-12T09:23:37.371059161Z {
2021-05-12T09:23:37.371129762Z 2021-05-12T09:23:37.371137263Z 2021-05-12T09:23:37.371142763Z 2021-05-12T09:23:37.371147363Z 2021-05-12T09:23:37.371152263Z },
2021-05-12T09:23:37.371156763Z 2021-05-12T09:23:37.371702276Z 2021-05-12T09:23:37.371716276Z 2021-05-12T09:23:37.371721476Z 2021-05-12T09:23:37.371726476Z },
2021-05-12T09:23:37.371756577Z 2021-05-12T09:23:37.371762977Z 2021-05-12T09:23:37.371768077Z }
2021-05-12T09:23:37.371772477Z
2021-05-12T09:23:37.411029982Z 2021-05-12T09:23:37.411059883Z 2021-05-12T09:23:37.411069083Z 2021-05-12T09:23:37.411073683Z 2021-05-12T09:23:37.411084183Z 2021-05-12T09:23:37.411089483Z 2021-05-12T09:23:37.411104584Z 2021-05-12T09:23:37.411108984Z 2021-05-12T09:23:37.411112684Z 2021-05-12T09:23:37.411116884Z 2021-05-12T09:23:37.411125684Z 2021-05-12T09:23:37.411129884Z 2021-05-12T09:23:37.411133884Z 2021-05-12T09:23:37.411137585Z 2021-05-12T09:23:37.411141485Z 2021-05-12T09:23:37.411145385Z 2021-05-12T09:23:37.411149785Z 2021-05-12T09:23:37.411153585Z 2021-05-12T09:23:37.411157485Z 2021-05-12T09:23:37.411161585Z 2021-05-12T09:23:37.411165485Z 2021-05-12T09:23:37.411169285Z 2021-05-12T09:23:37.411173285Z 2021-05-12T09:23:37.411177085Z 2021-05-12T09:23:37.411181386Z 2021-05-12T09:23:37.411193286Z 2021-05-12T09:23:37.411197286Z 2021-05-12T09:23:37.411201386Z 2021-05-12T09:23:37.411205286Z 2021-05-12T09:23:37.411209286Z 2021-05-12T09:23:37.411213086Z 2021-05-12T09:23:37.411216886Z 2021-05-12T09:23:37.411220786Z 2021-05-12T09:23:37.411224787Z “app”: {
“name”: “Squidex”,
“version”: “5.4.0.0”,
“sessionId”: “3359323f-77e3-4e5b-832f-2f6f09776f83”
“timestamp”: “2021-05-12T09:23:06Z”
“logLevel”: “Information”,
“initializedSystem”: “Squidex.Assets.AzureBlobAssetStore”,
“app”: {
“name”: “Squidex”,
“version”: “5.4.0.0”,
“sessionId”: “3359323f-77e3-4e5b-832f-2f6f09776f83”
“timestamp”: “2021-05-12T09:23:07Z”
“logLevel”: “Error”,
“message”: “QueueWorkItem was called on a non-null context [SystemTarget: S127.0.0.1:11111:358507384*stg/13/0000000d@S0000000d] but there is no valid WorkItemGroup for it.”,
“eventId”: {
“id”: 101231
“app”: {
“name”: “Squidex”,
“version”: “5.4.0.0”,
“sessionId”: “3359323f-77e3-4e5b-832f-2f6f09776f83”
“timestamp”: “2021-05-12T09:23:37Z”,
“category”: “Orleans.Runtime.Scheduler.OrleansTaskScheduler”
Unhandled exception. Squidex.Infrastructure.ConfigurationException: MongoDb connection failed to connect to database Squidex
—> System.TimeoutException: A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : “1”, ConnectionMode : “Automatic”, Type : “Unknown”, State : “Disconnected”, Servers : [{ ServerId: “{ ClusterId : 1, EndPoint : “Unspecified/surfacelove-new-dev-mongodb.westeurope.azurecontainer.io:27017” }”, EndPoint: “Unspecified/surfacelove-new-dev-mongodb.westeurope.azurecontainer.io:27017”, ReasonChanged: “Heartbeat”, State: “Disconnected”, ServerVersion: , TopologyVersion: , Type: “Unknown”, HeartbeatException: “MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server.
—> MongoDB.Driver.MongoConnectionException: An exception occurred while receiving a message from the server.
—> System.TimeoutException: The operation has timed out.
at MongoDB.Driver.Core.Misc.StreamExtensionMethods.ReadAsync(Stream stream, Byte[] buffer, Int32 offset, Int32 count, TimeSpan timeout, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Misc.StreamExtensionMethods.ReadBytesAsync(Stream stream, Byte[] buffer, Int32 offset, Int32 count, TimeSpan timeout, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync(CancellationToken cancellationToken)
— End of inner exception stack trace —
at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessageAsync(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
at MongoDB.Driver.Core.WireProtocol.CommandUsingQueryMessageWireProtocol1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken) 2021-05-12T09:23:37.411121684Z at MongoDB.Driver.Core.Connections.IsMasterHelper.GetResultAsync(IConnection connection, CommandWireProtocol1 isMasterProtocol, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.ConnectionInitializer.InitializeConnectionAsync(IConnection connection, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
— End of inner exception stack trace —
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Servers.ServerMonitor.InitializeConnectionAsync(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Servers.ServerMonitor.HeartbeatAsync(CancellationToken cancellationToken)”, LastHeartbeatTimestamp: “2021-05-12T09:23:36.3114319Z”, LastUpdateTimestamp: “2021-05-12T09:23:36.3114323Z” }] }.
at MongoDB.Driver.Core.Clusters.Cluster.ThrowTimeoutException(IServerSelector selector, ClusterDescription description)
at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedHelper.HandleCompletedTask(Task completedTask)
at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedAsync(IServerSelector selector, ClusterDescription description, Task descriptionChangedTask, TimeSpan timeout, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Clusters.Cluster.SelectServerAsync(IServerSelector selector, CancellationToken cancellationToken)
at MongoDB.Driver.MongoClient.AreSessionsSupportedAfterSeverSelctionAsync(CancellationToken cancellationToken)
at MongoDB.Driver.MongoClient.AreSessionsSupportedAsync(CancellationToken cancellationToken)
at MongoDB.Driver.MongoClient.StartImplicitSessionAsync(CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollectionImpl1.UsingImplicitSessionAsync[TResult](Func2 funcAsync, CancellationToken cancellationToken)
at Squidex.Infrastructure.EventSourcing.MongoEventStore.SetupCollectionAsync(IMongoCollection1 collection, CancellationToken ct) in /src/src/Squidex.Infrastructure.MongoDb/EventSourcing/MongoEventStore.cs:line 57 2021-05-12T09:23:37.411188986Z at Squidex.Infrastructure.MongoDb.MongoRepositoryBase1.InitializeAsync(CancellationToken ct) in /src/src/Squidex.Infrastructure.MongoDb/MongoDb/MongoRepositoryBase.cs:line 111
— End of inner exception stack trace —
at Squidex.Infrastructure.MongoDb.MongoRepositoryBase`1.InitializeAsync(CancellationToken ct) in /src/src/Squidex.Infrastructure.MongoDb/MongoDb/MongoRepositoryBase.cs:line 115
at Squidex.Config.Startup.InitializerHost.StartAsync(ISemanticLog log, CancellationToken ct) in /src/src/Squidex/Config/Startup/InitializerHost.cs:line 31
at Squidex.Config.Startup.SafeHostedService.StartAsync(CancellationToken cancellationToken) in /src/src/Squidex/Config/Startup/SafeHostedService.cs:line 27
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
at Squidex.Program.Main(String[] args) in /src/src/Squidex/Program.cs:line 23

Sebastian · May 12, 2021, 10:12am

I am not sure how I can help you there. It is definitely not a bug, more a hosting issue and I have too few information to help you there. Have you checked the MongoDB status?

Navya · May 12, 2021, 10:34am

strange! the mongodb is connected and working again. I dont know why this problem is reoccuring and how it is to be fixed. How can i make sure this doesnt happen again. Please guide.
This is the connections we have for cms webapp. We are using secret key vault
[
{
“name”: “ASSETSTORE__AZUREBLOB__CONNECTIONSTRING”,
“value”: “@Microsoft.KeyVault(VaultName=SurfaceLove-New-DEV;SecretName=ConnectionStrings–StorageAccount;SecretVersion=)”,
“slotSetting”: false
},
{
“name”: “ASSETSTORE__AZUREBLOB__CONTAINERNAME”,
“value”: “cms-assets”,
“slotSetting”: false
},
{
“name”: “ASSETSTORE__TYPE”,
“value”: “AzureBlob”,
“slotSetting”: false
},
{
“name”: “EVENTSTORE__MONGODB__CONFIGURATION”,
“value”: “@Microsoft.KeyVault(VaultName=SurfaceLove-New-DEV;SecretName=ConnectionStrings–MongoDB;SecretVersion=)”,
“slotSetting”: false
},
{
“name”: “IDENTITY__ADMINEMAIL”,
“value”: “@Microsoft.KeyVault(VaultName=SurfaceLove-New-DEV;SecretName=CMSAdminLogin;SecretVersion=)”,
“slotSetting”: false
},
{
“name”: “IDENTITY__ADMINPASSWORD”,
“value”: “@Microsoft.KeyVault(VaultName=SurfaceLove-New-DEV;SecretName=CMSAdminLoginPassword;SecretVersion=)”,
“slotSetting”: false
},
{
“name”: “IDENTITY__ADMINRECREATE”,
“value”: “true”,
“slotSetting”: false
},
{
“name”: “IDENTITY__GITHUBCLIENT”,
“value”: “”,
“slotSetting”: false
},
{
“name”: “IDENTITY__GITHUBSECRET”,
“value”: “”,
“slotSetting”: false
},
{
“name”: “IDENTITY__GOOGLECLIENT”,
“value”: “”,
“slotSetting”: false
},
{
“name”: “IDENTITY__GOOGLESECRET”,
“value”: “”,
“slotSetting”: false
},
{
“name”: “IDENTITY__MICROSOFTCLIENT”,
“value”: “”,
“slotSetting”: false
},
{
“name”: “IDENTITY__MICROSOFTSECRET”,
“value”: “”,
“slotSetting”: false
},
{
“name”: “IDENTITY__oidcAuthority”,
“value”: “https://login.microsoftonline.com/7e34a101-4f3e-45c7-943e-1bfeb9530099/v2.0”,
“slotSetting”: false
},
{
“name”: “IDENTITY__oidcClient”,
“value”: “@Microsoft.KeyVault(VaultName=SurfaceLove-New-DEV;SecretName=CmsOIDCClientId;SecretVersion=)”,
“slotSetting”: false
},
{
“name”: “IDENTITY__oidcName”,
“value”: “Microsoft”,
“slotSetting”: false
},
{
“name”: “IDENTITY__oidcSecret”,
“value”: “@Microsoft.KeyVault(VaultName=SurfaceLove-New-DEV;SecretName=CmsOIDCClientSecret;SecretVersion=)”,
“slotSetting”: false
},
{
“name”: “STORE__MONGODB__CONFIGURATION”,
“value”: “@Microsoft.KeyVault(VaultName=SurfaceLove-New-DEV;SecretName=ConnectionStrings–MongoDB;SecretVersion=)”,
“slotSetting”: false
},
{
“name”: “URLS__BASEURL”,
“value”: “https://surfacelove-new-dev-cms.azurewebsites.net”,
“slotSetting”: false
}
]

Sebastian · May 12, 2021, 10:46am

I need more infos about your deployment before I can help. Where do you host MongoDB? Just assume that I have no idea about your hosting setup.