Rules webhooks not working (Clustering issue)

Support
1

Hi

I’ve recently migrated to MongoAtlas to store my data and webhooks are no longer working bu are always in a ‘Pending’ state. I’m running v1.15.0 in Docker.

These logs are coming from the docker process:

  },
  "app": {
"name": "Squidex",
"version": "1.0.0.0",
"sessionId": "8caf1ce9-0345-4b2a-9e95-124ce06cfd6b"
  },
  "timestamp": "2019-02-14T21:16:59.3814284Z",
  "category": "Orleans.Providers.MongoDB.Membership.MongoMembershipTable"
}

{
  "logLevel": "Error",
  "message": "MongoMembershipTable.ReadAll failed. Exception=Object reference not set to an instance of an object.",
  "eventId": {
"id": 900200
  },
  "exception": {
"type": "System.NullReferenceException",
"message": "Object reference not set to an instance of an object.",
"stackTrace": "   at Orleans.Providers.MongoDB.Membership.MongoMembershipTable.<ReadAll>b__8_0()\n   at Orleans.Providers.MongoDB.Membership.MongoMembershipTable.DoAndLog[T](String actionName, Func`1 action)"
  },
  "app": {
"name": "Squidex",
"version": "1.0.0.0",
"sessionId": "8caf1ce9-0345-4b2a-9e95-124ce06cfd6b"
  },
  "timestamp": "2019-02-14T21:16:59.3833482Z",
  "category": "Orleans.Providers.MongoDB.Membership.MongoMembershipTable"
}

{
  "logLevel": "Error",
  "message": "Caught and ignored exception: System.NullReferenceException with message: Object reference not set to an instance of an object. thrown from timer callback GrainTimer. TimerCallbackHandler:OrleansDashboard.DashboardGrain->System.Threading.Tasks.Task Callback(System.Object)",
  "eventId": {
"id": 101413
  },
  "exception": {
"type": "System.NullReferenceException",
"message": "Object reference not set to an instance of an object.",
"stackTrace": "   at Orleans.Providers.MongoDB.Membership.MongoMembershipTable.<ReadAll>b__8_0()\n   at Orleans.Providers.MongoDB.Membership.MongoMembershipTable.DoAndLog[T](String actionName, Func`1 action)\n   at Orleans.Runtime.Management.ManagementGrain.GetHosts(Boolean onlyActive)\n   at Orleans.Runtime.Management.ManagementGrain.GetTotalActivationCount()\n   at Orleans.Runtime.OrleansCodeGenManagementGrainMethodInvoker.Invoke(IAddressable grain, InvokeMethodRequest request)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at Squidex.Infrastructure.Orleans.LocalCacheFilter.Invoke(IIncomingGrainCallContext context) in /src/src/Squidex.Infrastructure/Orleans/LocalCacheFilter.cs:line 37\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at OrleansDashboard.Metrics.GrainProfiler.Invoke(IIncomingGrainCallContext context)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke()\n   at Orleans.Runtime.InsideRuntimeClient.Invoke(IAddressable target, IInvokable invokable, Message message)\n   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)\n   at OrleansDashboard.DashboardGrain.Callback(Object _)\n   at Orleans.Runtime.GrainTimer.ForwardToAsyncCallback(Object state)"
  },
  "app": {
"name": "Squidex",
"version": "1.0.0.0",
"sessionId": "8caf1ce9-0345-4b2a-9e95-124ce06cfd6b"
  },
  "timestamp": "2019-02-14T21:16:59.3886739Z",
  "category": "Orleans.Runtime.GrainTimer"
}

This is my dockerfile, should that help:

version: '2.1'
services:
  squidex_squidex:
    image: "squidex/squidex:v1.15.0"
    ports:
      - "80:80"
    environment:
      - URLS__BASEURL=${SQUIDEX_PROTOCOL}://${SQUIDEX_DOMAIN}/
      - URLS__ENFORCEHTTPS=${SQUIDEX_FORCE_HTTPS}
      - EVENTSTORE__CONSUME=true
      - EVENTSTORE__MONGODB__CONFIGURATION=mongodb+srv://XXX@host/?retryWrites=true
      - STORE__MONGODB__CONFIGURATION=mongodb+srv://XXX@host/?retryWrites=true
      - IDENTITY__ADMINEMAIL=${SQUIDEX_ADMINEMAIL}
      - IDENTITY__ADMINPASSWORD=${SQUIDEX_ADMINPASSWORD}
      - IDENTITY__GOOGLECLIENT=${SQUIDEX_GOOGLECLIENT}
      - IDENTITY__GOOGLESECRET=${SQUIDEX_GOOGLESECRET}
      - IDENTITY__MICROSOFTCLIENT=${SQUIDEX_MICROSOFTCLIENT}
      - IDENTITY__MICROSOFTSECRET=${SQUIDEX_MICROSOFTSECRET}
      - LETSENCRYPT_HOST=${SQUIDEX_DOMAIN}
      - LETSENCRYPT_EMAIL=${SQUIDEX_ADMINEMAIL}
    volumes:
      - /etc/squidex/assets:/app/Assets
    networks:
      - internal
    restart: unless-stopped

networks:
  internal:
    driver: bridge

Any idea? Thanks in advance! My db user is admin over all databases and has r/w access to all databases.

2

The logs are confusing but not related directly. Are you running Squidex in a cluster?

Can you provide me the full logs?

3

I’m not running it in a cluster. Here’s the full txt log of the docker container:
https://mega.nz/#!tZBUlQLb!WWaegs1qFodQeugb2k7GLMdeKN9NSYV5n0QGUOgHbQQ

4

Have you checked this:

{
  "logLevel": "Error",
  "message": "MembershipFailedToStart",
  "eventId": {
    "id": 100646
  },
  "exception": {
    "type": "MongoDB.Driver.MongoConfigurationException",
    "message": "Host for mongodb+srv scheme cannot specify a port.",
    "stackTrace": "   at MongoDB.Driver.Core.Configuration.ConnectionString.ExtractHosts(Match match)\n   at MongoDB.Driver.MongoUrlBuilder.Parse(String url)\n   at MongoDB.Driver.MongoUrl..ctor(String url)\n   at MongoDB.Driver.MongoClient..ctor(String connectionString)\n   at Orleans.Providers.MongoDB.Utils.MongoClientPool.<>c.<Instance>b__1_0(String cs)\n   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)\n   at Orleans.Providers.MongoDB.Utils.MongoClientPool.Instance(String connectionString)\n   at Orleans.Providers.MongoDB.Utils.CollectionBase`1..ctor(String connectionString, String databaseName)\n   at Orleans.Providers.MongoDB.Membership.Store.MongoMembershipCollection..ctor(String connectionString, String databaseName, String collectionPrefix)\n   at Orleans.Providers.MongoDB.Membership.MongoMembershipTable.InitializeMembershipTable(Boolean tryInitTableVersion)\n   at Orleans.Runtime.MembershipService.MembershipOracle.Start()"
  },
  "app": {
    "name": "Squidex",
    "version": "1.0.0.0",
    "sessionId": "8caf1ce9-0345-4b2a-9e95-124ce06cfd6b"
  },
  "timestamp": "2019-02-14T21:09:01.3412731Z",
  "category": "Orleans.Runtime.MembershipService.MembershipOracleData"
}

I am not sure why the process even starts, I will have a look. But something is wrong with the connection string.

5

Everything else seems to work fine tho. Mhm.

6

Yes, but you are probably not communicating with a healthy cluster. I guess you have two disconnected nodes instead.

7

I could reproduce the problem. I think it is a bug in the Mongo c# driver. Btw: The latest version should prevent your process from starting if this happens, at least it did it in my case.

I will wait for the answer from the MongoDB team. Perhaps there is a workaround for this issue.

8

I pushed a potential fix.

9

Alright, should I run the dev version or what should I do?

10

Yes, please…if you want to test it

11

Issue seems to be fixed, webhooks are working again! Great! thanks for the support as always!