Squidex unresponsive and lots of Orleans.Messaging.Drop.Expired messages

Hosting (OnPremise)
1

Hi

We’re having some problems with our self-hosted version of Squidex in Azure. What we’re seeing is once a day or so the squidex web api starts to fail e.g. unable to load schemas, content etc. At the time this starts to happen the logs start to fill up with this event, that repeats every few seconds:

2021-07-12T10:07:33.753149352Z: [INFO]  {
2021-07-12T10:07:33.753152952Z: [INFO]    "logLevel": "Warning",
2021-07-12T10:07:33.753156852Z: [INFO]    "message": "Dropping expired message Status Response S127.0.0.1:11111:363115450*grn/Squidex.Domain.Apps.Entities.Schemas.DomainObject.SchemaDomainObjectGrain/0\u002B722f9229-65ef-494f-9aac-b66c50c62c37--4f704ced-ebbe-49c1-a89f-d31b5e91ce93@4873d6bc-\u003ES127.0.0.1:11111:363115450*cli/eab536a2@e047f684 #3982180 at phase Send",
2021-07-12T10:07:33.753161452Z: [INFO]    "eventId": {
2021-07-12T10:07:33.753165252Z: [INFO]      "id": 101038,
2021-07-12T10:07:33.753169052Z: [INFO]      "name": "Orleans.Messaging.Drop.Expired"
2021-07-12T10:07:33.753172952Z: [INFO]    },
2021-07-12T10:07:33.753177152Z: [INFO]    "message": "Status Response S127.0.0.1:11111:363115450*grn/Squidex.Domain.Apps.Entities.Schemas.DomainObject.SchemaDomainObjectGrain/0\u002B722f9229-65ef-494f-9aac-b66c50c62c37--4f704ced-ebbe-49c1-a89f-d31b5e91ce93@4873d6bc-\u003ES127.0.0.1:11111:363115450*cli/eab536a2@e047f684 #3982180",
2021-07-12T10:07:33.753182452Z: [INFO]    "phase": "Send",
2021-07-12T10:07:33.753186352Z: [INFO]    "timestamp": "2021-07-12T10:07:33Z",
2021-07-12T10:07:33.753190152Z: [INFO]    "app": {
2021-07-12T10:07:33.753193952Z: [INFO]      "name": "Squidex",
2021-07-12T10:07:33.753197752Z: [INFO]      "version": "5.6.0.0",
2021-07-12T10:07:33.753201552Z: [INFO]      "sessionId": "6c9e84fd-d970-436f-b938-566921171c0a"
2021-07-12T10:07:33.753205552Z: [INFO]    },
2021-07-12T10:07:33.753209252Z: [INFO]    "category": "Orleans.Messaging"
2021-07-12T10:07:33.753213052Z: [INFO]  }

If we restart the web app running the docker container then everything starts working again and this event stops repeating in the logs.

Do you have any thoughts on what might cause this please?
Thanks
Andrew

2

Azure web apps do not support clustering.

3

Is turning clustering off an option?
Or is it required and therefore we have to choose a different hosting model?

4

It should be turned off by default, but do you have more than one instance? I host the cloud with kubernetes and it works fine.

5

No, at the moment we have a single instance of the app service plan running - its a B1 scale unit, running Linux (if that helps in anyway)
Is there an app setting I can check or change?

6

Okay, thats weird. I do not have an answer. I have experienced it before that the apps sometimes just stops there and I don’t know why.

7

Ok thanks. In that case, would you recommend Kubernetes as a better option for a production deployment, where it needs to be self-hosted?

8

In general I recommend and prefer setups that are agnostic from the cloud provider.

I have a very CI / devops engineer who will have a look into this.

1 Like