MongoDB restore Out of memory

Support
1

Hey!
When I’m trying to restore a mongodb backup to other instance I have a problem when squidex is starting: it crashes with the message Out of memory

I’ve tried to increase requests and limits for the squidex instance but it didn’t help.

Some previous logs... 
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "94ae39cd-2226-49e4-8dcc-0e94b2f54b92",
  "elapsedMs": 5,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:03Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "57e89daa-e1fa-4285-8fa0-44b75b02a977",
  "elapsedMs": 6,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:03Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "6e682617-eabe-47fd-9740-f1e9a4602245",
  "elapsedMs": 4,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:03Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "a4754a08-06fb-4ea0-b87f-d78125cc097a",
  "elapsedMs": 10,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:04Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "b0bfdbab-ceb9-4dd4-b0d6-6c045021b10a",
  "elapsedMs": 11,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:04Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "b245dbcd-1f0f-469f-b332-568d16d74d4d",
  "elapsedMs": 6,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:05Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "4b570cb8-e296-45fd-9382-88efbd148dcc",
  "elapsedMs": 5,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:09Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "417b0c3a-ea5e-4d3e-ac5e-1d80961c2f4a",
  "elapsedMs": 54,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:10Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "2dbb7800-43a5-4b46-a818-a3c58eaeed14",
  "elapsedMs": 3,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:10Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "07fc016e-0417-4ae0-ae27-30250e522a84",
  "elapsedMs": 2,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:10Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "5e943db4-433e-4416-88a6-3ed1b919a621",
  "elapsedMs": 2,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:10Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "cff40d82-3ecc-4a52-846a-413f19455db0",
  "elapsedMs": 2,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:10Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "1052b990-596c-4049-a6b9-4b8581ec5701",
  "elapsedMs": 2,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:10Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "81124932-903c-40bf-8d04-df657ac8c523",
  "elapsedMs": 2,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:10Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "5b780e5a-d693-4de3-8b1a-ae1ca6fbf5cc",
  "elapsedMs": 2,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:10Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "fab87f24-4b27-4c5c-9d90-27581f6067a8",
  "elapsedMs": 2,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:10Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "fdc9b8ed-c015-4684-b5ce-aa47ae6e9172",
  "elapsedMs": 2,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:10Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "97291eca-d92d-4827-8a86-70846223df93",
  "elapsedMs": 3,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:10Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "090d3393-f224-4cd6-830f-84c98f91289e",
  "elapsedMs": 38,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:10Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "12948849-9a6e-4c53-914d-6c775cfbe19e",
  "elapsedMs": 7,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:11Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "960460f8-8cea-4ea5-bc21-ecdcb0b66db3",
  "elapsedMs": 7,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:11Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "2f5bcc34-272c-4374-857f-91f6c9bba397",
  "elapsedMs": 7,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:11Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "f57752e3-66bb-47e1-a41e-3e7ea76ec475",
  "elapsedMs": 3,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:11Z"
}
{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "65bf018a-d404-4f24-89d8-7a4fde7d7762",
  "elapsedMs": 4,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:12Z"
}

{
  "logLevel": "Information",
  "action": "ActivateDomainObject",
  "domainObjectType": "SchemaDomainObject",
  "domainObjectKey": "b57e9899-f790-4353-a589-a71a82db7721",
  "elapsedMs": 4,
  "app": {
    "name": "Squidex",
    "version": "4.5.1.0",
    "sessionId": "e7c76733-dce0-428a-a66d-c4e1f1d4c281"
  },
  "timestamp": "2021-05-21T12:06:12Z"
}
Out of memory.

I’m submitting a…

  • [ ] Regression (a behavior that stopped working in a new release)
  • [ x ] Bug report
  • [ ] Performance issue
  • [ ] Documentation issue or request

Current behavior

Squidex crashes on first run from a restored mongodb database

Expected behavior

Squidex should not crash during startup

Minimal reproduction of the problem

I have no idea how to reproduce it without my mongodb backup :frowning:

Environment

  • [ x ] Self hosted with docker
  • [ ] Self hosted with IIS
  • [ ] Self hosted with other version
  • [ ] Cloud version

Version: 4.5.1

Browser:

  • [ x ] Chrome (desktop)
  • [ ] Chrome (Android)
  • [ ] Chrome (iOS)
  • [ ] Firefox
  • [ ] Safari (desktop)
  • [ ] Safari (iOS)
  • [ ] IE
  • [ ] Edge

Others:

2

Just send me your backup.

3

I cannot reproduce a general memory issue. How many memory does your process consume when you have this issue?

5

Usage is around 6gb while instance is starting as I can see

image
image1211×391 32.1 KB

6

I’ve fixed! just removed the 10GB limit for memory
probably while starting the squidex tried to use more than 10GB

7

When I was running the migration in Squidex cloud it was using less than 6GB with far more data. Perhaps your limit is not forwarded to Squidex and the runtime allocates more resources than it should be and therefore it fails. Hard to say, but I was profiling your backup and even though it consumed a lot of memory it always went down a lot when I forced garbage collection.

8

Could you tell please how can I force the grabage collection?
Because after recovery, squidex consumes a lot of memory:

image
image1210×770 46 KB

9

And also could you tell me how can I provide the limit to squidex? I didn’t find any settings in appsettings.json about the memory limits

10

It is not possible to configure a limit in Squidex or to force the garbage collection (only when profiling), but Squidex uses the limits of the Operation system (or cgroups for docker).