Backup Restore fails with NullReferenceException

Support
1

Hi Sebastian, we have a self hosted Squidex with a single app that is showing degrading performance. In my research, I found the option to rebuild everything with the CONTENTS__OPTIMIZEFORSELFHOSTING flag from Squidex 7.0 Release Candidate 3 released - Squidex CMS which would hopefully save us.

But before I do this on Prod, I want to dry-run it on a fresh Docker container with the data from the backup. So I created a backup in our staging squidex instance and set up an instance (slightly edited docker compose from here) on my developer machine to restore the backup. But unfortunately this fails with an internal error. I think it must be a missing “AppId” but I cant figure out where exactly. I specified the optional app name and the app name is not in use. Can you identify the problem and help me?

Many thanks in advance!

Arguments:
{
  "appId": "5ed5d059-56f9-445f-9590-f0142886c9f0",
  "appName": "abcd"
}

Log:
2025-08-27 Total events: 807328, assets: 87

2025-08-27T22:50:46Z: Started. The restore process has the following steps:
2025-08-27T22:50:46Z: * Download backup
2025-08-27T22:50:46Z: * Restore events and attachments.
2025-08-27T22:50:46Z: * Restore all objects like app, schemas and contents
2025-08-27T22:50:46Z: * Complete the restore operation for all objects
2025-08-27T22:50:46Z: Downloading Backup
2025-08-27T22:50:55Z: Downloaded Backup
2025-08-27T22:50:55Z: Failed with internal error.

{
  "logLevel": "Information",
  "message": "Handling JobStart { OwnerId = 00000000-0000-0000-0000-000000000000, Request = JobRequest { Actor = subject:68af82b2467393e24eb9f03e, TaskName = restore, Arguments = Squidex.Infrastructure.Collections.ReadonlyDictionary\u00602[System.String,System.String], AppId =  } } for ChannelName { Name = backup.restore, Type = Queue } with System.Func\u00603[System.Object,System.Threading.CancellationToken,System.Threading.Tasks.Task].",
  "message": "JobStart { OwnerId = 00000000-0000-0000-0000-000000000000, Request = JobRequest { Actor = subject:68af82b2467393e24eb9f03e, TaskName = restore, Arguments = Squidex.Infrastructure.Collections.ReadonlyDictionary\u00602[System.String,System.String], AppId =  } }",
  "channel": "ChannelName { Name = backup.restore, Type = Queue }",
  "handler": "System.Func\u00603[System.Object,System.Threading.CancellationToken,System.Threading.Tasks.Task]",
  "timestamp": "2025-08-27T22:50:46Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "category": "Squidex.Messaging.Implementation.DelegatingConsumer"
}

{
  "logLevel": "Information",
  "message": "Starting new backup with backup id \u00272b80945c-1bb9-4280-be7b-cbc9893eb917\u0027 for owner 00000000-0000-0000-0000-000000000000.",
  "backupId": "2b80945c-1bb9-4280-be7b-cbc9893eb917",
  "ownerId": "00000000-0000-0000-0000-000000000000",
  "timestamp": "2025-08-27T22:50:46Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "category": "Squidex.Domain.Apps.Entities.Jobs.JobProcessor"
}

{
  "logLevel": "Information",
  "message": "Backup with job id 2b80945c-1bb9-4280-be7b-cbc9893eb917 with from URL \u0027https://example.com/api/apps/jobs/87244ecc-df95-46a2-92ae-1ff188c118e9?appId=5ed5d059-56f9-445f-9590-f0142886c9f0\u0027 started.",
  "backupId": "2b80945c-1bb9-4280-be7b-cbc9893eb917",
  "url": "https://example.com/api/apps/jobs/87244ecc-df95-46a2-92ae-1ff188c118e9?appId=5ed5d059-56f9-445f-9590-f0142886c9f0",
  "timestamp": "2025-08-27T22:50:46Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "category": "Squidex.Domain.Apps.Entities.Backup.RestoreJob"
}

{
  "logLevel": "Information",
  "message": "Start processing HTTP request GET https://example.com/api/apps/jobs/87244ecc-df95-46a2-92ae-1ff188c118e9?appId=5ed5d059-56f9-445f-9590-f0142886c9f0",
  "eventId": {
    "id": 100,
    "name": "RequestPipelineStart"
  },
  "httpMethod": "GET",
  "uri": "https://example.com/api/apps/jobs/87244ecc-df95-46a2-92ae-1ff188c118e9?appId=5ed5d059-56f9-445f-9590-f0142886c9f0",
  "timestamp": "2025-08-27T22:50:46Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "category": "System.Net.Http.HttpClient.Backup.LogicalHandler"
}

{
  "logLevel": "Information",
  "message": "Sending HTTP request GET https://example.com/api/apps/jobs/87244ecc-df95-46a2-92ae-1ff188c118e9?appId=5ed5d059-56f9-445f-9590-f0142886c9f0",
  "eventId": {
    "id": 100,
    "name": "RequestStart"
  },
  "httpMethod": "GET",
  "uri": "https://example.com/api/apps/jobs/87244ecc-df95-46a2-92ae-1ff188c118e9?appId=5ed5d059-56f9-445f-9590-f0142886c9f0",
  "timestamp": "2025-08-27T22:50:46Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "category": "System.Net.Http.HttpClient.Backup.ClientHandler"
}

{
  "logLevel": "Information",
  "message": "Received HTTP response headers after 330.6351ms - 200",
  "eventId": {
    "id": 101,
    "name": "RequestEnd"
  },
  "elapsedMilliseconds": "330.6351",
  "statusCode": "200",
  "timestamp": "2025-08-27T22:50:46Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "category": "System.Net.Http.HttpClient.Backup.ClientHandler"
}

{
  "logLevel": "Information",
  "message": "End processing HTTP request after 333.9667ms - 200",
  "eventId": {
    "id": 101,
    "name": "RequestPipelineEnd"
  },
  "elapsedMilliseconds": "333.9667",
  "statusCode": "200",
  "timestamp": "2025-08-27T22:50:46Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "category": "System.Net.Http.HttpClient.Backup.LogicalHandler"
}

{
  "logLevel": "Information",
  "message": "HTTP request executed.",
  "elapsedRequestMs": 3,
  "filters": {
    "userId": "68af82b2467393e24eb9f03e",
    "clientId": "squidex-frontend",
    "costs": 0
  },
  "timestamp": "2025-08-27T22:50:47Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "web": {
    "requestId": "00-22f86737b1bf0c4eded32acc015b22fe-b8127ecf1ee49383-01",
    "requestPath": "/api/apps/restore",
    "requestMethod": "GET"
  }
}

{
  "logLevel": "Information",
  "message": "HTTP request executed.",
  "elapsedRequestMs": 1,
  "filters": {
    "userId": "68af82b2467393e24eb9f03e",
    "clientId": "squidex-frontend",
    "costs": 0
  },
  "timestamp": "2025-08-27T22:50:49Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "web": {
    "requestId": "00-277d164e27e4169a25c33b1018739f77-fbe5497bec910515-01",
    "requestPath": "/api/apps/restore",
    "requestMethod": "GET"
  }
}

{
  "logLevel": "Information",
  "message": "HTTP request executed.",
  "elapsedRequestMs": 1,
  "filters": {
    "userId": "68af82b2467393e24eb9f03e",
    "clientId": "squidex-frontend",
    "costs": 0
  },
  "timestamp": "2025-08-27T22:50:51Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "web": {
    "requestId": "00-fdc940a3c61c78c2fd5da5c02b9a5b95-3cba539467963c54-01",
    "requestPath": "/api/apps/restore",
    "requestMethod": "GET"
  }
}

{
  "logLevel": "Information",
  "message": "HTTP request executed.",
  "elapsedRequestMs": 2,
  "filters": {
    "userId": "68af82b2467393e24eb9f03e",
    "clientId": "squidex-frontend",
    "costs": 0
  },
  "timestamp": "2025-08-27T22:50:53Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "web": {
    "requestId": "00-e1aa6507d6d3c0b6dd92b04da736b476-f659702ab7986480-01",
    "requestPath": "/api/apps/restore",
    "requestMethod": "GET"
  }
}

{
  "logLevel": "Information",
  "message": "HTTP request executed.",
  "elapsedRequestMs": 2,
  "filters": {
    "userId": "68af82b2467393e24eb9f03e",
    "clientId": "squidex-frontend",
    "costs": 0
  },
  "timestamp": "2025-08-27T22:50:55Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "web": {
    "requestId": "00-fc321ef9442f3eec4f81de061a5e946b-92c0df33b5f65ec2-01",
    "requestPath": "/api/apps/restore",
    "requestMethod": "GET"
  }
}

{
  "logLevel": "Error",
  "message": "Backup with job id 2b80945c-1bb9-4280-be7b-cbc9893eb917 from URL \u0027https://example.com/api/apps/jobs/87244ecc-df95-46a2-92ae-1ff188c118e9?appId=5ed5d059-56f9-445f-9590-f0142886c9f0\u0027 failed.",
  "backupId": "2b80945c-1bb9-4280-be7b-cbc9893eb917",
  "url": "https://example.com/api/apps/jobs/87244ecc-df95-46a2-92ae-1ff188c118e9?appId=5ed5d059-56f9-445f-9590-f0142886c9f0",
  "timestamp": "2025-08-27T22:50:55Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "category": "Squidex.Domain.Apps.Entities.Backup.RestoreJob",
  "exception": {
    "type": "System.NullReferenceException",
    "message": "Object reference not set to an instance of an object.",
    "stackTrace": "   at Squidex.Domain.Apps.Entities.Backup.RestoreJob.HandleEventAsync(JobRunContext run, State state, String stream, Envelope\u00601 event, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/Backup/RestoreJob.cs:line 336\n   at Squidex.Domain.Apps.Entities.Backup.RestoreJob.HandleEventsAsync(JobRunContext run, State state, CancellationToken ct)\u002BMoveNext() in /src/src/Squidex.Domain.Apps.Entities/Backup/RestoreJob.cs:line 300\n   at Squidex.Domain.Apps.Entities.Backup.RestoreJob.HandleEventsAsync(JobRunContext run, State state, CancellationToken ct)\u002BMoveNext() in /src/src/Squidex.Domain.Apps.Entities/Backup/RestoreJob.cs:line 298\n   at Squidex.Domain.Apps.Entities.Backup.RestoreJob.HandleEventsAsync(JobRunContext run, State state, CancellationToken ct)\u002BSystem.Threading.Tasks.Sources.IValueTaskSource\u003CSystem.Boolean\u003E.GetResult()\n   at Squidex.Infrastructure.CollectionExtensions.Batch[T](IAsyncEnumerable\u00601 source, Int32 size, CancellationToken ct)\u002BMoveNext() in /src/src/Squidex.Infrastructure/CollectionExtensions.cs:line 136\n   at Squidex.Infrastructure.CollectionExtensions.Batch[T](IAsyncEnumerable\u00601 source, Int32 size, CancellationToken ct)\u002BMoveNext() in /src/src/Squidex.Infrastructure/CollectionExtensions.cs:line 136\n   at Squidex.Infrastructure.CollectionExtensions.Batch[T](IAsyncEnumerable\u00601 source, Int32 size, CancellationToken ct)\u002BSystem.Threading.Tasks.Sources.IValueTaskSource\u003CSystem.Boolean\u003E.GetResult()\n   at Squidex.Infrastructure.Tasks.TaskExtensions.\u003C\u003Ec__DisplayClass3_0\u00601.\u003C\u003CBuffered\u003Eb__0\u003Ed.MoveNext() in /src/src/Squidex.Infrastructure/Tasks/TaskExtensions.cs:line 74\n--- End of stack trace from previous location ---\n   at Squidex.Infrastructure.Tasks.TaskExtensions.\u003C\u003Ec__DisplayClass3_0\u00601.\u003C\u003CBuffered\u003Eb__0\u003Ed.MoveNext() in /src/src/Squidex.Infrastructure/Tasks/TaskExtensions.cs:line 74\n--- End of stack trace from previous location ---\n   at Squidex.Infrastructure.Tasks.TaskExtensions.Buffered[T](IAsyncEnumerable\u00601 source, Int32 capacity, CancellationToken ct)\u002BMoveNext() in /src/src/Squidex.Infrastructure/Tasks/TaskExtensions.cs:line 102\n   at Squidex.Infrastructure.Tasks.TaskExtensions.Buffered[T](IAsyncEnumerable\u00601 source, Int32 capacity, CancellationToken ct)\u002BMoveNext() in /src/src/Squidex.Infrastructure/Tasks/TaskExtensions.cs:line 111\n   at Squidex.Infrastructure.Tasks.TaskExtensions.Buffered[T](IAsyncEnumerable\u00601 source, Int32 capacity, CancellationToken ct)\u002BSystem.Threading.Tasks.Sources.IValueTaskSource\u003CSystem.Boolean\u003E.GetResult()\n   at System.Threading.Tasks.Parallel.\u003C\u003Ec__57\u00601.\u003C\u003CForEachAsync\u003Eb__57_0\u003Ed.MoveNext()\n--- End of stack trace from previous location ---\n   at Squidex.Domain.Apps.Entities.Backup.RestoreJob.ReadEventsAsync(JobRunContext run, State state, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/Backup/RestoreJob.cs:line 256\n   at Squidex.Domain.Apps.Entities.Backup.RestoreJob.RunAsync(JobRunContext context, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/Backup/RestoreJob.cs:line 118"
  }
}

{
  "logLevel": "Error",
  "message": "Failed to run job with ID 2b80945c-1bb9-4280-be7b-cbc9893eb917.",
  "jobId": "2b80945c-1bb9-4280-be7b-cbc9893eb917",
  "timestamp": "2025-08-27T22:50:55Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "category": "Squidex.Domain.Apps.Entities.Jobs.JobProcessor",
  "exception": {
    "type": "System.NullReferenceException",
    "message": "Object reference not set to an instance of an object.",
    "stackTrace": "   at Squidex.Domain.Apps.Entities.Backup.RestoreJob.HandleEventAsync(JobRunContext run, State state, String stream, Envelope\u00601 event, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/Backup/RestoreJob.cs:line 336\n   at Squidex.Domain.Apps.Entities.Backup.RestoreJob.HandleEventsAsync(JobRunContext run, State state, CancellationToken ct)\u002BMoveNext() in /src/src/Squidex.Domain.Apps.Entities/Backup/RestoreJob.cs:line 300\n   at Squidex.Domain.Apps.Entities.Backup.RestoreJob.HandleEventsAsync(JobRunContext run, State state, CancellationToken ct)\u002BMoveNext() in /src/src/Squidex.Domain.Apps.Entities/Backup/RestoreJob.cs:line 298\n   at Squidex.Domain.Apps.Entities.Backup.RestoreJob.HandleEventsAsync(JobRunContext run, State state, CancellationToken ct)\u002BSystem.Threading.Tasks.Sources.IValueTaskSource\u003CSystem.Boolean\u003E.GetResult()\n   at Squidex.Infrastructure.CollectionExtensions.Batch[T](IAsyncEnumerable\u00601 source, Int32 size, CancellationToken ct)\u002BMoveNext() in /src/src/Squidex.Infrastructure/CollectionExtensions.cs:line 136\n   at Squidex.Infrastructure.CollectionExtensions.Batch[T](IAsyncEnumerable\u00601 source, Int32 size, CancellationToken ct)\u002BMoveNext() in /src/src/Squidex.Infrastructure/CollectionExtensions.cs:line 136\n   at Squidex.Infrastructure.CollectionExtensions.Batch[T](IAsyncEnumerable\u00601 source, Int32 size, CancellationToken ct)\u002BSystem.Threading.Tasks.Sources.IValueTaskSource\u003CSystem.Boolean\u003E.GetResult()\n   at Squidex.Infrastructure.Tasks.TaskExtensions.\u003C\u003Ec__DisplayClass3_0\u00601.\u003C\u003CBuffered\u003Eb__0\u003Ed.MoveNext() in /src/src/Squidex.Infrastructure/Tasks/TaskExtensions.cs:line 74\n--- End of stack trace from previous location ---\n   at Squidex.Infrastructure.Tasks.TaskExtensions.\u003C\u003Ec__DisplayClass3_0\u00601.\u003C\u003CBuffered\u003Eb__0\u003Ed.MoveNext() in /src/src/Squidex.Infrastructure/Tasks/TaskExtensions.cs:line 74\n--- End of stack trace from previous location ---\n   at Squidex.Infrastructure.Tasks.TaskExtensions.Buffered[T](IAsyncEnumerable\u00601 source, Int32 capacity, CancellationToken ct)\u002BMoveNext() in /src/src/Squidex.Infrastructure/Tasks/TaskExtensions.cs:line 102\n   at Squidex.Infrastructure.Tasks.TaskExtensions.Buffered[T](IAsyncEnumerable\u00601 source, Int32 capacity, CancellationToken ct)\u002BMoveNext() in /src/src/Squidex.Infrastructure/Tasks/TaskExtensions.cs:line 111\n   at Squidex.Infrastructure.Tasks.TaskExtensions.Buffered[T](IAsyncEnumerable\u00601 source, Int32 capacity, CancellationToken ct)\u002BSystem.Threading.Tasks.Sources.IValueTaskSource\u003CSystem.Boolean\u003E.GetResult()\n   at System.Threading.Tasks.Parallel.\u003C\u003Ec__57\u00601.\u003C\u003CForEachAsync\u003Eb__57_0\u003Ed.MoveNext()\n--- End of stack trace from previous location ---\n   at Squidex.Domain.Apps.Entities.Backup.RestoreJob.ReadEventsAsync(JobRunContext run, State state, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/Backup/RestoreJob.cs:line 256\n   at Squidex.Domain.Apps.Entities.Backup.RestoreJob.RunAsync(JobRunContext context, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/Backup/RestoreJob.cs:line 118\n   at Squidex.Domain.Apps.Entities.Backup.RestoreJob.RunAsync(JobRunContext context, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/Backup/RestoreJob.cs:line 172\n   at Squidex.Domain.Apps.Entities.Jobs.JobProcessor.ProcessAsync(JobRunContext context, IJobRunner runner, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/Jobs/JobProcessor.cs:line 213"
  }
}

{
  "logLevel": "Information",
  "message": "Handled JobStart { OwnerId = 00000000-0000-0000-0000-000000000000, Request = JobRequest { Actor = subject:68af82b2467393e24eb9f03e, TaskName = restore, Arguments = Squidex.Infrastructure.Collections.ReadonlyDictionary\u00602[System.String,System.String], AppId =  } } for ChannelName { Name = backup.restore, Type = Queue } with System.Func\u00603[System.Object,System.Threading.CancellationToken,System.Threading.Tasks.Task].",
  "message": "JobStart { OwnerId = 00000000-0000-0000-0000-000000000000, Request = JobRequest { Actor = subject:68af82b2467393e24eb9f03e, TaskName = restore, Arguments = Squidex.Infrastructure.Collections.ReadonlyDictionary\u00602[System.String,System.String], AppId =  } }",
  "channel": "ChannelName { Name = backup.restore, Type = Queue }",
  "handler": "System.Func\u00603[System.Object,System.Threading.CancellationToken,System.Threading.Tasks.Task]",
  "timestamp": "2025-08-27T22:50:55Z",
  "app": {
    "name": "Squidex",
    "version": "7.21.0.0",
    "sessionId": "8e6049ef-3b64-4ac1-99ed-d041fd2b3968"
  },
  "category": "Squidex.Messaging.Implementation.DelegatingConsumer"
}

I have…

  • Read the following guideline: Troubleshooting and Support | Squidex. I understand that my support request might get deleted if I do not follow the guideline.
  • Used code blocks with ``` to format my code examples like JSON or logs properly.

I’m submitting a…

  • Regression (a behavior that stopped working in a new release)
  • Bug report
  • Performance issue
  • Documentation issue or request

Current behavior

Restoring a backup fails with a System.NullReferenceException that does not state the empty variable

Expected behavior

Restore is working or/and the error tells what variable is missing.

Minimal reproduction of the problem

Environment

App Name: abcd

  • Self hosted with docker
  • Self hosted with IIS
  • Self hosted with other version
  • Cloud version

Version: 7.21.0

Browser:

  • Chrome (desktop)
  • Chrome (Android)
  • Chrome (iOS)
  • Firefox
  • Safari (desktop)
  • Safari (iOS)
  • IE
  • Edge

Others: