Backups & memory usage

Hosting (OnPremise)
1

Hi,

When a Squidex API container runs with a memory limit (e.g 1Gi), enough for handling production traffic, backups will fail with an OutOfMemory exception as the process stores json & assets in memory.
The only solution I found was to allocate 4Gi for Squidex API pods at all time. Is there a way to change this behavior and optimize ressource usage? Or could it be a feature request to stream in temp files instead of ram?

Thanks a lot!

Backups stacktrace for reference

"exception": {
    "type": "System.OutOfMemoryException",
    "message": "Exception of type 'System.OutOfMemoryException' was thrown.",
    "stackTrace": "   at System.GC.AllocateNewArray(IntPtr typeHandle, Int32 length, GC_ALLOC_FLAGS flags)
   at System.GC.<AllocateUninitializedArray>g__AllocateNewUninitializedArray|66_0[T](Int32 length, Boolean pinned)
   at System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1.Rent(Int32 minimumLength)
   at System.Text.Json.PooledByteBufferWriter.CheckAndResizeBuffer(Int32 sizeHint)
   at System.Text.Json.PooledByteBufferWriter.GetMemory(Int32 sizeHint)
   at System.Text.Json.Utf8JsonWriter.Grow(Int32 requiredSize)
   at System.Text.Json.Utf8JsonWriter.WriteStringMinimized(ReadOnlySpan`1 escapedValue)
   at System.Text.Json.Utf8JsonWriter.WriteStringEscapeValue(ReadOnlySpan`1 value, Int32 firstEscapeIndexVal)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.WriteCoreAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.JsonSerializer.WriteCore[TValue](Utf8JsonWriter writer, TValue& value, JsonTypeInfo jsonTypeInfo, WriteStack& state)
   at System.Text.Json.JsonSerializer.WriteStream[TValue](Stream utf8Json, TValue& value, JsonTypeInfo jsonTypeInfo)
   at Squidex.Infrastructure.Json.System.SystemJsonSerializer.Serialize(Object value, Type type, Stream stream, Boolean leaveOpen) in /src/src/Squidex.Infrastructure/Json/System/SystemJsonSerializer.cs:line 106
   at Squidex.Domain.Apps.Entities.Backup.BackupWriter.WriteEvent(StoredEvent storedEvent, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/Backup/BackupWriter.cs:line 104
   at Squidex.Domain.Apps.Entities.Backup.BackupProcessor.ProcessAsync(Run run, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/Backup/BackupProcessor.cs:line 201
   at Squidex.Domain.Apps.Entities.Backup.BackupProcessor.ProcessAsync(Run run, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/Backup/BackupProcessor.cs:line 201
   at Squidex.Domain.Apps.Entities.Backup.BackupProcessor.ProcessAsync(Run run, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/Backup/BackupProcessor.cs:line 201"

and

2

Hi,

the backup writer should directly write to a temporary stream. I am not sure what the memory is used for. It could just be memory pools that are created by the json serializer or something similar. MongoDB also maintains some pools.

But I have created gcdump into Squidex: https://github.com/Squidex/squidex/blob/3d3391a78305daeae5a19e19777cfb366582a2dc/backend/src/Squidex/Areas/Api/Controllers/Diagnostics/DiagnosticsController.cs#L60

So while you make a backup you should be able to trigger this endpoint and then the process will create a gcdump and store it in the asset store.

Then you could either analyze it yourself or send it to me.

3

Thanks a lot, I’ll have a look at it :slight_smile:

1 Like