[SOLVED] Upload Amazon S3 Bucket failed

Andreas · January 16, 2020, 7:41am

I’m submitting a…

[ ] Regression (a behavior that stopped working in a new release)
[X] Bug report
[ ] Performance issue
[ ] Documentation issue or request

Environment

[ ] Self hosted with docker
[X] Self hosted with IIS
[ ] Self hosted with other version
[ ] Cloud version

Version: Last Master

I wanted to move the storage to Amazon S3 because the data is getting too big for the server.

The settings work and most files are loaded into the bucket without any problems.

However, I have some ZIP files and larger files that generate an error.

Error message:

“type”: “System.InvalidOperationException”,

"message": "Base stream of PartialWrapperStream must be seekable",

"stackTrace": "   at Amazon.Runtime.Internal.Util.PartialWrapperStream..ctor(Stream stream, Int64 partSize)\r\n   at Amazon.S3.Model.Internal.MarshallTransformations.UploadPartRequestMarshaller.Marshall(UploadPartRequest uploadPartRequest)\r\n   at Amazon.S3.Model.Internal.MarshallTransformations.UploadPartRequestMarshaller.Marshall(AmazonWebServiceRequest input)\r\n   at Amazon.Runtime.Internal.Marshaller.PreInvoke(IExecutionContext executionContext)\r\n   at Amazon.Runtime.Internal.Marshaller.InvokeAsync[T](IExecutionContext executionContext)\r\n   at Amazon.Runtime.Internal.PipelineHandler.InvokeAsync[T](IExecutionContext executionContext)\r\n   at Amazon.S3.Internal.AmazonS3PreMarshallHandler.InvokeAsync[T](IExecutionContext executionContext)\r\n   at Amazon.Runtime.Internal.PipelineHandler.InvokeAsync[T](IExecutionContext executionContext)\r\n   at Amazon.Runtime.Internal.CallbackHandler.\u003C\u003En__0[T](IExecutionContext executionContext)\r\n   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)\r\n   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)\r\n   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)\r\n   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)\r\n   at Amazon.S3.Transfer.Internal.MultipartUploadCommand.UploadPartAsync(UploadPartRequest uploadRequest, CancellationTokenSource internalCts, SemaphoreSlim asyncThrottler)\r\n   at Amazon.S3.Transfer.Internal.BaseCommand.WhenAllOrFirstExceptionAsync[T](List\u00601 pendingTasks, CancellationToken cancellationToken)\r\n   at Amazon.S3.Transfer.Internal.MultipartUploadCommand.ExecuteAsync(CancellationToken cancellationToken)\r\n   at Squidex.Infrastructure.Assets.AmazonS3AssetStore.UploadAsync(String fileName, Stream stream, Boolean overwrite, CancellationToken ct) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Infrastructure.Amazon\\Assets\\AmazonS3AssetStore.cs:line 151\r\n   at Squidex.Domain.Apps.Entities.Assets.AssetCommandMiddleware.EnrichWithHashAndUploadAsync(UploadAssetCommand command, String tempFile) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Domain.Apps.Entities\\Assets\\AssetCommandMiddleware.cs:line 147\r\n   at Squidex.Domain.Apps.Entities.Assets.AssetCommandMiddleware.HandleAsync(CommandContext context, Func\u00601 next) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Domain.Apps.Entities\\Assets\\AssetCommandMiddleware.cs:line 58\r\n   at Squidex.Domain.Apps.Entities.Apps.AppCommandMiddleware.HandleAsync(CommandContext context, Func\u00601 next) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Domain.Apps.Entities\\Apps\\AppCommandMiddleware.cs:line 55\r\n   at Squidex.Domain.Apps.Entities.Schemas.Indexes.SchemasIndex.HandleAsync(CommandContext context, Func\u00601 next) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Domain.Apps.Entities\\Schemas\\Indexes\\SchemasIndex.cs:line 128\r\n   at Squidex.Domain.Apps.Entities.Rules.Indexes.RulesIndex.HandleAsync(CommandContext context, Func\u00601 next) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Domain.Apps.Entities\\Rules\\Indexes\\RulesIndex.cs:line 77\r\n   at Squidex.Domain.Apps.Entities.Apps.Indexes.AppsIndex.HandleAsync(CommandContext context, Func\u00601 next) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Domain.Apps.Entities\\Apps\\Indexes\\AppsIndex.cs:line 200\r\n   at Squidex.Domain.Apps.Entities.Apps.Invitation.InviteUserCommandMiddleware.HandleAsync(CommandContext context, Func\u00601 next) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Domain.Apps.Entities\\Apps\\Invitation\\InviteUserCommandMiddleware.cs:line 43\r\n   at Squidex.Infrastructure.Commands.CustomCommandMiddlewareRunner.HandleAsync(CommandContext context, Func\u00601 next) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Infrastructure\\Commands\\CustomCommandMiddlewareRunner.cs:line 33\r\n   at Squidex.Web.CommandMiddlewares.EnrichWithSchemaIdCommandMiddleware.HandleAsync(CommandContext context, Func\u00601 next) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Web\\CommandMiddlewares\\EnrichWithSchemaIdCommandMiddleware.cs:line 54\r\n   at Squidex.Web.CommandMiddlewares.ETagCommandMiddleware.HandleAsync(CommandContext context, Func\u00601 next) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Web\\CommandMiddlewares\\ETagCommandMiddleware.cs:line 55\r\n   at Squidex.Infrastructure.Commands.InMemoryCommandBus.PublishAsync(ICommand command) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Infrastructure\\Commands\\InMemoryCommandBus.cs:line 42\r\n   at Squidex.Areas.Api.Controllers.Assets.AssetsController.InvokeCommandAsync(String app, ICommand command) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex\\Areas\\Api\\Controllers\\Assets\\AssetsController.cs:line 302\r\n   at Squidex.Areas.Api.Controllers.Assets.AssetsController.PostAsset(String app, Guid parentId, List\u00601 file) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex\\Areas\\Api\\Controllers\\Assets\\AssetsController.cs:line 194\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeActionMethodAsync\u003Eg__Awaited|12_0(ControllerActionInvoker invoker, ValueTask\u00601 actionResultValueTask)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeNextActionFilterAsync\u003Eg__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State\u0026 next, Scope\u0026 scope, Object\u0026 state, Boolean\u0026 isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeInnerFilterAsync\u003Eg__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeNextExceptionFilterAsync\u003Eg__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ExceptionContextSealed context)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State\u0026 next, Scope\u0026 scope, Object\u0026 state, Boolean\u0026 isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeNextResourceFilter\u003Eg__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State\u0026 next, Scope\u0026 scope, Object\u0026 state, Boolean\u0026 isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeFilterPipelineAsync\u003Eg__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeAsync\u003Eg__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\r\n   at Microsoft.AspNetCore.Routing.EndpointMiddleware.\u003CInvoke\u003Eg__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\r\n   at NSwag.AspNetCore.Middlewares.OpenApiDocumentMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\r\n   at Squidex.Web.Pipeline.EnforceHttpsMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Web\\Pipeline\\EnforceHttpsMiddleware.cs:line 28\r\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.\u003C\u003Ec__DisplayClass5_1.\u003C\u003CUseMiddlewareInterface\u003Eb__1\u003Ed.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Squidex.Web.Pipeline.LocalCacheMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Web\\Pipeline\\LocalCacheMiddleware.cs:line 30\r\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.\u003C\u003Ec__DisplayClass5_1.\u003C\u003CUseMiddlewareInterface\u003Eb__1\u003Ed.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Squidex.Web.Pipeline.RequestLogPerformanceMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\\Offline Daten\\Source\\squidex\\backend\\src\\Squidex.Web\\Pipeline\\RequestLogPerformanceMiddleware.cs:line 37\r\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.\u003C\u003Ec__DisplayClass5_1.\u003C\u003CUseMiddlewareInterface\u003Eb__1\u003Ed.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT\u00601.ProcessRequestAsync()"

Sebastian · January 16, 2020, 8:27am

Oh, that sucks. I use the upload stream to calculate the file hash while I upload it. seems I have to change that.

Thank you very much.

Andreas · January 16, 2020, 8:33am

OK
for me it’s not that bad. So far I have only tested Amazon locally.

then switch if it works

Sebastian · January 16, 2020, 8:33am

It is bad because it means I have to read the stream twice, which hurts performance. Lets see if there is another solution.

Sebastian · January 16, 2020, 1:34pm

What means bigger? I have an indicator, it is a strange behavior in Amazon S3 SDK, that can be corrected by a fake seek stream (I hope)

Andreas · January 16, 2020, 1:46pm

is a bit strange.

Sometimes there are ZIP files that are 18mb that have this error message. other files that were so big, the message came too big.

An example of the error was the backup of an Squidex app.

Sebastian · January 16, 2020, 2:09pm

Hi, I fixed it with my fake seekable stream

Andreas · January 16, 2020, 2:29pm

now the upload works perfectly.

Thank you very much

Sebastian · January 16, 2020, 4:54pm

Sebastian · October 24, 2020, 8:08pm