Squidex CLI sync out fails with error

I have…

I’m submitting a…

  • [ ] Regression (a behavior that stopped working in a new release)
  • [x] Bug report
  • [ ] Performance issue
  • [ ] Documentation issue or request

Current behavior

I’m running the Squidex CLI to sync out an old (self-hosted) installation of Squidex. I want to migrate to using the cloud instead.

I get an error when it’s exporting one of the schemas:

Expected behavior

Squidex CLI should be able to sync out the DB.

Minimal reproduction of the problem

Environment

App Name:

  • [ ] Self hosted with docker
  • [x] Self hosted with IIS
  • [ ] Self hosted with other version
  • [ ] Cloud version

Version: 1.16.2

Browser:

  • [ ] Chrome (desktop)
  • [ ] Chrome (Android)
  • [ ] Chrome (iOS)
  • [ ] Firefox
  • [ ] Safari (desktop)
  • [ ] Safari (iOS)
  • [ ] IE
  • [ ] Edge

Others:

Log can be downloaded here: https://drive.google.com/file/d/1G4KFgVQf6fxtOWOFtlYXxHu82cDu6j1B/view?usp=sharing

Hi,

I think the related error is this one:


  "logLevel": "Error",
  "message": "Connection ID \"7421932191812169915\", Request ID \"80003905-0001-6700-b63f-84710c7967bb\": An unhandled exception was thrown by the application.",
  "eventId": {
    "id": 2,
    "name": "ApplicationError"
  },
  "connectionId": "7421932191812169915",
  "traceIdentifier": "80003905-0001-6700-b63f-84710c7967bb",
  "exception": {
    "type": "MongoDB.Driver.MongoCommandException",
    "message": "Command find failed: error while multiplanner was selecting best plan :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in..",
    "stackTrace": "   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.ProcessResponse(ConnectionId connectionId, CommandMessage responseMessage)\r\n   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocolAsync[TResult](IWireProtocol`1 protocol, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocolAsync(IChannelSource channelSource, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.Core.Operations.FindCommandOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.Core.Operations.FindOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)\r\n   at MongoDB.Driver.IAsyncCursorSourceExtensions.ToListAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)\r\n   at Squidex.Domain.Apps.Entities.MongoDb.Contents.MongoContentCollection.QueryAsync(IAppEntity app, ISchemaEntity schema, Query query, List`1 ids, Status[] status, Boolean useDraft) in E:\\7peaks\\avinor-community-backend\\src\\Squidex.Domain.Apps.Entities.MongoDb\\Contents\\MongoContentCollection.cs:line 88\r\n   at Squidex.Domain.Apps.Entities.MongoDb.Contents.MongoContentRepository.QueryAsync(IAppEntity app, ISchemaEntity schema, Status[] status, Query query) in E:\\7peaks\\avinor-community-backend\\src\\Squidex.Domain.Apps.Entities.MongoDb\\Contents\\MongoContentRepository.cs:line 81\r\n   at Squidex.Domain.Apps.Entities.Contents.ContentQueryService.QueryAsync(QueryContext context, String schemaIdOrName, Q query) in E:\\7peaks\\avinor-community-backend\\src\\Squidex.Domain.Apps.Entities\\Contents\\ContentQueryService.cs:line 130\r\n   at Squidex.Areas.Api.Controllers.Contents.ContentsController.GetContents(String app, String name, String ids, Boolean archived) in E:\\7peaks\\avinor-community-backend\\src\\Squidex\\Areas\\Api\\Controllers\\Contents\\ContentsController.cs:line 170\r\n   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\r\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()\r\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()\r\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)\r\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()\r\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextExceptionFilterAsync()\r\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ExceptionContext context)\r\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()\r\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)\r\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()\r\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()\r\n   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)\r\n   at NSwag.AspNetCore.Middlewares.SwaggerDocumentMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\r\n   at Squidex.Web.Pipeline.EnforceHttpsMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in E:\\7peaks\\avinor-community-backend\\src\\Squidex.Web\\Pipeline\\EnforceHttpsMiddleware.cs:line 42\r\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Squidex.Web.Pipeline.LocalCacheMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in E:\\7peaks\\avinor-community-backend\\src\\Squidex.Web\\Pipeline\\LocalCacheMiddleware.cs:line 30\r\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Squidex.Web.Pipeline.RequestLogPerformanceMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in E:\\7peaks\\avinor-community-backend\\src\\Squidex.Web\\Pipeline\\RequestLogPerformanceMiddleware.cs:line 33\r\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()"
  },
  "app": {
    "name": "Squidex",
    "version": "1.0.0.0",
    "sessionId": "2e248985-0c3a-40d0-bdf5-6c984c5e50ee"
  },
  "web": {
    "requestId": "c4f6833d-8ba7-47fa-81f9-4522cada5662",
    "requestPath": "/api/content/avinor-community/notificationslog/",
    "requestMethod": "GET"
  },
  "timestamp": "2023-06-28T20:42:12Z",
  "category": "Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer"
}

Are you sure that you are using Squidex 1.16.2? There is no real solution to that. You have to update your instance first or try to create a backup.

Well, I’m not 100% sure. All I know is that the endpoint /api/info doesn’t work and the modified date on the Squidex.dll is May 1st 2019. The installation has not been updated since.

We have tried to upgraded it a couple times, but without succeeding. The backup is not working.

We don’t have to have all the events. The most important thing is the actual data.

Here is what you can do:

  1. Enable the profiler for the content database: https://studio3t.com/knowledge-base/articles/mongodb-query-performance. This allows us to see the queries.
  2. Run the CLI. Now you should be able to see a db.profiler collection (or similar name). Try to find the slowest query.
  3. Send me this query. Perhaps I can recommend you an index to create.

The content schema for notificationsLog was not important so I decided to delete it all together. I’m almost through, but now it’s stopping me when exporting rules.

The log can be downloaded here: https://drive.google.com/file/d/105T1fzro3_L1SJG-9C8mgTZycpl7O7Wn/view?usp=sharing

Could you open the rules page in the Squidex UI and post the result from the browser network tab? Please remember to remove sensitive data.

GET /api/apps/avinor-community/rules returns empty array: []

Thanks. I think you are just using a super old Squidex version and the CLI is not compatible to that. I think I have implemented a few backwards compatibility things, but I am not sure.

I have pushed a new version of the CLI (build should be done soon), which enables the compatibility mode for rules.

Thank you! Sorry for being so late with my replies. The update e-mails keeps on getting into my spam folder :slight_smile:

I will test the new CLI build tomorrow!

Tried the latest CLI (v11.3.0), but with the same result.

Hi,

I can not really test it, because I don’t have the environment. If you provide me your own (readonly) client id and secret it would help a lot (please use PM!) :wink: