[NOT_SQUIDEX] Can't create rule for some reason

Bugs Solved Bugs
1

I’m submitting a…

[ ] Regression (a behavior that stopped working in a new release)
[X] Bug report
[ ] Performance issue
[ ] Documentation issue or request

Current behavior

Hangs and fails to create a webhook.

Expected behavior

Create the webhook against CONTENT.

Minimal reproduction of the problem

Not entirely sure. Log dump:

"requestPath": "/apps/engineering/rules",
  "action": "PostRule",
  "controller": "Rules"
"domainObjectType": "RuleGrain",
"requestPath": "/apps/engineering/rules",
  "action": "PostRule",
  "controller": "Rules"
"requestPath": "/api/apps/engineering/rules",
"stackTrace": "   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.ExecuteCommandAsync(TCommand typedCommand) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 41\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 29\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Domain.Apps.Entities.Apps.Indexes.AppsByNameIndexCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Domain.Apps.Entities/Apps/Indexes/AppsByNameIndexCommandMiddleware.cs:line 50\n   at Squidex.Domain.Apps.Entities.Apps.Indexes.AppsByNameIndexCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Domain.Apps.Entities/Apps/Indexes/AppsByNameIndexCommandMiddleware.cs:line 66\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Domain.Apps.Entities.Assets.AssetCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Domain.Apps.Entities/Assets/AssetCommandMiddleware.cs:line 102\n   at Squidex.Pipeline.CommandMiddlewares.EnrichWithSchemaIdCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex/Pipeline/CommandMiddlewares/EnrichWithSchemaIdCommandMiddleware.cs:line 54\n   at Squidex.Pipeline.CommandMiddlewares.ETagCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex/Pipeline/CommandMiddlewares/ETagCommandMiddleware.cs:line 47\n   at Squidex.Infrastructure.Commands.InMemoryCommandBus.PublishAsync(ICommand command) in /src/Squidex.Infrastructure/Commands/InMemoryCommandBus.cs:line 42\n   at Squidex.Areas.Api.Controllers.Rules.RulesController.PostRule(String app, CreateRuleDto request) in /src/Squidex/Areas/Api/Controllers/Rules/RulesController.cs:line 128\n   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextExceptionFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ExceptionContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()\n   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)\n   at NSwag.AspNetCore.Middlewares.AspNetCoreToSwaggerMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\n   at Squidex.Pipeline.RequestLogPerformanceMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/RequestLogPerformanceMiddleware.cs:line 32\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Squidex.Pipeline.EnforceHttpsMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/EnforceHttpsMiddleware.cs:line 29\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)\n   at Squidex.Pipeline.LocalCacheMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/LocalCacheMiddleware.cs:line 30\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)"


"requestPath": "/api/apps/engineering/rules",


"requestPath": "/api/rules/actions",


"requestPath": "/api/rules/triggers",


"AppProvider/GetRulesAsync": {


"requestPath": "/api/apps/engineering/rules",


"requestPath": "/api/apps/engineering/rules",


"stackTrace": "   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.ExecuteCommandAsync(TCommand typedCommand) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 41\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 29\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Domain.Apps.Entities.Apps.Indexes.AppsByNameIndexCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Domain.Apps.Entities/Apps/Indexes/AppsByNameIndexCommandMiddleware.cs:line 50\n   at Squidex.Domain.Apps.Entities.Apps.Indexes.AppsByNameIndexCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Domain.Apps.Entities/Apps/Indexes/AppsByNameIndexCommandMiddleware.cs:line 66\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Domain.Apps.Entities.Assets.AssetCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Domain.Apps.Entities/Assets/AssetCommandMiddleware.cs:line 102\n   at Squidex.Pipeline.CommandMiddlewares.EnrichWithSchemaIdCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex/Pipeline/CommandMiddlewares/EnrichWithSchemaIdCommandMiddleware.cs:line 54\n   at Squidex.Pipeline.CommandMiddlewares.ETagCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex/Pipeline/CommandMiddlewares/ETagCommandMiddleware.cs:line 47\n   at Squidex.Infrastructure.Commands.InMemoryCommandBus.PublishAsync(ICommand command) in /src/Squidex.Infrastructure/Commands/InMemoryCommandBus.cs:line 42\n   at Squidex.Areas.Api.Controllers.Rules.RulesController.PostRule(String app, CreateRuleDto request) in /src/Squidex/Areas/Api/Controllers/Rules/RulesController.cs:line 128\n   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextExceptionFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ExceptionContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()\n   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)\n   at NSwag.AspNetCore.Middlewares.AspNetCoreToSwaggerMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\n   at Squidex.Pipeline.RequestLogPerformanceMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/RequestLogPerformanceMiddleware.cs:line 32\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Squidex.Pipeline.EnforceHttpsMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/EnforceHttpsMiddleware.cs:line 29\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)\n   at Squidex.Pipeline.LocalCacheMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/LocalCacheMiddleware.cs:line 30\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)"


"requestPath": "/api/apps/engineering/rules",


"requestPath": "/apps/engineering/rules",


  "action": "PostRule",


  "controller": "Rules"


 "domainObjectType": "RuleGrain",


"requestPath": "/apps/engineering/rules",


  "action": "PostRule",


  "controller": "Rules"


"requestPath": "/api/apps/engineering/rules",


"stackTrace": "   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.ExecuteCommandAsync(TCommand typedCommand) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 41\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 29\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Domain.Apps.Entities.Apps.Indexes.AppsByNameIndexCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Domain.Apps.Entities/Apps/Indexes/AppsByNameIndexCommandMiddleware.cs:line 50\n   at Squidex.Domain.Apps.Entities.Apps.Indexes.AppsByNameIndexCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Domain.Apps.Entities/Apps/Indexes/AppsByNameIndexCommandMiddleware.cs:line 66\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Domain.Apps.Entities.Assets.AssetCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Domain.Apps.Entities/Assets/AssetCommandMiddleware.cs:line 102\n   at Squidex.Pipeline.CommandMiddlewares.EnrichWithSchemaIdCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex/Pipeline/CommandMiddlewares/EnrichWithSchemaIdCommandMiddleware.cs:line 54\n   at Squidex.Pipeline.CommandMiddlewares.ETagCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex/Pipeline/CommandMiddlewares/ETagCommandMiddleware.cs:line 47\n   at Squidex.Infrastructure.Commands.InMemoryCommandBus.PublishAsync(ICommand command) in /src/Squidex.Infrastructure/Commands/InMemoryCommandBus.cs:line 42\n   at Squidex.Areas.Api.Controllers.Rules.RulesController.PostRule(String app, CreateRuleDto request) in /src/Squidex/Areas/Api/Controllers/Rules/RulesController.cs:line 128\n   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextExceptionFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ExceptionContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()\n   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)\n   at NSwag.AspNetCore.Middlewares.AspNetCoreToSwaggerMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\n   at Squidex.Pipeline.RequestLogPerformanceMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/RequestLogPerformanceMiddleware.cs:line 32\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Squidex.Pipeline.EnforceHttpsMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/EnforceHttpsMiddleware.cs:line 29\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)\n   at Squidex.Pipeline.LocalCacheMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/LocalCacheMiddleware.cs:line 30\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)"


"requestPath": "/api/apps/engineering/rules",

Environment

  • [X] Self hosted with docker
  • [ ] Self hosted with IIS
  • [ ] Self hosted with other version
  • [ ] Cloud version

Browser:

  • [X] Chrome (desktop)
  • [ ] Chrome (Android)
  • [ ] Chrome (iOS)
  • [ ] Firefox
  • [ ] Safari (desktop)
  • [ ] Safari (iOS)
  • [ ] IE
  • [ ] Edge

Others:

2

I have absolutely no idea what is going wrong, sorry.

3

My guess is Orleans – constant Orleans errors and production “Orlease Membership” collection had 14 entries even though there is only 1 replica (instance) of Squidex running in the swarm. Clearing that whole collection… hopefully it starts running correctly.

4

Cleared the collection, still seeing issues:

{
  "logLevel": "Error",
  "message": "Connection id \"0HLI8M21F9ESU\", Request id \"0HLI8M21F9ESU:00000002\": An unhandled exception was thrown by the application.",
  "eventId": {
    "id": 13,
    "name": "ApplicationError"
  },
  "connectionId": "0HLI8M21F9ESU",
  "traceIdentifier": "0HLI8M21F9ESU:00000002",
  "exception": {
    "type": "System.TimeoutException",
    "message": "Response did not arrive on time in 00:00:30 for message: NewPlacement Request     S10.0.4.45:11111:279731222*cli/3fc9759e@f644759d->S10.0.4.45:11111:279731222*grn/3AEBD331/d5444688@b87f0698 #2072: . Target History is: <    S10.0.4.45:11111:279731222:*grn/3AEBD331/d5444688:@b87f0698>.",
    "stackTrace": "   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)\n   at     Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.ExecuteCommandAsync(TCommand typedCommand) in /src/Squidex.Infrastructure/Commands/    GrainCommandMiddleware.cs:line 41\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/    Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 29\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext     context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at     Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/    GrainCommandMiddleware.cs:line 34\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/    Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext     context, Func`1 next) in /src/Squidex.Infrastructure/Commands/GrainCommandMiddleware.cs:line 34\n   at     Squidex.Domain.Apps.Entities.Apps.Indexes.AppsByNameIndexCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/    Squidex.Domain.Apps.Entities/Apps/Indexes/AppsByNameIndexCommandMiddleware.cs:line 50\n   at     Squidex.Domain.Apps.Entities.Apps.Indexes.AppsByNameIndexCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/    Squidex.Domain.Apps.Entities/Apps/Indexes/AppsByNameIndexCommandMiddleware.cs:line 66\n   at     Squidex.Infrastructure.Commands.GrainCommandMiddleware`2.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex.Infrastructure/Commands/    GrainCommandMiddleware.cs:line 34\n   at Squidex.Domain.Apps.Entities.Assets.AssetCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/    Squidex.Domain.Apps.Entities/Assets/AssetCommandMiddleware.cs:line 102\n   at     Squidex.Pipeline.CommandMiddlewares.EnrichWithSchemaIdCommandMiddleware.HandleAsync(CommandContext context, Func`1 next) in /src/Squidex/Pipeline/    CommandMiddlewares/EnrichWithSchemaIdCommandMiddleware.cs:line 54\n   at Squidex.Pipeline.CommandMiddlewares.ETagCommandMiddleware.HandleAsync(CommandContext     context, Func`1 next) in /src/Squidex/Pipeline/CommandMiddlewares/ETagCommandMiddleware.cs:line 47\n   at     Squidex.Infrastructure.Commands.InMemoryCommandBus.PublishAsync(ICommand command) in /src/Squidex.Infrastructure/Commands/InMemoryCommandBus.cs:line 42\n   at     Squidex.Areas.Api.Controllers.Rules.RulesController.PostRule(String app, CreateRuleDto request) in /src/Squidex/Areas/Api/Controllers/Rules/    RulesController.cs:line 128\n   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper,     ObjectMethodExecutor executor, Object controller, Object[] arguments)\n   at     Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()\n   at     Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()\n   at     Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)\n   at     Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at     Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()\n   at     Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextExceptionFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(    ExceptionContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at     Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(    ResourceExecutedContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean&     isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()\n   at     Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()\n   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)\n   at     NSwag.AspNetCore.Middlewares.AspNetCoreToSwaggerMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(    HttpContext context)\n   at Squidex.Pipeline.RequestLogPerformanceMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/    RequestLogPerformanceMiddleware.cs:line 32\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>    b__1>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Squidex.Pipeline.EnforceHttpsMiddleware.InvokeAsync(    HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/EnforceHttpsMiddleware.cs:line 29\n   at     Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\n--- End of stack trace from previous     location where exception was thrown ---\n   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)\n   at     Squidex.Pipeline.LocalCacheMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/LocalCacheMiddleware.cs:line 30\n   at     Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\n--- End of stack trace from previous     location where exception was thrown ---\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[    TContext](IHttpApplication`1 application)"
  },

  "app": {
    "name": "Squidex",
    "version": "1.0.0.0",
    "sessionId": "c2de95e8-5f1b-4272-96e3-707d08811c41"
  },


  "web": {
    "requestId": "4493d749-cd32-43c2-8883-902df5b029eb",
    "requestPath": "/api/apps/engineering/rules",
    "requestMethod": "POST"
  },

  "timestamp": "2018-11-12T15:10:57.4459404Z",
  "category": "Microsoft.AspNetCore.Server.Kestrel"
}
5

Do I need to setup eventPublishers? I have a production RabbitMQ instance.

6

No, it should be fine.

Can you check the health at /healthz or the cluster state at /orleans

7

Update: when I restored the app backup in my dev swarm (v12) – it restored all the rules I had attempted to add in production but had failed :thinking:

8