[OBSOLETE] The target silo is no longer active

I’m submitting a…

[ ] Regression (a behavior that stopped working in a new release)
[ X] Bug report
[ ] Performance issue
[ ] Documentation issue or request

Current behavior

Site has been workin fine for the last couple of weeks, today when I was going to check it in Chrome web browser in Chrome the site had crashed.

Also when checking Squidex, I could log in, see the dashboard and navigation but when clicking on Schemas or Content I get a “Failed to load schemas. Please reload.”. And I cant load schemas or content. Assets/Images are working though.

Expected behavior

Schemas and Content should load.

Minimal reproduction of the problem

I havent touched the site in 24 hrs, it crashed by its own.

Environment

  • [ ] Self hosted with docker
  • [ ] Self hosted with IIS
  • [ x] Self hosted with other version (Azure AKS)
  • [ ] Cloud version

Browser:

  • [ x] Chrome (desktop)
  • [x ] Chrome (Android)
  • [ ] Chrome (iOS)
  • [ ] Firefox
  • [ ] Safari (desktop)
  • [ ] Safari (iOS)
  • [ ] IE
  • [ ] Edge

Others:

Here are the logs from the Squidex pod.
I click on “Schemas” or “Content” and this is what gets logged:

{
  "logLevel": "Error",
  "message": "Connection id \"0HLIPHEGEM9I9\", Request id \"0HLIPHEGEM9I9:00000001\": An unhandled exception was thrown by the application.",
  "eventId": {
    "id": 13,
    "name": "ApplicationError"
  },
  "connectionId": "0HLIPHEGEM9I9",
  "traceIdentifier": "0HLIPHEGEM9I9:00000001",
  "exception": {
    "type": "Orleans.Runtime.OrleansMessageRejectionException",
    "message": "The target silo is no longer active: target was S10.244.0.201:11111:280351504, but this silo is S10.244.0.201:11111:281584002. The rejected message is Request S10.244.0.201:11111:281584002*cli/fbb2c2a2@da00fe9e->S10.244.0.201:11111:280351504*grn/528B4978/0282879d@da13e991 #255503: .",
    "stackTrace": "   at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)\n   at Squidex.Domain.Apps.Entities.AppProvider.GetSchemaIdAsync(Guid appId, String name) in /src/Squidex.Domain.Apps.Entities/AppProvider.cs:line 215\n   at Squidex.Domain.Apps.Entities.AppProvider.<>c__DisplayClass5_0.<<GetSchemaAsync>b__0>d.MoveNext() in /src/Squidex.Domain.Apps.Entities/AppProvider.cs:line 99\n--- End of stack trace from previous location where exception was thrown ---\n   at Squidex.Infrastructure.Caching.RequestCacheExtensions.GetOrCreateAsync[T](ILocalCache cache, Object key, Func`1 task)\n   at Squidex.Domain.Apps.Entities.Contents.ContentQueryService.GetSchemaAsync(QueryContext context, String schemaIdOrName) in /src/Squidex.Domain.Apps.Entities/Contents/ContentQueryService.cs:line 267\n   at Squidex.Domain.Apps.Entities.Contents.ContentQueryService.QueryAsync(QueryContext context, String schemaIdOrName, Q query) in /src/Squidex.Domain.Apps.Entities/Contents/ContentQueryService.cs:line 111\n   at Squidex.Areas.Api.Controllers.Contents.ContentsController.GetContents(String app, String name, Boolean archived, String ids) in /src/Squidex/Areas/Api/Controllers/Contents/ContentsController.cs:line 130\n   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextExceptionFilterAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ExceptionContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()\n   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)\n   at NSwag.AspNetCore.Middlewares.AspNetCoreToSwaggerMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\n   at Squidex.Pipeline.EnforceHttpsMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/EnforceHttpsMiddleware.cs:line 43\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)\n   at Squidex.Pipeline.LocalCacheMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/LocalCacheMiddleware.cs:line 30\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Squidex.Pipeline.RequestLogPerformanceMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/Squidex/Pipeline/RequestLogPerformanceMiddleware.cs:line 32\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_1.<<UseMiddlewareInterface>b__1>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)"
  },
  "app": {
    "name": "Squidex",
    "version": "1.0.0.0",
    "sessionId": "977d8cd5-5b62-4bee-a5b0-2dd2f5a84516"
  },
  "web": {
    "requestId": "5751725a-0780-4ebc-8178-3545dd5a8401",
    "requestPath": "/api/content/my-app/my-schema/",
    "requestMethod": "GET"
  },
  "timestamp": "2018-12-04T15:29:03.7585838Z",
  "category": "Microsoft.AspNetCore.Server.Kestrel"
}

FYI: Killed the two Squidex pods and now the site is working again.

Are you sure that no node has been restarted by k8?

Closed due to inactivity.