[INACTIVE] Issues getting squidex back up and running

Bugs Stale or unverified Bugs
1

At some point our Squidex instance stopped working, it started out with timeout to mongodb, and now we get the following exceptions:

2019-12-12T11:31:58.281871446Z “logLevel”: “Fatal”,
2019-12-12T11:31:58.281876046Z “action”: “Migration”,
2019-12-12T11:31:58.281879646Z “status”: “Failed”,
2019-12-12T11:31:58.281883146Z “migrator”: “Migrate_01.Migrations.StopEventConsumers”,
2019-12-12T11:31:58.281886646Z “exception”: {
2019-12-12T11:31:58.281897446Z “type”: “System.TimeoutException”,
2019-12-12T11:31:58.283005453Z “message”: "Response did not arrive on time in 00:00:30 for message: NewPlacement Request S127.0.0.1:11111:313846255cli/d39ba38e@d29e3f9c-\u003ES127.0.0.1:11111:313846255grn/89DA8F81/00000000\u002BDefault@4839ac9e #13: . Target History is: \u003CS127.0.0.1:11111:313846255:grn/89DA8F81/00000000\u002BDefault:@4839ac9e\u003E.",
2019-12-12T11:31:58.283018853Z “stackTrace”: " at Squidex.Infrastructure.Migrations.Migrator.MigrateAsync(CancellationToken ct) in /src/src/Squidex.Infrastructure/Migrations/Migrator.cs:line 78"
2019-12-12T11:31:58.283023353Z },
2019-12-12T11:31:58.283026653Z “app”: {
2019-12-12T11:31:58.283029953Z “name”: “Squidex”,
2019-12-12T11:31:58.283033353Z “version”: “4.0.3.0”,
2019-12-12T11:31:58.283036653Z “sessionId”: “a1799e39-c46b-41db-bea8-23fb0c34af6e”
2019-12-12T11:31:58.283040053Z },
2019-12-12T11:31:58.283043253Z “timestamp”: “2019-12-12T11:31:58Z”
2019-12-12T11:31:58.283046753Z }
2019-12-12T11:31:58.283049953Z
2019-12-12T11:31:58.287768282Z {
2019-12-12T11:31:58.287780982Z “logLevel”: “Error”,
2019-12-12T11:31:58.287784982Z “action”: “GrainInvoked”,
2019-12-12T11:31:58.287788082Z “status”: “Failed”,
2019-12-12T11:31:58.287791382Z “grain”: “Squidex.Infrastructure.EventSourcing.Grains.EventConsumerManagerGrain”,
2019-12-12T11:31:58.287794782Z “grainMethod”: “System.Threading.Tasks.Task StopAllAsync()”,
2019-12-12T11:31:58.287798182Z “exception”: {
2019-12-12T11:31:58.289873795Z “type”: “System.TimeoutException”,
2019-12-12T11:31:58.289888895Z “message”: "Response did not arrive on time in 00:00:30 for message: NewPlacement Request S127.0.0.1:11111:313846255grn/89DA8F81/00000000\u002BDefault@4839ac9e-\u003ES127.0.0.1:11111:313846255grn/647D2954/00000000\u002BTextIndexer@285a358f #22: . Target History is: \u003CS127.0.0.1:11111:313846255:grn/647D2954/00000000\u002BTextIndexer:@285a358f\u003E.",
2019-12-12T11:31:58.289904495Z “stackTrace”: " at Orleans.Internal.OrleansTaskExtentions.\u003CToTypedTask\u003Eg__ConvertAsync|4_0[T](Task\u00601 asyncTask)\n at Squidex.Infrastructure.EventSourcing.Grains.OrleansCodeGenEventConsumerManagerGrainMethodInvoker.Invoke(IAddressable grain, InvokeMethodRequest request) in /src/src/Squidex.Infrastructure/obj/Release/netcoreapp3.0/Squidex.Infrastructure.orleans.g.cs:line 299\n at Orleans.Runtime.GrainMethodInvoker.Invoke()\n at Squidex.Infrastructure.Orleans.StateFilter.Invoke(IIncomingGrainCallContext context) in /src/src/Squidex.Infrastructure/Orleans/StateFilter.cs:line 27\n at Orleans.Runtime.GrainMethodInvoker.Invoke()\n at Squidex.Infrastructure.Orleans.LoggingFilter.Invoke(IIncomingGrainCallContext context) in /src/src/Squidex.Infrastructure/Orleans/LoggingFilter.cs:line 46"
2019-12-12T11:31:58.289911695Z },
2019-12-12T11:31:58.289930695Z “app”: {
2019-12-12T11:31:58.289935095Z “name”: “Squidex”,
2019-12-12T11:31:58.289939295Z “version”: “4.0.3.0”,
2019-12-12T11:31:58.289943395Z “sessionId”: “a1799e39-c46b-41db-bea8-23fb0c34af6e”
2019-12-12T11:31:58.289947595Z },
2019-12-12T11:31:58.289951595Z “timestamp”: “2019-12-12T11:31:58Z”
2019-12-12T11:31:58.289955695Z }
2019-12-12T11:31:58.289959595Z
2019-12-12T11:31:58.294305422Z {
2019-12-12T11:31:58.294319922Z “logLevel”: “Warning”,
2019-12-12T11:31:58.294325022Z “message”: "Dropped expired message during Respond phase. Message: NewPlacement Request S127.0.0.1:11111:313846255cli/d39ba38e@d29e3f9c-\u003ES127.0.0.1:11111:313846255grn/89DA8F81/00000000\u002BDefault@4839ac9e #13: ",
2019-12-12T11:31:58.295255228Z “eventId”: {
2019-12-12T11:31:58.295284628Z “id”: 101038
2019-12-12T11:31:58.295288828Z },
2019-12-12T11:31:58.295292428Z “phase”: “Respond”,
2019-12-12T11:31:58.295296228Z “message”: "NewPlacement Request S127.0.0.1:11111:313846255cli/d39ba38e@d29e3f9c-\u003ES127.0.0.1:11111:313846255grn/89DA8F81/00000000\u002BDefault@4839ac9e #13: ",
2019-12-12T11:31:58.295300528Z “app”: {
2019-12-12T11:31:58.295304128Z “name”: “Squidex”,
2019-12-12T11:31:58.295315828Z “version”: “4.0.3.0”,
2019-12-12T11:31:58.295319628Z “sessionId”: “a1799e39-c46b-41db-bea8-23fb0c34af6e”
2019-12-12T11:31:58.295323528Z },
2019-12-12T11:31:58.295327028Z “timestamp”: “2019-12-12T11:31:58Z”,
2019-12-12T11:31:58.295332028Z “category”: “Orleans.Runtime.InsideRuntimeClient”
2019-12-12T11:31:58.295335928Z }
2019-12-12T11:31:58.295339428Z
2019-12-12T11:31:58.397836157Z Unhandled exception. Squidex.Infrastructure.Migrations.MigrationFailedException: Failed to run migration ‘Migrate_01.Migrations.StopEventConsumers’
2019-12-12T11:31:58.397872757Z —> System.TimeoutException: Response did not arrive on time in 00:00:30 for message: NewPlacement Request S127.0.0.1:11111:313846255cli/d39ba38e@d29e3f9c->S127.0.0.1:11111:313846255grn/89DA8F81/00000000+Default@4839ac9e #13: . Target History is: S127.0.0.1:11111:313846255:*grn/89DA8F81/00000000+Default:@4839ac9e.
2019-12-12T11:31:58.397878657Z at Squidex.Infrastructure.Migrations.Migrator.MigrateAsync(CancellationToken ct) in /src/src/Squidex.Infrastructure/Migrations/Migrator.cs:line 78
2019-12-12T11:31:58.397889957Z — End of inner exception stack trace —
2019-12-12T11:31:58.397893457Z at Squidex.Infrastructure.Migrations.Migrator.MigrateAsync(CancellationToken ct) in /src/src/Squidex.Infrastructure/Migrations/Migrator.cs:line 88
2019-12-12T11:31:58.397896857Z at Squidex.Infrastructure.Migrations.Migrator.MigrateAsync(CancellationToken ct) in /src/src/Squidex.Infrastructure/Migrations/Migrator.cs:line 97
2019-12-12T11:31:58.397899957Z at Squidex.Config.Startup.SafeHostedService.StartAsync(CancellationToken cancellationToken) in /src/src/Squidex/Config/Startup/SafeHostedService.cs:line 30
2019-12-12T11:31:58.397903157Z at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
2019-12-12T11:31:58.397906357Z at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
2019-12-12T11:31:58.397909657Z at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
2019-12-12T11:31:58.397974458Z at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
2019-12-12T11:31:58.400730174Z at Squidex.Program.Main(String[] args) in /src/src/Squidex/Program.cs:line 23

I have…

  • [x] Checked the logs and have provided uploaded a log file and provided a link because I found something suspicious there.

I’m submitting a…

  • [ ] Regression (a behavior that stopped working in a new release)
  • [x] Bug report
  • [ ] Performance issue
  • [ ] Documentation issue or request

Current behavior

Expected behavior

Minimal reproduction of the problem

Environment

  • [x] Self hosted with docker
  • [ ] Self hosted with IIS
  • [ ] Self hosted with other version
  • [ ] Cloud version

Version: [4.0.3]

Browser:

  • [ ] Chrome (desktop)
  • [ ] Chrome (Android)
  • [ ] Chrome (iOS)
  • [ ] Firefox
  • [ ] Safari (desktop)
  • [ ] Safari (iOS)
  • [ ] IE
  • [ ] Edge
2

We have done a rollback, but i have dump of the corrupted db’s if you want it for troubleshooting

3

Rollback to what? The Db would be helpful, my assumption is that you have a few problematic entries in your membership table.

4

We did a rollback to a mongodb backup we did

5
6