I have…
- [X ] Checked the logs and have provided uploaded a log file and provided a link because I found something suspicious there.
I’m submitting a…
- [ ] Regression (a behavior that stopped working in a new release)
- [X ] Bug report
- [X ] Performance issue
- [ ] Documentation issue or request
Current behavior
We running Squidex in the GKE and at some stage the pod has been failed with a CrashLoopBackOff Status.
Then I deleted the pod to recreate a new one to start, and it won’t be starting anymore, it keeps get failed until CrashLoopBackOff…
-squidex-cms-59dd6bd66f-lbf9d 0/1 CrashLoopBackOff 6 9m9s
Please see the pod log for more details.
Expected behavior
After deleted the failed pod, when a new pod is created, it should be just started a new instance and running
Minimal reproduction of the problem
Using JMeter to run the load test for 30 mins then the pod won’t be response quickly and crashedLoopBackOff in the end. And after that, even new pod won’t work.
Is it to do dangling data in mongodb and where I can check that?
I noticed the Migration object has a lock:true, will it be the issue?
Environment
- [X ] Self hosted with docker
- [ ] Self hosted with IIS
- [ ] Self hosted with other version
- [ ] Cloud version
Version: [VERSION]
Browser:
- [ ] Chrome (desktop)
- [ ] Chrome (Android)
- [ ] Chrome (iOS)
- [ ] Firefox
- [ ] Safari (desktop)
- [ ] Safari (iOS)
- [ ] IE
- [ ] Edge
Others:
Logs
Now listening on: http://[::]:80
Application started. Press Ctrl+C to shut down.
{
“logLevel”: “Information”,
“filters”: {
“costs”: 0.0
},
“elapsedRequestMs”: 1027,
“app”: {
“name”: “Squidex”,
“version”: “3.3.0.0”,
“sessionId”: “0d7ed7dc-70ce-487b-9707-366d67fb44fc”
},
“web”: {
“requestId”: “112bfac8-2024-44cd-ab3f-0bcd226f6e5f”,
“requestPath”: “/build/index.html”,
“requestMethod”: “GET”
},
“timestamp”: “2019-10-02T01:56:51Z”
}
{
“logLevel”: “Warning”,
“message”: “UnregisterManyAsync 43 failed.”,
“eventId”: {
“id”: 100502
},
“exception”: {
“type”: “System.InvalidOperationException”,
“message”: “Grain directory is stopping”,
“stackTrace”: " at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)\n at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterOrPutInForwardList(IEnumerable1 addresses, UnregistrationCause cause, Int32 hopCount, Dictionary2& forward, List1 tasks, String context)\n at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterManyAsync(List1 addresses, UnregistrationCause cause, Int32 hopCount)\n at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()\n at Orleans.Runtime.Catalog.FinishDestroyActivations(List`1 list, Int32 number, MultiTaskCompletionSource tcs)"
},
“app”: {
“name”: “Squidex”,
“version”: “3.3.0.0”,
“sessionId”: “0d7ed7dc-70ce-487b-9707-366d67fb44fc”
},
“timestamp”: “2019-10-02T01:56:51Z”,
“category”: “Orleans.Runtime.Catalog”
}
{
“logLevel”: “Warning”,
“message”: “UnregisterManyAsync 1 failed.”,
“eventId”: {
“id”: 100502
},
“exception”: {
“type”: “System.InvalidOperationException”,
“message”: “Grain directory is stopping”,
“stackTrace”: " at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)\n at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterOrPutInForwardList(IEnumerable1 addresses, UnregistrationCause cause, Int32 hopCount, Dictionary2& forward, List1 tasks, String context)\n at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterManyAsync(List1 addresses, UnregistrationCause cause, Int32 hopCount)\n at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()\n at Orleans.Runtime.Catalog.FinishDestroyActivations(List`1 list, Int32 number, MultiTaskCompletionSource tcs)"
},
“app”: {
“name”: “Squidex”,
“version”: “3.3.0.0”,
“sessionId”: “0d7ed7dc-70ce-487b-9707-366d67fb44fc”
},
“timestamp”: “2019-10-02T01:56:51Z”,
“category”: “Orleans.Runtime.Catalog”
}
{
“logLevel”: “Warning”,
“message”: “Socket error on new socket during ReceiveAsync with RemoteEndPoint: OperationAborted”,
“eventId”: {
“id”: 101013
},
“app”: {
“name”: “Squidex”,
“version”: “3.3.0.0”,
“sessionId”: “0d7ed7dc-70ce-487b-9707-366d67fb44fc”
},
“timestamp”: “2019-10-02T01:56:53Z”,
“category”: “Orleans.Runtime.Messaging.IncomingMessageAcceptor”
}
{
“logLevel”: “Error”,
“message”: “RunClientMessagePump has thrown exception”,
“eventId”: {
“id”: 100326
},
“exception”: {
“type”: “System.OperationCanceledException”,
“message”: “The operation was canceled.”,
“stackTrace”: " at System.Collections.Concurrent.BlockingCollection1.TryTakeWithNoTimeValidation(T& item, Int32 millisecondsTimeout, CancellationToken cancellationToken, CancellationTokenSource combinedTokenSource)\n at System.Collections.Concurrent.BlockingCollection1.TryTake(T& item, Int32 millisecondsTimeout, CancellationToken cancellationToken)\n at System.Collections.Concurrent.BlockingCollection`1.Take(CancellationToken cancellationToken)\n at Orleans.Runtime.HostedClient.RunClientMessagePump()"
},
“app”: {
“name”: “Squidex”,
“version”: “3.3.0.0”,
“sessionId”: “0d7ed7dc-70ce-487b-9707-366d67fb44fc”
},
“timestamp”: “2019-10-02T01:56:53Z”,
“category”: “Orleans.Runtime.HostedClient”
}
{
“logLevel”: “Warning”,
“message”: “Failed to update status to dead in the alotted time during shutdown”,
“app”: {
“name”: “Squidex”,
“version”: “3.3.0.0”,
“sessionId”: “0d7ed7dc-70ce-487b-9707-366d67fb44fc”
},
“timestamp”: “2019-10-02T01:56:58Z”,
“category”: “Orleans.Runtime.MembershipService.MembershipTableManager”
}
{
“logLevel”: “Warning”,
“message”: “Lifecycle stop operations canceled by request.”,
“app”: {
“name”: “Squidex”,
“version”: “3.3.0.0”,
“sessionId”: “0d7ed7dc-70ce-487b-9707-366d67fb44fc”
},
“timestamp”: “2019-10-02T01:56:58Z”,
“category”: “Orleans.Runtime.SiloLifecycleSubject”
}
Application is shutting down…