Hi Sebastian,
Recognize the following error message?
I’ve got a bunch of them and I would like to get rid of them.
Got any tips on where I should look first?
I haven’t migrated. I have only used dev-xxxx Docker image and I have continuosly been upgrading my pod with the latest version. So maybe I do need to do the migration fix documented here?
Or do I need to reset some event consumers under Administration?
I’m using squidex/squidex:dev-5061 and MongoDB 4.2.6 as a replica set.
{
"logLevel": "Error",
"message": "Caught and ignored exception: Squidex.Infrastructure.States.InconsistentStateException with message: Requested version 81602, but found 81739. thrown from timer callback GrainTimer. TimerCallbackHandler:Squidex.Domain.Apps.Entities.Rules.UsageTracking.UsageTrackerGrain-\u003ESystem.Threading.Tasks.Task \u003COnActivateAsync\u003Eb__5_0(System.Object)",
"eventId": {
"id": 101413
},
"app": {
"name": "Squidex",
"version": "4.0.0.0",
"sessionId": "cf7b3a3e-5160-4ebe-8665-05d9f28af75d"
},
"timestamp": "2021-01-25T19:33:41Z",
"category": "Orleans.Runtime.GrainTimer",
"exception": {
"type": "Squidex.Infrastructure.States.InconsistentStateException",
"message": "Requested version 81602, but found 81739.",
"stackTrace": " at Squidex.Infrastructure.MongoDb.MongoExtensions.UpsertVersionedAsync[TEntity,TKey](IMongoCollection\u00601 collection, TKey key, Int64 oldVersion, Int64 newVersion, Func\u00602 updater) in /src/src/Squidex.Infrastructure.MongoDb/MongoDb/MongoExtensions.cs:line 123\n at Squidex.Infrastructure.States.MongoSnapshotStore\u00602.WriteAsync(TKey key, T value, Int64 oldVersion, Int64 newVersion) in /src/src/Squidex.Infrastructure.MongoDb/States/MongoSnapshotStore.cs:line 68\n at Squidex.Infrastructure.States.Persistence\u00602.WriteSnapshotAsync(TSnapshot state) in /src/src/Squidex.Infrastructure/States/Persistence{TSnapshot,TKey}.cs:line 143\n at Squidex.Domain.Apps.Entities.Rules.UsageTracking.UsageTrackerGrain.CheckUsagesAsync() in /src/src/Squidex.Domain.Apps.Entities/Rules/UsageTracking/UsageTrackerGrain.cs:line 107\n at Orleans.Runtime.GrainTimer.ForwardToAsyncCallback(Object state)"
}
}
Are you running in cluster mode or with multiple nodes? If not a simple restart should help.
What do you mean by cluster mode?
Currently I’m using 1 node.
I have been scaling up and down my cluster a few times the last couple of days to fine tune resource requests and limits for my containers. The plan is to go with ONE node for the next year or so.
Sometimes this happens when people run multiple nodes without clustering enabled. Clustering connects all nodes to one cluster where grains like micro-services are distributed and managed.
I see.
When you say a restart would help… do you mean restarting Squidex Pod, MongoDB pods, or both?
This has nothing to do with Squidex or MongoDB versions, correct?
Only the pod, it happens when the in-memory state of a grain (actor) does not match to the database.
Okay. Which pod? MongoDB?
Squidex pod, sorry for the confusion.
Okay, will restart it now.
Trying some funky asset operations again, e.g. uploading tons of large images. Which might have been the cause in the first place 30 mins ago when I did the exact same thing and I spotted those error messages.
Or it could have just been a completely different thing. I really can’t tell. Would have to test/troubleshoot more to really understand what is going on.
Still no errors in the Squidex pod though 
It’s been up for 12 mins now.
Tried to upload 20 images again ~80MB, the larger files being 5-8 MB each.
16 images uploaded successfully. The other four gave me the following error in Chrome console:
21:31:39.695 /api/apps/db/assets?parentId=92298888-0db3-43ab-950d-d5799313c375:1 Failed to load resource: the server responded with a status of 504 ()
21:31:48.061 /api/apps/db/assets?parentId=92298888-0db3-43ab-950d-d5799313c375:1 Failed to load resource: the server responded with a status of 504 ()
21:31:49.888 /api/apps/db/assets?parentId=92298888-0db3-43ab-950d-d5799313c375:1 Failed to load resource: the server responded with a status of 504 ()
21:31:53.140 /api/apps/db/assets?parentId=92298888-0db3-43ab-950d-d5799313c375:1 Failed to load resource: the server responded with a status of 504 ()
Sorry, I know this one is off-topic.
Interesting. I went back one folder and then went back to my new folder where I uploaded the files and now there’s 20 files in it. And the Thumbnails for those four large files got generated while I was in the folder idling. It’s like it was never finished processing the images and Squidex decided to throw an error instead. Maybe something timed out or something?
Could be, it is restricting the number of parallel resizes.
Okay. Yes, seem like that could have something to do with it.
Tested with a new folder and still gives me (1) upload error. The error messages in Chrome Console look inconsistent. Don’t see any clear pattern.
Do you want me to send you a zip with the image files so that you can test it in your environment?
Here’s the other error:
21:50:59.101 cat-iheojs-c-scale-w-7680.jpg:1 GET https://www.example.com/api/assets/db/2c3aff37-599a-43d9-8db2-0f4d0847ab0c/cat-iheojs-c-scale-w-7680.jpg?version=0&width=68&height=66&mode=Pad&nofocus 504
Image (async)
setProperty @ app.js?99874b64951c967171e2:2
setProperty @ app.js?99874b64951c967171e2:2
t.setImageSource @ app.js?99874b64951c967171e2:2
t.resize @ app.js?99874b64951c967171e2:2
t.ngAfterViewInit @ app.js?99874b64951c967171e2:2
tr @ app.js?99874b64951c967171e2:2 .........................
In the first test:
16/20 successful uploads.
4 thumbnails were not rendered.
4 red alerts I think.
In this last test:
19/20 successful uploads.
3 thumbnails still not rendered.
1 red alert error about “Asset could not be uploaded. Please reload”.
Yes, please send them to me.
@Sebastian I’ve uploaded the zip file with all of the files.
I have no problem with these assets, but I found a bug with error handling for large assets, because the behavior of ASP.NET Core became a little bit weird.