We are facing a strange issue. All webhooks and History stopped working when one of the team member deleted 64 records from one schema. This schema is little different than other as it also have assets linked. Looks like We have some assets now which are not liked anymore to any records due to this delete.
This delete was done by team member using Squidex UI. I can see history is still working for other sceham contents. It shows nothing for any record in this schema. Not sure how to recover from this failure. Entire system stopped working as lots of webhooks are linked to this schema for whole system to work smoothly.
We have latest version of Squidex deployed and we are using self hosting.
Hi,
if you go to the administration settings you can check the status of the consumers (background processes)
I can see some of them are paused
Just check the bugs (the icon) and post it here and start them.
You can also use the following endpoint, if you want to monitor them:
they are getting stopped even if I start them. Looks like some exception happening.
Squidex.Infrastructure.Json.JsonException: After parsing a value an unexpected character was encountered: :. Path ‘data.DDDD.iv[146].DDDD’, line 1, position 359789.
—> Newtonsoft.Json.JsonReaderException: After parsing a value an unexpected character was encountered: :. Path ‘data.Sections.iv[146].Name’, line 1, position 359789.
at Newtonsoft.Json.JsonTextReader.ParsePostValue(Boolean ignoreComments)
at Newtonsoft.Json.JsonTextReader.Read()
at Squidex.Infrastructure.Json.Newtonsoft.JsonValueConverter.ReadJson(JsonReader reader) in C:\src\src\Squidex.Infrastructure\Json\Newtonsoft\JsonValueConverter.cs:line 49
When I click bug icon, I can see above Error
Can you post the latest event from the Events2 collection? Or the one after the position.
Not sure what you mean. I can see which two records might be causing this to happen. Not sure how it went to that state but I am not able to do anything with them. I can’t even delete them.
I can see below status bar with No text when I try to delete that record.
Other record says soemthing like below when I try to delete
Since this is test environment and this was newly created schema, I can drop the schema and create it again so all records technically go away. Will that fix the issue?
No. please make a backup of your database and send it to me, then I can have a look. Somehow it fails to deserialize some json, but I don’t know why, because the data in the database MUST be valid json.
Not sure If I am allowed to provide a backup. I need to check with my manager.
Can we have screen share meeting if you are available sometime tomorrow?
No, sorry…not before end of next week. I have to debug it, I will probably not able to help via screen sharing.
Looks like that section (146 Index) has some chars like below which might be causing this error. Here is text for that item
“ Tubular goods should be adequately chocked with wooden blocks or other suitable materials on level racks. When moving tubular goods with a hydraulic hoist, lifting lines should have two wraps around the drill pipe with the hook facing downward. Personnel should not walk between or beneath tubular goods that are being hoisted. Personnel should always roll pipe from the ends in case the pipe gets out of control. Pipe racks should be worked from the ground at all times. Pipe racks should be kept level and in a safe and orderly condition. Flagging signals should be used when moving pipe with cranes and hydraulic hoists if the operator’s view is blocked. Tail or tag lines are required to maintain good control when lifting loads or moving drill pipe. Note: Knots should not be tied on the ends of tag lines because the line could get hung up between objects. This could create a “slingshot” effect to the rope when it is pulled loose by the crane.”
I am fine with screen share next week.
Thx, but I need the original. You have to understand the flow:
- A json stream comes in, into the API.
- The json is successfully deserialized to an object and handled.
- The object is successfully serialized to json again to write it to the DB.
- Then it fails to read from the DB.
It is super weird.
The flow you mentioned is not correct. I can provide recent logs from server if that helps when this issue occurs.
I can still read the record but can’t update or delete anymore.