[SOLVED] Lost data upon addition of a new field to schema

Hi Sebastian,

we need your support in order to identify an issue and hopefully restore lost data with our schema.

Schema:
https://cloud.squidex.io/app/matmatch-cms/schemas/edu-content

What was done:
Around 14.30 (Berlin time), to the schema (edu-content) definition were added two new field (ogImage and sideContentReferences)

After that old requests to Squidex API are started ending up in following error:

Squidex response {“errors”:[{“message”:“Cannot query field “internalIdTags” on type “EduContentDataDto”.”,“locations”:[{“line”:30,“column”:13}]},{“message”:“Cannot query field “materialCategoryTags” on type “EduContentDataDto”.”,“locations”:[{“line”:33,“column”:13}]}]}

The error means that internalIdTags and materialCategoryTags fields are not existing.
We definitely had these fields defined and in comparison to other fields were added quite recently (around 1 month ago).

In order to restore our website we had to define there two fields (internalIdTags and materialCategoryTags).
This solved the issues with the request to Squidex.

However (obviously) the content that were previously in the recreated fields were not restored.

The question is: is it possible to restore the data? Despite the changes in the schema from today (that isn’t important yet, since it was adding new fields that are not in use yet) we touched content of this schema on February-14.

Some update:

  1. ogImage field were created at ‘2019-02-20T12:08:24Z’ (got it from backup events/3/3423.json)
  2. several FieldUpdatedEvent and SchemaFieldsReorderedEvent events for this field
  3. sideContentReferences added at ‘2019-02-20T12:11:32Z’ (events/3/3430.json)
  4. several FieldUpdatedEvent and SchemaFieldsReorderedEvent events for sideContentReferences

    the issue happened
  5. we started readding old field at ‘2019-02-20T13:35:19Z’

so if it’s possible to restore the state until ‘events/3/3423.json’ than it should be ok

On the other hand I still don’t know what happened with the two existing fields

Hi Egvenjy,

I cannot find these fields in the history. I will send you a PM. The problem is that content is not stored with the field name but with the field id. Therefore it is not easy to restore these changes.

Somehow the ogImage field got the same ID like the old internalIdTags event. I don’t know why. But this is the reason for the trouble.

I will take the history and try to reproduce it on my machine. I hope it is okay when I take the time first.

Yes, it’s ok with us. Thank you for your help.

I understand the problem now. It is related to this problem :Is cloud server down now?

In each schema I create and increment a counter for all fields. Whenever a field is added an event is created and for each FieldAddedEvent this counter is incremented. But when I fixed your schema I had to remove 2 fields from the history and therefore the counter has the wrong value. So the new field got an old id and has overwritten your existing fields.

What I can do: I can “fix” the history by reassigning new ids to the new fields. I will use the ids of the deleted fields then it should match again.

Do you know when it is a good time to do it? I could fix it today at around 9pm.

Let’s have it today at 9pm.

Can I also delete your new fields? It would be easier to provide a fix for the problem and then delete all changes from today.

I have deployed a fix. Please create a new field so that I can check if it works. Unfortunately I have no access to your app.

Hi Sebastian,

Unfortunately, the results of the issue remained: the data that were originally in ‘internalIdTags’ and ‘materialCategoryTags’ fields are not there.

Here is an example of the fields from backup events/3/3424.json (2019-02-13T15:52:07Z):

“internalIdTags”:{“iv”:[“minfm50389”,“vdmm016”,“minfm67141”,“come056”,“minfc273”,“grae010”,“minfm21427”,“come050”,“come054”,“come037”,“vdmm018”,“sver40”,“braz007”,“sver3”,“sver25”,“fico008”,“alco007”,“minfc31837”,“minfm33714”,“minfm53026”,“minfm50856”,“minfc37596”,“minfm67216”,“minfm53087”,“minfm39812”]},“materialCategoryTags”:{“iv”:[“All Metal”]}},

It’s just an example showing that we had data in the internalIdTags and materialCategoryTags fields. However, as of now non of the content articles in the schema have internalIdTags and materialCategoryTags available.

What can be done with this?

Thank you, I will dig in further.

Can you add me to your app: My username is Sebastian (squidex), just search for sebastian@squidex.io

I have added you as a contributor to the project:

Can you give me developer permissions? Need to see the schema editor.

Done, you have Developer role

Can you have another look?

It looks good now. Thank you for your help!

Just a quick question: can we proceed with adding new fields to the schema?

Yes, it should work. Please add a not here if you have added a field. I will have another look to the DB then