Schemas and data gone from UI after an upgrade to latest version

Support
1

Hey guys, this morning I noticed that some linked assets were not rendering on our website correctly. Even after saving it didnt link correctly.

Tried an upgrade to squidex:latest and all the schemas and data was gone from the UI.

Logged into the mongo db and the data still seems to be there (I can query it in the SquidexContent collection). Tried re-running the docker with the environment variable REBUILD__SCHEMAS=true but no luck.

Please advice?

I have…

  • [x] Checked the logs and have provided the logs if I found something suspicious there

I’m submitting a…

  • [x ] Regression (a behavior that stopped working in a new release)

Current behavior

Probably after an upgrade, but not sure since its running kubernetes as squidex:latest

Expected behavior

Have my data appear :slight_smile:

Minimal reproduction of the problem

Environment

  • [x ] Self hosted with docker

Version: latest

Browser:

  • [x ] Chrome (desktop)

Others:

2

Just a side note, my apps are still available; so its able to reach those correctly, squidex just can not see any schema and or data belonging to those schema anymore

3

Please make an backup of your database with mongodb restore and send it to me.

4

Just sent it to mail2stehle@gmail.com (which was the address I found here on the forums)

5

Please send it to sebastian@squidex.io

6

done, please check your mail

7

I am not sure why and how you have done it, but you have deployed the latest dev version with changes from yesterday.

8

I actually know how, I read in another thread that squidex:latest went to 5.0, so I just tried upgrading to latest dev version

Is my installation hosed, eg start from scratch with re-entering data or is this salvable?

9

The problem is, that I made a big change to the migration and the beta I was about the launch was not supposed to be compatible with previous betas.

But I have made a fix to keep the compatibility. I am just testing it with another dataset now again.

There is a collection called “Migration” with a single document. This document holds the database version which is currently 22.

If you change it to 21, the migration will run again and fixes your issue.

image
image.png872×318 19.5 KB

I keep you updated when the fix is out.

10

Turned into this exception:

{

05/10/2020 11:06:22 "logLevel": "Fatal",

05/10/2020 11:06:22 "action": "Migration",

05/10/2020 11:06:22 "status": "Failed",

05/10/2020 11:06:22 "migrator": "Migrations.Migrations.ConvertEventStore",

05/10/2020 11:06:22 "app": {

05/10/2020 11:06:22 "name": "Squidex",

05/10/2020 11:06:22 "version": "4.0.0.0",

05/10/2020 11:06:22 "sessionId": "61590431-47de-49fb-b261-4a89582dddca"

05/10/2020 11:06:22 },

05/10/2020 11:06:22 "timestamp": "2020-10-05T09:06:22Z",

05/10/2020 11:06:22 "exception": {

05/10/2020 11:06:22 "type": "System.InvalidCastException",

05/10/2020 11:06:22 "message": "Unable to cast object of type \u0027MongoDB.Bson.BsonDocument\u0027 to type \u0027MongoDB.Bson.BsonString\u0027.",

05/10/2020 11:06:22 "stackTrace": " at Migrations.Migrations.ConvertEventStore.\u003C\u003Ec__DisplayClass2_0.\u003C\u003CUpdateAsync\u003Eb__1\u003Ed.MoveNext() in /src/src/Migrations/Migrations/ConvertEventStore.cs:line 54\n--- End of stack trace from previous location where exception was thrown ---\n at MongoDB.Driver.IAsyncCursorExtensions.ForEachAsync[TDocument](IAsyncCursor\u00601 source, Func\u00603 processor, CancellationToken cancellationToken)\n at MongoDB.Driver.IAsyncCursorSourceExtensions.ForEachAsync[TDocument](IAsyncCursorSource\u00601 source, Func\u00602 processor, CancellationToken cancellationToken)\n at Migrations.Migrations.ConvertEventStore.UpdateAsync() in /src/src/Migrations/Migrations/ConvertEventStore.cs:line 65\n at Squidex.Infrastructure.Migrations.Migrator.MigrateAsync(CancellationToken ct) in /src/src/Squidex.Infrastructure/Migrations/Migrator.cs:line 78"

05/10/2020 11:06:22 }

05/10/2020 11:06:22 }

05/10/2020 11:06:22

05/10/2020 11:06:22 Unhandled exception. Squidex.Infrastructure.Migrations.MigrationFailedException: Failed to run migration 'Migrations.Migrations.ConvertEventStore'

05/10/2020 11:06:22 ---> System.InvalidCastException: Unable to cast object of type 'MongoDB.Bson.BsonDocument' to type 'MongoDB.Bson.BsonString'.

05/10/2020 11:06:22 at Migrations.Migrations.ConvertEventStore.<>c__DisplayClass2_0.<<UpdateAsync>b__1>d.MoveNext() in /src/src/Migrations/Migrations/ConvertEventStore.cs:line 54

05/10/2020 11:06:22 --- End of stack trace from previous location where exception was thrown ---

05/10/2020 11:06:22 at MongoDB.Driver.IAsyncCursorExtensions.ForEachAsync[TDocument](IAsyncCursor`1 source, Func`3 processor, CancellationToken cancellationToken)

05/10/2020 11:06:22 at MongoDB.Driver.IAsyncCursorSourceExtensions.ForEachAsync[TDocument](IAsyncCursorSource`1 source, Func`2 processor, CancellationToken cancellationToken)

05/10/2020 11:06:22 at Migrations.Migrations.ConvertEventStore.UpdateAsync() in /src/src/Migrations/Migrations/ConvertEventStore.cs:line 65

05/10/2020 11:06:22 at Squidex.Infrastructure.Migrations.Migrator.MigrateAsync(CancellationToken ct) in /src/src/Squidex.Infrastructure/Migrations/Migrator.cs:line 78

05/10/2020 11:06:22 --- End of inner exception stack trace ---

05/10/2020 11:06:22 at Squidex.Infrastructure.Migrations.Migrator.MigrateAsync(CancellationToken ct) in /src/src/Squidex.Infrastructure/Migrations/Migrator.cs:line 88

05/10/2020 11:06:22 at Squidex.Infrastructure.Migrations.Migrator.MigrateAsync(CancellationToken ct) in /src/src/Squidex.Infrastructure/Migrations/Migrator.cs:line 97

05/10/2020 11:06:22 at Squidex.Config.Startup.SafeHostedService.StartAsync(CancellationToken cancellationToken) in /src/src/Squidex/Config/Startup/SafeHostedService.cs:line 29

05/10/2020 11:06:22 at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)

05/10/2020 11:06:22 at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)

05/10/2020 11:06:22 at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)

05/10/2020 11:06:22 at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)

05/10/2020 11:06:22 at Squidex.Program.Main(String[] args) in /src/src/Squidex/Program.cs:line 24
11

Do I need to wait for the fix before I run this?

12

Additional question: whats the best version for me to be at, at this point: latest or beta?

Thanks Sebastian, appreciate your help

13

Right now it is too late for latest (aka 4.X versions). You need to run for my fix. Are you sure you have changed the version to 21?

14

Ok i;ll wait for your update, yes the version was changed successfully like this:

 db.Migration.update({ _id: "Default" }, {"Version": 21})
 WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
 db.Migration.find()
 { "_id" : "Default", "Version" : 21 }
15

We hit the same problem using :latest. Data seem to be there in mongo and you can see the apps but no content or schema.

16

@devstator82 You can try dev-4967.

@Marcus_Lambert Perhaps you have old tag, but as you can see, :latest currently points to 4.7.3: https://hub.docker.com/repository/docker/squidex/squidex/tags?page=1 (they have the same digest).

In general, I recommend never ever to use :latest. Imagine you are using a version :3 and then after a year you add a second instance and the docker image is not cached on this server, then you have :3 and :5 in your cluster. It will fuck everything, even if the version does not contain a bug and is super stable.

If you want you can also send me your database and I can have a look.

17

Thanks for the update, upgraded to dev-4967, still crash (copy/pasting stacktrace below).

I’ll try to revert to the database I sent you and then run the upgrade again, maybe i changed something in the meanwhile which messed things up.

{

05/10/2020 16:03:30 "logLevel": "Fatal",

05/10/2020 16:03:30 "action": "Migration",

05/10/2020 16:03:30 "status": "Failed",

05/10/2020 16:03:30 "migrator": "Migrations.Migrations.ConvertEventStore",

05/10/2020 16:03:30 "app": {

05/10/2020 16:03:30 "name": "Squidex",

05/10/2020 16:03:30 "version": "4.0.0.0",

05/10/2020 16:03:30 "sessionId": "84965cad-7695-4f96-b14d-8d01c496aa20"

05/10/2020 16:03:30 },

05/10/2020 16:03:30 "timestamp": "2020-10-05T14:03:30Z",

05/10/2020 16:03:30 "exception": {

05/10/2020 16:03:30 "type": "System.InvalidCastException",

05/10/2020 16:03:30 "message": "Unable to cast object of type \u0027MongoDB.Bson.BsonDocument\u0027 to type \u0027MongoDB.Bson.BsonString\u0027.",

05/10/2020 16:03:30 "stackTrace": " at Migrations.Migrations.ConvertEventStore.\u003C\u003Ec__DisplayClass2_0.\u003C\u003CUpdateAsync\u003Eb__1\u003Ed.MoveNext() in /src/src/Migrations/Migrations/ConvertEventStore.cs:line 54\n--- End of stack trace from previous location where exception was thrown ---\n at MongoDB.Driver.IAsyncCursorExtensions.ForEachAsync[TDocument](IAsyncCursor\u00601 source, Func\u00603 processor, CancellationToken cancellationToken)\n at MongoDB.Driver.IAsyncCursorSourceExtensions.ForEachAsync[TDocument](IAsyncCursorSource\u00601 source, Func\u00602 processor, CancellationToken cancellationToken)\n at Migrations.Migrations.ConvertEventStore.UpdateAsync() in /src/src/Migrations/Migrations/ConvertEventStore.cs:line 65\n at Squidex.Infrastructure.Migrations.Migrator.MigrateAsync(CancellationToken ct) in /src/src/Squidex.Infrastructure/Migrations/Migrator.cs:line 78"

05/10/2020 16:03:30 }

05/10/2020 16:03:30 }

05/10/2020 16:03:30

05/10/2020 16:03:30 Unhandled exception. Squidex.Infrastructure.Migrations.MigrationFailedException: Failed to run migration 'Migrations.Migrations.ConvertEventStore'

05/10/2020 16:03:30 ---> System.InvalidCastException: Unable to cast object of type 'MongoDB.Bson.BsonDocument' to type 'MongoDB.Bson.BsonString'.

05/10/2020 16:03:30 at Migrations.Migrations.ConvertEventStore.<>c__DisplayClass2_0.<<UpdateAsync>b__1>d.MoveNext() in /src/src/Migrations/Migrations/ConvertEventStore.cs:line 54

05/10/2020 16:03:30 --- End of stack trace from previous location where exception was thrown ---

05/10/2020 16:03:30 at MongoDB.Driver.IAsyncCursorExtensions.ForEachAsync[TDocument](IAsyncCursor`1 source, Func`3 processor, CancellationToken cancellationToken)

05/10/2020 16:03:30 at MongoDB.Driver.IAsyncCursorSourceExtensions.ForEachAsync[TDocument](IAsyncCursorSource`1 source, Func`2 processor, CancellationToken cancellationToken)

05/10/2020 16:03:30 at Migrations.Migrations.ConvertEventStore.UpdateAsync() in /src/src/Migrations/Migrations/ConvertEventStore.cs:line 65

05/10/2020 16:03:30 at Squidex.Infrastructure.Migrations.Migrator.MigrateAsync(CancellationToken ct) in /src/src/Squidex.Infrastructure/Migrations/Migrator.cs:line 78

05/10/2020 16:03:30 --- End of inner exception stack trace ---

05/10/2020 16:03:30 at Squidex.Infrastructure.Migrations.Migrator.MigrateAsync(CancellationToken ct) in /src/src/Squidex.Infrastructure/Migrations/Migrator.cs:line 88

05/10/2020 16:03:30 at Squidex.Infrastructure.Migrations.Migrator.MigrateAsync(CancellationToken ct) in /src/src/Squidex.Infrastructure/Migrations/Migrator.cs:line 97

05/10/2020 16:03:30 at Squidex.Config.Startup.SafeHostedService.StartAsync(CancellationToken cancellationToken) in /src/src/Squidex/Config/Startup/SafeHostedService.cs:line 29

05/10/2020 16:03:30 at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)

05/10/2020 16:03:30 at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)

05/10/2020 16:03:30 at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)

05/10/2020 16:03:30 at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)

05/10/2020 16:03:30 at Squidex.Program.Main(String[] args) in /src/src/Squidex/Program.cs:line 24
18

Please format your logs properly.

The ConvertEventStore is a very old migration. It seems that it is running, but this can only happen when the version field is not correct in the migration.

19

Sorry about that :slight_smile:

So here is what I tried

  • Dropped current database
  • Imported the database that I have sent to you by email using mongorestore
> use Squidex;
switched to db Squidex
> db.Migration.find()
{ "_id" : "Default", "IsLocked" : false, "Version" : 22 }
> db.Migration.update({ _id: "Default" }, {"Version": 21})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

Start container for version squidex/squidex:dev-4967

> db.Migration.find()
{ "_id" : "Default", "Version" : 0, "IsLocked" : false }

I think you should be able to reproduce this on your end as well.

Any suggestions?

20

But why is “Version” 0 now?

I am not using the mongo shell, I am lazy and use a visual tool for that :wink:

I have imported your database, then changed the version to 21 and then started the application and the migration was running properly afterwards.

But I have to make another fix again (which has nothing to do with the exception you see).