[SOLVED] OUTAGE: Our production website is down and it seems not to have any data

Maintenance
3

Pleased that we aren’t the only ones effected. The status page is showing as green :confused:. https://status.squidex.io/

image
image2296×1458 228 KB

4

Can confirm, something is wrong on hosted Squidex. Started a few minutes ago. Queries return with 200, but query results are empty.

5

We also got the same issue.

6

Yes, the data is there and has not been deleted. And I also see the squidex website data. What is your app name? And have you had an app with the same name before?

7

Our application name is asap-hub

8

Having the same issue too, all requests to schema data return 200 but no items.

9

Yep same here.

None of my schemas are showing ay data at this point.

10

We have the exact same problem. We did not add any new app or any new schema and barely touched content today. There’s no visible trace of anyone deleting anything on our side.

12

I am really sorry. This was pure human error. A new version of Squidex has been deployed to act as migrator for apps and schemas. Unfortunately also rebuilding of content was turned on and it has been stopped in between. The content is not lost, but not available. I have rerun the migrator to rebuild all contents now. I will update you.

3 Likes
13

After this is done, is it possible to invalidate caches on the CDN as well so that CDN doesn’t serve empty data for the next hours? @Sebastian And many thanks for the quick response over here! :hugs:

14

Of course I can purge CDN

1 Like
15

Data is coming back step by step…

But I don’t know how long it takes because i do not know the total content. I guess we have 60% now.

1 Like
16

If we have added data in the meantime is this an issue? It’s only a small amount but worth knowing.

17

No, it should be catched. Only if you add it at the very end of the migration process.

18

Looking good, responses contain data again. :smiley: Many thanks, @Sebastian!

19

It is solved now, I am very, very sorry.

1 Like
20

no problem sebastian - all looking good now

1 Like
21

Don’t worry about it. Things happen, mistakes are being made, we are all human. :man_shrugging:

Looking forward: is there a way this could be catched by automated tests on the server side? Something which also triggers the status page to show that there is an issue in such a case?

22

Yes, definitely, but it would not have helped in this case. The error would have been detected 10 minutes earlier or so.

23

Visibility always helps, at the least with expectation management. :sweat_smile: