[SOLVED] OUTAGE: Our production website is down and it seems not to have any data

kmaid · December 3, 2021, 11:38am

I have…

[x] Read the following guideline: https://docs.squidex.io/01-getting-started/installation/troubleshooting-and-support. I understand that my support request might get deleted if I do not follow the guideline.

I’m submitting a…

[ ] Regression (a behavior that stopped working in a new release)
[x] Bug report
[ ] Performance issue
[ ] Documentation issue or request

Current behavior

Our app asap-hub application is currently showing no “content” but there are schemas. We noticed this problem after running a sync schema via our CI pipeline. The CMS either shows content types at 404 or an empty.

* STEP 0 of 2: Schemas

[90](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L90)* STEP 1 of 2: Rules

[91](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L91)--------------------------------------------------------

[92](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L92)* STEP 0 of 2: Importing Schemas started

[93](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L93)Schema migrations updating... failed: Application is in readonly mode at the moment.

[94](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L94)Schema dashboard updating... failed: Application is in readonly mode at the moment.

[95](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L95)Schema events updating... failed: Application is in readonly mode at the moment.

[96](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L96)Schema research-outputs updating... failed: Application is in readonly mode at the moment.

[97](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L97)Schema users updating... failed: Application is in readonly mode at the moment.

[98](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L98)Schema external-authors updating... failed: Application is in readonly mode at the moment.

[99](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L99)Schema labs updating... failed: Application is in readonly mode at the moment.

[100](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L100)Schema news-and-events updating... failed: Application is in readonly mode at the moment.

[101](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L101)Schema teams updating... failed: Application is in readonly mode at the moment.

[102](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L102)Schema groups updating... failed: Application is in readonly mode at the moment.

[103](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L103)Schema discover updating... failed: Application is in readonly mode at the moment.

[104](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L104)Schema calendars updating... failed: Application is in readonly mode at the moment.

[105](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L105)Schema pages updating... failed: Application is in readonly mode at the moment.

[106](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L106)* STEP 0 of 2: Importing Schemas completed

[107](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L107)--------------------------------------------------------

[108](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L108)--------------------------------------------------------

[109](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L109)* STEP 1 of 2: Importing Rules started

[110](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L110)Rule 'Fetch user ORCID on update OR c...failed: Squidex.ClientLibrary.SquidexException: Squidex Request failed: {"message":"Application is in readonly mode at the moment.","traceId":"00-451747b9054db2478e869f67c2a77578-20408269bdfa4760-01","type":"https://tools.ietf.org/html/rfc7231#section-6.5.1","statusCode":400}

[111](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L111) at Squidex.ClientLibrary.Utils.SquidexClientBase.EnsureResponseIsValidAsync(HttpResponseMessage response)

[112](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L112) at Squidex.ClientLibrary.Utils.SquidexClientBase.RequestJsonAsync[T](HttpMethod method, String path, HttpContent content, QueryContext context, CancellationToken ct)

[113](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L113) at Squidex.CLI.Commands.Implementation.Sync.Rules.RulesSynchronizer.<>c__DisplayClass7_3.<<ImportAsync>b__7>d.MoveNext()

[114](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L114)--- End of stack trace from previous location ---

[115](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L115) at Squidex.CLI.Commands.Implementation.LogExtensions.DoVersionedAsync(ILogger log, String process, Int64 version, Func`1 action).

[116](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L116)ERROR: Squidex.ClientLibrary.SquidexException: Squidex Request failed: {"message":"Application is in readonly mode at the moment.","traceId":"00-451747b9054db2478e869f67c2a77578-20408269bdfa4760-01","type":"https://tools.ietf.org/html/rfc7231#section-6.5.1","statusCode":400}

[117](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L117) at Squidex.CLI.Commands.Implementation.LogExtensions.HandleException(Exception ex, Action`1 error)

[118](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L118) at Squidex.CLI.Commands.Implementation.LogExtensions.DoVersionedAsync(ILogger log, String process, Int64 version, Func`1 action)

[119](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L119) at Squidex.CLI.Commands.Implementation.Sync.Rules.RulesSynchronizer.ImportAsync(ISyncService sync, SyncOptions options, ISession session)

[120](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L120) at Squidex.CLI.Commands.Implementation.Sync.Synchronizer.<>c__DisplayClass5_0.<<ImportAsync>b__0>d.MoveNext()

[121](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L121)--- End of stack trace from previous location ---

[122](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L122) at Squidex.CLI.Commands.Implementation.Extensions.Foreach[T](IEnumerable`1 source, Func`3 action)

[123](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L123) at Squidex.CLI.Commands.Implementation.Sync.Synchronizer.ImportAsync(String path, SyncOptions options, ISession session)

[124](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L124) at Squidex.CLI.Commands.App.Sync.In(InArguments arguments)

[125](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L125) at CommandDotNet.Execution.InvocationResultExtensions.GetResultCodeAsync(Object value)

[126](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L126) at CommandDotNet.AppRunner.HandleException(Exception ex, IConsole console, CommandContext commandContext)

[127](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L127) at CommandDotNet.AppRunner.Run(String[] args)

[128](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L128) at Squidex.CLI.Program.Main(String[] args)

[130](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L130)Cleaning up project directory and file based variables00:01

[132](https://gitlab.com/yldio/asap-hub/-/jobs/1846531212#L132)ERROR: Job failed: exit code 1

Expected behavior

There to be content and the application not to be in read only mode.

Minimal reproduction of the problem

Environment

[ ] Self hosted with docker
[ ] Self hosted with IIS
[ ] Self hosted with other version
[x] Cloud version

Version: [VERSION]

Vasili_Fedotov · December 3, 2021, 11:48am

I have the same issue

kmaid · December 3, 2021, 11:52am

Pleased that we aren’t the only ones effected. The status page is showing as green . https://status.squidex.io/

ubuntudroid · December 3, 2021, 12:01pm

Can confirm, something is wrong on hosted Squidex. Started a few minutes ago. Queries return with 200, but query results are empty.

haihninit · December 3, 2021, 12:03pm

We also got the same issue.

Sebastian · December 3, 2021, 12:04pm

Yes, the data is there and has not been deleted. And I also see the squidex website data. What is your app name? And have you had an app with the same name before?

kmaid · December 3, 2021, 12:04pm

Our application name is asap-hub

Jose_Brito · December 3, 2021, 12:11pm

Having the same issue too, all requests to schema data return 200 but no items.

jonhilt1 · December 3, 2021, 12:12pm

Yep same here.

None of my schemas are showing ay data at this point.

Michal_Simka_Houzy · December 3, 2021, 12:12pm

We have the exact same problem. We did not add any new app or any new schema and barely touched content today. There’s no visible trace of anyone deleting anything on our side.

Sebastian · December 3, 2021, 12:16pm

I am really sorry. This was pure human error. A new version of Squidex has been deployed to act as migrator for apps and schemas. Unfortunately also rebuilding of content was turned on and it has been stopped in between. The content is not lost, but not available. I have rerun the migrator to rebuild all contents now. I will update you.

ubuntudroid · December 3, 2021, 12:17pm

After this is done, is it possible to invalidate caches on the CDN as well so that CDN doesn’t serve empty data for the next hours? @Sebastian And many thanks for the quick response over here!

Sebastian · December 3, 2021, 12:22pm

Of course I can purge CDN

Sebastian · December 3, 2021, 12:39pm

Data is coming back step by step…

But I don’t know how long it takes because i do not know the total content. I guess we have 60% now.

Seb_Smith · December 3, 2021, 12:44pm

If we have added data in the meantime is this an issue? It’s only a small amount but worth knowing.

Sebastian · December 3, 2021, 12:51pm

No, it should be catched. Only if you add it at the very end of the migration process.

ubuntudroid · December 3, 2021, 12:53pm

Looking good, responses contain data again. Many thanks, @Sebastian!

Sebastian · December 3, 2021, 12:59pm

It is solved now, I am very, very sorry.

Seb_Smith · December 3, 2021, 1:00pm

no problem sebastian - all looking good now

ubuntudroid · December 3, 2021, 1:02pm

Don’t worry about it. Things happen, mistakes are being made, we are all human.

Looking forward: is there a way this could be catched by automated tests on the server side? Something which also triggers the status page to show that there is an issue in such a case?