[SOLVED] 500 error when getting /apps

Bugs Solved Bugs
#1

I have…

  • [ x ] Checked the logs and have uploaded a log file and provided a link because I found something suspicious there. Please do not post the log file in the topic because very often something important is missing.

I’m submitting a…

  • [ ] Regression (a behavior that stopped working in a new release)
  • [x ] Bug report
  • [ ] Performance issue
  • [ ] Documentation issue or request

Current behavior

This morning when I opended squidex, I got a 500 error when tried to access the /apps endpoint
There has been 0 activity over the weekend, and was working fine when I left work friday.

Expected behavior

Able to get the /apps endpoint - I have the logs this time from the app-service - I’ll PM them if requested

Minimal reproduction of the problem

I think it’s quite hard to reproduce - I’ve done nothing extra ordinary lately to the system…

Environment

  • [ x ] Self hosted with docker
  • [ ] Self hosted with IIS
  • [ ] Self hosted with other version
  • [ ] Cloud version

Version: 5.0.0

Browser:

  • [ x ] Chrome (desktop)
  • [ ] Chrome (Android)
  • [ ] Chrome (iOS)
  • [ ] Firefox
  • [ ] Safari (desktop)
  • [ ] Safari (iOS)
  • [ ] IE
  • [ ] Edge

Others:

I updated the container image to 5.2.0 and restarted the app-service, and was able to fetch the /apps endpoint now, and get to the squidex dashboard/overview

#2

I tend to get a lot of these time-out exceptions:

[…] Response did not arrive on time in 00:00:30 for message: NewPlacement Request S127.0.0.1:11111:341397831*cli/ […]

where the endpoint is:

“requestPath”: “/apps/app-name-here/ui/settings/me”,

#3

What happens when you restart your container? Btw: I always ask for the logs :frowning:

#4

https://pastebin.com/KKCR4P6F - This is what I could gather in the app-service

After I restarted the container/app-service (and in this case updated the docker image to 5.2.0) - it’s working again

#5

Okay, there is not that much I can do about it, right now…

#6

I’m still getting this error from time to time. I haven’t added any new apps since the last error I got. This morning we got the same error again where we couldn’t query the /apps endpoint

We can still query the content though. Do you have the slightest idea on how to improve this? It’s hosted on Azure if you need to know

If I restart the webapp, it then again works

#7

I have never seen it, it could be someting like a sleep mode which causes the app not to wake up properly.

#8

Thanks for getting back.
We already have this option enabled on our app:

image

Any other suggestions?

#9

I think this setting could make it even worse. In IIS this prevents the web server from shutting down your application if there are no requests.

But there is also a sleep mode or so on the machine level. But it should not happen. You have only one node, right?

#10

Oh. Should I try to disable it?

Yes, only one node. Hosted as a container webapp on Azure, and a mongodb container instance hooked up to it.

Current version is 5.2.0.

#11

I am not 100% sure what happens. I have posted it in Github: https://github.com/dotnet/orleans/issues/6820

You can try a monitoring service and just query your service every 5 minutes or so to keep it alive. Just an idea.

#12

That’s great! Just ping me on github if I need to answer or provide anything!

Funny that you mention it, I’ve already set up such a service. :slight_smile:

Let’s see how this goes for now. Thanks a lot for the service!

1 Like
#13

Hi again!

Since last, I’ve set up a uptime robot that pings both the mongodb container instance, and the Squidex web-app every minute.

This morning I noticed that a ping on the mongodb port was timed out. So the container instance was unresponsive for maximum of 1 min - Properly less, since the next ping was OK.

Then I checked the Squidex app - It throws the same 500 error as I have previously reported in this thread.
So I guess it has something to do with the db connection. I don’t know which call that causes Squidex to go into a failed state - but if the request fails, I have to restart the Squidex app so I can use the squidex app again. The good thing is that all the content is still query-able.

I checked the logs for the mongodb container instance, and I couldn’t see anything that stood out.

So, a feature request for me is; Could you make some kind of retry policy for these requests that fails, so it does not goes into a failed state if a request fails? I do not have enough insight in Squidex to say, if it is Orleans or Squidex that should be pathed for this - Either way, it will improve the system to have this feature I think :slight_smile:

In the meantime I think I will migrate the mongodb instance to another cloud provider. I do not have the time to investigate further why the mongodb becomes unresponsive at random times. :smile:

:: Update - I didn’t restart the web app right away. I tried to clear localstorage, and was able to access the /app endpoint after the login attempt. Strange…

#14

Sure, I could do that, but I would also think about a liveness probe in your case. This would solve the problem I guess.

#15

So a liveness prope as part of the mongodb container instance?

#16

No, it should observe squidex and then it will just restart it.

#17

Just an update if anybody else experience this strange error.

I migrated the mongoDB instance to a vm on Google Cloud (e2-medium (2 vCPUs, 4 GB memory)), and this seemed to have fixed the problems we’re having.

The previous deployment was a 2gig container instance on Azure. MongoDB is pretty memory hungry, so I just decided to scale up to 4gig.

1 Like
#18

This topic was automatically closed after 2 days. New replies are no longer allowed.