Squidex fails to start-up

Support
#1

I have…

I’m submitting a…

  • [ ] Regression (a behavior that stopped working in a new release)
  • [x] Bug report
  • [ ] Performance issue
  • [ ] Documentation issue or request

Current behavior

Expected behavior

Application / docker container should start-up properly after redeploying.

Minimal reproduction of the problem

After redeploying squidex we are getting the following error at startup, preventing access ( HTTP Error 502.5 - ANCM Out-Of-Process Startup Failure)

The target silo became unavailable for message: Request S10.53.131.25:11114:389282555stg/Catalog/0000000e@S0000000e->S10.53.132.27:11114:389289388stg/DirectoryService/0000000a@S0000000a InvokeMethodRequest Orleans.Runtime.IRemoteGrainDirectory:LookupAsync #34369. Target History is: S10.53.132.27:11114:389289388:*stg/DirectoryService/0000000a:@S0000000a. See https://aka.ms/orleans-troubleshooting for troubleshooting help. About to break its promise.

Environment

App Name:

  • [x] Self hosted with docker
  • [ ] Self hosted with IIS
  • [ ] Self hosted with other version
  • [ ] Cloud version

Version: commit#377fe120d74085bac86607674b244855cab0215b

Browser:

  • [x] Chrome (desktop)
  • [x] Chrome (Android)
  • [x] Chrome (iOS)
  • [x] Firefox
  • [x] Safari (desktop)
  • [x] Safari (iOS)
  • [x] IE
  • [x] Edge

Others:

#2

Are you sure, that this is the only error?

#3

I found some more errors in the logs… seems to be mostly Orleans connectivity issues

exception.message
Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.53.132.27:11114:389372401. See InnerException
—> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.53.132.27:11114. Error: ConnectionRefused
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in //src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 59
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in //src/Orleans.Core/Networking/ConnectionFactory.cs:line 56
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task1 task, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 410 at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 293 --- End of inner exception stack trace --- at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 293 at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 139 at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 139 at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask1 c, Message m) in //src/Orleans.Runtime/Messaging/OutboundMessageQueue.cs:line 130
at Orleans.Runtime.Messaging.OutboundMessageQueue.g__SendAsync|10_0(ValueTask`1 c, Message m) in //src/Orleans.Runtime/Messaging/OutboundMessageQueue.cs:line 130

exception.stackTrace
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object ) in //src/Orleans.Runtime/Placement/DeploymentLoadPublisher.cs:line 117

exception.type
Orleans.Runtime.OrleansMessageRejectionException

errormessage
An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.

eventId.id
101,802

exception.message
Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.53.132.27:11114:389372401. See InnerException
—> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.53.132.27:11114. Error: ConnectionRefused
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in //src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 59
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in //src/Orleans.Core/Networking/ConnectionFactory.cs:line 56
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task1 task, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 410 at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 293 --- End of inner exception stack trace --- at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 293 at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 139 at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 139 at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask1 c, Message m) in //src/Orleans.Runtime/Messaging/OutboundMessageQueue.cs:line 130
at Orleans.Runtime.Messaging.OutboundMessageQueue.g__SendAsync|10_0(ValueTask`1 c, Message m) in //src/Orleans.Runtime/Messaging/OutboundMessageQueue.cs:line 130

exception.stackTrace
at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object ) in //src/Orleans.Runtime/Placement/DeploymentLoadPublisher.cs:line 117

#4

Do you use clustering? Do you have more than one instance / node?

#5

We have 2 instances (2 legged load balanced environment).

#6

Perhaps this helps: https://docs.squidex.io/01-getting-started/installation/troubleshooting-and-support#my-cluster-fails-to-start

Just added it.

#7

Thank you, Sebastian. We will give this a try and update here if it fixes the startup errors.