IIS Hosting Orleans Failure

Hosting (OnPremise)
1

Every couple of days, I’m getting an orleans error saying it can’t open the port it needs to. This has now happened three times (over 6 days). I’ve changed the port on the appsettings to 21111 to see if that helps. I’m assuming another asp.net application tries to use that port dynamically and then it fails when orleans tries to grab it.

Any thoughts?

In the event log, the first error I get is:

Application: w3wp.exe
CoreCLR Version: 4.700.19.60701
.NET Core Version: 3.1.1
Description: The process was terminated due to an unhandled exception.
Exception Info: System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Squidex.Program.Main(String[] args) in /src/src/Squidex/Program.cs:line 23

and then

Application: w3wp.exe
CoreCLR Version: 4.700.19.60701
.NET Core Version: 3.1.1
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.AspNetCore.Connections.AddressInUseException: Only one usage of each socket address (protocol/network address/port) is normally permitted.
 ---> System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted.
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Orleans.Networking.Shared.SocketConnectionListener.Bind()
   --- End of inner exception stack trace ---
   at Orleans.Networking.Shared.SocketConnectionListener.Bind()
   at Orleans.Networking.Shared.SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionListener.BindAsync(CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.SiloConnectionListener.OnRuntimeInitializeStart(CancellationToken cancellationToken)
   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)
   at Orleans.LifecycleSubject.<OnStart>g__CallOnStart|7_0(Int32 stage, OrderedObserver observer, CancellationToken cancellationToken)
   at Orleans.LifecycleSubject.OnStart(CancellationToken ct)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken)
   at Orleans.Hosting.SiloWrapper.StartAsync(CancellationToken cancellationToken)
   at Orleans.Hosting.SiloHostedService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Squidex.Program.Main(String[] args) in /src/src/Squidex/Program.cs:line 23
2

Have you checked the port usage with powershell?

3

By the time I get to the box to look at it, I’m not showing anything listening on that port. I have searched through the event logs, though and don’t see any IIS process that says it started on that port.

I had the app pool set to be OnDemand prior to this, so I’ve set it to AlwaysRunning so we will see if that helps as well. I’m guessing maybe when it stops, it temporarily releases the port and then tries to get it back and it’s not available?

4

Yes, sounds like a good idea to turn OnDemand off. I mean nobody wants to have this behavior anyway

5

I’ll update you if the problem happens again. Just wanted to see if you maybe saw something else that could be causing this.

It’s 2 a.m. so I’m going back to bed for now :slight_smile:

1 Like
6

That helped for a while, but it all of a sudden happened again today. Site shutdown and then threw the same error. It seems it has to do with the Orleans service. It’s trying to grab a port that is already in use.

7

Thx, I am not using IIS myself anymore, so I do not have advices from production usage. Perhaps you can configure a grace period or so, so that in case of a restart it waits 2 minutes to restart. My only idea is that the process is still hanging in your use case, because Orleans takes the same port all the time. Perhaps you can find it in the task manager.