Web site down because of IIS Exception

lcotte · November 16, 2023, 3:55pm

I have…

[ x] Read the following guideline: https://docs.squidex.io/01-getting-started/installation/troubleshooting-and-support. I understand that my support request might get deleted if I do not follow the guideline.
[ x] Used code blocks with ``` to format my code examples like JSON or logs properly.

I’m submitting a…

Regression (a behavior that stopped working in a new release)
Bug report
Performance issue
Documentation issue or request

Current behavior

For some reason, IIS on our server has an exception that will freeze all web sites and requires us to restart the IIS service.

Expected behavior

Does not rewquire a restart of IIS

Minimal reproduction of the problem

Environment

App Name:

Self hosted with docker
Self hosted with IIS
Self hosted with other version
Cloud version

Version: 4.4.0

Browser:

Chrome (desktop)
Chrome (Android)
Chrome (iOS)
Firefox
Safari (desktop)
Safari (iOS)
IE
Edge

Others:
Here are some stack traces from the event viewer when iis crashes

Application: w3wp.exe
CoreCLR Version: 4.700.20.11803
.NET Core Version: 3.1.3
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.AspNetCore.Connections.AddressInUseException: Only one usage of each socket address (protocol/network address/port) is normally permitted.
 ---> System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted.
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Orleans.Networking.Shared.SocketConnectionListener.Bind()
   --- End of inner exception stack trace ---
   at Orleans.Networking.Shared.SocketConnectionListener.Bind()
   at Orleans.Networking.Shared.SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionListener.BindAsync(CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.SiloConnectionListener.OnRuntimeInitializeStart(CancellationToken cancellationToken)
   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)
   at Orleans.LifecycleSubject.OnStart(CancellationToken ct)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken)
   at Orleans.Hosting.SiloWrapper.StartAsync(CancellationToken cancellationToken)
   at Orleans.Hosting.SiloHostedService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Squidex.Program.Main(String[] args) in /src/src/Squidex/Program.cs:line 25

{
  "logLevel": "Error",
  "message": "GatewayConnectionListener failed to start due to errors at stage 1999: Microsoft.AspNetCore.Connections.AddressInUseException: Only one usage of each socket address (protocol/network address/port) is normally permitted.\r\n ---\u003E System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted.\r\n   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)\r\n   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)\r\n   at System.Net.Sockets.Socket.Bind(EndPoint localEP)\r\n   at Orleans.Networking.Shared.SocketConnectionListener.Bind()\r\n   --- End of inner exception stack trace ---\r\n   at Orleans.Networking.Shared.SocketConnectionListener.Bind()\r\n   at Orleans.Networking.Shared.SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)\r\n   at Orleans.Runtime.Messaging.ConnectionListener.BindAsync(CancellationToken cancellationToken)\r\n   at Orleans.Runtime.Messaging.GatewayConnectionListener.OnRuntimeInitializeStart(CancellationToken cancellationToken)\r\n   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)",
  "eventId": {
    "id": 100450
  },
  "name": "GatewayConnectionListener",
  "stage": "1999",
  "app": {
    "name": "Squidex",
    "version": "4.4.0.0",
    "sessionId": "353dcb0c-8157-402b-b180-4791bea344bd"
  },
  "timestamp": "2023-11-10T11:00:05Z",
  "category": "Orleans.Runtime.SiloLifecycleSubject",
  "exception": {
    "type": "Microsoft.AspNetCore.Connections.AddressInUseException",
    "message": "Only one usage of each socket address (protocol/network address/port) is normally permitted.",
    "stackTrace": "   at Orleans.Networking.Shared.SocketConnectionListener.Bind()\r\n   at Orleans.Networking.Shared.SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)\r\n   at Orleans.Runtime.Messaging.ConnectionListener.BindAsync(CancellationToken cancellationToken)\r\n   at Orleans.Runtime.Messaging.GatewayConnectionListener.OnRuntimeInitializeStart(CancellationToken cancellationToken)\r\n   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)"
  }
}

{
  "logLevel": "Error",
  "message": "SiloConnectionListener failed to start due to errors at stage 1999: Microsoft.AspNetCore.Connections.AddressInUseException: Only one usage of each socket address (protocol/network address/port) is normally permitted.\r\n ---\u003E System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted.\r\n   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)\r\n   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)\r\n   at System.Net.Sockets.Socket.Bind(EndPoint localEP)\r\n   at Orleans.Networking.Shared.SocketConnectionListener.Bind()\r\n   --- End of inner exception stack trace ---\r\n   at Orleans.Networking.Shared.SocketConnectionListener.Bind()\r\n   at Orleans.Networking.Shared.SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)\r\n   at Orleans.Runtime.Messaging.ConnectionListener.BindAsync(CancellationToken cancellationToken)\r\n   at Orleans.Runtime.Messaging.SiloConnectionListener.OnRuntimeInitializeStart(CancellationToken cancellationToken)\r\n   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)",
  "eventId": {
    "id": 100450
  },
  "name": "SiloConnectionListener",
  "stage": "1999",
  "app": {
    "name": "Squidex",
    "version": "4.4.0.0",
    "sessionId": "353dcb0c-8157-402b-b180-4791bea344bd"
  },
  "timestamp": "2023-11-10T11:00:05Z",
  "category": "Orleans.Runtime.SiloLifecycleSubject",
  "exception": {
    "type": "Microsoft.AspNetCore.Connections.AddressInUseException",
    "message": "Only one usage of each socket address (protocol/network address/port) is normally permitted.",
    "stackTrace": "   at Orleans.Networking.Shared.SocketConnectionListener.Bind()\r\n   at Orleans.Networking.Shared.SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)\r\n   at Orleans.Runtime.Messaging.ConnectionListener.BindAsync(CancellationToken cancellationToken)\r\n   at Orleans.Runtime.Messaging.SiloConnectionListener.OnRuntimeInitializeStart(CancellationToken cancellationToken)\r\n   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)"
  }
}

{
  "logLevel": "Error",
  "message": "Lifecycle start canceled due to errors at stage 1999: Microsoft.AspNetCore.Connections.AddressInUseException: Only one usage of each socket address (protocol/network address/port) is normally permitted.\r\n ---\u003E System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted.\r\n   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)\r\n   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)\r\n   at System.Net.Sockets.Socket.Bind(EndPoint localEP)\r\n   at Orleans.Networking.Shared.SocketConnectionListener.Bind()\r\n   --- End of inner exception stack trace ---\r\n   at Orleans.Networking.Shared.SocketConnectionListener.Bind()\r\n   at Orleans.Networking.Shared.SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)\r\n   at Orleans.Runtime.Messaging.ConnectionListener.BindAsync(CancellationToken cancellationToken)\r\n   at Orleans.Runtime.Messaging.SiloConnectionListener.OnRuntimeInitializeStart(CancellationToken cancellationToken)\r\n   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)\r\n   at Orleans.LifecycleSubject.OnStart(CancellationToken ct)",
  "eventId": {
    "id": 100450
  },
  "stage": "1999",
  "app": {
    "name": "Squidex",
    "version": "4.4.0.0",
    "sessionId": "353dcb0c-8157-402b-b180-4791bea344bd"
  },
  "timestamp": "2023-11-10T11:00:05Z",
  "category": "Orleans.Runtime.SiloLifecycleSubject",
  "exception": {
    "type": "Microsoft.AspNetCore.Connections.AddressInUseException",
    "message": "Only one usage of each socket address (protocol/network address/port) is normally permitted.",
    "stackTrace": "   at Orleans.Networking.Shared.SocketConnectionListener.Bind()\r\n   at Orleans.Networking.Shared.SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)\r\n   at Orleans.Runtime.Messaging.ConnectionListener.BindAsync(CancellationToken cancellationToken)\r\n   at Orleans.Runtime.Messaging.SiloConnectionListener.OnRuntimeInitializeStart(CancellationToken cancellationToken)\r\n   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)\r\n   at Orleans.LifecycleSubject.OnStart(CancellationToken ct)"
  }
}

{
  "logLevel": "Error",
  "message": "Exception during Silo.Start",
  "eventId": {
    "id": 100439
  },
  "app": {
    "name": "Squidex",
    "version": "4.4.0.0",
    "sessionId": "353dcb0c-8157-402b-b180-4791bea344bd"
  },
  "timestamp": "2023-11-10T11:00:05Z",
  "category": "Orleans.Runtime.Silo",
  "exception": {
    "type": "Microsoft.AspNetCore.Connections.AddressInUseException",
    "message": "Only one usage of each socket address (protocol/network address/port) is normally permitted.",
    "stackTrace": "   at Orleans.Networking.Shared.SocketConnectionListener.Bind()\r\n   at Orleans.Networking.Shared.SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)\r\n   at Orleans.Runtime.Messaging.ConnectionListener.BindAsync(CancellationToken cancellationToken)\r\n   at Orleans.Runtime.Messaging.SiloConnectionListener.OnRuntimeInitializeStart(CancellationToken cancellationToken)\r\n   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)\r\n   at Orleans.LifecycleSubject.OnStart(CancellationToken ct)\r\n   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()\r\n   at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken)"
  }
}

{
  "logLevel": "Error",
  "message": "QueueWorkItem was called on a non-null context [SystemTarget: S127.0.0.1:11111:437310003*stg/13/0000000d@S0000000d] but there is no valid WorkItemGroup for it.",
  "eventId": {
    "id": 101231
  },
  "app": {
    "name": "Squidex",
    "version": "4.4.0.0",
    "sessionId": "353dcb0c-8157-402b-b180-4791bea344bd"
  },
  "timestamp": "2023-11-10T11:00:05Z",
  "category": "Orleans.Runtime.Scheduler.OrleansTaskScheduler"
}

Unhandled exception. Microsoft.AspNetCore.Connections.AddressInUseException: Only one usage of each socket address (protocol/network address/port) is normally permitted.
 ---> System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted.
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Orleans.Networking.Shared.SocketConnectionListener.Bind()
   --- End of inner exception stack trace ---
   at Orleans.Networking.Shared.SocketConnectionListener.Bind()
   at Orleans.Networking.Shared.SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionListener.BindAsync(CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.SiloConnectionListener.OnRuntimeInitializeStart(CancellationToken cancellationToken)
   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)
   at Orleans.LifecycleSubject.OnStart(CancellationToken ct)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken)
   at Orleans.Hosting.SiloWrapper.StartAsync(CancellationToken cancellationToken)
   at Orleans.Hosting.SiloHostedService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Squidex.Program.Main(String[] args) in /src/src/Squidex/Program.cs:line 25

We realize that version 4.4.0 is very old and that we need to update to the latest version. We have tried in the past to update our Squidex incrementally, but we were not successful.

Do you think that updating Squdex to the latest version will solve our problem?
Do you have any recommendations on how to update from version 4.4.0 to version 7.9.0?

Sebastian · November 17, 2023, 11:03am

Probably not, I don’t understand why you have multiple applications listening on the same port.

I would just create a staging environment and try it out.