Forbidden error on startup when hosting in Kubernetes in Azure

Andrew · November 18, 2021, 3:22pm

Hello

We’ve been successfully running Squidex v5.7.0 in Kubernetes in Azure for a few months now. But we’ve hit problems when creating a new deployment using the latest helm charts (appVersion 5.9.0). The squidex container won’t start and we are seeing the following in the logs:

{"logLevel":"Error","message":"Exception during Silo.Start","eventId":    {"id":100439},"timestamp":"2021-11-18T15:06:13Z","app":{"name":"Squidex","version":"5.9.0.0","sessionId":"1f7b775b-f319-49e7-a7f0-caa60cc1c4b1"},"category":"Orleans.Runtime.Silo","exception":{"type":"Microsoft.Rest.HttpOperationException","message":"Operation returned an invalid status code \u0027Forbidden\u0027","stackTrace":"   at k8s.Kubernetes.ListNamespacedPodWithHttpMessagesAsync(String namespaceParameter, Nullable\u00601 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable\u00601 limit, String resourceVersion, Nullable\u00601 timeoutSeconds, Nullable\u00601 watch, String pretty, Dictionary\u00602 customHeaders, CancellationToken cancellationToken)\n   at k8s.KubernetesExtensions.ListNamespacedPodAsync(IKubernetes operations, String namespaceParameter, Nullable\u00601 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable\u00601 limit, String resourceVersion, Nullable\u00601 timeoutSeconds, Nullable\u00601 watch, String pretty, CancellationToken cancellationToken)\n   at Orleans.Hosting.Kubernetes.KubernetesClusterAgent.OnRuntimeInitializeStart(CancellationToken cancellation)\n   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)\n   at Orleans.LifecycleSubject.OnStart(CancellationToken ct)\n   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()\n   at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken)"}}
Unhandled exception. Microsoft.Rest.HttpOperationException: Operation returned an invalid status code 'Forbidden'
   at k8s.Kubernetes.ListNamespacedPodWithHttpMessagesAsync(String namespaceParameter, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.KubernetesExtensions.ListNamespacedPodAsync(IKubernetes operations, String namespaceParameter, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, CancellationToken cancellationToken)
   at Orleans.Hosting.Kubernetes.KubernetesClusterAgent.OnRuntimeInitializeStart(CancellationToken cancellation)
   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)
   at Orleans.LifecycleSubject.OnStart(CancellationToken ct)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken)
   at Orleans.Hosting.SiloHost.StartAsync(CancellationToken cancellationToken)
   at Orleans.Hosting.SiloHostedService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Squidex.Program.Main(String[] args) in /src/src/Squidex/Program.cs:line 22

We’re a bit stumped as to what is causing this/how to fix.

Thanks
Andrew

Sebastian · November 18, 2021, 4:43pm

It is this setting in the helm chart that activates kubernetes support:

github.com

Squidex/squidex/blob/master/backend/src/Squidex/appsettings.json#L380



"orleans": {
    // Define the clustering type.
    // 
    // SUPPORTED: MongoDB, Development
    "clustering": "Development",

    // Tell Orleans it is running in kubernetes.
    // 
    // Read more: https://dotnet.github.io/orleans/docs/deployment/kubernetes.html?q=kubernetes
    "kubernetes": false,

    // The port is used to share messages between all cluster members. Must be accessible within your cluster or network.
    "siloPort": "11111",

    // The ports used by Orleans to connect to external clients. Not used.
    "gatewayPort": "40000",

    // The advertised IP address. Usually not needed.
    "ipAddress": "",

You can turn it off with

ORLEANS__KUBERNETES=false

or give the permissions: