Error in DigitalOcean Kubernetes Cluster

philippdrebes · July 22, 2019, 8:58pm

Hey,

for a while now I’ve been trying to run Squidex in a kubernetes cluster on DigitalOcean. The pod always starts with the following error An error occurred while reading the key ring. Also there are a lot of those errors A timeout occured after 30000ms selecting a server using CompositeServerSelector.... After a while the app shuts down and the pod is recreated. The mongodb log shows an accepted connection.

To run Squidex in k8s I’ve copied the example from the squidex-docker github repo and only changed some names, env vars and the storageClass. The link below contains the mongo.yaml, squidex.yaml and the complete log from the Squidex pod.

I hope someone can help as I’m running out of ideas.

Thanks,
Philipp

gist.github.com

https://gist.github.com/philippdrebes/1f7009627f7049f7d01fe9d5587010fb

mongo.yaml

apiVersion: v1
kind: Service
metadata:
  name: squidex-mongo
  labels:
    name: squidex-mongo
spec:
  ports:
  - port: 27017
    targetPort: 27017

This file has been truncated. show original

squidex.yaml

kind: Secret
apiVersion: v1
metadata:
  name: squidex-secrets
stringData:
  IDENTITY__ADMINEMAIL: "my-email@gmail.com"
  IDENTITY__ADMINPASSWORD: "some-password"
  IDENTITY__GOOGLECLIENT: "clientid.apps.googleusercontent.com"
  IDENTITY__GOOGLESECRET: "my-secret"
---

This file has been truncated. show original

squidex_pod_log.json

{
    "logLevel": "Error",
    "message": "An error occurred while reading the key ring.",
    "eventId": {
      "id": 48
    },
    "exception": {
      "type": "System.AggregateException",
      "message": "One or more errors occurred. (A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : \"1\", ConnectionMode : \"Automatic\", Type : \"ReplicaSet\", State : \"Connected\", Servers : [{ ServerId: \"{ ClusterId : 1, EndPoint : \"Unspecified/squidex-mongo-0.squidex-mongo:27017\" }\", EndPoint: \"Unspecified/squidex-mongo-0.squidex-mongo:27017\", State: \"Connected\", Type: \"ReplicaSetGhost\", WireVersionRange: \"[0, 5]\", LastUpdateTimestamp: \"2019-07-22T16:58:02.9843129Z\" }, { ServerId: \"{ ClusterId : 1, EndPoint : \"Unspecified/squidex-mongo-1.squidex-mongo:27017\" }\", EndPoint: \"Unspecified/squidex-mongo-1.squidex-mongo:27017\", State: \"Connected\", Type: \"ReplicaSetGhost\", WireVersionRange: \"[0, 5]\", LastUpdateTimestamp: \"2019-07-22T16:58:02.6639114Z\" }, { ServerId: \"{ ClusterId : 1, EndPoint : \"Unspecified/squidex-mongo-2.squidex-mongo:27017\" }\", EndPoint: \"Unspecified/squidex-mongo-2.squidex-mongo:27017\", State: \"Connected\", Type: \"ReplicaSetGhost\", WireVersionRange: \"[0, 5]\", LastUpdateTimestamp: \"2019-07-22T16:58:02.7141994Z\" }] }.)",
      "stackTrace": "   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)\n   at Squidex.Domain.Users.DefaultXmlRepository.GetAllElements() in /src/src/Squidex.Domain.Users/DefaultXmlRepository.cs:line 45\n   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.GetAllKeys()\n   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(DateTimeOffset now, IKey keyJustAdded)\n   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow)"

This file has been truncated. show original

Sebastian · July 22, 2019, 9:32pm

Looks like it cannot connect to MongoDB.

philippdrebes · July 23, 2019, 6:42pm

Thanks, this helped me to pinpoint the problem.
The mongodb replicaset would, for some reason, not initiate, so I used the mongodb helm chart in order to set up the db -> https://github.com/helm/charts/tree/master/stable/mongodb

Also, I had to change the connection strings in the Squidex appsettings to FQDN’s.