How to upgrade from Squidex 5.9.0 to 7.8.2

Hi Sebastian,

I’m preparing for an upgrade from Squidex version 5.9.0 to the latest stable Squidex version as of today which is 7.8.2.

You mentioned “single instance” in your latest reply of this question:

Yes, the changelog mentions long running migrations, that run automatically, but cause a temporary downtime. If you have a single instance there is not that much to do. Just upgrade the version in docker engine or kubernetes.

I was just wondering what exactly classifies as a “single instance”?

In my PROD Kubernetes setup I have the following resources for Squidex:

MongoDB:

apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  name: mongodb
  namespace: mongodb
spec:
  members: 3
  type: ReplicaSet
  version: "4.2.6"
  security:
    authentication:
      modes: ["SCRAM"]
  users:
    - name: REDACTED
      db: REDACTED
      passwordSecretRef:
        name: REDACTED
      roles:
        - name: REDACTED
          db: REDACTED
      scramCredentialsSecretName: REDACTED
  statefulSet:
    spec:
      template:
        spec:
          containers:
            - name: mongod
              resources:
                requests:
                  cpu: 50m
                  memory: 1Gi
                limits:
                  cpu: 2000m
                  memory: 4Gi
            - name: mongodb-agent
              resources:
                requests:
                  cpu: 50m
                  memory: 1Gi
                limits:
                  cpu: 2000m
                  memory: 2Gi

Squidex:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: squidex
  namespace: REDACTED
spec:
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      name: squidex
  template:
    metadata:
      labels:
        name: squidex
    spec:
      containers:
        - name: squidex
          image: squidex/squidex:5.9.0
          resources:
            requests:
              cpu: 40m
              memory: 700Mi
            limits:
              cpu: 1000m
              memory: 1Gi
          env:
            - name: URLS__BASEURL
              value: "REDACTED"
            - name: ASSETSTORE__TYPE
              value: MongoDb
            - name: ASSETSTORE__MONGODB__CONFIGURATION
              valueFrom:
                secretKeyRef:
                  name: REDACTED
                  key: REDACTED
            - name: EVENTSTORE__MONGODB__CONFIGURATION
              valueFrom:
                secretKeyRef:
                  name: REDACTED
                  key: REDACTED
            - name: STORE__MONGODB__CONFIGURATION
              valueFrom:
                secretKeyRef:
                  name: REDACTED
                  key: REDACTED
            - name: IDENTITY__ADMINEMAIL
              valueFrom:
                secretKeyRef:
                  name: REDACTED
                  key: REDACTED
            - name: IDENTITY__ADMINPASSWORD
              valueFrom:
                secretKeyRef:
                  name: REDACTED
                  key: REDACTED
            - name: ASSETS__MAXSIZE
              value: "52428800"
            - name: LOGGING__LEVEL
              value: "Error"
            - name: ROBOTS__TEXT
              value: "User-agent: *\nAllow: /api/assets/\nDisallow: /"

So as you can see I have 3 replicas for MongoDB. Would this mean I have “three instances” and not a “single instance” like you said?

I was just talking about the number of replicas for Squidex.

Do you mean the number of pods?

Yes, exactly…the replicas setting.

Okay.

I only use one pod for the Squidex deployment.

And 3 for Mongodb.

So I should be okay then.

1 Like

Hi Sebastian,

In my STAGING environment using my PRODUCTION database I’m performing a migration, i.e., I’m bumping my docker tag from 5.9.0 to 6.0.0.

When I apply my Kubernetes resource the Squidex pod terminates and a new one is created and the migration should start.

I can see in the Squidex database, in the Migration collection that the version has been updated from 25 to 26. IsLocked is set to false.

I then open up Squidex UI in my browser but I only see the default view and all my apps are gone; it says: You are not collaborating on any apps yet

I have 35 documents in the Squidex.States_Apps collection. And when I browse the documents the data seems intact.

Do you have any idea what might be going on? Why are my apps not showing? And how can I tell the migration process is done?

I got the following error:

{

  "logLevel": "Error",

  "message": "An unexpected exception has occurred.",

  "timestamp": "2023-10-22T21:10:12Z",

  "app": {

    "name": "Squidex",

    "version": "6.3.0.0",

    "sessionId": "509a304c-4a08-4dfc-804a-2ee300a00121"

  },

  "web": {

    "requestId": "00-b7c78ada414ba73cc353633f75b557ed-3567f819e5f1f91d-01",

    "requestPath": "/apps",

    "requestMethod": "GET",

    "routeValues": {

      "area": "Api",

      "action": "GetApps",

      "controller": "Apps"

    }

  },

  "exception": {

    "type": "System.ArgumentException",

    "message": "An item with the same key has already been added. Key: homepage",

    "stackTrace": "   at System.Collections.Generic.Dictionary\u00602.TryInsert(TKey key, TValue value, InsertionBehavior behavior)\n  

    at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable\u00601 source, Func\u00602 keySelector, Func\u00602 elementSelector, IEqualityComparer\u00601 comparer)\n  

    at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable\u00601 source, Func\u00602 keySelector, Func\u00602 elementSelector)\n  

    at Squidex.Domain.Apps.Entities.MongoDb.Apps.MongoAppRepository.QueryAsync(IFindFluent\u00602 find, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities.MongoDb/Apps/MongoAppRepository.cs:line 77\n  

    at Squidex.Domain.Apps.Entities.MongoDb.Apps.MongoAppRepository.QueryIdsAsync(String contributorId, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities.MongoDb/Apps/MongoAppRepository.cs:line 57\n  

    at Squidex.Domain.Apps.Entities.Apps.Indexes.AppsIndex.GetAppIdsByUserAsync(String userId) in /src/src/Squidex.Domain.Apps.Entities/Apps/Indexes/AppsIndex.cs:line 125\n  

    at Squidex.Domain.Apps.Entities.Apps.Indexes.AppsIndex.GetAppsForUserAsync(String userId, PermissionSet permissions, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/Apps/Indexes/AppsIndex.cs:line 59\n  

    at Squidex.Caching.LocalCacheExtensions.GetOrCreateAsync[T](ILocalCache cache, Object key, Func\u00601 task)\n  

    at Squidex.Domain.Apps.Entities.AppProvider.GetUserAppsAsync(String userId, PermissionSet permissions, CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/AppProvider.cs:line 147\n  

    at Squidex.Areas.Api.Controllers.Apps.AppsController.GetApps() in /src/src/Squidex/Areas/Api/Controllers/Apps/AppsController.cs:line 76\n  

    at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n  

    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeActionMethodAsync\u003Eg__Logged|12_1(ControllerActionInvoker invoker)\n  

    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeNextActionFilterAsync\u003Eg__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n  

    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State\u0026 next, Scope\u0026 scope, Object\u0026 state, Boolean\u0026 isCompleted)\n  

    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeInnerFilterAsync\u003Eg__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n  

    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeNextExceptionFilterAsync\u003Eg__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)"

  }

}

I also noticed that I had two apps in my Squidex.States_Apps collection with the same name, “homepage”, with different guids, but both had isDeleted: false which I think is incorrect. In the GUI I can obviously only see one app called “homepage”.

The query I used to query the collection:

{_an: 'homepage','Doc.isDeleted': false}

I manually edited the isDeleted property for the “homepage” app that should be in a deleted state.

I then upgraded to 6.4.0 and I got my apps back. But I think the fix was not the change I did in the database but this (or maybe both?):

6.4.0 Changelog notes:

  • Database: Fix to support duplicate entries in the database (database is corrupt).

Documenting all of this for my future self. Going to do some more testing tomorrow just to be sure everything is OK.

Sebastian, I shouldn’t need to manually change the isDeleted, correct? Only an upgrade to 6.4.0 should be enough, yes?

I tried the following and now everything seems to work:

  1. Recreate MongoDB StatefulSet
  2. Restore the db using mongorestore
  3. Apply squidex deployment using version 5.9.0
  4. Sanity check Squidex UI
  5. Apply squidex deployment using version 6.4.0
  6. Sanity check Squidex UI
  7. Everything looks OK

I had to skip the Squidex versions in between 5.9.0 and 6.4.0 because they were giving me issues (Apps failed to load, please try again)

1 Like

Okay. Last post.

I finally upgraded my PROD Squidex from 5.9.0 to 7.8.2 :partying_face:

Summarizing my last step for my future self and others that might run into the same issues:

This is what I did:

In dev I upgraded Squidex one minor version at a time. Then in PROD I skipped intermediary minor releases.

For 7.0.0 (Orleans removed) I made sure I only had 1 node in my Kubernetes cluster.

Upgrade order in DEV:

5.9.0 (Current)
6.4.0 OK
6.6.0 OK
6.7.0 OK
6.8.0 OK
6.9.0 OK
6.10.0 OK
6.11.0 OK
6.13.0 OK
7.0.0 (Use 1 node) OK
7.0.2 OK
7.0.3 OK
7.1.0 OK
7.2.0 OK
7.3.0 OK
7.4.0 OK
7.5.0 OK
7.6.0 OK
7.6.1 OK
7.7.0 OK
7.8.0 OK
7.8.1 OK
7.8.2 OK

Upgrade order in PROD:

5.9.0 (Current)
6.4.0 OK
6.13.0 OK
7.0.0 (Use 1 node) OK
7.8.2 OK

Sebastian, for peace of mind, I would like to know if still need to worry about that duplicate key?

"message": "An item with the same key has already been added. Key: homepage",

Where is this message coming from? There was really no need to go from one version to another. But if it makes you feel better…

Yes, I know. But I like to upgrade and test my stuff gradually. It’s a built-in mechanism that I have :wink:

I’ve written all the details in this post: How to upgrade from Squidex 5.9.0 to 7.8.2

1 Like

Sebastian, it should be okay to delete the following collections now that I’m on 7.8.2?

  • Orleans_OrleansMembershipV2
  • Orleans_OrleansReminderV2

Yes, you can remove them. But there are like 20 items or so in the collections.

Okay, great. Will delete them then. I like when all my things are nice and tidy. :wink:

Thanks for all the support btw, really appreciate it.

1 Like

Sebastian, should I delete the duplicate document? Or set ‘Doc.isDeleted’: false? Or should I just leave it be?

It’s this post here: How to upgrade from Squidex 5.9.0 to 7.8.2

As you wish. I would mark it deleted.

Okay. Will just update the document then. Thanks.

Just for the record, I had to update the _dl field as well.
If you only change isDeleted the UI will not let you click on the app and you will get an error in the devtools console.

1 Like

Are you getting HTTP 408 Error sometimes?

Don’t think so.
Where?