[IMPLEMENTED PARTLY] Migrate schemas and rules between apps

#1

Hi, We’ve been struggeling with setting up Squidex in our environment which has seperate dev,test,staging and production environments.

Ideally i would like to have a different app (or Squidex server) per environment. But migrating changes to our schema’s and rules between these environments is quite a difficult task.

Would a system that links the Squidex apps be a solution for this? For example:

  • Do some changes to the schema’s of website-app-development
  • Have a linked website-app-test which schemas cannot be changed directly
  • Have a controlled way to run the changes that we have done to the development version to the test environment
  • Etc…
#2

Sounds like a useful feature. I am also working on the CLI from time to time and this is something the CLI could handle. A simple command like schema sync woujld be fine I guess.

And the same for rules of course.

#3

That would also be a brilliant solution! That could be triggered by our deployment tools.

I kind of compare the challenge with migrations in Entity Framework. With deployment of a version of my front-end i’m also, somehow, able to define the schema version that should come with it in Squidex.

#4

I have started with the sync logic a little bit: https://github.com/Squidex/squidex-samples/blob/sync/cli/Squidex.CLI/Squidex.CLI/Commands/App_Schemas.cs#L70

With the CLI you can manage your configs, so you could use it like this:

sq config add ... -label myapp-prod
sq config add ... -label myapp-dev

// Use dev config
sq config use myapp-dev
// Save schema in file
sq schema get my-schema >> temp.json
// Switch to ´production
sq config use myapp-prod
// Sync schema in
sq schema sync temp.json

Do you want to continue and provide a PR? :wink:

#5

Man you’re fast! I hope to find some time this week to follow up.

#6

But it is not ready yet, I just started with the development. But the idea should be clear.

#7

It will be released this week.

#8

Looking forward to it :-). We’ve been struggeling to find time to contribute to this.

#9

No problem. I solved it in a totally different way now. it is integrated directly into the api.

#10

Hi, there is a no sync endpoint: https://cloud.squidex.io/api/docs#operation/Schemas_PutSchemaSync

Here is how to you use it: https://github.com/Squidex/squidex-samples/blob/master/cli/Squidex.CLI/Squidex.CLI/Commands/App_Schemas.cs#L69

#11

Hey @Sebastian

How does this API call work?

It looks like it will create the schema if it doesn’t exist on the target, which is great. How is it picking up the base application? I don’t see where that is specified.

I am new to the CLI and haven’t even got it running yet, so appreciate your patience. Any documentation you may have to get started with it would be great.

#12

Hi @blakecodes

the API works on specific schema, so it must exist. If it does not exist, you have to create a new schema first, but the json schema for the request body is the same.

So what you can do:

  1. Read schema1 and save the json to file.
  2. Read file and forward it to schema2 on the same or another app.

Squidex will then make all the changes that are necessary to keep both schemas in sync.

The CLI manages connections to multiple apps. First you have to register the app:

sq config add <app> <app:client> <secret>

You can switch between apps with

sq config use <app>

Then you have to download the schema from one app and store it to a file, e.g.

// Connect to app1
sq config use app1
// Download schema1 from app1
sq schemas get schema1 > schema.json
// Connect to app2
sq config use app2
// Sync the schema with the schema name from the json
sq schemas sync schema.json
// OR use another schema name
sq schemas sync schema.json -name schema2

Documentation is on my todo list, but I wanted to wait for the first responses about the CLI first.

#13

@Sebastian

I’ll give it a shot over the weekend, thanks for the insight.

#14

Do you have an example .configuration I can use?

#15

What configuration do you mean?

#16

If you click the connect button for a client you will get a dialog with a command to connect the cli to the app with this client:

#17

Wow, never seen that. Awesome!

#18

Hi Sebastian,

Works like a charm. We’ve packed the CLI tool in an internal Nuget package which is included in our Visual Studio projects. This way we can easily update a local copy of the schemas in our solution. This way we always have a matching schema in json within our solution.

On deployment using Octopus Deploy we use these json files to update the corresponding Squidex app along with installing our webapp.

Bottom line. Changes to schemas are now part of our regular automated deployment process. Many thanks for that!

On thing i’ve noticed is that fields with references are not transfered proberly (because the schemaId will be different). It’s not such a problem to change these afterwards. But downside is that manual changes will be overwritten with the next sync. Any thoughts or directions to fix this?

Many thanks,
Marcel

#19

Yes, I am aware of the problem with the schema id. It is on my todo list :wink:

The problem is that right now the CLI tool does not have the necessary information. It either needs all schemas from the source and target app or the name of the referenced schemas. I tend to use the second approach.

#20

Using the name should work making sure not to overwrite manual changes.

Something like:

  • If name is found in target > reference that target schema
  • If name is not found in target but other schema is referenced in target > leave reference as it is (might be manually corrected)
  • Else target = null.