Migration Freezing at RebuildContents

Support
1

I have…

  • [ x ] Checked the logs and have uploaded a log file and provided a link because I found something suspicious there. Please do not post the log file in the topic because very often something important is missing.

I’m submitting a…

  • [ x ] Regression (a behavior that stopped working in a new release)
  • [ ] Bug report
  • [ ] Performance issue
  • [ ] Documentation issue or request

Current behavior

Hi Sebastian,

I have spent over a week trying to investigate this issue and not entirely sure if it is a bug or not, but wondering if you have seen anything similar before or know anything that might help?

We are attempting to deploy the latest Master of Squidex (version 25 of migration) to our application that has 4 million records (running on version 21 of migration),

When we run the deployment on a pre-live environment, the migration starts and when it gets to the ‘RebuildContents’ step it rebuilt 300 contents before getting stuck and throwing the following logs (it doesn’t actually fail the migration, but it doesn’t continue rebuilding contents):

Migration RebuildContents Issue
Migration RebuildContents Issue3679×124 34 KB

(Edit:

I just noticed that there is an error in that failed orleans log:

image
image1664×35 3.37 KB

Still not sure however how one would rectify this

)

This issue has happened at multiple stages of the migration for this same step.

We tried to reproduce locally but can’t seem to get the same behaviour. This made us think this might be an environmental problem with Kubernetes or Mongo Atlas, but it seems like it could be a code issue since it failed after just 300 contents on pre-live.

Any help would be appreciated please, could this be an issue with Orleans during the migration? or is it more likely an environmental issue?

Thanks,
Jason

Expected behavior

Migration goes from version 21 to 25 without freezing and all 4 million contents items are migrated successfully.

Minimal reproduction of the problem

We have only managed to reproduce this on a pre-live environment. We can’t seem to reproduce this issue locally with the same amount of data.

Environment

  • [ x ] Self hosted with docker
  • [ ] Self hosted with IIS
  • [ ] Self hosted with other version
  • [ ] Cloud version

Version: [VERSION]

Browser:

  • [ x ] Chrome (desktop)
  • [ ] Chrome (Android)
  • [ ] Chrome (iOS)
  • [ ] Firefox
  • [ ] Safari (desktop)
  • [ ] Safari (iOS)
  • [ ] IE
  • [ ] Edge

Others:

2

It seems that you have not read the hints how to write a good support package, because posting screenshots for logs is a horrible idea :smiley:

Besides that: Your database is corrupt. The exception says (I wish I could copy and paste :wink: ) that you events are in non-consecutive order, for example, meaning there are events missing in the chain or the order is fucked up.

I have never seen this before, but you have to fix this.

3

No worries, feel free to close the thread and i’ll continue to investigate