Event Sourcing (Events2 collection), mongodb size, DB is full

Support
1

I have…

I’m submitting a…

  • [ ] Regression (a behavior that stopped working in a new release)
  • [ ] Bug report
  • [ x ] Performance issue
  • [ x ] Documentation issue or request

Current behavior

Thank you for a good CMS!

Interested in the details of the Event Sourcing concept. The Events2 collection is taking up more and more space. I plan to support the project for years.
I have already changed the tariff plan 2 times to increase the volume of the database. And this is disturbing. Now the volume is 5 GB. Data size 2.3GB and continue to grow.

Is there any mechanism to clean up redundant change history?
For example, store history only for a while or clear it if necessary.
For the RequestLog collection, I see we can minimize the logging level or disable saving completely.

If I clear the Events2 collection, how will this affect the application?

Screenshot_277
Screenshot_2771903×989 108 KB

Expected behavior

Should continue to work and not fill the database indefinitely ofcourse))

Minimal reproduction of the problem

Lots of content updates for some time(less than a year)

Environment

App Name:

  • [ x ] Self hosted with docker
  • [ ] Self hosted with IIS
  • [ ] Self hosted with other version
  • [ ] Cloud version

Version: 6.4.0

Others:
Interested in links to other mechanisms for managing performance and data storage volume in squidex

2

Never ever delete Events2. Squidex is not designed to remove data from this collection. Today 5 GB is no size, just get more disk space.

I will provide a detailed answer later.

1 Like
3

Longer answer:

Architecture

Squidex was designed to never delete data. A few exceptions have been introduced (more about this later), but in general it is still the policy. The reason is that storage is just cheap and we get a lot of benefits.

For example when the database structure is changed we can completely rebuild everything from the events. Therefore events are never deleted and when you delete them manually a lot of things would not work anymore, because Squidex has some consistency checks when working with events.

So the basic tip is: Get a cheap server with 100GB storage and you are safe for a long time. If you store large documents as assets and not as contents it is very unlikely that ever allocate so much data that you will get an issue.

Exceptions

As mentioned above, there are a few exceptions, that come from the following requirements:

  1. User create test apps and then stop using them (especially in the cloud).
  2. User create apps for CI and automated testing regularly and don’t need them after the tests.
  3. Some data needs to be deleted permanently because of data privacy reasons.

The exceptions:

  1. When you delete and asset or content you can add a query string permanent=true to delete them and all events as well.
  2. You can set the following setting to true, to let Squidex remove all data for deleted apps in the background: https://github.com/Squidex/squidex/blob/master/backend/src/Squidex/appsettings.json
  3. You can set the following setting to true, to let Squidex delete asset files in the background when the asset object is deleted: https://github.com/Squidex/squidex/blob/master/backend/src/Squidex/appsettings.json#L256
  4. You can set the following setting to true, to let Squidex delete all child asset folders and assets in the background when a folder is deleted: https://github.com/Squidex/squidex/blob/master/backend/src/Squidex/appsettings.json#L253 (works together with the previous setting).
2 Likes