Rule Events's execution frequency

bourderalex · December 2, 2024, 9:37am

I have…

Read the following guideline: Rules | Squidex. I understand that my support request might get deleted if I do not follow the guideline.
Used code blocks with ``` to format my code examples like JSON or logs properly.

I’m submitting a…

Regression (a behavior that stopped working in a new release)
Bug report
Performance issue
Documentation issue or request

Current behavior

Hi, Sebastian. I have several rules configured to be executed on content change. Rules put events into Kafka. Sometimes I make several hundreds changes during several minutes via API. And I see rules being pended for about 10 seconds. E.g. - they seem to be proceeded not instantly, but after some timeout.

Expected behavior

I’d like to configure the frequency of proceeding rule events (if possible).
I’d like to know, is there some setting to control this period? I tried to set this environment to 0:

github.com

Squidex/squidex/blob/master/backend/src/Squidex/appsettings.json#L124


      
          "languages": {
              // Use custom languages where the key is the language code and the value is the english name.
              "custom": ""
          },
          
          "rules": {
              // The timeout to execute rule actions.
              "executionTimeoutInSeconds": 10,
          
              // The cache duration for rules.
              "rulesCacheDuration": "00:00:10"
          },
          
          "ui": {
              // Regex suggestions for the UI
              "regexSuggestions": {
                  // Regex for emails.
                  "Email": "^[a-zA-Z0-9.!#$%&’*+\\/=?^_`{|}~-]+@[a-zA-Z0-9-]+(?:.[a-zA-Z0-9-]+)*$",
                  // Regex for phone numbers.
                  "Phone": "^\\(*\\+*[1-9]{0,3}\\)*-*[1-9]{0,3}[-. /]*\\(*[2-9]\\d{2}\\)*[-. /]*\\d{3}[-. /]*\\d{4} *e*x*t*\\.* *\\d{0,4}$",
                  // Regex for slugs (e.g. hello-world).

But it seems not to make any sense.

Minimal reproduction of the problem

Configure rule to put event into Kafka on content Change. Make changes on approximately 5 contents every 2 seconds. And I’ll observe rules queuing (in state pending), but not proceeded instantly, after some time. Furthermore, it happens like a batch proceeding. I see queue being filled, and after a few seconds all rules switch into “Success” proceeding status.

Environment

App Name: content-catalog

Self hosted with docker
Self hosted with IIS
Self hosted with other version
Cloud version

Version: [7.9.0]

Browser:

Chrome (desktop)
Chrome (Android)
Chrome (iOS)
Firefox
Safari (desktop)
Safari (iOS)
IE
Edge

Others:

Sebastian · December 2, 2024, 9:04pm

Hi,

Squidex works in this way:

There is a listener who subscribes to events, the events do not necessarily show up instantly there.
The listener is creating enriched events and then jobs from these events.
Another listener takes these job from the database and executes them “one by one”. This is controlled by a timer and is triggered every 10 seconds. (It is not configurable).
The execution system has 32 logical partitions and each partition is executed sequentially. The partition key is usually derived from the entity of the entity.

All these polling things obviously increase the latency (and a little bit the throughput), but are needed for a reliability.

This means: Step 3 is the culprit here for latency. I think I can make that configurable. Furthermore. If you are unlucky most of your changes end up in one partition, which would decrease the throughput.

Sebastian · December 7, 2024, 6:20pm

Hi,
I have added a new setting to reduce the 10 second timer.