Full sync from integrations

arnfasto · February 21, 2020, 1:05pm

Is there a way to force a full rerun of rules on all schemas?

Let’s say i’ve setup a webhook rule, but need to do a full load before receiving incremental changes, is it possible?

Sebastian · February 21, 2020, 4:01pm

When you self host you can rerun the rule enqueuer. But there is a hard coded constant to ignore old events:

Squidex/squidex/blob/master/backend/src/Squidex.Domain.Apps.Core.Operations/HandleRules/RuleService.cs#L112




var now = clock.GetCurrentInstant();


var eventTime =
    @event.Headers.ContainsKey(CommonHeaders.Timestamp) ?
    @event.Headers.Timestamp() :
    now;


var expires = eventTime.Plus(Constants.ExpirationTime);


if (eventTime.Plus(Constants.StaleTime) < now)
{
    return result;
}


if (!triggerHandler.Trigger(typed.Payload, rule.Trigger, ruleId))
{
    return result;
}


var appEventEnvelope = @event.To<AppEvent>();

You could provide a PR to move this setting to the rule options.

arnfasto · February 24, 2020, 3:37pm

Thanks for the reply, my suggestion would be to add a button to push all models current state through the rules and to force a full sync. Is that your idea as well ?

Sebastian · February 24, 2020, 3:38pm

The button is already there for self hosting:

You can just click the “Restart” button for the RuleEnqueuer to replay all events, but this ignores all old events, therefore we just need the setting.

arnfasto · February 27, 2020, 6:29pm

I only need the latest version of the object, so i guess that would fit my need

arnfasto · June 22, 2020, 11:42am

Hi Sebastian

I tried what you suggested, but getting this error here:

MongoDB.Driver.MongoCommandException: Command find failed: Executor error during find command :: caused by :: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit..
   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.ProcessResponse(ConnectionId connectionId, CommandMessage responseMessage)
   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocolAsync[TResult](IWireProtocol`1 protocol, ICoreSession session, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableReadOperationExecutor.ExecuteAsync[TResult](IRetryableReadOperation`1 operation, RetryableReadContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.ExecuteAsync(RetryableReadContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.FindCommandOperation`1.ExecuteAsync(RetryableReadContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.FindOperation`1.ExecuteAsync(RetryableReadContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.FindOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
   at Squidex.Infrastructure.MongoDb.MongoExtensions.ForEachPipelineAsync[TDocument](IAsyncCursorSource`1 source, Func`2 processor, CancellationToken cancellationToken) in /src/src/Squidex.Infrastructure.MongoDb/MongoDb/MongoExtensions.cs:line 183
   at Squidex.Infrastructure.EventSourcing.MongoEventStore.QueryAsync(Func`2 callback, StreamPosition lastPosition, FilterDefinition`1 filterDefinition, EventPredicate filterExpression, CancellationToken ct) in /src/src/Squidex.Infrastructure.MongoDb/EventSourcing/MongoEventStore_Reader.cs:line 187
   at Squidex.Infrastructure.EventSourcing.PollingSubscription.<>c__DisplayClass1_0.<<-ctor>b__0>d.MoveNext() in /src/src/Squidex.Infrastructure/EventSourcing/PollingSubscription.cs:line 45

Could you advice me what i need to do?

Sebastian · June 22, 2020, 12:24pm

This has been solved in the current dev version.

arnfasto · June 22, 2020, 12:35pm

Ahh okay cool thanks ! do you know when there will be a release version of it? or potentially something that we can do short term?

Sebastian · June 22, 2020, 1:56pm

You need a ascending index on the timestamp field.