[NOT_REPRODUCIBLE] Content not allowing status changes

Bugs Stale or unverified Bugs
1

I have…

  • [x] Checked the logs and have uploaded a log file and provided a link because I found something suspicious there. Please do not post the log file in the topic because very often something important is missing.

I’m submitting a…

  • [ ] Regression (a behavior that stopped working in a new release)
  • [x] Bug report
  • [ ] Performance issue
  • [ ] Documentation issue or request

Current behavior

Have content that shows as ‘Published’ and I cannot change the version as it gives me an error stating it already exists. Weirdly in the ‘History’ it is still showing these status changes, yet ‘Inspect’ shows that these updates are not actually being applied.

image
image862×587 35.9 KB

Just found another example where some content is in Draft status and cannot have it’s status changed, so this is a general issue, not just changing from Published.

Expected behavior

I should be able to change the status of the content, or at least create a ‘New Draft’.

Minimal reproduction of the problem

I have no idea how it got into this odd state. Apparently this is happening consistently for all content however only in our Live environment so I cannot test it that vigorously.

Environment

  • [x] Self hosted with docker
  • [ ] Self hosted with IIS
  • [ ] Self hosted with other version
  • [ ] Cloud version

Version: 6.9.0

Browser:

  • [x] Chrome (desktop)
  • [ ] Chrome (Android)
  • [ ] Chrome (iOS)
  • [x] Firefox
  • [ ] Safari (desktop)
  • [ ] Safari (iOS)
  • [ ] IE
  • [ ] Edge

Others:
This has gotten us quite perplexed!

From looking at the code I think that error message is only displayed when Upsert decides to Insert instead of Update, but obviously in this context that doesn’t make sense as the content clearly does already exist.

Log entries:

Squidex.Infrastructure.States.InconsistentStateException: Failed to repair snapshot for domain object of type Squidex.Domain.Apps.Entities.Contents.DomainObject.ContentDomainObject with ID 00afb0e2-f6fd-4a89-9e34-5ee9a39cfd98–10f312ae-d786-45fc-b473-0958f184e06f.
at Squidex.Infrastructure.MongoDb.MongoExtensions.UpsertVersionedAsync[T,TKey](IMongoCollection1 collection, TKey key, Int64 oldVersion, Int64 newVersion, T document, CancellationToken ct) in C:\src\src\Squidex.Infrastructure.MongoDb\MongoDb\MongoExtensions.cs:line 150 at Squidex.Domain.Apps.Entities.MongoDb.Contents.MongoContentRepository.UpsertDraftContentAsync(SnapshotWriteJob1 job, CancellationToken ct) in C:\src\src\Squidex.Domain.Apps.Entities.MongoDb\Contents\MongoContentRepository_SnapshotStore.cs:line 140
at Squidex.Domain.Apps.Entities.MongoDb.Contents.MongoContentRepository.Squidex.Infrastructure.States.ISnapshotStore<Squidex.Domain.Apps.Entities.Contents.DomainObject.ContentDomainObject.State>.WriteAsync(SnapshotWriteJob1 job, CancellationToken ct) in C:\src\src\Squidex.Domain.Apps.Entities.MongoDb\Contents\MongoContentRepository_SnapshotStore.cs:line 84 at Squidex.Infrastructure.States.Persistence1.WriteSnapshotAsync(T state, CancellationToken ct) in C:\src\src\Squidex.Infrastructure\States\Persistence.cs:line 192
at Squidex.Infrastructure.Commands.DomainObject`1.ReadAsync() in C:\src\src\Squidex.Infrastructure\Commands\DomainObject.cs:line 347

and then:

Squidex.Infrastructure.DomainObjectConflictException: Entity (00afb0e2-f6fd-4a89-9e34-5ee9a39cfd98–10f312ae-d786-45fc-b473-0958f184e06f) already exists.
at Squidex.Infrastructure.Commands.DomainObject1.UpsertCoreAsync[TCommand](TCommand command, Func2 handler, Boolean isCreation) in C:\src\src\Squidex.Infrastructure\Commands\DomainObject.cs:line 252
at Squidex.Infrastructure.Commands.DomainObject1.UpdateReturnAsync[TCommand](TCommand command, Func2 handler) in C:\src\src\Squidex.Infrastructure\Commands\DomainObject.Execute.cs:line 55
at Squidex.Infrastructure.Commands.DomainObjectGrain2.ExecuteAsync(J1 request) in C:\src\src\Squidex.Infrastructure\Commands\DomainObjectGrain.cs:line 45
at Squidex.Domain.Apps.Entities.Contents.DomainObject.OrleansCodeGenContentGrainMethodInvoker.Invoke(IAddressable grain, InvokeMethodRequest request) in C:\src\src\Squidex.Domain.Apps.Entities\obj\Release\net6.0\Squidex.Domain.Apps.Entities.orleans.g.cs:line 1485
at Orleans.Runtime.GrainMethodInvoker.Invoke() in //src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 104
at Squidex.Infrastructure.Orleans.StateFilter.Invoke(IIncomingGrainCallContext context) in C:\src\src\Squidex.Infrastructure\Orleans\StateFilter.cs:line 48
at Orleans.Runtime.GrainMethodInvoker.Invoke() in //src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 104
at Squidex.Infrastructure.Orleans.LoggingFilter.Invoke(IIncomingGrainCallContext context) in C:\src\src\Squidex.Infrastructure\Orleans\LoggingFilter.cs:line 47
at Orleans.Runtime.GrainMethodInvoker.Invoke() in //src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 104
at Squidex.Infrastructure.Orleans.LocalCacheFilter.Invoke(IIncomingGrainCallContext context) in C:\src\src\Squidex.Infrastructure\Orleans\LocalCacheFilter.cs:line 35
at Orleans.Runtime.GrainMethodInvoker.Invoke() in //src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 104
at Orleans.Runtime.GrainMethodInvoker.Invoke() in //src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 104
at Squidex.Infrastructure.Orleans.ActivityPropagationGrainCallFilter.ProcessNewActivity(IGrainCallContext context, String activityName, ActivityKind activityKind, ActivityContext activityContext) in C:\src\src\Squidex.Infrastructure\Orleans\ActivityPropagationGrainCallFilter.cs:line 69
at Orleans.Runtime.GrainMethodInvoker.Invoke() in //src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 104
at Squidex.Infrastructure.Orleans.ExceptionWrapperFilter.Invoke(IIncomingGrainCallContext context) in C:\src\src\Squidex.Infrastructure\Orleans\ExceptionWrapperFilter.cs:line 33
at Orleans.Runtime.GrainMethodInvoker.Invoke() in //src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 104
at OrleansDashboard.Metrics.GrainProfilerFilter.Invoke(IIncomingGrainCallContext context)
at Orleans.Runtime.GrainMethodInvoker.Invoke() in //src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 104
at Orleans.Runtime.InsideRuntimeClient.Invoke(IAddressable target, IInvokable invokable, Message message) in //src/Orleans.Runtime/Core/InsideRuntimeClient.cs:line 469
at Orleans.Runtime.OutgoingCallInvoker.Invoke() in //src/Orleans.Core/Runtime/OutgoingCallInvoker.cs:line 113
at Orleans.Runtime.OutgoingCallInvoker.Invoke() in //src/Orleans.Core/Runtime/OutgoingCallInvoker.cs:line 113
at Orleans.Runtime.GrainReferenceRuntime.InvokeWithFilters(GrainReference reference, InvokeMethodRequest request, InvokeMethodOptions options) in //src/Orleans.Core/Runtime/GrainReferenceRuntime.cs:line 116
at Orleans.Internal.OrleansTaskExtentions.g__ConvertAsync|4_0[T](Task1 asyncTask) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 114 at Squidex.Infrastructure.Commands.GrainCommandMiddleware2.ExecuteCommandAsync(TCommand typedCommand) in C:\src\src\Squidex.Infrastructure\Commands\GrainCommandMiddleware.cs:line 49
at Squidex.Infrastructure.Commands.GrainCommandMiddleware2.ExecuteCommandAsync(CommandContext context) in C:\src\src\Squidex.Infrastructure\Commands\GrainCommandMiddleware.cs:line 32 at Squidex.Infrastructure.Commands.GrainCommandMiddleware2.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Infrastructure\Commands\GrainCommandMiddleware.cs:line 23
at Squidex.Domain.Apps.Entities.Contents.DomainObject.ContentsBulkUpdateCommandMiddleware.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Domain.Apps.Entities\Contents\DomainObject\ContentsBulkUpdateCommandMiddleware.cs:line 136
at Squidex.Infrastructure.Commands.GrainCommandMiddleware2.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Infrastructure\Commands\GrainCommandMiddleware.cs:line 25 at Squidex.Domain.Apps.Entities.Assets.DomainObject.AssetCommandMiddleware.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Domain.Apps.Entities\Assets\DomainObject\AssetCommandMiddleware.cs:line 62 at Squidex.Domain.Apps.Entities.Assets.DomainObject.AssetsBulkUpdateCommandMiddleware.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Domain.Apps.Entities\Assets\DomainObject\AssetsBulkUpdateCommandMiddleware.cs:line 127 at Squidex.Infrastructure.Commands.GrainCommandMiddleware2.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Infrastructure\Commands\GrainCommandMiddleware.cs:line 25
at Squidex.Domain.Apps.Entities.Apps.DomainObject.AppCommandMiddleware.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Domain.Apps.Entities\Apps\DomainObject\AppCommandMiddleware.cs:line 42
at Squidex.Domain.Apps.Entities.Schemas.Indexes.SchemasIndex.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Domain.Apps.Entities\Schemas\Indexes\SchemasIndex.cs:line 134
at Squidex.Domain.Apps.Entities.Rules.Indexes.RulesIndex.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Domain.Apps.Entities\Rules\Indexes\RulesIndex.cs:line 50
at Squidex.Domain.Apps.Entities.Apps.Indexes.AppsIndex.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Domain.Apps.Entities\Apps\Indexes\AppsIndex.cs:line 172
at Squidex.Domain.Apps.Entities.Apps.Invitation.InviteUserCommandMiddleware.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Domain.Apps.Entities\Apps\Invitation\InviteUserCommandMiddleware.cs:line 55
at Squidex.Domain.Apps.Entities.Apps.Plans.RestrictAppsCommandMiddleware.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Domain.Apps.Entities\Apps\Plans\RestrictAppsCommandMiddleware.cs:line 34
at Squidex.Domain.Apps.Entities.Apps.AlwaysCreateClientCommandMiddleware.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Domain.Apps.Entities\Apps\AlwaysCreateClientCommandMiddleware.cs:line 19
at Squidex.Domain.Apps.Entities.Apps.Templates.TemplateCommandMiddleware.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Domain.Apps.Entities\Apps\Templates\TemplateCommandMiddleware.cs:line 49
at Squidex.Infrastructure.Commands.CustomCommandMiddlewareRunner.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Infrastructure\Commands\CustomCommandMiddlewareRunner.cs:line 26
at Squidex.Web.CommandMiddlewares.ETagCommandMiddleware.HandleAsync(CommandContext context, NextDelegate next) in C:\src\src\Squidex.Web\CommandMiddlewares\ETagCommandMiddleware.cs:line 51
at Squidex.Infrastructure.Commands.InMemoryCommandBus.PublishAsync(ICommand command) in C:\src\src\Squidex.Infrastructure\Commands\InMemoryCommandBus.cs:line 65
at Squidex.Domain.Apps.Entities.Contents.DomainObject.ContentsBulkUpdateCommandMiddleware.ExecuteCommandAsync(BulkTaskCommand bulkCommand) in C:\src\src\Squidex.Domain.Apps.Entities\Contents\DomainObject\ContentsBulkUpdateCommandMiddleware.cs:line 146

I’d be grateful for any ideas or solutions you may have, and please do let us know if there is any other information we can provide to help narrow down the issue. Atm the only next step I can think of is deleting and recreating the content, but this problem could still persist. Also as second error message mentions Orleans I will use this as more leverage to convince others we should migrate to 7.x asap…

2

I can have a look, but I expect that it will be difficult without the original database.

3

So far I cannot reproduce it

4

Thanks for trying, I have the feeling it was some kind of random one off issue, perhaps edits were made during a deployment or something. Will have another look and see if I can get it reproducible with different content. If not then I will aim to find the events for the content in Mongo and see if they shed any light on the situation.

1 Like
5