[SOLVED] Container upgrade deleted all S3 assets

I have…

I’m submitting a…

  • [ ] Regression (a behavior that stopped working in a new release)
  • [ x] Bug report
  • [ ] Performance issue
  • [ ] Documentation issue or request

Current behavior

Was running container tag “dev-6578” in AWS ECS using S3 asset storage. Created a new task definition to use “6.5.0” and launched. All S3 assets have been deleted from S3. Squidex has record of them in the db but no asset content.

Expected behavior

Launching new container version would retain S3 assets.

Minimal reproduction of the problem

  1. Launch ECS using docker.io/squidex/squidex:dev-6578 (using S3 asset storage).
  2. Upload assets, confirm files in S3
  3. Create new task revision, modify only the image name docker.io/squidex/squidex:6.5.0
  4. Open existing app and view Assets.
  5. Confirm in S3 bucket that assets are no longer there.

Environment

App Name: [self hosted app]

  • [x ] Self hosted with docker
  • [ ] Self hosted with IIS
  • [ ] Self hosted with other version
  • [ ] Cloud version

Version: 6.5

Browser:

  • [x ] Chrome (desktop)
  • [ ] Chrome (Android)
  • [ ] Chrome (iOS)
  • [ ] Firefox
  • [ ] Safari (desktop)
  • [ ] Safari (iOS)
  • [ ] IE
  • [ ] Edge

Others:
Relevant Environment Variables:
ASSETS__DELETEPERMANENT - true
ASSETS__DELETERECURSIVE - true
ASSETS__FOLDERPERAPP - true
ASSETSTORE__TYPE - AmazonS3

Can you do a few things:

  1. If you go to Profile -> Administration -> EventConsumers you should see that there is an entry with the name AssetPermanentDeleter. What is the status of this entry?

  2. Check the status in MongoDB, how many asset deletion events do you have? Use the folllowing query on the Events2 collection: { "Events.Type": "AssetDeletedEvent" }

  3. Are you sure that the file have ever existed in S3? Can you check S3 logs?

Hi Sebastian,

  1. AssetPermanentDeleter 1485 1646163378-161-0-1 (the app in question, after I restored it has approximately 1311 assets). However, I don’t know how accurate this will be since I’ve deleted/restored apps on this database several times.
  2. This count is the same 1485
  3. I’m certain the files existed. I was modifying them before the upgrade and they were visible in the Squidex UI. Now they are not visible in the UI. I had verified their existence in S3 after I initially restored the application into the S3 environment.

I will completely reset my scenario to confirm and get back to you.

Sebastian,
I’ve done this a couple of times now. Here’s what I’m able to reproduce. It’s not actually related to the upgrade at all. It just seems like a really bad bug.

  1. Launch a container using dev-6578.

  2. Create an app and upload some assets.

  3. Confirm the assets are created in S3 in their own app folder.

  4. Delete a single asset from Squidex.

  5. The entire folder for that app in S3 gets deleted.

  6. Launch a container using 6.5.0.

  7. Create an app and upload some assets.

  8. Confirm the assets are created in S3 in their own app folder.

  9. Delete a single asset from Squidex.

  10. The entire folder for that app in S3 gets deleted.

The Consumers “AssetPermanentDelete” only has a single count for each of the single asset deletes I’ve done. It does not account for the 30 or so assets that have been removed between my 3 tests.

The S3 logs aren’t shedding much light. In fact, 30 minutes after I’ve deleted an app, I’m still seeing log entries like this
REST.PUT.OBJECT squidex-assets/d2f0a7c7-315c-4440-b9ae-cfa5f6e888bf/215d7261-93d7-482e-ae09-744e37174663_0_68_66_Pad
where d2f0a7c7-315c-4440-b9ae-cfa5f6e888bf is an app folder that was removed several iterations ago.

I’m happy to try to track down specific information for you. Or DM/Teams call.

Thanks,
Jason

You are right, it is an evil bug:
https://github.com/Squidex/squidex/blob/master/backend/src/Squidex.Domain.Apps.Entities/Assets/DefaultAssetFileStore.cs#L118

There are tests for that… :frowning:

I have pushed a fix for that.

1 Like

This topic was automatically closed after 2 days. New replies are no longer allowed.