Squidex CDN API calls returning empty data intermittently

Support
1

I have…

  • Read the following guideline: 01-getting-started/installation/troubleshooting-and-support. I understand that my support request might get deleted if I do not follow the guideline.
  • Used code blocks with ``` to format my code examples like JSON or logs properly.

I’m submitting a…

  • Regression (a behavior that stopped working in a new release)
  • Bug report
  • Performance issue
  • Documentation issue or request

Current behavior

Squidex CDN API calls intermittently returning null response and causing lot of content missing issues on our production website.

https://contents.squidex.io/cargocrew-prod/

We are in situation of manually identifying the pages with component issues and had to republish it manually throughout the day.

In some cases, combo products are not showing data resulting in customers unable to place orders. As this is impacting sales, please look into this on top priority.

We have started seeing this behavior from 25th of Jan. But from 29th Jan onwards, the frequency of content missing has increased a lot.

Expected behavior

All the components data queried through Squidex CDN API calls should return correct data from the Squidex CMS.

Minimal reproduction of the problem

Environment

App Name: cargocrew-prod

  • Self hosted with docker
  • Self hosted with IIS
  • Self hosted with other version
  • Cloud version

Version: latest

Browser:

  • Chrome (desktop)
  • Chrome (Android)
  • Chrome (iOS)
  • Firefox
  • Safari (desktop)
  • Safari (iOS)
  • IE
  • Edge

Others:
Please reach out to indusekhar.sajja@overdose.digital (or) manjeera.tsadipirala@overdose.digital for any additional information.

2

Hi Team,

Any update on this issue. This is impacting our revenues, please do the needful.

3

Do you know if this also happens without CDN?

it would also be great to have concrete samples.

4

We are using non CDN API URL only when new products are created which is very rare.

In most of the cases, we are getting intermittent blank (or) null response with 200 status code when pulling content for multiple components using single API call . Example mentioned below.

e.g. https://contents.squidex.io/cargocrew-prod/?ids=68d1dfb2-83ef-4879-b76c-2bb43dd20913,0f57bef0-4720-499f-8ad6-577db5ece3b1,6ac75f6e-16ae-4f35-b39f-2e679c3f0712,0dfc49a8-90f4-4a2c-9a0b-9984014dfee0

5

But this example seems to work, at least for me.

6

Yes… If we run manually it works… but intermittently in our batch jobs it is not working (or) not returning any component data resulting in content issue in the website.

7

I am not sure what to do here, I have checked:

  • Fastly status
  • Fastly cache invalidation log
  • Squidex cache invalidations
  • Squidex logs

There was also no deployment in January and I do not have a concrete sample or code or so to investigate that.

8

Is there something you can check from cloud infrastructure perspective for our application.

9

I can download the logs and we can have a look if we find it there.

10

I have downloaded your logs and I am not sure if I can find your requests:

Header
Timestamp, RequestPath, RequestMethod, RequestElapsedMs, Costs, AuthClientId, AuthUserId, Bytes, CacheHits, CacheServer, CacheStatus, CacheTTL, StatusCode
10 Most frequent paths.
/api/content/cargocrew-prod/: 834412
/api/apps/cargocrew-prod/history: 166514
/api/assets/cargocrew-prod/ffc6b481-d169-4d74-8d5a-19c4af47cb76: 41394
/api/content/cargocrew-prod/apparelproduct/: 37134
/api/assets/cargocrew-prod/e702c61a-bf1e-4829-ba95-a640bbda908d: 29157
/api/assets/cargocrew-prod/6e210d92-0af7-4587-9ac5-5c86770e8dcf: 28754
/api/assets/cargocrew-prod/a5781f11-19a1-4d9b-936c-ad5c29100572: 25920
/api/assets/cargocrew-prod/c6c459ff-1fc7-41d5-8182-e3194dc7ff2b: 25757
/api/assets/cargocrew-prod/1d38591c-3575-442e-8e7f-79064a4ed895: 25237
/api/assets/cargocrew-prod/f24df82f-43bf-4aa7-946b-39d9e439f0f4: 24966

Total size for /api/content/cargocrew-prod.
         1KB to 10KB: 1566060
       10KB to 100KB: 438761
              < 512b: 220803
        100KB to 1MB: 173119
         512b to 1KB: 48

Here is the code, if you want to test it…Logs · GitHub

11

We have observed 60+ API calls returned empty responses between
2/1/2024, 11:29 PM - 2/1/2024, 11:52 PM UTC and details mentioned below.

timestamp [UTC] message
2/1/2024, 11:29:25.583 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=62c991ab-1d69-4744-a9a9-489203edc7cc
2/1/2024, 11:29:26.250 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=552459a8-8c25-4ac6-b9c0-c1868214a7f4
2/1/2024, 11:29:26.266 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=7001c0db-5e43-4593-9772-0a088892927b
2/1/2024, 11:29:46.599 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=b1b011b6-9f1c-4d21-baa7-3ab88576d3d4
2/1/2024, 11:29:46.953 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=68d1dfb2-83ef-4879-b76c-2bb43dd20913,1d27ed8c-0fe1-42c8-b18b-239d3851bad7,45c4cf9a-a747-43a9-bb31-101b835686e9,853716fc-5554-4ae6-8ff2-f3438079f94e,93f895ff-ce00-4796-94d9-fe8e654ff66f,fc2edf9b-dd3a-479e-a573-69934f968243,9876b685-cbdd-4d2a-bf44-92cb316f86e8,fe4cdefe-a233-4002-932f-5716b1dda6ae
2/1/2024, 11:29:47.552 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=7deae5f7-5008-4222-9236-dc71b3d939c5
2/1/2024, 11:29:47.568 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=e58d7ce3-faa7-4b23-96bb-ad563760e12c
2/1/2024, 11:29:47.802 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=4af0e102-4be2-47c8-9c18-b665efbecd44,2344583a-6c12-4c74-bfa2-3caa674411b1,9a0aa652-2761-4e42-89d6-3fd95d2bd13f,aa63e3a8-cc66-483b-81b3-3eda15f5f82b,84e3f0b8-5701-4ac4-b813-2e198133cda4,08ba9083-855d-4e8b-8f0a-bdaae6381ff9,af490c58-543a-4631-83d5-36742114a5c2,96a153ce-6144-495d-ad10-4d9205a2b236,c12ae3d4-cf3e-4e57-98a1-4e92027a9735,14ba97dd-7ef4-4f4e-a235-ad22d9a98c09,04cd1e95-53d7-4aa8-9967-b69c3dc44a76,5f446d44-ca31-4504-97e6-1f14b2a95010,70e052cb-6e82-4ebc-8f52-66d9e7cf42f8,eafd16cd-f0f9-4108-9532-8a6ddc6c846e,655cce3b-c2a3-4edf-ab72-de260f48b21d,72a5c39b-55f8-41a8-89ea-ef8710289944,ce87baf5-a51c-4539-b548-13a34c853da4,5ea9401a-c23c-455a-b725-701b86178b60,8395d3ef-7045-4e87-aa3f-80a140c74cc8,634daea1-c28c-4f86-914b-f1d1cf55e6fc,c664c089-371b-47c5-9006-03c36657cc9d,52530391-1932-460f-8f0a-0c52923e860a,29020408-8bf4-42a4-b86c-26e7657bc123,1504e75e-ff61-4dc9-87ec-459a7290f6a0,2d39513c-ff0b-4066-ad8c-5587afeacc06,a751d786-f593-40bf-997a-411e730245c0,d7cf29fd-c7b0-4ce6-95da-18055ccaad9e,519c0353-4d6a-46fa-b295-3d8221905dba
2/1/2024, 11:30:08.825 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=b136859e-fde0-4643-b765-16047773cc89,6094f263-61f0-48bf-b1dd-203bcfeb60ef,37c069a9-e76c-4411-83e2-0a951bedbfc9
2/1/2024, 11:30:08.840 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=d406418d-4bbf-44a3-9754-a8b35ea3e8a7
2/1/2024, 11:30:09.186 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=3d8f63b6-22f3-4281-b951-4da62c2178e3
2/1/2024, 11:30:09.560 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=53686b08-a191-4071-96da-83538d585a60,4e70e873-dc8e-4122-bc1d-7f4585e57656
2/1/2024, 11:30:29.868 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=6094f263-61f0-48bf-b1dd-203bcfeb60ef
2/1/2024, 11:30:30.212 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=7001c0db-5e43-4593-9772-0a088892927b
2/1/2024, 11:30:30.790 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=5c76797e-8ff1-43fe-9202-3c50691ac238
2/1/2024, 11:30:31.121 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=4af0e102-4be2-47c8-9c18-b665efbecd44,3d686d25-16ed-4842-87a8-576166e774e5,5f446d44-ca31-4504-97e6-1f14b2a95010,9a0aa652-2761-4e42-89d6-3fd95d2bd13f,2344583a-6c12-4c74-bfa2-3caa674411b1,aa63e3a8-cc66-483b-81b3-3eda15f5f82b,19a6dc1a-1fb6-4c15-8a38-da0d8c588646,af490c58-543a-4631-83d5-36742114a5c2,c12ae3d4-cf3e-4e57-98a1-4e92027a9735,96a153ce-6144-495d-ad10-4d9205a2b236,14ba97dd-7ef4-4f4e-a235-ad22d9a98c09,04cd1e95-53d7-4aa8-9967-b69c3dc44a76,08ba9083-855d-4e8b-8f0a-bdaae6381ff9,cb19c956-4307-47e0-b70e-0c3c5bd2e3a9,eafd16cd-f0f9-4108-9532-8a6ddc6c846e,655cce3b-c2a3-4edf-ab72-de260f48b21d,72a5c39b-55f8-41a8-89ea-ef8710289944,cfbf0863-5cb6-4291-ae15-293673ed4fd6,634daea1-c28c-4f86-914b-f1d1cf55e6fc,b0a4ebb8-c3d3-4d82-ae09-9d3e50763785,a12557c3-a364-481b-b302-652bb75d0143,1504e75e-ff61-4dc9-87ec-459a7290f6a0,c664c089-371b-47c5-9006-03c36657cc9d,29020408-8bf4-42a4-b86c-26e7657bc123,52530391-1932-460f-8f0a-0c52923e860a,a751d786-f593-40bf-997a-411e730245c0,519c0353-4d6a-46fa-b295-3d8221905dba,d8b54643-e30b-4336-9806-f819ae1cc69f,8b7d02bb-0bfc-4405-9901-a1b2f7e0fb45,f28253c3-9717-4e85-a8a4-9b79d7e51655,4de41cfc-24f9-4cee-8d23-6b026245ea84,8166439e-4e7c-4891-8044-7c49325132d0,b1c33fef-f5f8-495f-8f1e-c2d5889fae33,d7cf29fd-c7b0-4ce6-95da-18055ccaad9e,918df411-1361-4da0-8447-839ee38cfa8a,7e971c6f-a1e6-4e76-9b22-60f7ea28e714
2/1/2024, 11:30:51.533 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=e58d7ce3-faa7-4b23-96bb-ad563760e12c
2/1/2024, 11:30:52.151 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=8cdeca86-837b-4b5e-8e5b-a7f11677e169,0a3265b7-d7ce-4b91-a330-01ad0e89dd06,63737314-b0ca-4bf9-be6b-cf30f6341e32
2/1/2024, 11:30:52.341 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=cd68b887-516a-4598-89b8-f1d99c5398cb
2/1/2024, 11:31:13.123 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=3d8f63b6-22f3-4281-b951-4da62c2178e3
2/1/2024, 11:31:13.185 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=63737314-b0ca-4bf9-be6b-cf30f6341e32
2/1/2024, 11:31:13.654 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=4902a168-ee7e-466e-a76a-86195d7e43f9
2/1/2024, 11:31:33.232 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=e9de5f47-5214-4740-8d96-7ca6e56e857e,fc2edf9b-dd3a-479e-a573-69934f968243,2a39e5ed-d8c2-4d20-aa14-ee59a85d48fb,4512dea2-b936-4a8d-add9-658c6d014342,6d7874d9-147a-444d-a66e-0e9ee2059753,05c2ea26-2d18-4a20-b109-3e9976731ade,381b9492-57f6-4922-97d6-f400580e85c8,a3ad4176-94d9-427e-9f90-5a2532c83a29,667f8d9c-6f7c-455a-b8e9-2d3e70410c83,8b6054e7-fa88-459e-b618-78c50d15b5ff,0a4eb5d0-3f1b-4752-b581-7449f3518a29,2939c8f8-6e74-4a95-a088-8c5d86e1e4c9,04b553c2-07f2-479c-9340-393b242336a6
2/1/2024, 11:31:34.443 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=5c76797e-8ff1-43fe-9202-3c50691ac238
2/1/2024, 11:31:34.708 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=3efb1158-3b79-4280-8728-6d44aceba744
2/1/2024, 11:32:20.209 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=e58d7ce3-faa7-4b23-96bb-ad563760e12c
2/1/2024, 11:32:20.616 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=c2b81fce-ccfe-47ff-989a-e75112a003df,f5abcbb5-d672-4e45-a0b4-1548f6c234f0,83899206-b530-49ac-88f6-5eda6386132c,abd67c76-18f5-4cf8-a40c-57c359efcad3,57557e65-1c63-489f-b188-b76f1b16d316,b8969023-f823-40ef-ad7e-c5763c9aa421,7b784d4c-a920-4103-b06b-5d8179b14ab3,3486c30e-1f94-4f7e-afb1-8374b8b4f5ea,f214ef24-2f14-48fb-a3b5-a98be8c67776,ddbaff36-ac90-4fd8-a5ef-1f7fe3403c5e,3c14563e-fe2d-4226-89ce-42357dbdcc71
2/1/2024, 11:32:20.944 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=552459a8-8c25-4ac6-b9c0-c1868214a7f4
2/1/2024, 11:32:26.333 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=5379dbc0-3802-461b-b8e6-b168e20ce9d1,01bbb759-6424-4cfc-bdaf-d43372ac22a2,5a9ef822-bee6-4b64-8454-d4dd7bde629f,47a34a0e-421d-41e7-9eb9-e22dec69104a,93621fd2-524c-47bb-bd8e-b9881610b1f1,ec5bf3fc-7a85-46fd-b44e-d55e213feb19,2035f6d8-d29f-4eeb-87e2-9aacedfddf01
2/1/2024, 11:32:41.511 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=3d8f63b6-22f3-4281-b951-4da62c2178e3
2/1/2024, 11:32:41.964 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=4af0e102-4be2-47c8-9c18-b665efbecd44,8166439e-4e7c-4891-8044-7c49325132d0,2344583a-6c12-4c74-bfa2-3caa674411b1,41ec540c-50d6-457a-aa93-eeea911223a4,b1c33fef-f5f8-495f-8f1e-c2d5889fae33,5ec784b7-eb5b-4d17-9ca2-7e254c733886,a751d786-f593-40bf-997a-411e730245c0,ec23743c-42f3-428e-868f-91dbe30c15f7,5f446d44-ca31-4504-97e6-1f14b2a95010,9a0aa652-2761-4e42-89d6-3fd95d2bd13f,aa63e3a8-cc66-483b-81b3-3eda15f5f82b,7f8949cf-84e2-4f8c-993c-af059b03c116,b5092c42-7c9a-4a81-af90-e144f3f6f82e,cf05c664-a40a-461c-a18a-0da1e96c18ce,55e19f7b-83a4-47a5-85c8-3bda1cbccb52,4ad65311-6f92-4c72-96f0-a720e295c804,14ba97dd-7ef4-4f4e-a235-ad22d9a98c09,04cd1e95-53d7-4aa8-9967-b69c3dc44a76,19a6dc1a-1fb6-4c15-8a38-da0d8c588646,634daea1-c28c-4f86-914b-f1d1cf55e6fc,a0b04342-d56d-4bf3-8a52-ac3bc777b95e,cfbf0863-5cb6-4291-ae15-293673ed4fd6,5fa8c96d-ce58-4dd7-a61c-8db64e905a2a,a2da0529-880d-433f-b86b-cc1224fb2ff2,fe0e2c68-a174-4131-b055-d4095d5ef361,a99e0469-0b90-420e-a615-42c5e4fde8f0,eafd16cd-f0f9-4108-9532-8a6ddc6c846e,380766cd-de72-40f1-bb96-b3b1c18d2899,7f677f7a-f9a6-4bfb-afbd-465dd60fb3bd,08ba9083-855d-4e8b-8f0a-bdaae6381ff9,cf0cf243-5983-48d4-ab3d-09959e17b41c,5f4da36a-5fc0-43be-99fc-a88109c1d65f,8b7d02bb-0bfc-4405-9901-a1b2f7e0fb45,1a032f29-0cff-4f85-a8f1-0c1c10dab8be,5ea9401a-c23c-455a-b725-701b86178b60,84e3f0b8-5701-4ac4-b813-2e198133cda4,0030ac2a-0f3d-4232-ad9f-a7325eac1fbe,918df411-1361-4da0-8447-839ee38cfa8a,d7cf29fd-c7b0-4ce6-95da-18055ccaad9e,7e971c6f-a1e6-4e76-9b22-60f7ea28e714
2/1/2024, 11:32:42.261 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=7deae5f7-5008-4222-9236-dc71b3d939c5
2/1/2024, 11:32:43.242 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=8cdeca86-837b-4b5e-8e5b-a7f11677e169,0a3265b7-d7ce-4b91-a330-01ad0e89dd06,63737314-b0ca-4bf9-be6b-cf30f6341e32
2/1/2024, 11:32:47.415 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=40befad6-0b1a-4697-a059-1244222f2030,d59b2fd4-ffd1-4a15-8199-3f33afc107a2,e6885a43-63c7-41e3-a042-bc144e259ed7
2/1/2024, 11:34:28.024 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=f3463a32-6582-41da-ae92-2e8d8383bc67
2/1/2024, 11:34:28.101 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=e58d7ce3-faa7-4b23-96bb-ad563760e12c
2/1/2024, 11:34:28.335 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=4902a168-ee7e-466e-a76a-86195d7e43f9
2/1/2024, 11:34:28.445 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=d406418d-4bbf-44a3-9754-a8b35ea3e8a7
2/1/2024, 11:34:39.788 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=87b830c9-9665-45ea-ac22-ec33cbe449ed,2a8409a6-1d9a-4051-8c0d-11099d4e0c74,2e1a9662-f03a-4acc-bfd4-33f84b7dbc94
2/1/2024, 11:34:49.414 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=3d8f63b6-22f3-4281-b951-4da62c2178e3
2/1/2024, 11:34:49.633 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=3efb1158-3b79-4280-8728-6d44aceba744
2/1/2024, 11:34:49.742 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=7001c0db-5e43-4593-9772-0a088892927b
2/1/2024, 11:35:01.559 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=16fe14c9-f759-42ee-9241-adc52331095a,be0bf5a6-ff26-4204-a433-f497734304d3
2/1/2024, 11:35:10.728 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=c3729715-c009-427e-9835-4489b75ac8be,8a3c3bb6-924a-4f53-97cd-8b8331611ffa,45c4cf9a-a747-43a9-bb31-101b835686e9,6d7874d9-147a-444d-a66e-0e9ee2059753,131fbe9e-513d-4237-9593-a45c7ad8bd77,8b6054e7-fa88-459e-b618-78c50d15b5ff,df8f42dc-ea00-4375-9c69-5b2bdac900d7,5a9ef822-bee6-4b64-8454-d4dd7bde629f,c1dafdae-68ed-468c-b809-0c51fe78e80c,eaa5d170-bc64-43ca-98fd-98513a789bba
2/1/2024, 11:35:10.728 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=5c76797e-8ff1-43fe-9202-3c50691ac238
2/1/2024, 11:35:11.071 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=e58d7ce3-faa7-4b23-96bb-ad563760e12c
2/1/2024, 11:35:12.833 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=4af0e102-4be2-47c8-9c18-b665efbecd44,8166439e-4e7c-4891-8044-7c49325132d0,a5bd98a7-2850-4228-ae2d-f6acc384e8e9,ec23743c-42f3-428e-868f-91dbe30c15f7,5f446d44-ca31-4504-97e6-1f14b2a95010,8395d3ef-7045-4e87-aa3f-80a140c74cc8,89e2795c-65f8-4c57-88c4-e61dbc71f7ff,a751d786-f593-40bf-997a-411e730245c0,634daea1-c28c-4f86-914b-f1d1cf55e6fc,c664c089-371b-47c5-9006-03c36657cc9d,a2da0529-880d-433f-b86b-cc1224fb2ff2,1504e75e-ff61-4dc9-87ec-459a7290f6a0,29020408-8bf4-42a4-b86c-26e7657bc123,52530391-1932-460f-8f0a-0c52923e860a,5ea9401a-c23c-455a-b725-701b86178b60,70e052cb-6e82-4ebc-8f52-66d9e7cf42f8,d7cf29fd-c7b0-4ce6-95da-18055ccaad9e,519c0353-4d6a-46fa-b295-3d8221905dba
2/1/2024, 11:35:22.605 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=d7ee02d4-1256-4ee6-ae0e-d1a97dee2886,08aa2b52-3ad2-4e68-8edd-15ea43c6ab1e,3625fc38-a076-45d9-a6ad-dccaa07b993a,c4c4f9a1-f355-455c-83df-70edfa4bde50,c258080c-96da-48fa-8f7d-2975243052c8,9d4b07b7-e7d6-464a-a71d-a25c9250ae40,502fb2bc-191f-4228-945c-4445008b618e,cda3a738-9f82-4ce7-bbfb-c88e44137589,d3173a34-27cb-4e0f-86f0-ba4a020651b8,7511b855-ed68-4cc0-b7ca-67fdec8a9339
2/1/2024, 11:35:32.052 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=cd68b887-516a-4598-89b8-f1d99c5398cb
2/1/2024, 11:35:32.369 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=3d8f63b6-22f3-4281-b951-4da62c2178e3
2/1/2024, 11:35:33.852 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=aea5f023-295e-4c37-bd25-4bf6bd59712e,0a6038c4-c6f1-4909-a7ab-10cb2c1b9c78,be3b9954-c440-4113-b724-cb6c384b79c2,5ac5cb6e-4535-4f1b-8b71-f76b653f33f3,5212f1e4-5be8-4869-89ae-a337a18e2316,928888df-770a-4466-9d05-9906365970d2,99b8cc59-cab2-47c8-a1d5-5413ed3a657b,b3195adc-d816-4ef5-9858-cd457cfd9482,0cdbb687-aa2a-48d2-878e-b8b49e707732,2b26fda6-41c3-4d00-a4fe-d9d04bc532ad,ec791dd2-eb75-4542-92e2-ebb289ea06c3
2/1/2024, 11:35:53.390 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=4902a168-ee7e-466e-a76a-86195d7e43f9
2/1/2024, 11:35:53.672 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=5c76797e-8ff1-43fe-9202-3c50691ac238
2/1/2024, 11:36:14.687 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=3efb1158-3b79-4280-8728-6d44aceba744
2/1/2024, 11:36:14.989 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=cd68b887-516a-4598-89b8-f1d99c5398cb
2/1/2024, 11:36:35.734 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=cdb8b78a-36ba-4b05-a8f2-b650fb471baf
2/1/2024, 11:36:36.312 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=4902a168-ee7e-466e-a76a-86195d7e43f9
2/1/2024, 11:36:52.290 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=265890ab-bfba-4b74-9981-0aed64e27ada,fc2edf9b-dd3a-479e-a573-69934f968243,e9de5f47-5214-4740-8d96-7ca6e56e857e,2688f4be-888e-431e-bb08-258d0eef275d,0a4eb5d0-3f1b-4752-b581-7449f3518a29,d028c5cc-e3f2-485b-9230-d74677868c1a,f186ded3-a263-4fea-8bde-5c65ae035162
2/1/2024, 11:37:30.338 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=4af0e102-4be2-47c8-9c18-b665efbecd44,3d686d25-16ed-4842-87a8-576166e774e5,2344583a-6c12-4c74-bfa2-3caa674411b1,9a0aa652-2761-4e42-89d6-3fd95d2bd13f,aa63e3a8-cc66-483b-81b3-3eda15f5f82b,af490c58-543a-4631-83d5-36742114a5c2,96a153ce-6144-495d-ad10-4d9205a2b236,c12ae3d4-cf3e-4e57-98a1-4e92027a9735,14ba97dd-7ef4-4f4e-a235-ad22d9a98c09,04cd1e95-53d7-4aa8-9967-b69c3dc44a76,5f446d44-ca31-4504-97e6-1f14b2a95010,70e052cb-6e82-4ebc-8f52-66d9e7cf42f8,eafd16cd-f0f9-4108-9532-8a6ddc6c846e,08ba9083-855d-4e8b-8f0a-bdaae6381ff9,655cce3b-c2a3-4edf-ab72-de260f48b21d,72a5c39b-55f8-41a8-89ea-ef8710289944,5ea9401a-c23c-455a-b725-701b86178b60,634daea1-c28c-4f86-914b-f1d1cf55e6fc,feba6004-e434-4637-af32-168bd1bea039,a751d786-f593-40bf-997a-411e730245c0,5f4da36a-5fc0-43be-99fc-a88109c1d65f,d8b54643-e30b-4336-9806-f819ae1cc69f,8b7d02bb-0bfc-4405-9901-a1b2f7e0fb45,8166439e-4e7c-4891-8044-7c49325132d0,d7cf29fd-c7b0-4ce6-95da-18055ccaad9e,19a6dc1a-1fb6-4c15-8a38-da0d8c588646,918df411-1361-4da0-8447-839ee38cfa8a,7e971c6f-a1e6-4e76-9b22-60f7ea28e714
2/1/2024, 11:37:30.408 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=0a4eb5d0-3f1b-4752-b581-7449f3518a29,6d7874d9-147a-444d-a66e-0e9ee2059753,25f1496b-85fa-4281-9474-8cb068c2f442,c1dafdae-68ed-468c-b809-0c51fe78e80c,1114beb1-3060-45c7-b32e-7234080831fb,f5be5f18-20f0-40f0-8056-200ba5d01890,24a477ec-17ac-42f1-b765-2c48d4e0aef0,67ad366f-a117-454d-9251-7f9cfbcd0c54,fc2edf9b-dd3a-479e-a573-69934f968243,5a9ef822-bee6-4b64-8454-d4dd7bde629f,667f8d9c-6f7c-455a-b8e9-2d3e70410c83
2/1/2024, 11:37:31.319 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=ceccc5fa-a4a4-4cc4-aae1-80eabc03fdbb,eef29d28-15e7-4bc7-a118-9842b9edb2e7,86cfd948-14d8-45d3-bbaa-7579519a4bfa,6adafced-a2cc-4ad7-acdd-2683c9485cc6
2/1/2024, 11:37:33.711 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=ac33949d-ab8b-40a8-8d1c-462da3561505,5763f2ff-8092-4f36-bcf9-0245a06bf9a1,5956f0a9-b8b8-4f05-b87a-b3d51b4a7004,3f4c5e2c-d4f5-4d04-a861-6028df6843ae,1c49fbde-1bd9-4eab-ba66-af56ea7830c7
2/1/2024, 11:52:12.724 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=d1e64219-e71a-46fc-9b21-bd9689202ee9
2/1/2024, 11:52:13.147 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=3efb1158-3b79-4280-8728-6d44aceba744
2/1/2024, 11:52:44.455 PM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=5c76797e-8ff1-43fe-9202-3c50691ac238
12

We have observed another set of empty API responses (80+ calls) between
2/2/2024, 4:41 AM and 2/2/2024, 4:49 AM UTC time.

Few API calls that returned empty response mentioned below.

timestamp [UTC] message
2/2/2024, 4:41:05.110 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=6094f263-61f0-48bf-b1dd-203bcfeb60ef
2/2/2024, 4:41:05.305 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=987d25ee-e6cd-4e57-af49-a50a9a0a629e,0fe8fcd1-10b3-4b2c-9ab6-53ceb31e4463,7b9a10f1-047b-4cb7-b8f8-df44906d16e1,866883a5-05fb-4ca5-a965-5959796b296b,c4f03af5-a0ed-4b12-a20c-fff9294db984,c159b037-98c4-4533-b74f-7d39a5d8523d,1b459790-8ae2-4e81-9f47-17d939edafdb,eaa5d170-bc64-43ca-98fd-98513a789bba,08ee6fdb-73a1-484d-8c7b-42356ebbf899,78b12b5a-c0bd-4529-a1f4-8dbbd63ac568,58177d06-ff9d-4dd1-b2a8-09c76b28f37c
2/2/2024, 4:41:06.970 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=405aaa63-b3c4-442f-aa14-82864ec9b8ef,bcb9b426-70e3-4b27-b60c-66ff7403d7ee,1036c187-7380-4d87-b2ae-bd1de4acc5e6,fe93927d-30c6-4c71-8188-74c13ee0e238,6b31cc2d-b12b-429c-b1e3-a109cf2bcd38,c0e7a823-af4b-4478-ab73-440272cc8a18,c591f36b-dfab-47d1-a47b-5b19950c9675,3e8dc785-8826-478a-ae25-06a60b876b6a,da222da6-e097-4060-9b04-f7a6f578055b,2726eb9c-73ca-4a74-80d0-d4eb420c2861,b5c438b5-348f-4973-8310-e82c6f224a2a,459be935-070a-436f-acfc-caa7ac951928,4105d70f-7ae8-43fb-8edd-34eba72702fc,60fffbb0-ede1-4f26-ba64-03661f86b8f0,c8fe032e-644c-4be1-b060-eb873b36a0fa,87f288a5-f7a5-4345-bc9e-b97cae44b000,7866bd2a-16be-46cd-a0a4-2fcda260486a,229b4552-74ad-4ad5-ae9c-f5056c2d2cf2,c02f4f33-9b05-4e11-93bc-3ef91b20b810,b45b59c7-79c0-4792-9473-fd438d689cae,2348e955-529f-4f2c-be19-97122b9c43a7,6d3c3ac2-5e66-4275-b9be-9cea6e6bd170,6f94f59c-3366-42ed-a760-4617c73adb43,31bbe0a7-d234-440c-a9fe-2ba0124374ee,c8e72bf4-3f24-466e-8e39-92fef8f2bbdb,fe95c43c-36f3-4fca-af04-f9283ea72bef,7b37afe4-710e-4a67-99e5-f05f56c38b93,faa4d2e2-b409-42b4-bb86-f7ff1d8befb0
2/2/2024, 4:41:07.516 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=e6a8781d-2fb3-49ec-96b9-b6137733543b
2/2/2024, 4:41:07.688 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=9e04f17b-6994-4f76-8f35-2ff605864bed,4d1eff4b-0c7f-40f4-94fb-d26846bad0ca
2/2/2024, 4:41:15.742 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=4af0e102-4be2-47c8-9c18-b665efbecd44,5f446d44-ca31-4504-97e6-1f14b2a95010,08ba9083-855d-4e8b-8f0a-bdaae6381ff9,b0a4ebb8-c3d3-4d82-ae09-9d3e50763785,b1c33fef-f5f8-495f-8f1e-c2d5889fae33,adad1926-5e4b-4453-8a5f-53ed622c8e93,5ea9401a-c23c-455a-b725-701b86178b60,634daea1-c28c-4f86-914b-f1d1cf55e6fc,a0b04342-d56d-4bf3-8a52-ac3bc777b95e,ffc58eda-240c-4e38-b25c-8869acde7989,a751d786-f593-40bf-997a-411e730245c0,519c0353-4d6a-46fa-b295-3d8221905dba,d8b54643-e30b-4336-9806-f819ae1cc69f,8b7d02bb-0bfc-4405-9901-a1b2f7e0fb45,fe0e2c68-a174-4131-b055-d4095d5ef361,d7cf29fd-c7b0-4ce6-95da-18055ccaad9e,19a6dc1a-1fb6-4c15-8a38-da0d8c588646,918df411-1361-4da0-8447-839ee38cfa8a,7e971c6f-a1e6-4e76-9b22-60f7ea28e714
2/2/2024, 4:47:30.418 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=083cbf0c-949c-4c52-8f30-57f4a6a9567b,1727ab8e-4d0f-4eac-a417-0c021d894d2c,64e720a3-c61d-407f-9edb-7ed057759214,50dd0316-1408-4b9a-9955-5f7c37ed1442
2/2/2024, 4:47:30.593 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=cd68b887-516a-4598-89b8-f1d99c5398cb
2/2/2024, 4:47:33.519 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=5ef2fb01-28a9-4f44-b36e-2a829d4d9668,91c29c99-2cf5-443a-8d06-d954ab2f69d8,b9d0988a-8a71-4338-9704-57323bfe17ed
2/2/2024, 4:47:39.129 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=b7c7fc5b-4672-493b-bfda-95a2a79bbbf2,6905a705-3189-43ec-9735-c4aa67069a47,67ad366f-a117-454d-9251-7f9cfbcd0c54,866883a5-05fb-4ca5-a965-5959796b296b,20d49e14-6c5e-4085-af2f-c03d92e03e69,a0149017-f916-4e9b-adb5-d047297bc384,3f1f6e2f-6a14-421b-a2a6-463a52d433cd,0a4eb5d0-3f1b-4752-b581-7449f3518a29
2/2/2024, 4:48:41.230 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=7bb75162-a650-4bca-8f2f-b68590942c9a,0dfc49a8-90f4-4a2c-9a0b-9984014dfee0,038f5865-518d-4287-8d97-34bbe7709b0b,65478e44-18f3-4107-8e52-1fdd6d2bce82,c0574264-83ab-49f7-a404-688f4ca94fc9,5f617d06-409e-459c-9779-056bf2568161,184b28c7-4e08-49b7-b55f-cab299ed8062,fcffa0c2-f31c-4362-8ea0-a0d8ef2c3415,ecec4b9d-cfe4-4e40-a1de-13ed214573ab,3e5923d6-be29-45ca-b960-ae68412d9f69,4dfb867c-2e65-4a67-8044-37e8afef8926,93d8418c-94df-4973-b570-9c7e1e930495,b8b58053-ea6a-48a4-906e-1ca1279845e6,c4f03af5-a0ed-4b12-a20c-fff9294db984,18d97ee5-6e86-47cf-ad4e-bfc8d19dab56,01bbb759-6424-4cfc-bdaf-d43372ac22a2,8aaa1c66-7b8b-40db-88ab-813c7efd3fd1
2/2/2024, 4:48:41.355 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=b2555096-007e-4256-9373-f31f4110fe01
2/2/2024, 4:48:41.480 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=5c76797e-8ff1-43fe-9202-3c50691ac238
2/2/2024, 4:48:42.330 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=420c4ea2-c3f4-4401-8fde-6e030cdd51ac
2/2/2024, 4:48:43.114 AM Squidex CDN - Empty response : https://contents.squidex.io/cargocrew-prod/?ids=d70a2e71-4795-4687-bc39-7b2fef831d3b
13

None of them looks empty in my case…

So what would help:

  1. What is your definition of “empty”.
  2. Whenever you see an empty response, you could make the same request without CDN to verify the root cause.
  3. How is your code? Are you sure, that it is not just a swallowed exception or something like that?
  4. Whenever you have only one ID, you could use the endpoint without the query string. Like /<ID>. Because then this request is an extra entry in the logs and we could verify it like that. Query strings are not logged.
14

Hi Sebastian,

We have added additional logs to check the response code and getting status code as Faulted.

Also, we have added fallback code to get data from squidex content API if the CDN API call fails.

Please check below screenshot, the CDN call took 21 sec to execute and returned “Faulted” response. The subsequent content API call is successful (fallback code we added today). We are observing same behavior with all failed CDN calls.

image
image1178×847 81.4 KB

.Net code screenshot attached below.

image
image1128×671 114 KB

_squidexUrl - https://contents.squidex.io/cargocrew-prod/
_squidexApiUrl -https://cloud.squidex.io/api/content/cargocrew-prod/

15

Faulted is not a http status response. Can you also get something specific?

16

Btw:

  1. Always configure a timeout.
  2. Have you considered to use something like Polly?

Both approaches together could probably solve your issue right now.