Random Error 404 when creating content

Support
1

I have…

  • [x] Checked the logs and have provided the logs if I found something suspicious there

I’m submitting a…

  • [x] Regression (a behavior that stopped working in a new release)
  • [ ] Bug report
  • [ ] Performance issue
  • [ ] Documentation issue or request

Current behavior

We use the content API to create and publish data points in one of our schemas. We have a script that deletes the schema, creates it again, and starts filling it with contents.
Recently (the last month to be precise), it started to respond with a 404 Not Found error to a great amount of the content creation requests. All the requests are done to https://cloud.squidex.io/api/content/my-test-app/data/?publish=true with different JSON bodies.

It seems that the error is not related to specific content items because the first time we execute the script, it creates the items, then in the next execution, it starts to fail on some of them. After several retries, it might succeed on the previously failed items and err on other ones.

Expected behavior

The content creation request should not fail with a 404 error if there is no issue in the provided data and request params. Previously we did not experience such behavior.

Minimal reproduction of the problem

In our case it happens after these steps performed by the API:

  1. Create a new schema
  2. Post and publish new contents to the schema (~200 items)
  3. Delete the schema
  4. Again Post and publish the same contents to the schema (~200 items)

It might need another loop of these steps to actually start getting the errors on some of the requests.

Environment

  • [ ] Self hosted with docker
  • [ ] Self hosted with IIS
  • [ ] Self hosted with other version
  • [x] Cloud version

Version: [VERSION]

Browser:

  • [ ] Chrome (desktop)
  • [ ] Chrome (Android)
  • [ ] Chrome (iOS)
  • [ ] Firefox
  • [ ] Safari (desktop)
  • [ ] Safari (iOS)
  • [ ] IE
  • [ ] Edge

Others:

2

I have never experienced this. Perhaps it is an issue with authentication, because it can return a 404 in some scenarios as well so that you cannot try to discover app names.

3

I see. Has something changed recently in the authentication flow? Because it’s something we haven’t encountered before also.
In the beginning we obtain a token from https://cloud.squidex.io/identity-server/connect/token with squidex-api scope, and use it in all subsequent requests. Strange thing is that after some failures it starts to succeed with the same token. Might it somehow be related to the frequency of the requests or used traffic?
What do you recommend to try or provide for faster resolution?

4

This is fine. The simplest approach is to reuse the token as long as possible and just retry the request with a new token when you get a 401. But for example if your server is connected to multiple apps and the token is a cached without the app name, it could get mixed up or so.

5

Well, we obtain a new token on each script execution, so we never get 401. But we do use the same clientId/secret from the owner user account for all of the apps. The scripts for different apps are not executed simultaneously even they do not run during the same day.
Can this be a problem?

6

I don’t understand which client id and secret you actually use.

Can you be sure that token for App A does not get to App B sometimes?

7

We use the one that is in the user profile and not in the apps. So we do use the same client/secret for all of our apps.

8

Okay, this is almost never used, I will have a look if something is broken there.

9

Thanks, we use it as we have a lot of apps with the same user and a single script that is used to manage them. Easier to use a single clientId than to generate and manage for each app.

10

Yes, I see…the main reason it was introduced was the to be able to create apps from the API and then use the individual client ids and secrets.

11

Can you send me an access token of a failed request and the user that you use?

12

Can you also send me the code how to create the token? I have a lot of strange logs in my log store.

Especially this line is confusing:

"Invalid reference token. { "ValidateLifetime": true, "AccessTokenType": "Reference", "ExpectedScope": "squidex-api", "TokenHandle": "undefined" }"

A reference token is a special self contained token, but I have never implemented it and to implement it you have to implement a token store. The default implementation is in-memory, so if you somehow created a reference token, you would perhaps see the 404. Just an idea.

13

No idea how to create a reference token :slight_smile: Here is how we get the token we use in all subsequent requests and never do anything to it:

const rq = require('request-promise')
const auth = async () => {
    var options = {
        method: 'POST',
        uri: 'https://cloud.squidex.io/identity-server/connect/token',
        form: {
            grant_type: 'client_credentials',
            client_id: 'my_client_id',
            client_secret: 'my_secret,
            scope: 'squidex-api'
        },
        json: true
    }
    try {
        const token_response = await rq(options)
        return token_response.access_token
    }
    catch(error) {
        console.log(error) //we never get this actually
    }
}

One of the latest failed tokens was:

TOKEN DELETED

And the user is registered with this email: info.mappr@gmail.com

14

Can you also send me your full script / code? Perhaps as PM. I only see a 404 if you are trying to access an app with an anonymous user (no token).

15

Here is a more recent token. Tried just now.

eyJhbGciOiJSUzI1NiIsImtpZCI6IlYzV0d4cU5rcWlpNl9OM2l4Wk9xY3ciLCJ0eXAiOiJhdCtqd3QifQ.eyJuYmYiOjE2MDAzMzY3MzgsImV4cCI6MTYwMjkyODczOCwiaXNzIjoiaHR0cHM6Ly9jbG91ZC5zcXVpZGV4LmlvL2lkZW50aXR5LXNlcnZlciIsImNsaWVudF9pZCI6IjVkZTUyMmYwM2E1MDVjMDAwMTA5Y2Y5OCIsInN1YiI6IjVkZTUyMmYwM2E1MDVjMDAwMTA5Y2Y5OCIsImp0aSI6Ijk2NjExMUVBODI2MkIyMjk1M0U5REVFM0M4MDk1Q0JEIiwiaWF0IjoxNjAwMzM2NzM4LCJzY29wZSI6WyJzcXVpZGV4LWFwaSJdfQ.MhYGK7M6KbN99A-B-AeHncVmAzcCnuPbtzNHjlNs7Yo_Wpj22zlY2ZZXr8OrLR-jC5kbkC7tXRgHYPGGYJWR0QlU4fY5nvHcke2NXe3UOxdgVmXku4d9ZBUiorMwGuCA5wq1cTyOeK2kS0QDQxvjRSnBdr9sNNN4mpv0Q0oX5hHloH830P0LKSwLQb8lguPUgNFTggB_yFUsWoNuU0zviWVdy9anHtmGH_i0_h2TqpFsvrM7TqZpyvveTayhI-sPszDFKKx2sG-Qdpp0GWg1Ng1hozLNyebxcrNzaJMrai3-sH6uHq1IOpGQYspgjtlvOe2vc9jLkAEWjpcANPtMOw

Also as a hint, can it be related to the fact that we perform the auth request several times in subsequent script executions. Like I did 2 auth requests within 5 minutes.

16

Are you doing the content creations just after something else? e.g. you create an app, then you insert content and then you get this problem?

17

Yes we do. I’ll send the code in a PM.

1 Like
18

I have turned off an internal caching layer. Perhaps this fixes it.

19

Hey Sebastian. We’re running into something very similar to this. We also use the root user’s credentials, and encounter 404’s that feel extremely random. The difference is that we’re not using Squidex cloud, we run Squidex on-premises. Is there anything we could do about this? Perhaps disable caching, like you did here?

20

You can try, if it helps, the keys is CACHING__REPLICATED__ENABLED=false.

A few questions:

  1. Which version do you use?
  2. Do you run a Mongo replica set?
  3. Do you run a Squidex cluster with clustering enabled?
  4. Do you see any exceptions in your logs?