[SOLVED] Mysterious 404 (NotFound) issue

I have…

I’m submitting a…

  • [ ] Regression (a behavior that stopped working in a new release)
  • [ ] Bug report
  • [ ] Performance issue
  • [x] Documentation issue or request

Current behavior

I am using squidex:5.4.0 docker with the following configuration:

    marketing-cms:
    image: squidex/squidex:5.4.0
    restart: always
    environment:
      URLS__BASEURL: "https://marketing-cms"
      EVENTSTORE__TYPE: MongoDB
      EVENTSTORE__MONGODB__CONFIGURATION: mongodb://marketing-cms-datastore
      STORE__MONGODB__CONFIGURATION: mongodb://marketing-cms-datastore
      IDENTITY__ADMINEMAIL: "test@test.de"
      IDENTITY__ADMINPASSWORD: "xxx"
      UI__REDIRECTTOLOGIN: "true"
      IDENTITY__adminClientId: "admin"
      IDENTITY__adminClientSecret: "xxx"
      IDENTITY__googleClient: ""
      IDENTITY__googleSecret: ""
      IDENTITY__githubClient: ""
      IDENTITY__githubSecret: ""
      IDENTITY__microsoftClient: ""
      IDENTITY__microsoftSecret: ""
    depends_on:
      - marketing-cms-datastore
    volumes:
      - ./volumes/marketing-cms:/app/Assets
    labels:
      traefik.enable: true
      traefik.http.routers.marketing-cms.rule: Host(`marketing-cms`)
      traefik.http.routers.marketing-cms.tls: true
      traefik.http.services.marketing-cms.loadbalancer.server.port: 80

I am using a read-only anonymous client to access my content.

I created some schemas and content and initially, everything ran fine.

But now I am getting sometimes mysterious 404 responses for the same endpoint that sometimes successfully answers my request.

I even can reproduce that when I refresh the Squidex UI in my browser then immediately afterward the request (in Postman) succeeds. But after some time the same request returns 404 again (in Postman).

The Squidex UI seems to work properly and I never get a 404.
Only one thing is strange the cluster overview in the UI gives me a 500 response.

{
  "logLevel": "Information",
  "message": "Application started",
  "environment": {
    "applicationname": "Squidex",
    "aspnetcore_urls": "http://\u002B:80",
    "aspnet_version": "5.0.0",
    "assets:defaultpagesize": "200",
    "assets:defaultpagesizegraphql": "20",
    "assets:deleterecursive": "True",
    "assets:maxresults": "200",
    "assets:maxsize": "5242880",
    "assetstore:amazons3:accesskey": "\u003CMY_KEY\u003E",
    "assetstore:amazons3:bucket": "squidex-test",
    "assetstore:amazons3:bucketfolder": "squidex-assets",
    "assetstore:amazons3:forcepathstyle": "False",
    "assetstore:amazons3:regionname": "eu-central-1",
    "assetstore:amazons3:secretkey": "\u003CMY_SECRET\u003E",
    "assetstore:amazons3:serviceurl": "",
    "assetstore:azureblob:connectionstring": "UseDevelopmentStorage=true",
    "assetstore:azureblob:containername": "squidex-assets",
    "assetstore:exposesourceurl": "False",
    "assetstore:folder:path": "Assets",
    "assetstore:ftp:password": "",
    "assetstore:ftp:path": "Assets",
    "assetstore:ftp:serverhost": "",
    "assetstore:ftp:serverport": "21",
    "assetstore:ftp:username": "",
    "assetstore:googlecloud:bucket": "squidex-assets",
    "assetstore:mongodb:bucket": "fs",
    "assetstore:mongodb:configuration": "mongodb://localhost",
    "assetstore:mongodb:database": "SquidexAssets",
    "assetstore:type": "Folder",
    "caching:maxsurrogatekeyssize": "0",
    "caching:replicated:enable": "True",
    "caching:strongetag": "False",
    "contentroot": "/app",
    "contents:defaultpagesize": "200",
    "contents:defaultpagesizegraphql": "20",
    "contents:maxresults": "200",
    "dotnet_running_in_container": "true",
    "dotnet_version": "5.0.0",
    "email:notifications:existinguserbody": "Dear User,\r\n\r\n$ASSIGNER_NAME ($ASSIGNER_EMAIL) has invited you to join App $APP_NAME at Squidex Headless CMS.\r\n\r\nLogin or reload the Management UI to see the App.\r\n\r\nThank you very much,\r\nThe Squidex Team\r\n\r\n\u003C\u003CStart now!\u003E\u003E [$UI_URL]",
    "email:notifications:existingusersubject": "[Squidex CMS] You have been invited to join App $APP_NAME",
    "email:notifications:newuserbody": "Welcome to Squidex\r\nDear User,\r\n\r\n$ASSIGNER_NAME ($ASSIGNER_EMAIL) has invited you to join Project (also called an App) $APP_NAME at Squidex Headless CMS. Login with your Github, Google or Microsoft credentials to create a new user account and start editing content now.\r\n\r\nThank you very much,\r\nThe Squidex Team\r\n\r\n\u003C\u003CStart now!\u003E\u003E [$UI_URL]",
    "email:notifications:newusersubject": "You have been invited to join Project $APP_NAME at Squidex CMS",
    "email:notifications:usagebody": "Dear User,\r\n\r\nYou you are about to reach your usage limit for App $APP_NAME at Squidex Headless CMS.\r\n\r\nYou have already used $API_CALLS of your monthy limit of $API_CALLS_LIMIT API calls.\r\n\r\nPlease check your clients or upgrade your plan!\r\n\r\n\u003C\u003CGo to Squidex!\u003E\u003E [$UI_URL]",
    "email:notifications:usagesubject": "[Squidex CMS] You you are about to reach your usage limit for App $APP_NAME",
    "email:smtp:enablessl": "True",
    "email:smtp:password": "",
    "email:smtp:port": "587",
    "email:smtp:sender": "hello@squidex.io",
    "email:smtp:server": "",
    "email:smtp:username": "",
    "eventpublishers:alltorabbitmq:configuration": "amqp://guest:guest@localhost/",
    "eventpublishers:alltorabbitmq:enabled": "False",
    "eventpublishers:alltorabbitmq:eventsfilter": ".*",
    "eventpublishers:alltorabbitmq:exchange": "squidex",
    "eventpublishers:alltorabbitmq:type": "RabbitMq",
    "eventstore:cosmosdb:configuration": "https://localhost:8081",
    "eventstore:cosmosdb:database": "Squidex",
    "eventstore:cosmosdb:masterkey": "C2y6yDjf5/R\u002Bob0N8A7Cgv30VRDJIWEHLM\u002B4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==",
    "eventstore:geteventstore:configuration": "ConnectTo=tcp://admin:changeit@localhost:1113; HeartBeatTimeout=500; MaxReconnections=-1",
    "eventstore:geteventstore:prefix": "squidex",
    "eventstore:geteventstore:projectionhost": "localhost",
    "eventstore:mongodb:configuration": "mongodb://marketing-cms-datastore",
    "eventstore:mongodb:database": "Squidex",
    "eventstore:type": "MongoDB",
    "exposedconfiguration:version": "squidex:version",
    "fulltext:elastic:configuration": "http://localhost:9200",
    "fulltext:elastic:indexname": "squidex",
    "fulltext:type": "default",
    "healthz:gc:threshold": "4096",
    "home": "/root",
    "hostname": "aea3ea478ba3",
    "identity:adminclientid": "admin",
    "identity:adminclientsecret": "xxx",
    "identity:adminemail": "test@test.de",
    "identity:adminpassword": "xxx",
    "identity:adminrecreate": "False",
    "identity:allowpasswordauth": "True",
    "identity:githubclient": "",
    "identity:githubsecret": "",
    "identity:googleclient": "",
    "identity:googlesecret": "",
    "identity:lockautomatically": "False",
    "identity:microsoftclient": "",
    "identity:microsoftsecret": "",
    "identity:microsofttenant": "",
    "identity:oidcauthority": "",
    "identity:oidcclient": "",
    "identity:oidcname": "OIDC",
    "identity:oidcscopes:0": "email",
    "identity:oidcsecret": "",
    "identity:privacyurl": "https://squidex.io/privacy",
    "identity:showpii": "True",
    "kafka:bootstrapservers": "",
    "languages:custom": "",
    "logging:colors": "True",
    "logging:datadog": "False",
    "logging:human": "True",
    "logging:level": "Information",
    "logging:logprofiler": "False",
    "logging:logrequests": "True",
    "logging:storeretentationindays": "90",
    "mode:isreadonly": "False",
    "news:appname": "squidex-website",
    "news:clientid": "squidex-website:default",
    "news:clientsecret": "QGgqxd7bDHBTEkpC6fj8sbdPWgZrPrPfr3xzb3LKoec=",
    "orleans:clustering": "Development",
    "orleans:gatewayport": "40000",
    "orleans:ipaddress": "",
    "orleans:siloport": "11111",
    "path": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "plugins:0": "Squidex.Extensions.dll",
    "rebuild:apps": "False",
    "rebuild:assetfiles": "False",
    "rebuild:assets": "False",
    "rebuild:contents": "False",
    "rebuild:indexes": "False",
    "rebuild:rules": "False",
    "rebuild:schemas": "False",
    "robots:text": "User-agent: *\nAllow: /api/assets/*",
    "rules:executiontimeoutinseconds": "10",
    "running_in_container": "true",
    "store:mongodb:configuration": "mongodb://marketing-cms-datastore",
    "store:mongodb:contentdatabase": "SquidexContent",
    "store:mongodb:database": "Squidex",
    "store:type": "MongoDb",
    "translations:deepl:authkey": "",
    "translations:googlecloud:projectid": "",
    "twitter:clientid": "QZhb3HQcGCvE6G8yNNP9ksNet",
    "twitter:clientsecret": "Pdu9wdN72T33KJRFdFy1w4urBKDRzIyuKpc0OItQC2E616DuZD",
    "ui:disablescheduledchanges": "False",
    "ui:google:analyticsid": "UA-99989790-2",
    "ui:hidedatebuttons": "False",
    "ui:hidedatetimemodebutton": "False",
    "ui:hidenews": "False",
    "ui:hideonboarding": "False",
    "ui:map:googlemaps:key": "AIzaSyB_Z8l3nwUxZhMJykiDUJy6bSHXXlwcYMg",
    "ui:map:type": "OSM",
    "ui:onlyadminscancreateapps": "False",
    "ui:redirecttologin": "true",
    "ui:referencesdropdownitemcount": "100",
    "ui:regexsuggestions:email": "^[a-zA-Z0-9.!#$%\u0026\u2019*\u002B\\/=?^_\u0060{|}~-]\u002B@[a-zA-Z0-9-]\u002B(?:.[a-zA-Z0-9-]\u002B)*$",
    "ui:regexsuggestions:phone": "^\\(*\\\u002B*[1-9]{0,3}\\)*-*[1-9]{0,3}[-. /]*\\(*[2-9]\\d{2}\\)*[-. /]*\\d{3}[-. /]*\\d{4} *e*x*t*\\.* *\\d{0,4}$",
    "ui:regexsuggestions:slug": "^[a-z0-9]\u002B(\\-[a-z0-9]\u002B)*$",
    "ui:regexsuggestions:url": "^(?:http(s)?:\\/\\/)?[\\w.-]\u002B(?:\\.[\\w\\.-]\u002B)\u002B[\\w\\-\\._~:\\/?#%[\\]@!\\$\u0026\u0027\\(\\)\\*\\\u002B,;=.]\u002B$",
    "ui:showinfo": "False",
    "urls": "http://\u002B:80",
    "urls:baseurl": "https://marketing-cms",
    "urls:enableforwardheaders": "True",
    "urls:enforcehost": "False",
    "urls:enforcehttps": "False",
    "version": "5.0.0"
  },
  "app": {
    "name": "Squidex",
    "version": "5.4.0.0",
    "sessionId": "e716c961-4f9e-4fd8-9165-b075595f48dc"
  },
  "timestamp": "2021-02-01T17:49:01Z"
}

{
  "logLevel": "Information",
  "filters": {
    "costs": 0
  },
  "elapsedRequestMs": 26,
  "app": {
    "name": "Squidex",
    "version": "5.4.0.0",
    "sessionId": "e716c961-4f9e-4fd8-9165-b075595f48dc"
  },
  "web": {
    "requestId": "00-400abfb20a0d4d489ed7b5178cec0920-05f138ede3a74444-00",
    "requestPath": "/orleans",
    "requestMethod": "GET"
  },
  "timestamp": "2021-02-01T18:01:21Z"
}

{
  "logLevel": "Error",
  "message": "An unexpected exception has occurred.",
  "app": {
    "name": "Squidex",
    "version": "5.4.0.0",
    "sessionId": "e716c961-4f9e-4fd8-9165-b075595f48dc"
  },
  "web": {
    "requestId": "00-400abfb20a0d4d489ed7b5178cec0920-05f138ede3a74444-00",
    "requestPath": "/orleans",
    "requestMethod": "GET"
  },
  "timestamp": "2021-02-01T18:01:21Z",
  "exception": {
    "type": "System.InvalidOperationException",
    "message": "IDX20803: Unable to obtain configuration from: \u0027https://marketing-cms/identity-server/.well-known/openid-configuration\u0027.",
    "stackTrace": "   at Microsoft.IdentityModel.Protocols.ConfigurationManager\u00601.GetConfigurationAsync(CancellationToken cancel)\n   at Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler.HandleChallengeAsyncInternal(AuthenticationProperties properties)\n   at Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler.HandleChallengeAsync(AuthenticationProperties properties)\n   at Microsoft.AspNetCore.Authentication.AuthenticationHandler\u00601.ChallengeAsync(AuthenticationProperties properties)\n   at Microsoft.AspNetCore.Authentication.AuthenticationService.ChallengeAsync(HttpContext context, String scheme, AuthenticationProperties properties)\n   at Squidex.Areas.OrleansDashboard.Middlewares.OrleansDashboardAuthenticationMiddleware.InvokeAsync(HttpContext context) in /src/src/Squidex/Areas/OrleansDashboard/Middlewares/OrleansDashboardAuthenticationMiddleware.cs:line 47\n   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\n   at Squidex.Web.Pipeline.LocalCacheMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/src/Squidex.Web/Pipeline/LocalCacheMiddleware.cs:line 32\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.\u003C\u003Ec__DisplayClass6_1.\u003C\u003CUseMiddlewareInterface\u003Eb__1\u003Ed.MoveNext()\n--- End of stack trace from previous location ---\n   at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)\n   at Squidex.Web.Pipeline.UsageMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/src/Squidex.Web/Pipeline/UsageMiddleware.cs:line 91\n   at Squidex.Web.Pipeline.UsageMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/src/Squidex.Web/Pipeline/UsageMiddleware.cs:line 91\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.\u003C\u003Ec__DisplayClass6_1.\u003C\u003CUseMiddlewareInterface\u003Eb__1\u003Ed.MoveNext()\n--- End of stack trace from previous location ---\n   at Squidex.Web.Pipeline.RequestLogPerformanceMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/src/Squidex.Web/Pipeline/RequestLogPerformanceMiddleware.cs:line 61\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.\u003C\u003Ec__DisplayClass6_1.\u003C\u003CUseMiddlewareInterface\u003Eb__1\u003Ed.MoveNext()\n--- End of stack trace from previous location ---\n   at Squidex.Web.Pipeline.RequestExceptionMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in /src/src/Squidex.Web/Pipeline/RequestExceptionMiddleware.cs:line 69"
  }
}

I am not sure whether the problem is that the client uses “anonymous” access (maybe I can test this afterward).

Any ideas on how to debug this? The logs do not show any errors, except for the Cluster problem ( Regarding this: I can access the identity server at https://marketing-cms/identity-server/.well-known/openid-configuration without problem in the browser).

Expected behavior

Minimal reproduction of the problem

  • Create app, schemas, content, and a read-only API client with anonymous access
  • Make a GET request with Postman to access content => 404 (NotFound)
  • Refresh the Squidex UI in Chrome Browser and make the same GET request in Postman afterward => 200 (Success)
  • Wait some seconds and make the same GET request in Postman => 404 (NotFound)

If you are interested I can send you the mongodb folder (7mb) which can be used to reproduce the problem surely.

Environment

  • [x] Self hosted with docker
  • [ ] Self hosted with IIS
  • [ ] Self hosted with other version
  • [ ] Cloud version

Version: [5.4.0]

Browser:

  • [x] Chrome (desktop)
  • [ ] Chrome (Android)
  • [ ] Chrome (iOS)
  • [ ] Firefox
  • [ ] Safari (desktop)
  • [ ] Safari (iOS)
  • [ ] IE
  • [ ] Edge

Others:

thank you very much for your detailed support request.

The exception is not relevant. You can try to turn this setting off caching:replicated:enable. Do you use multiple instances of this container?

I have pushed a new version (dev-5489) with some logging for 404.

The build is not done yet, but it would be great if you can test it and check the logs for warnings: https://build.squidex.io/Squidex/squidex/5489/1/2

Thank you for your great and fast support!

  • I am not using multiple instances.
  • The caching__replicated__enable: "false" environment variable seems to fix the issue with the 404 partially. Means: now the same GET request with Postman seems to work.
  • But I have a setup container where I am using PowerShell to setup Squidex via REST API (create app, schemas,… if they don’t exist)). There I am using an admin (oauth) client (IDENTITY__adminClientId: “admin”. IDENTITY__adminClientSecret:“xxx”): I can successfully acquire the access token but a GET to /api/apps gives me an empty, 200 response, although my app exists. The setup container worked before running into this state.
  • The cluster overview in Squidex UI shows the same 500 problem.

Your dev build did not pass the tests, that’s why I think I cant find the tag dev-5489. If you try to publish a new debug version I would be glad to test it.
If you are interested I can send you a complete setup (docker-compose + mongo-db volume) where you can also reproduce.

Thanks again!

Is the image pushed? I can not pull squidex/squidex:dev-5490. Or shall I build the image locally?

Sorry, it was a PR, I have to merge it.

I tried to reproduce it. Can you create a step by step explanation what exactly you did and can you reproduce it? Btw: 5499 is pushed and can be tested.

Ok, I have interesting test results.

I used the image squidex/squidex:dev with caching__replicated__enable: "true" and recorded a log from first a successful Request with Postman to /api/content/<<myapp>>/slideshow (appname “obfuscated”) and after that (in the end) a 404 request with Postman (same endpoint).
There is only one relevant log entry:

{
    "logLevel": "Warning",
    "message": "Cannot find app with the given name.",
    "appId": "404",
    "appName": "",
    "timestamp": "2021-02-02T11:37:33Z",
    "app": {
      "name": "Squidex",
      "version": "4.0.0.0",
      "sessionId": "90d8547a-ee10-495c-8fa2-a3f1c6fb312e"
    },
    "web": {
      "requestId": "00-2a5db701054f69438860b7ebab1cc4ae-580a9e19c59bd24f-00",
      "requestPath": "/content/<<myapp>>/slideshow",
      "requestMethod": "GET",
      "routeValues": {
        "area": "Api",
        "action": "GetContents",
        "controller": "Contents"
      }
    }
  }

The interesting information:
There seems to be a side effect in combination with the setup container. This means If I do not start the setup container the 404 problem does not occur anymore.
What am I doing in the setup container:
I setup app, schema, via the Squidex Rest Api with the oauth client (with the credentials defined in identity:adminClientId, identity:adminClientSecret).
Basically currently it does the following:

  1. Health Check (GET /healthz)
  2. Aquire-AccessToken, after squidex healthy (POST /identity-server/connect/token)
  3. Get-Available-Apps (GET /api/apps) (returns currently an empty set)
  4. Create-App if not exists (POST /api/apps) => results in a crash because the app already exists => restart Setup Container and start with 1.

The setup container ran fine for some days, now it restarts in a loop due to the 404 problem.

I still do not understand it. Can you make this container available or the code…I need something I can reproduce. I guess you never delete the app in the container.

Of course. Link has been removed…

It contains a docker-compose with volumes. You need to add a Dns name mapping in your etc/hosts file for : marketing-cms -> 127.0.0.1 . It can be started via docker-compose up -d. Passwords are in the compose file. I tested with a GET request on https://marketing-cms/api/content/<<myadpp>>/slideshow

Great. Btw: It is probably much easier if you write a simple c# program with the SDK :wink:

Next time …

You are right, found the Sdk after my setup script.
Are you able to reproduce using this setup?

Not yet, but I had other things to do as well.

It just works for me. What I tried is to create a new app with your powershell script and then access it with a anonymous client over postman and it just works.

Did you try it with the mongo db volume? As I said for several days it worked for me too, but then I ran in this state, which is only reproducible if you use the mongo-db volume.

Not yet, which mongodb version do you use? If I do not have the exact version it does not start :frowning:

In the download, I send to you was a docker-compose file with a volumes folder. When you start the compose via docker-compose up Squidex, MongoDb starts with the versions defined in this docker-compose and with the volume from the volumes folder also defined there.

Sorry, I have not seen it. But this also does not work. I get errors from MongoDB that it cannot start.

Maybe a permission problem, sorry I have worked on windows with these files, if you have linux/mac you will need to chown the mongo-db folder to the correct user/group.