[IMPLEMENTED] Custom robots.txt

[spacecat]

I’ve been running some mobile validation tools in the Google Search Console and it appears it doesn’t like that Squidex is not allowing the Google bot to crawl and index assets (images).

“Googlebot blocked by robots.txt”

So, in addition to

[IMPLEMENTED] How to add robots.txt to a Squidex installation

Hi, After deploying Squidex I would like to prevent Google and other crawlers from crawling and indexing my Squidex web app installation. Is there an easy way of doing this without compiling and redeploying the app? Maybe this can be done through the shell - by creating a new robots.txt file in each running pod/container? Note: One prereq is that I am using the squidex:dev image on Docker Hub and don’t want to manually build Squidex - I want to keep using the squidex:dev image.

I would like to request a “custom robots.txt” feature. One idea is to have a UI/page under settings where you can edit the content of robots.txt which then is saved in mongo. This way we don’t need to think about writing to the file system and/or backup strategy of the same.

The solution I’m looking for looks something like this:

User-agent: *
Allow: /api/assets/
Disallow: /

robots.txt to disallow all pages except one? Do they override and cascade?

[Sebastian]

Good point, thank you for the input. I would just add it to the settings, then it is a 20 minute task.

[Sebastian]

See: https://github.com/Squidex/squidex/blob/master/src/Squidex/appsettings.json#L56