Trying to add custom templates repos

d-Pixie · June 15, 2023, 1:59pm

I have…

[x] Read the following guideline: https://docs.squidex.io/01-getting-started/installation/troubleshooting-and-support. I understand that my support request might get deleted if I do not follow the guideline.
[x] Used code blocks with ``` to format my code examples like JSON or logs properly.

I’m submitting a…

[ ] Regression (a behavior that stopped working in a new release)
[ ] Bug report
[ ] Performance issue
[x] Documentation issue or request

Current behavior

I’m running into problems trying to use a separate repo for the custom templates. I cloned the official templates repo to https://github.com/accodeing/squidex-templates and configured the container to use:

env.TEMPLATES__REPOSITORIES__CONTENTURL: https://raw.githubusercontent.com/accodeing/squidex-templates/main
env.TEMPLATES__REPOSITORIES__GITURL: https://github.com/accodeing/squidex-templates.git

But I get an error while loading the site about it not being able to fetch the templates. Any ideas as to what is wrong here @Sebastian?

Screenshot 2023-06-13 at 12.50.12

Expected behavior

Cloning the official repo and replacing the URLs using ENV config should result in templates being loaded from the cloned repo.

Minimal reproduction of the problem

Setup a new installation of Squidex with the above config and launch. The error presents as soon as the dashboard is loaded.

Environment

App Name:

[x] Self hosted with docker
[ ] Self hosted with IIS
[ ] Self hosted with other version
[ ] Cloud version

Version: 7.0.2

Browser:

[x] Chrome (desktop)
[ ] Chrome (Android)
[ ] Chrome (iOS)
[ ] Firefox
[ ] Safari (desktop)
[ ] Safari (iOS)
[ ] IE
[ ] Edge

Others:
The app is deployed on GCP managed Kubernetes, using the v7 Helm chart.

Sebastian · June 15, 2023, 2:51pm

Do you see something in the logs?

d-Pixie · June 16, 2023, 7:00am

Should ofc have attached them in the original request… I get a generic 500 error in the browser console when the dashboard loads and the server side logs for that request is attached.

One thing I note is that the request it says it makes is for https://raw.githubusercontent.com/Squidex/templates/main/README.md and not, as I would expect: https://raw.githubusercontent.com/accodeing/squidex-templates/main/README.md. Not sure if this is a hardcoded log message, I set the wrong ENV var or it is not picked up correctly …

They seem to be correctly configured if I inspect the running container:

Screenshot 2023-06-16 at 9.57.05 1

But I’m not 100% sure since the documentation for how to do this is … sparse

I’d be happy to contribute a part to the docs on how to set this up, once we get it working.

Logs

[
  {
    "insertId": "f7v6l93nt5e2gexq",
    "jsonPayload": {
      "app": {
        "sessionId": "2755c75e-4d0e-4dbb-94dc-381eac528d9f",
        "name": "Squidex",
        "version": "7.0.2.0"
      },
      "web": {
        "requestPath": "/images/add-app.svg",
        "requestMethod": "GET",
        "requestId": "00-4820f23cc07f4f35f054d635cf4594ce-b02d913d8ef858fe-01"
      },
      "filters": {
        "costs": 0,
        "userId": "648724738a1f03e8bc35dd9b"
      },
      "logLevel": "Information",
      "timestamp": "2023-06-16T06:24:01Z",
      "message": "HTTP request executed.",
      "elapsedRequestMs": 0
    },
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "squidex",
        "location": "europe-north1-a",
        "pod_name": "squidex-squidex7-d8c7ccc6-hjxnp",
        "project_id": "accodeing-saas",
        "container_name": "squidex7",
        "cluster_name": "primary"
      }
    },
    "timestamp": "2023-06-16T06:24:01.749599835Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app_kubernetes_io/role": "api",
      "compute.googleapis.com/resource_name": "gke-primary-high-mem-pool-fbf58d2b-5pxg",
      "k8s-pod/pod-template-hash": "d8c7ccc6",
      "k8s-pod/app_kubernetes_io/instance": "squidex",
      "k8s-pod/app_kubernetes_io/component": "squidex",
      "k8s-pod/app_kubernetes_io/name": "squidex7"
    },
    "logName": "projects/accodeing-saas/logs/stdout",
    "receiveTimestamp": "2023-06-16T06:24:02.781368426Z"
  },
  {
    "insertId": "h4e9iheo17l38a6w",
    "jsonPayload": {
      "uri": "https://raw.githubusercontent.com/Squidex/templates/main/README.md",
      "eventId": {
        "name": "RequestPipelineStart",
        "id": 100
      },
      "httpMethod": "GET",
      "web": {
        "requestId": "00-9c8593c9fd3cff4ebfaf968722ac9361-de127ee93f398f0e-01",
        "routeValues": {
          "controller": "Templates",
          "area": "api",
          "action": "GetTemplates"
        },
        "requestPath": "/api/templates",
        "requestMethod": "GET"
      },
      "app": {
        "version": "7.0.2.0",
        "name": "Squidex",
        "sessionId": "2755c75e-4d0e-4dbb-94dc-381eac528d9f"
      },
      "logLevel": "Information",
      "timestamp": "2023-06-16T06:24:01Z",
      "category": "System.Net.Http.HttpClient.Default.LogicalHandler",
      "message": "Start processing HTTP request GET https://raw.githubusercontent.com/Squidex/templates/main/README.md"
    },
    "resource": {
      "type": "k8s_container",
      "labels": {
        "cluster_name": "primary",
        "container_name": "squidex7",
        "location": "europe-north1-a",
        "pod_name": "squidex-squidex7-d8c7ccc6-hjxnp",
        "project_id": "accodeing-saas",
        "namespace_name": "squidex"
      }
    },
    "timestamp": "2023-06-16T06:24:01.834836747Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app_kubernetes_io/role": "api",
      "compute.googleapis.com/resource_name": "gke-primary-high-mem-pool-fbf58d2b-5pxg",
      "k8s-pod/app_kubernetes_io/name": "squidex7",
      "k8s-pod/app_kubernetes_io/instance": "squidex",
      "k8s-pod/pod-template-hash": "d8c7ccc6",
      "k8s-pod/app_kubernetes_io/component": "squidex"
    },
    "logName": "projects/accodeing-saas/logs/stdout",
    "receiveTimestamp": "2023-06-16T06:24:02.781368426Z"
  },
  {
    "insertId": "kgmmxupne05men6h",
    "jsonPayload": {
      "timestamp": "2023-06-16T06:24:01Z",
      "httpMethod": "GET",
      "logLevel": "Information",
      "category": "System.Net.Http.HttpClient.Default.ClientHandler",
      "message": "Sending HTTP request GET https://raw.githubusercontent.com/Squidex/templates/main/README.md",
      "app": {
        "version": "7.0.2.0",
        "name": "Squidex",
        "sessionId": "2755c75e-4d0e-4dbb-94dc-381eac528d9f"
      },
      "web": {
        "requestMethod": "GET",
        "requestId": "00-9c8593c9fd3cff4ebfaf968722ac9361-de127ee93f398f0e-01",
        "requestPath": "/api/templates",
        "routeValues": {
          "area": "api",
          "controller": "Templates",
          "action": "GetTemplates"
        }
      },
      "eventId": {
        "name": "RequestStart",
        "id": 100
      },
      "uri": "https://raw.githubusercontent.com/Squidex/templates/main/README.md"
    },
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "europe-north1-a",
        "pod_name": "squidex-squidex7-d8c7ccc6-hjxnp",
        "cluster_name": "primary",
        "project_id": "accodeing-saas",
        "container_name": "squidex7",
        "namespace_name": "squidex"
      }
    },
    "timestamp": "2023-06-16T06:24:01.835994538Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app_kubernetes_io/role": "api",
      "k8s-pod/app_kubernetes_io/instance": "squidex",
      "k8s-pod/app_kubernetes_io/component": "squidex",
      "compute.googleapis.com/resource_name": "gke-primary-high-mem-pool-fbf58d2b-5pxg",
      "k8s-pod/app_kubernetes_io/name": "squidex7",
      "k8s-pod/pod-template-hash": "d8c7ccc6"
    },
    "logName": "projects/accodeing-saas/logs/stdout",
    "receiveTimestamp": "2023-06-16T06:24:02.781368426Z"
  },
  {
    "insertId": "52yvhod4jinaas31",
    "jsonPayload": {
      "logLevel": "Information",
      "web": {
        "requestMethod": "GET",
        "requestId": "00-93ffea1ee3658d67aaa336be4b354d27-b4d8012b7dd47b58-01",
        "requestPath": "/api/users/648724738a1f03e8bc35dd9b/notifications"
      },
      "filters": {
        "clientId": "squidex-frontend",
        "costs": 0,
        "userId": "648724738a1f03e8bc35dd9b"
      },
      "app": {
        "name": "Squidex",
        "version": "7.0.2.0",
        "sessionId": "2755c75e-4d0e-4dbb-94dc-381eac528d9f"
      },
      "message": "HTTP request executed.",
      "elapsedRequestMs": 3,
      "timestamp": "2023-06-16T06:24:01Z"
    },
    "resource": {
      "type": "k8s_container",
      "labels": {
        "project_id": "accodeing-saas",
        "cluster_name": "primary",
        "container_name": "squidex7",
        "pod_name": "squidex-squidex7-d8c7ccc6-hjxnp",
        "namespace_name": "squidex",
        "location": "europe-north1-a"
      }
    },
    "timestamp": "2023-06-16T06:24:01.967023560Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "d8c7ccc6",
      "compute.googleapis.com/resource_name": "gke-primary-high-mem-pool-fbf58d2b-5pxg",
      "k8s-pod/app_kubernetes_io/name": "squidex7",
      "k8s-pod/app_kubernetes_io/role": "api",
      "k8s-pod/app_kubernetes_io/component": "squidex",
      "k8s-pod/app_kubernetes_io/instance": "squidex"
    },
    "logName": "projects/accodeing-saas/logs/stdout",
    "receiveTimestamp": "2023-06-16T06:24:02.781368426Z"
  },
  {
    "insertId": "3rqkuro33vf9vem3",
    "jsonPayload": {
      "web": {
        "requestPath": "/api/templates",
        "routeValues": {
          "action": "GetTemplates",
          "area": "api",
          "controller": "Templates"
        },
        "requestMethod": "GET",
        "requestId": "00-9c8593c9fd3cff4ebfaf968722ac9361-de127ee93f398f0e-01"
      },
      "message": "Received HTTP response headers after 552.9585ms - 200",
      "timestamp": "2023-06-16T06:24:02Z",
      "app": {
        "version": "7.0.2.0",
        "name": "Squidex",
        "sessionId": "2755c75e-4d0e-4dbb-94dc-381eac528d9f"
      },
      "statusCode": "200",
      "eventId": {
        "id": 101,
        "name": "RequestEnd"
      },
      "logLevel": "Information",
      "elapsedMilliseconds": "552.9585",
      "category": "System.Net.Http.HttpClient.Default.ClientHandler"
    },
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "squidex",
        "pod_name": "squidex-squidex7-d8c7ccc6-hjxnp",
        "location": "europe-north1-a",
        "project_id": "accodeing-saas",
        "container_name": "squidex7",
        "cluster_name": "primary"
      }
    },
    "timestamp": "2023-06-16T06:24:02.397157145Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app_kubernetes_io/name": "squidex7",
      "k8s-pod/pod-template-hash": "d8c7ccc6",
      "compute.googleapis.com/resource_name": "gke-primary-high-mem-pool-fbf58d2b-5pxg",
      "k8s-pod/app_kubernetes_io/role": "api",
      "k8s-pod/app_kubernetes_io/instance": "squidex",
      "k8s-pod/app_kubernetes_io/component": "squidex"
    },
    "logName": "projects/accodeing-saas/logs/stdout",
    "receiveTimestamp": "2023-06-16T06:24:02.781368426Z"
  },
  {
    "insertId": "ph5igbzv6glx7qh5",
    "jsonPayload": {
      "statusCode": "200",
      "category": "System.Net.Http.HttpClient.Default.LogicalHandler",
      "elapsedMilliseconds": "602.0516",
      "app": {
        "version": "7.0.2.0",
        "sessionId": "2755c75e-4d0e-4dbb-94dc-381eac528d9f",
        "name": "Squidex"
      },
      "logLevel": "Information",
      "eventId": {
        "id": 101,
        "name": "RequestPipelineEnd"
      },
      "timestamp": "2023-06-16T06:24:02Z",
      "message": "End processing HTTP request after 602.0516ms - 200",
      "web": {
        "routeValues": {
          "area": "api",
          "action": "GetTemplates",
          "controller": "Templates"
        },
        "requestId": "00-9c8593c9fd3cff4ebfaf968722ac9361-de127ee93f398f0e-01",
        "requestPath": "/api/templates",
        "requestMethod": "GET"
      }
    },
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "squidex",
        "container_name": "squidex7",
        "location": "europe-north1-a",
        "pod_name": "squidex-squidex7-d8c7ccc6-hjxnp",
        "cluster_name": "primary",
        "project_id": "accodeing-saas"
      }
    },
    "timestamp": "2023-06-16T06:24:02.399081288Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app_kubernetes_io/instance": "squidex",
      "k8s-pod/app_kubernetes_io/component": "squidex",
      "k8s-pod/pod-template-hash": "d8c7ccc6",
      "k8s-pod/app_kubernetes_io/role": "api",
      "k8s-pod/app_kubernetes_io/name": "squidex7",
      "compute.googleapis.com/resource_name": "gke-primary-high-mem-pool-fbf58d2b-5pxg"
    },
    "logName": "projects/accodeing-saas/logs/stdout",
    "receiveTimestamp": "2023-06-16T06:24:02.781368426Z"
  },
  {
    "insertId": "w3xkbzeyd1c60ijn",
    "jsonPayload": {
      "category": "Squidex.Web.ApiExceptionFilterAttribute",
      "exception": {
        "type": "System.NullReferenceException",
        "message": "Object reference not set to an instance of an object.",
        "stackTrace": "   at Squidex.Domain.Apps.Entities.Apps.Templates.TemplatesClient.GetTemplatesAsync(CancellationToken ct) in /src/src/Squidex.Domain.Apps.Entities/Apps/Templates/TemplatesClient.cs:line 62\n   at Squidex.Areas.Api.Controllers.Templates.TemplatesController.GetTemplates() in /src/src/Squidex/Areas/Api/Controllers/Templates/TemplatesController.cs:line 42\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)"
      },
      "message": "An unexpected exception has occurred.",
      "web": {
        "routeValues": {
          "action": "GetTemplates",
          "area": "api",
          "controller": "Templates"
        },
        "requestMethod": "GET",
        "requestPath": "/api/templates",
        "requestId": "00-9c8593c9fd3cff4ebfaf968722ac9361-11d126bf7bec99cf-01"
      },
      "timestamp": "2023-06-16T06:24:02Z",
      "logLevel": "Error",
      "app": {
        "sessionId": "2755c75e-4d0e-4dbb-94dc-381eac528d9f",
        "name": "Squidex",
        "version": "7.0.2.0"
      }
    },
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "europe-north1-a",
        "cluster_name": "primary",
        "pod_name": "squidex-squidex7-d8c7ccc6-hjxnp",
        "namespace_name": "squidex",
        "container_name": "squidex7",
        "project_id": "accodeing-saas"
      }
    },
    "timestamp": "2023-06-16T06:24:02.429341208Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app_kubernetes_io/instance": "squidex",
      "k8s-pod/app_kubernetes_io/name": "squidex7",
      "k8s-pod/pod-template-hash": "d8c7ccc6",
      "k8s-pod/app_kubernetes_io/component": "squidex",
      "k8s-pod/app_kubernetes_io/role": "api",
      "compute.googleapis.com/resource_name": "gke-primary-high-mem-pool-fbf58d2b-5pxg"
    },
    "logName": "projects/accodeing-saas/logs/stdout",
    "receiveTimestamp": "2023-06-16T06:24:02.781368426Z"
  },
  {
    "insertId": "y4leharbknfhv5hc",
    "jsonPayload": {
      "filters": {
        "costs": 0,
        "userId": "648724738a1f03e8bc35dd9b",
        "clientId": "squidex-frontend"
      },
      "timestamp": "2023-06-16T06:24:02Z",
      "elapsedRequestMs": 680,
      "logLevel": "Information",
      "app": {
        "version": "7.0.2.0",
        "sessionId": "2755c75e-4d0e-4dbb-94dc-381eac528d9f",
        "name": "Squidex"
      },
      "message": "HTTP request executed.",
      "web": {
        "requestPath": "/api/templates",
        "requestId": "00-9c8593c9fd3cff4ebfaf968722ac9361-11d126bf7bec99cf-01",
        "requestMethod": "GET"
      }
    },
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "squidex",
        "cluster_name": "primary",
        "pod_name": "squidex-squidex7-d8c7ccc6-hjxnp",
        "location": "europe-north1-a",
        "container_name": "squidex7",
        "project_id": "accodeing-saas"
      }
    },
    "timestamp": "2023-06-16T06:24:02.432430054Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/app_kubernetes_io/role": "api",
      "compute.googleapis.com/resource_name": "gke-primary-high-mem-pool-fbf58d2b-5pxg",
      "k8s-pod/app_kubernetes_io/component": "squidex",
      "k8s-pod/app_kubernetes_io/name": "squidex7",
      "k8s-pod/pod-template-hash": "d8c7ccc6",
      "k8s-pod/app_kubernetes_io/instance": "squidex"
    },
    "logName": "projects/accodeing-saas/logs/stdout",
    "receiveTimestamp": "2023-06-16T06:24:02.781368426Z"
  },
  {
    "insertId": "tyillajk64vb4djb",
    "jsonPayload": {
      "message": "HTTP request executed.",
      "logLevel": "Information",
      "elapsedRequestMs": 845,
      "filters": {
        "userId": "648724738a1f03e8bc35dd9b",
        "clientId": "squidex-frontend",
        "costs": 0
      },
      "web": {
        "requestPath": "/api/news/features",
        "requestId": "00-aca562f8d8da947d917d43be5a4fe129-c0c5c5402e8482d5-01",
        "requestMethod": "GET"
      },
      "timestamp": "2023-06-16T06:24:02Z",
      "app": {
        "sessionId": "2755c75e-4d0e-4dbb-94dc-381eac528d9f",
        "version": "7.0.2.0",
        "name": "Squidex"
      }
    },
    "resource": {
      "type": "k8s_container",
      "labels": {
        "project_id": "accodeing-saas",
        "location": "europe-north1-a",
        "cluster_name": "primary",
        "namespace_name": "squidex",
        "container_name": "squidex7",
        "pod_name": "squidex-squidex7-d8c7ccc6-hjxnp"
      }
    },
    "timestamp": "2023-06-16T06:24:02.590066599Z",
    "severity": "INFO",
    "labels": {
      "k8s-pod/pod-template-hash": "d8c7ccc6",
      "k8s-pod/app_kubernetes_io/role": "api",
      "k8s-pod/app_kubernetes_io/component": "squidex",
      "k8s-pod/app_kubernetes_io/instance": "squidex",
      "compute.googleapis.com/resource_name": "gke-primary-high-mem-pool-fbf58d2b-5pxg",
      "k8s-pod/app_kubernetes_io/name": "squidex7"
    },
    "logName": "projects/accodeing-saas/logs/stdout",
    "receiveTimestamp": "2023-06-16T06:24:02.781368426Z"
  }
]

d-Pixie · June 16, 2023, 7:12am

After updating to 7.6.1, to helm chart defaulted to 7.0.2, the dashboard loads without issue. I still get the three default template options, but that might be because it is parsing the README.md instead of looking at the repo structure to figure out the available templates?

I did delete the default templates from the forked repo, but I haven’t updated the README …

d-Pixie · June 16, 2023, 10:05am

I tried editing the README to remove the bullets for the templates I had removed, and restarted the service but the same three templates show up anyway …

Sebastian · June 16, 2023, 12:47pm

The URL you use returns a 404: https://raw.githubusercontent.com/accodeing/squidex-templates/main/README.md.

Is it public?

d-Pixie · June 16, 2023, 2:12pm

Eh … For me that works fine - in an incognito window Yes, it is public. It is a fork of the official templates repo and has the same visibility as that - non changeable due to it being a fork.

Just to be on the safe side I have requested that it be disconnected from the upstream repo - the changes we are going to make will not be of interests to others anyway.

Strange that you can not load it though. Sure it wasn’t just during a period when Github was unreachable? It has happened a fair bit lately …

d-Pixie · June 16, 2023, 2:18pm

Also I get no errors in the logs now, the server requests and loads a readme:

Start processing HTTP request GET https://raw.githubusercontent.com/Squidex/templates/main/README.md

But it still looks like it is fetching the wrong URL, ignoring the ENV config.

Sebastian · June 16, 2023, 4:12pm

I think arrays use indexers in environment variables, e.g. REPOSITORIES__0__GITHUBURL

d-Pixie · June 19, 2023, 1:43pm

That seemed to solve the first issue, it’s loading the correct repo without error now. But no templates are showing up in the dashboard (except for the default new app one).

Sebastian · June 21, 2023, 12:58pm

The code reads the main readme and then looks for all other readme files that are referenced.

If the name starts with “Starter” it is shown on the main page, otherwise on the settings page only.

d-Pixie · June 28, 2023, 7:42am

Ok. All I have done is to clone the main repo, delete a few of the directories, and update the main README to remove the references to the removed template folders. But I still don’t get the “Starter Sample Profile” - the only one I left alone - on the dashboard.

You can check the repo I’m using - linked above - if you want to check that I didn’t break anything.

Sebastian · June 28, 2023, 8:09am

The name must start with “Starter”.