Did anything change with Identity server last couple of days?
My site went down again and now I can’t log in to the Squidex editorial interface anymore?
I click on “Login to Squidex” green button and then I just get an empty popup which says 502 Bad Gateway.
Checked the Squidex and Mongo logs but I don’t see any errors there.
Any ideas?
Also, I’m wondering if I can go back to a previous Squidex docker image? Currently I’m using the dev tag which always pull the latest Squidex.
Kind of. I recently unified some storages and one part was the XmlRepository, where keys are stored. Can you clear your cookies and try it again? But I have deployed it to the cloud and have not seen any problems there yet.
Okay.
Cleared all site data, including cookies, I get the popup but when I enter my credentials and hit Login I get the 502 again.
Can you also clear the cookies of the popup? There are isolated to the /identity-server path.
Did not work either.
Had time to rework the docker images tags btw? I’m thinking maybe I can script some devops thing where I take the dev image from your repo and retag it and push it to mine. Then from AKS I can pull from my private repo instead. That way I have more control of the image tags and also rollbacks will work properly in kubernetes. As a temporary solution. I still need to fix the current problem though because I have revisionHistoryLimit to 2 and both revisions are broken since I’ve killed the pods a couple of times now.
About docker tags:
What do you mean? What I could do is to also add the build version to the dev build., e.g. :dev-332
About your problem:
Is there nothing in the logs?
I have two warnings:
Hosting environment: Production
Content root path: /app
Now listening on: http://[::]:80
Application started. Press Ctrl+C to shut down.
{
"logLevel": "Warning",
"message": "Remote socket closed while receiving connection preamble data from endpoint 10.244.0.220:33677.",
"eventId": {
"id": 101307
},
"app": {
"name": "Squidex",
"version": "1.0.0.0",
"sessionId": "a7392fd3-43f1-4e85-ac73-be316c4cd895"
},
"timestamp": "2018-12-06T14:50:46.8567273Z",
"category": "Orleans.Runtime.Messaging.IncomingMessageAcceptor"
}
{
"logLevel": "Warning",
"message": "Exception getting a sending socket to endpoint S10.244.0.220:11111:281803837",
"eventId": {
"id": 101021
},
"exception": {
"type": "System.TimeoutException",
"message": "Connection to 10.244.0.220:11111 could not be established in 00:00:05",
"stackTrace": " at Orleans.Runtime.SocketManager.Connect(Socket s, IPEndPoint endPoint, TimeSpan connectionTimeout)\n at Orleans.Runtime.SocketManager.SendingSocketCreator(IPEndPoint target)\n at Orleans.Runtime.LRU`2.Get(TKey key)\n at Orleans.Runtime.Messaging.SiloMessageSender.GetSendingSocket(Message msg, Socket& socket, SiloAddress& targetSilo, String& error)"
},
"app": {
"name": "Squidex",
"version": "1.0.0.0",
"sessionId": "a7392fd3-43f1-4e85-ac73-be316c4cd895"
},
"timestamp": "2018-12-06T14:50:53.6180040Z",
"category": "Runtime.Messaging.SiloMessageSender/PingSender"
}
0 errors.
And the rest are "logLevel": "Information". Here are the authentication part:
"logLevel": "Information",
"elapsedRequestMs": 388,
"app": {
"name": "Squidex",
"version": "1.0.0.0",
"sessionId": "b1bb4b50-6df4-4f11-9582-7a507896d9eb"
},
"web": {
"requestId": "1449b7f2-4be0-4b49-a52c-f0ac1aa09033",
"requestPath": "/identity-server/connect/authorize",
"requestMethod": "GET"
},
"timestamp": "2018-12-06T14:51:08.0062848Z"
}
{
"logLevel": "Information",
"message": "Invoking IdentityServer endpoint: IdentityServer4.Endpoints.DiscoveryEndpoint for /.well-known/openid-configuration",
"endpointType": "IdentityServer4.Endpoints.DiscoveryEndpoint",
"url": "/.well-known/openid-configuration",
"app": {
"name": "Squidex",
"version": "1.0.0.0",
"sessionId": "b1bb4b50-6df4-4f11-9582-7a507896d9eb"
},
"web": {
"requestId": "b55b109b-ad14-471b-b3ab-e2759ddec954",
"requestPath": "/.well-known/openid-configuration",
"requestMethod": "GET"
},
"timestamp": "2018-12-06T14:58:19.1093140Z",
"category": "IdentityServer4.Hosting.IdentityServerMiddleware"
}
{
"logLevel": "Information",
"elapsedRequestMs": 1,
"app": {
"name": "Squidex",
"version": "1.0.0.0",
"sessionId": "b1bb4b50-6df4-4f11-9582-7a507896d9eb"
},
"web": {
"requestId": "b55b109b-ad14-471b-b3ab-e2759ddec954",
"requestPath": "/identity-server/.well-known/openid-configuration",
"requestMethod": "GET"
},
"timestamp": "2018-12-06T14:58:19.1096497Z"
}
{
"logLevel": "Information",
"message": "Invoking IdentityServer endpoint: IdentityServer4.Endpoints.AuthorizeEndpoint for /connect/authorize",
"endpointType": "IdentityServer4.Endpoints.AuthorizeEndpoint",
"url": "/connect/authorize",
"app": {
"name": "Squidex",
"version": "1.0.0.0",
"sessionId": "b1bb4b50-6df4-4f11-9582-7a507896d9eb"
},
"web": {
"requestId": "ff785c2a-9053-4124-aec1-d40899e882f0",
"requestPath": "/connect/authorize",
"requestMethod": "GET"
},
"timestamp": "2018-12-06T15:00:45.5149595Z",
"category": "IdentityServer4.Hosting.IdentityServerMiddleware"
}
{
"logLevel": "Information",
"message": "ValidatedAuthorizeRequest\n{\n \"ClientId\": \"squidex-frontend\",\n \"ClientName\": \"squidex-frontend\",\n \"RedirectUri\": \"https://squidex.mydomain.com/client-callback-popup\",\n \"AllowedRedirectUris\": [\n \"https://squidex.mydomain.com/login;\",\n \"https://squidex.mydomain.com/client-callback-silent\",\n \"https://squidex.mydomain.com/client-callback-popup\"\n ],\n \"SubjectId\": \"5bacce9c5935680001f6cba6\",\n \"ResponseType\": \"id_token token\",\n \"ResponseMode\": \"fragment\",\n \"GrantType\": \"implicit\",\n \"RequestedScopes\": \"squidex-api openid profile email squidex-profile role permissions\",\n \"State\": \"1da10ecc55054fb08e9d9386748155f7\",\n \"Nonce\": \"a8a0b32ca75d4f9e8350f4a14709d655\",\n \"DisplayMode\": \"popup\",\n \"SessionId\": \"cc102dadccf5a05a1020158768b4d7f0\",\n \"Raw\": {\n \"client_id\": \"squidex-frontend\",\n \"redirect_uri\": \"https://squidex.mydomain.com/client-callback-popup\",\n \"response_type\": \"id_token token\",\n \"scope\": \"squidex-api openid profile email squidex-profile role permissions\",\n \"state\": \"1da10ecc55054fb08e9d9386748155f7\",\n \"nonce\": \"a8a0b32ca75d4f9e8350f4a14709d655\",\n \"display\": \"popup\"\n }\n}",
"validationDetails": "{\n \"ClientId\": \"squidex-frontend\",\n \"ClientName\": \"squidex-frontend\",\n \"RedirectUri\": \"https://squidex.mydomain.com/client-callback-popup\",\n \"AllowedRedirectUris\": [\n \"https://squidex.mydomain.com/login;\",\n \"https://squidex.mydomain.com/client-callback-silent\",\n \"https://squidex.mydomain.com/client-callback-popup\"\n ],\n \"SubjectId\": \"5bacce9c5935680001f6cba6\",\n \"ResponseType\": \"id_token token\",\n \"ResponseMode\": \"fragment\",\n \"GrantType\": \"implicit\",\n \"RequestedScopes\": \"squidex-api openid profile email squidex-profile role permissions\",\n \"State\": \"1da10ecc55054fb08e9d9386748155f7\",\n \"Nonce\": \"a8a0b32ca75d4f9e8350f4a14709d655\",\n \"DisplayMode\": \"popup\",\n \"SessionId\": \"cc102dadccf5a05a1020158768b4d7f0\",\n \"Raw\": {\n \"client_id\": \"squidex-frontend\",\n \"redirect_uri\": \"https://squidex.mydomain.com/client-callback-popup\",\n \"response_type\": \"id_token token\",\n \"scope\": \"squidex-api openid profile email squidex-profile role permissions\",\n \"state\": \"1da10ecc55054fb08e9d9386748155f7\",\n \"nonce\": \"a8a0b32ca75d4f9e8350f4a14709d655\",\n \"display\": \"popup\"\n }\n}",
"app": {
"name": "Squidex",
"version": "1.0.0.0",
"sessionId": "b1bb4b50-6df4-4f11-9582-7a507896d9eb"
},
"web": {
"requestId": "ff785c2a-9053-4124-aec1-d40899e882f0",
"requestPath": "/connect/authorize",
"requestMethod": "GET"
},
"timestamp": "2018-12-06T15:00:45.5163785Z",
"category": "IdentityServer4.Endpoints.AuthorizeEndpoint"
}
{
"logLevel": "Information",
"message": "Authorize endpoint response\n{\n \"SubjectId\": \"5bacce9c5935680001f6cba6\",\n \"ClientId\": \"squidex-frontend\",\n \"RedirectUri\": \"https://squidex.mydomain.com/client-callback-popup\",\n \"State\": \"1da10ecc55054fb08e9d9386748155f7\",\n \"Scope\": \"openid profile email squidex-profile role permissions squidex-api\"\n}",
"response": "{\n \"SubjectId\": \"5bacce9c5935680001f6cba6\",\n \"ClientId\": \"squidex-frontend\",\n \"RedirectUri\": \"https://squidex.mydomain.com/client-callback-popup\",\n \"State\": \"1da10ecc55054fb08e9d9386748155f7\",\n \"Scope\": \"openid profile email squidex-profile role permissions squidex-api\"\n}",
"app": {
"name": "Squidex",
"version": "1.0.0.0",
"sessionId": "b1bb4b50-6df4-4f11-9582-7a507896d9eb"
},
"web": {
"requestId": "ff785c2a-9053-4124-aec1-d40899e882f0",
"requestPath": "/connect/authorize",
"requestMethod": "GET"
},
"timestamp": "2018-12-06T15:00:45.5222509Z",
"category": "IdentityServer4.Endpoints.AuthorizeEndpoint"
}About the tags:
dev-332 would work as long as each and every tag is unique.
First the good news: I have also added the build number to the docker tags.
Can you try an incognito window? Just to ensure that it is not a cookie problem.
502 Bad Gateway is very strange, does not really sound like an identity server problem I think.
Yes, I use an nginx ingress. Will check the nginx logs.
Incognito window did not work either.
Great with the docker tags!
With the new permission system, your claims could become larger. Could be an indicator why nginx is not happy about it.
Yes, I see that error message in nginx logs:
upstream sent too big header while reading response header from upstream
Thank you, Sebastian. Works now. It must have been the new permission system leading to larger header.
And for the next folks having the same problem:
Add this to your ingress manifest:
"nginx.ingress.kubernetes.io/proxy-buffer-size": "8k"
For us it’s still not working with this annotation
“nginx.ingress.kubernetes.io/proxy-buffer-size”: “8k”
We have tried increasing the size upto 8m but still didn’t work. It doesn’t give 502 but the page keeps refreshring and we see 401 on userinfo request.
I have no idea what it is. I think the docker-compose files with nginx and caddy work fine, so it must be something else.
@Sebastian resuming on Saurav’s post, the log in with identity server seems to be working fine now after adding the annotation to have an increased proxy buffer size, but it’s now logging us out as soon as we log in.
Do you see something in the logs?
