Login problems behind haproxy

I have…

  • [ ] Checked the logs and have provided the logs if I found something suspicious there

I’m submitting a…

  • [ ] Regression (a behavior that stopped working in a new release)
  • [x] Bug report
  • [ ] Performance issue
  • [ ] Documentation issue or request

Current behavior

I have set up squidex as a docker image running on Docker swarm. We are using traefik as the proxy for the swarm. It is also running as docker containers and it is terminating ssl. When I go through haproxy, I can get to the login page but when I click the ‘log in’ link, the pop up login window is blank. When I bypass haproxy by putting the IP for the docker swarm in /etc/hosts, squidex works as expected.

Already logged in users can get right to the /app page and everything works normally. API calls work normally. It’s only logging in that is a problem.

http 2 is enabled on haproxy and I’ve checked and re-checked the hostnames in all of the configurations.

Expected behavior

I would expect that putting squidex behind haproxy would not break squidex.

Minimal reproduction of the problem

Install squidex 1.16.2 via docker
Create users and applications in squidex
Place squidex behind haproxy 1.8.20

Environment

  • [x ] Self hosted with docker
  • [ ] Self hosted with IIS
  • [ ] Self hosted with other version
  • [ ] Cloud version

Version: [VERSION]

Browser:

  • [ x] Chrome (desktop)
  • [ ] Chrome (Android)
  • [ ] Chrome (iOS)
  • [x ] Firefox
  • [ ] Safari (desktop)
  • [ ] Safari (iOS)
  • [ ] IE
  • [ ] Edge

Squidex docker compose:
squidex:
image: squidex/squidex:v1.16.2
environment:
- URLS__BASEURL=https://squidex-qa.
- URLS__ENFORCEHTTPS=true
- EVENTSTORE__CONSUME=true
- EVENTSTORE__MONGODB__CONFIGURATION=mongodb://mongo-qa.
- STORE__MONGODB__CONFIGURATION=mongodb://mongo-qa.
- IDENTITY__ADMINEMAIL=webdev@
- IDENTITY__ADMINPASSWORD=
- VIRTUAL_HOST=squidex-qa.
- IDENTITY__GITHUBCLIENT=${SQUIDEX_GITHUBCLIENT}
- IDENTITY__GOOGLECLIENT=${SQUIDEX_GOOGLECLIENT}
- IDENTITY__MICROSOFTCLIENT=${SQUIDEX_MICROSOFTCLIENT}
volumes:
- /mnt/sasquidexqa/assets:/app/Assets
deploy:
labels:
- “traefik.backend.loadbalancer.method=drr”
- “traefik.frontend.rule=Host:squidex-qa.”
- “traefik.port=80”
- “traefik.docker.network=ops_proxy”
placement:
constraints:
- node.role == worker
mode: replicated
replicas: 2
restart_policy:
condition: on-failure
networks:
- internal
- proxy

haproxy config:
global
log 127.0.0.1 local2

chroot      /var/lib/haproxy
pidfile     /var/run/haproxy.pid
maxconn     4000
user        haproxy
group       haproxy
daemon

tune.ssl.default-dh-param 4096
ssl-default-bind-options no-sslv3 no-tls-tickets no-tlsv10 no-tlsv11
ssl-default-bind-ciphers ECDHE+aRSA+AES256+GCM+SHA384:ECDHE+aRSA+AES128+GCM+SHA256:ECDHE+aRSA+AE

S256+SHA384:ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:!aNULL:!MD5:!DSS

stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------

Defaults for all the ‘listen’ and ‘backend’ sections

#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option httpclose
option abortonclose
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 10m
timeout server 10m
timeout http-keep-alive 10s
timeout check 10s
maxconn 5000
#---------------------------------------------------------------------

Frontend which proxys to the backends

#---------------------------------------------------------------------
frontend localhost
bind *:80
bind *:443 ssl crt /path/to/cert1.pem crt /path/to/cert2.pem crt /path/to/cert3.pem alpn http/1.1,h2
mode http
rspidel Server

redirect scheme https if !{ ssl_fc }

use_backend squidex if { hdr(host) -i squidex-qa.<my domain> }

backend squidex
balance roundrobin
option forwardfor

server kbrdsmql01 10.200.1.181:443 check ssl verify none
server kbrdsmql02 10.200.1.182:443 check ssl verify none

Others:

DO you see if the X-FORWARED-PROTO header is set? Is there something in the logs?

Going through haproxy, these are the response headers:
HTTP/1.1 302 Found
Content-Length: 0
Date: Mon, 17 Jun 2019 21:47:34 GMT
Connection: close

Bypassing haproxy, these are the reponse headers:
HTTP/2.0 302 Found
date: Mon, 17 Jun 2019 21:56:02 GMT
location: https://squidex-qa./identity-server/Account/Login?ReturnUrl=%2Fidentity-server%2Fconnect%2Fauthorize%2Fcallback%3Fclient_id%3Dsquidex-frontend%26redirect_uri%3Dhttps%253A%252F%252Fsquidex-qa.%252Fclient-callback-popup%26response_type%3Did_token%2520token%26scope%3Dsquidex-api%2520openid%2520profile%2520email%2520squidex-profile%2520role%2520permissions%26state%3D4813d3101fb04394860198622c23e28f%26nonce%3D67222e250ae14d7cabc1612762d7b9f9%26display%3Dpopup
content-length: 0
X-Firefox-Spdy: h2

So it looks like even though I enabled http 2 in haproxy, it might still be a problem.

The X-FORWARDED-PROTO header is not anywhere that I see.

The logs don’t show any obvious problems. Here’s a section of log where it failed behind haproxy:
{
“logLevel”: “Information”,
“filters”: {
“costs”: 0.0
},
“elapsedRequestMs”: 0,
“app”: {
“name”: “Squidex”,
“version”: “1.0.0.0”,
“sessionId”: “1eaa5e9e-c1df-4450-af9f-7157fd257097”
},
“web”: {
“requestId”: “dfbb574e-e05a-47fd-a17b-e65844818f57”,
“requestPath”: “/build/index.html”,
“requestMethod”: “GET”
},
“timestamp”: “2019-06-17T22:19:08Z”
}

{
“logLevel”: “Information”,
“filters”: {
“costs”: 0.0
},
“elapsedRequestMs”: 0,
“app”: {
“name”: “Squidex”,
“version”: “1.0.0.0”,
“sessionId”: “1eaa5e9e-c1df-4450-af9f-7157fd257097”
},
“web”: {
“requestId”: “dc864cb5-77f0-4fc1-95d2-da214a426c66”,
“requestPath”: “/build/shims.js”,
“requestMethod”: “GET”
},
“timestamp”: “2019-06-17T22:19:08Z”
}

{
“logLevel”: “Information”,
“filters”: {
“costs”: 0.0
},
“elapsedRequestMs”: 0,
“app”: {
“name”: “Squidex”,
“version”: “1.0.0.0”,
“sessionId”: “1eaa5e9e-c1df-4450-af9f-7157fd257097”
},
“web”: {
“requestId”: “9cb98b2d-b080-4c01-be67-cff30f84da11”,
“requestPath”: “/images/loader.gif”,
“requestMethod”: “GET”
},
“timestamp”: “2019-06-17T22:19:08Z”
}

{
“logLevel”: “Information”,
“message”: “Invoking IdentityServer endpoint: IdentityServer4.Endpoints.DiscoveryEndpoint for /.well-known/openid-configuration”,
“endpointType”: “IdentityServer4.Endpoints.DiscoveryEndpoint”,
“url”: “/.well-known/openid-configuration”,
“app”: {
“name”: “Squidex”,
“version”: “1.0.0.0”,
“sessionId”: “1eaa5e9e-c1df-4450-af9f-7157fd257097”
},
“web”: {
“requestId”: “cf394ea9-92a9-4fda-804e-8c36e9a8e902”,
“requestPath”: “/.well-known/openid-configuration”,
“requestMethod”: “GET”
},
“timestamp”: “2019-06-17T22:19:11Z”,
“category”: “IdentityServer4.Hosting.IdentityServerMiddleware”
}

{
“logLevel”: “Information”,
“filters”: {
“costs”: 0.0
},
“elapsedRequestMs”: 0,
“app”: {
“name”: “Squidex”,
“version”: “1.0.0.0”,
“sessionId”: “1eaa5e9e-c1df-4450-af9f-7157fd257097”
},
“web”: {
“requestId”: “cf394ea9-92a9-4fda-804e-8c36e9a8e902”,
“requestPath”: “/identity-server/.well-known/openid-configuration”,
“requestMethod”: “GET”
},
“timestamp”: “2019-06-17T22:19:11Z”
}

{
“logLevel”: “Information”,
“message”: “Invoking IdentityServer endpoint: IdentityServer4.Endpoints.AuthorizeEndpoint for /connect/authorize”,
“endpointType”: “IdentityServer4.Endpoints.AuthorizeEndpoint”,
“url”: “/connect/authorize”,
“app”: {
“name”: “Squidex”,
“version”: “1.0.0.0”,
“sessionId”: “1eaa5e9e-c1df-4450-af9f-7157fd257097”
},
“web”: {
“requestId”: “75bcebba-5b20-4885-95ee-95519b0745ad”,
“requestPath”: “/connect/authorize”,
“requestMethod”: “GET”
},
“timestamp”: “2019-06-17T22:19:11Z”,
“category”: “IdentityServer4.Hosting.IdentityServerMiddleware”
}

{
“logLevel”: “Information”,
“message”: “ValidatedAuthorizeRequest\n{\n “ClientId”: “squidex-frontend”,\n “ClientName”: “squidex-frontend”,\n “RedirectUri”: “ht tps://squidex-qa./client-callback-popup”,\n “AllowedRedirectUris”: [\n “ht tps://squidex-qa./login;”,\n “ht tps://squidex-qa./client-callback-silent”,\n “ht tps://squidex-qa./client-callback-popup”\n ],\n “SubjectId”: “anonymous”,\n “ResponseType”: “id_token token”,\n “ResponseMode”: “fragment”,\n “GrantType”: “implicit”,\n “RequestedScopes”: “squidex-api openid profile email squidex-profile role permissions”,\n “State”: “a0a7c2b29ae743fa8e027e6dbd7300b4”,\n “Nonce”: “0033ac8c25bc48f5a40643ae103f6b8c”,\n “DisplayMode”: “popup”,\n “Raw”: {\n “client_id”: “squidex-frontend”,\n “redirect_uri”: “ht tps://squidex-qa./client-callback-popup”,\n “response_type”: “id_token token”,\n “scope”: “squidex-api openid profile email squidex-profile role permissions”,\n “state”: “a0a7c2b29ae743fa8e027e6dbd7300b4”,\n “nonce”: “0033ac8c25bc48f5a40643ae103f6b8c”,\n “display”: “popup”\n }\n}”,
@validationDetails”: “{\n “ClientId”: “squidex-frontend”,\n “ClientName”: “squidex-frontend”,\n “RedirectUri”: “https://squidex-qa./client-callback-popup”,\n “AllowedRedirectUris”: [\n “ht tps://squidex-qa./login;”,\n “ht tps://squidex-qa./client-callback-silent”,\n “ht tps://squidex-qa./client-callback-popup”\n ],\n “SubjectId”: “anonymous”,\n “ResponseType”: “id_token token”,\n “ResponseMode”: “fragment”,\n “GrantType”: “implicit”,\n “RequestedScopes”: “squidex-api openid profile email squidex-profile role permissions”,\n “State”: “a0a7c2b29ae743fa8e027e6dbd7300b4”,\n “Nonce”: “0033ac8c25bc48f5a40643ae103f6b8c”,\n “DisplayMode”: “popup”,\n “Raw”: {\n “client_id”: “squidex-frontend”,\n “redirect_uri”: “ht tps://squidex-qa./client-callback-popup”,\n “response_type”: “id_token token”,\n “scope”: “squidex-api openid profile email squidex-profile role permissions”,\n “state”: “a0a7c2b29ae743fa8e027e6dbd7300b4”,\n “nonce”: “0033ac8c25bc48f5a40643ae103f6b8c”,\n “display”: “popup”\n }\n}”,
“app”: {
“name”: “Squidex”,
“version”: “1.0.0.0”,
“sessionId”: “1eaa5e9e-c1df-4450-af9f-7157fd257097”
},
“web”: {
“requestId”: “75bcebba-5b20-4885-95ee-95519b0745ad”,
“requestPath”: “/connect/authorize”,
“requestMethod”: “GET”
},
“timestamp”: “2019-06-17T22:19:11Z”,
“category”: “IdentityServer4.Endpoints.AuthorizeEndpoint”
}

{
“logLevel”: “Information”,
“message”: “Showing login: User is not authenticated”,
“app”: {
“name”: “Squidex”,
“version”: “1.0.0.0”,
“sessionId”: “1eaa5e9e-c1df-4450-af9f-7157fd257097”
},
“web”: {
“requestId”: “75bcebba-5b20-4885-95ee-95519b0745ad”,
“requestPath”: “/connect/authorize”,
“requestMethod”: “GET”
},
“timestamp”: “2019-06-17T22:19:11Z”,
“category”: “IdentityServer4.ResponseHandling.AuthorizeInteractionResponseGenerator”
}

{
“logLevel”: “Information”,
“filters”: {
“costs”: 0.0
},
“elapsedRequestMs”: 1,
“app”: {
“name”: “Squidex”,
“version”: “1.0.0.0”,
“sessionId”: “1eaa5e9e-c1df-4450-af9f-7157fd257097”
},
“web”: {
“requestId”: “75bcebba-5b20-4885-95ee-95519b0745ad”,
“requestPath”: “/identity-server/connect/authorize”,
“requestMethod”: “GET”
},
“timestamp”: “2019-06-17T22:19:11Z”
}

**** I had to mangle the logged urls so that my post was allowed, they were totally normal before I added the spaces in https.

Do you use https from proxy to Squidex or http?. In case you use https you have to tell haproxy to add this header, otherwise you would get a redirect loop. Can restart Squidex, try it again and then send me the full logs as file?

browser <----> haproxy <----> traefik (docker swarm border proxy) <----> squidex

It’s https from the browser to traefik, and http from traefik to squidex.

I added X-FORWARDED-PROTO to the back end config, but it didn’t make a difference.

The log file is here, with attempts with and without the header change.

https://send.firefox.com/download/c3dc6dafa8bb860c/#xvH5Iikg4nhYN1Z5EfoAcQ

THe logs look good. In another case there was an issue that the header was skipped by the proxy because it was too large. Can you check that? Is there something in the logs of HAProxy?

Haproxy is not logging any errors or warnings. The more I dig into this, the more I wonder if it’s an issue of haproxy not speaking http2 on the backend causing problems for the build in identity server. Are there http2-only parts of the built in identity server?

I don’t think so. Identity server is just a few ASP.NET Core Middlewares, so it should speak Http2 with Kestrel.

If it would not speak http2 you would not see any requests at all, I guess. But they look okay.