Problem:
After upgrading from 2.1.4 to 2.2.2, our Foreman instance is no longer able to authenticate Kerberos SSO users. There is no failure message on the web interface when it displays the standard login page, but production.log shows SSO failed after the attempt.
Expected outcome:
Users are able to automatically log in using an active Kerberos ticket.
Foreman and Proxy versions:
foreman-2.2.2-1.el7
foreman-proxy-2.2.2-1.el7
Distribution and version:
CentOS 7.9.2009
Other relevant data:
Weāre using the standard Kerberos SSO config from Foreman :: Manual. This has been functional for many versions up until now.
In the past, errors with Kerberos SSO would show a small error message at the top left of the browser from the ErrorDocument 401 '<html><meta http-equiv="refresh" content="0; URL=/users/login"><body>Kerberos authentication did not pass.</body></html>' tag in /etc/httpd/conf.d/auth_kerb.conf. This no longer appears.
I tested after the upgrade before running foreman-installer and got a failed attempt. I also ran foreman-installer and reconfigured our TLS to no avail. Logging set to debug doesnāt provide much useful info. Weāre willing to upgrade again to 2.3.1 if thatās the fix.
This is possibly related to the switch from Passenger to Puma behind reverse proxy. It seems similar to Bug #30535: When using Puma with Foreman 2.1 FreeIPA external authentication does not work - Foreman which should be fixed in 2.2.2 already.
Can you confirm if you are running with Puma or Passenger as the webserver? Is there anything in the apache error logs? Which Kerberos provider are you using?
Interesting - Iām not specifying --foreman-passenger true so Iām assuming our instance has moved to Puma. If thereās a way to verify, let me know and I can do that.
One thing I see in the Apache config file (/etc/httpd/conf.d/05-foreman-ssl.conf) is these REMOTE USER unset lines:
I donāt see any reference to HTTP_REMOTE_USER as discussed in the issue you linked. Perhaps thatās related?
Weāre using AD for Kerberos auth. As for apache errors, there are some but none seem to be appearing as the failed login occurs. There are tons of these in foreman_error.log:
[Thu Jan 14 01:38:13.216559 2021] [proxy:error] [pid 1948] [client MY_IP:43926] AH00898: Error reading from remote server returned by /cgi-bin/forum_2.php
[Thu Jan 14 01:38:13.612230 2021] [proxy_http:error] [pid 5888] (20014)Internal error: [client MY_IP:44054] AH01102: error reading status line from remote server 127.0.0.1:3000
Thanks for the heads up. After changing the Apache LogLevel to debug and comparing to another instance where SSO is still working (2.1.4), this seems to be where things diverge:
[ssl:debug] [pid 15197] ssl_engine_kernel.c(225): [client CLIENT_IP:54259] AH02034: Initial (No.1) HTTPS request received for child 2 (server SERVER_FQDN:443)
[authz_core:debug] [pid 15197] mod_authz_core.c(809): [client CLIENT_IP:54259] AH01626: authorization result of Require all granted: granted
[authz_core:debug] [pid 15197] mod_authz_core.c(809): [client CLIENT_IP:54259] AH01626: authorization result of <RequireAny>: granted
[deflate:debug] [pid 15197] mod_deflate.c(849): [client CLIENT_IP:54259] AH01384: Zlib: Compressed 16103 to 4093 : URL /
Specifically the authz_core:debug lines. Everything up until that point looks the same on both servers. Unfortunately this doesnāt mean much to me - any thoughts on what authorization result: granted (no directives) might refer to?
First of all please try if the server has actually joined the realm by realm list.
If that is ok,
Could you compare the /etc/httpd/conf.d/05-foreman-ssl.d/ configs on both servers?
Maybe try the /var/log/httpd/foreman-ssl_access_ssl.log if thereās something interesting.
We arenāt using realmd in this config - just a keytab and an AD service account. Thatās been working for ages on multiple Foreman instances, but I can try joining it to the domain if you think that would help.
As for 05-foreman-ssl.conf, this entire block is present on 2.2.2 and not present on 2.1.4:
## Request header rules
## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader
RequestHeader set X_FORWARDED_PROTO "https"
RequestHeader set SSL_CLIENT_S_DN "%{SSL_CLIENT_S_DN}s"
RequestHeader set SSL_CLIENT_CERT "%{SSL_CLIENT_CERT}s"
RequestHeader set SSL_CLIENT_VERIFY "%{SSL_CLIENT_VERIFY}s"
RequestHeader unset REMOTE_USER
RequestHeader unset REMOTE_USER_EMAIL
RequestHeader unset REMOTE_USER_FIRSTNAME
RequestHeader unset REMOTE_USER_LASTNAME
RequestHeader unset REMOTE_USER_GROUPS
## SSL Proxy directives
SSLProxyEngine On
## Proxy rules
ProxyRequests Off
ProxyPreserveHost On
ProxyAddHeaders On
ProxyPass /pulp !
ProxyPass /pulp2 !
ProxyPass /streamer !
ProxyPass /pub !
ProxyPass /icons !
ProxyPass / http://127.0.0.1:3000/ retry=0
ProxyPassReverse / http://127.0.0.1:3000/
## Rewrite rules
RewriteEngine On
#Upgrade Websocket connections
RewriteCond %{HTTP:Upgrade} =websocket [NC]
RewriteRule /(.*) ws://127.0.0.1:3000/$1 [P,L]
From the issue linked in tbriskerās reply, it seems like there should be some header set in this file.
I have fixed this by changing app/services/sso/apache.rb to use HTTP_REMOTE_USER instead of REMOTE_USER and setting HTTP_REMOTE_USER in apache. I have created pull requests to foreman and puppet-foreman projects in case this is acceptable.
I donāt see any set lines for HTTP_REMOTE_USER in the config file, though our apache.rb does have references to HTTP_REMOTE_USER.
For what itās worth, neither of the below lines fixed the issue when added to the Apache config file:
RequestHeader set REMOTE_USER "%{REMOTE_USER}s"
RequestHeader set HTTP_REMOTE_USER "%{HTTP_REMOTE_USER}s"
It looks like this problem has not been solved yet. We also have the problem. Iāve also been watching this post since February and have tried since Foreman 2.2.2 through 2.3.3 to now 2.4.0. Unfortunately no success. We have been using the Kerberos config at ā¦/apache2/conf.d/05-foreman-ssl.d/auth_kerb.conf for almost 6 years now and the switch to PUMA probably broke it.
So at the moment we are running Foreman 2.4 on Debian 10 without IPA or AD.
If I let apache be Verbose, I also see āauthorization result: granted (no directives)ā.
But I canāt log in via Kerberos. If I change auth_kerb.conf so that no proper authentication is possible, I see at least the error from this line "ErrorDocument 401 āKerberos authentication did not pass.ā
". The authentication itself works in my eyes, but not the transfer of this information.
As a workaround, you can switch back to Passenger (for now). If you used the installer, itās --foreman-passenger true. However, we are planning to drop Passenger support so thatās a blocker to upgrade at some point.
Can you tell us how you upgraded? Did you use the installer?
Sadly I donāt have a Kerberos setup to replicate.
We use the debian repository. After the obligatory changes in the sources.list and the apt update && upgrade I run āforeman-installerā without Parameter. No error appear in the process. I checked that the foreman-installer answers ipa is set to no, to exclude this.
I have not yet returned to passenger because I have the time to wait until Puma can do it. Rather I help with troubleshooting, sometime I have to use it anyway.
Rolling back to Passenger didnāt fix this on our dev server. Iāve tried it both ways a few times. The --foreman-passenger true option on foreman-installer actually seems to leave a lot of the Puma config lines in the Apache config file; the RequestHeader lines, ProxyPass lines that refer to port 3000, etc. These lines arenāt present on our prod servers which are still using Passenger on 2.1.4.
We upgrade using the steps in the Upgrade section of the manual and foreman-installer.
No luck here, but we are using Puma; it seems like youāre using Passenger? Iām hesitant to get comfortable with a Passenger-based config since it will eventually be going away, but I can give it a try if youāre looking for additional data points.
Yes, Iām still on Passenger (I think itās because I come from a default Debian setup).
No, really itās not worth: if you are on Puma the env variablesā names (actually the headers because there is a reverse proxy, but anyhow) are just fine.
