Kerberos SSO Broken After 2.2.2 Upgrade

Problem:
After upgrading from 2.1.4 to 2.2.2, our Foreman instance is no longer able to authenticate Kerberos SSO users. There is no failure message on the web interface when it displays the standard login page, but production.log shows SSO failed after the attempt.

Expected outcome:
Users are able to automatically log in using an active Kerberos ticket.

Foreman and Proxy versions:
foreman-2.2.2-1.el7
foreman-proxy-2.2.2-1.el7

Distribution and version:
CentOS 7.9.2009

Other relevant data:

We’re using the standard Kerberos SSO config from Foreman :: Manual. This has been functional for many versions up until now.

In the past, errors with Kerberos SSO would show a small error message at the top left of the browser from the ErrorDocument 401 '<html><meta http-equiv="refresh" content="0; URL=/users/login"><body>Kerberos authentication did not pass.</body></html>' tag in /etc/httpd/conf.d/auth_kerb.conf. This no longer appears.

I tested after the upgrade before running foreman-installer and got a failed attempt. I also ran foreman-installer and reconfigured our TLS to no avail. Logging set to debug doesn’t provide much useful info. We’re willing to upgrade again to 2.3.1 if that’s the fix.

Thanks!

This is possibly related to the switch from Passenger to Puma behind reverse proxy. It seems similar to Bug #30535: When using Puma with Foreman 2.1 FreeIPA external authentication does not work - Foreman which should be fixed in 2.2.2 already.
Can you confirm if you are running with Puma or Passenger as the webserver? Is there anything in the apache error logs? Which Kerberos provider are you using?

2 Likes

Interesting - I’m not specifying --foreman-passenger true so I’m assuming our instance has moved to Puma. If there’s a way to verify, let me know and I can do that.

One thing I see in the Apache config file (/etc/httpd/conf.d/05-foreman-ssl.conf) is these REMOTE USER unset lines:

  RequestHeader unset REMOTE_USER
  RequestHeader unset REMOTE_USER_EMAIL
  RequestHeader unset REMOTE_USER_FIRSTNAME
  RequestHeader unset REMOTE_USER_LASTNAME
  RequestHeader unset REMOTE_USER_GROUPS

I don’t see any reference to HTTP_REMOTE_USER as discussed in the issue you linked. Perhaps that’s related?

We’re using AD for Kerberos auth. As for apache errors, there are some but none seem to be appearing as the failed login occurs. There are tons of these in foreman_error.log:

[Thu Jan 14 01:38:13.216559 2021] [proxy:error] [pid 1948] [client MY_IP:43926] AH00898: Error reading from remote server returned by /cgi-bin/forum_2.php
[Thu Jan 14 01:38:13.612230 2021] [proxy_http:error] [pid 5888] (20014)Internal error: [client MY_IP:44054] AH01102: error reading status line from remote server 127.0.0.1:3000

There’s not much interesting in error_log.

Thanks!

Hi, the interesting log in this case would be /var/log/httpd/foreman-ssl_error_ssl.log as there should be logged the SSO failure.

1 Like

Thanks for the heads up. After changing the Apache LogLevel to debug and comparing to another instance where SSO is still working (2.1.4), this seems to be where things diverge:

2.2.2 (Broken):

[ssl:debug] [pid 29922] ssl_engine_kernel.c(225): [client CLIENT_IP:58902] AH02034: Initial (No.1) HTTPS request received for child 5 (server SERVER_FQDN:443)
[authz_core:debug] [pid 29922] mod_authz_core.c(835): [client CLIENT_IP:58902] AH01628: authorization result: granted (no directives)
[proxy:debug] [pid 29922] mod_proxy.c(1123): [client CLIENT_IP:58902] AH01143: Running scheme http handler (attempt 0)

2.1.4 (Working)

[ssl:debug] [pid 15197] ssl_engine_kernel.c(225): [client CLIENT_IP:54259] AH02034: Initial (No.1) HTTPS request received for child 2 (server SERVER_FQDN:443)
[authz_core:debug] [pid 15197] mod_authz_core.c(809): [client CLIENT_IP:54259] AH01626: authorization result of Require all granted: granted
[authz_core:debug] [pid 15197] mod_authz_core.c(809): [client CLIENT_IP:54259] AH01626: authorization result of <RequireAny>: granted
[deflate:debug] [pid 15197] mod_deflate.c(849): [client CLIENT_IP:54259] AH01384: Zlib: Compressed 16103 to 4093 : URL /

Specifically the authz_core:debug lines. Everything up until that point looks the same on both servers. Unfortunately this doesn’t mean much to me - any thoughts on what authorization result: granted (no directives) might refer to?

That means nothing to me :thinking:

First of all please try if the server has actually joined the realm by realm list.

If that is ok,
Could you compare the /etc/httpd/conf.d/05-foreman-ssl.d/ configs on both servers?
Maybe try the /var/log/httpd/foreman-ssl_access_ssl.log if there’s something interesting.

We aren’t using realmd in this config - just a keytab and an AD service account. That’s been working for ages on multiple Foreman instances, but I can try joining it to the domain if you think that would help.

As for 05-foreman-ssl.conf, this entire block is present on 2.2.2 and not present on 2.1.4:

  ## Request header rules
  ## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader
  RequestHeader set X_FORWARDED_PROTO "https"
  RequestHeader set SSL_CLIENT_S_DN "%{SSL_CLIENT_S_DN}s"
  RequestHeader set SSL_CLIENT_CERT "%{SSL_CLIENT_CERT}s"
  RequestHeader set SSL_CLIENT_VERIFY "%{SSL_CLIENT_VERIFY}s"
  RequestHeader unset REMOTE_USER
  RequestHeader unset REMOTE_USER_EMAIL
  RequestHeader unset REMOTE_USER_FIRSTNAME
  RequestHeader unset REMOTE_USER_LASTNAME
  RequestHeader unset REMOTE_USER_GROUPS

  ## SSL Proxy directives
  SSLProxyEngine On

  ## Proxy rules
  ProxyRequests Off
  ProxyPreserveHost On
  ProxyAddHeaders On
  ProxyPass /pulp !
  ProxyPass /pulp2 !
  ProxyPass /streamer !
  ProxyPass /pub !
  ProxyPass /icons !
  ProxyPass / http://127.0.0.1:3000/ retry=0
  ProxyPassReverse / http://127.0.0.1:3000/
  ## Rewrite rules
  RewriteEngine On

  #Upgrade Websocket connections
  RewriteCond %{HTTP:Upgrade} =websocket [NC]
  RewriteRule /(.*) ws://127.0.0.1:3000/$1 [P,L]

From the issue linked in tbrisker’s reply, it seems like there should be some header set in this file.

I have fixed this by changing app/services/sso/apache.rb to use HTTP_REMOTE_USER instead of REMOTE_USER and setting HTTP_REMOTE_USER in apache. I have created pull requests to foreman and puppet-foreman projects in case this is acceptable.

I don’t see any set lines for HTTP_REMOTE_USER in the config file, though our apache.rb does have references to HTTP_REMOTE_USER.

For what it’s worth, neither of the below lines fixed the issue when added to the Apache config file:

  RequestHeader set REMOTE_USER "%{REMOTE_USER}s"
  RequestHeader set HTTP_REMOTE_USER "%{HTTP_REMOTE_USER}s"

It looks like this problem has not been solved yet. We also have the problem. I’ve also been watching this post since February and have tried since Foreman 2.2.2 through 2.3.3 to now 2.4.0. Unfortunately no success. We have been using the Kerberos config at …/apache2/conf.d/05-foreman-ssl.d/auth_kerb.conf for almost 6 years now and the switch to PUMA probably broke it.

So at the moment we are running Foreman 2.4 on Debian 10 without IPA or AD.
If I let apache be Verbose, I also see “authorization result: granted (no directives)”.
But I can’t log in via Kerberos. If I change auth_kerb.conf so that no proper authentication is possible, I see at least the error from this line "ErrorDocument 401 ‘Kerberos authentication did not pass.’
". The authentication itself works in my eyes, but not the transfer of this information.

Glad to hear it’s not just us - if you guys come across a solution, please share!

I somehow missed this.

As a workaround, you can switch back to Passenger (for now). If you used the installer, it’s --foreman-passenger true. However, we are planning to drop Passenger support so that’s a blocker to upgrade at some point.

Can you tell us how you upgraded? Did you use the installer?

Sadly I don’t have a Kerberos setup to replicate.

1 Like

We use the debian repository. After the obligatory changes in the sources.list and the apt update && upgrade I run ‘foreman-installer’ without Parameter. No error appear in the process. I checked that the foreman-installer answers ipa is set to no, to exclude this.

I have not yet returned to passenger because I have the time to wait until Puma can do it. Rather I help with troubleshooting, sometime I have to use it anyway. :slight_smile:

Rolling back to Passenger didn’t fix this on our dev server. I’ve tried it both ways a few times. The --foreman-passenger true option on foreman-installer actually seems to leave a lot of the Puma config lines in the Apache config file; the RequestHeader lines, ProxyPass lines that refer to port 3000, etc. These lines aren’t present on our prod servers which are still using Passenger on 2.1.4.

We upgrade using the steps in the Upgrade section of the manual and foreman-installer.

I have found a workaround for a similar issue:

https://community.theforeman.org/t/actualize-sso-external-auth-from-2-1-to-2-4/22930

If you could confirm it works in your setting it would be possible to open a bugfix request backed up with some experimental data.

No luck here, but we are using Puma; it seems like you’re using Passenger? I’m hesitant to get comfortable with a Passenger-based config since it will eventually be going away, but I can give it a try if you’re looking for additional data points.

Yes, I’m still on Passenger (I think it’s because I come from a default Debian setup).
No, really it’s not worth: if you are on Puma the env variables’ names (actually the headers because there is a reverse proxy, but anyhow) are just fine.

Remind me if you don’t mind, what version of Foreman are you running and on what OS?

Sure - our dev server is on Version 2.3.3 running on CentOS 7.9.2009.

Same here. No luck with the above solution.

The Foreman Version ist 2.4.0 on Debian 10.

Could you share the contents of your lookup_identity.conf file please?

<LocationMatch ^/users/(ext)?login$>
  LookupUserAttr email REMOTE_USER_EMAIL
  LookupUserAttr firstname REMOTE_USER_FIRSTNAME
  LookupUserAttr lastname REMOTE_USER_LASTNAME
  LookupUserGroups REMOTE_USER_GROUPS :
  LookupUserGroupsIter REMOTE_USER_GROUP

# Set headers for proxy requests
  RequestHeader set REMOTE_USER %{REMOTE_USER}e
  RequestHeader set REMOTE_USER_EMAIL %{REMOTE_USER_EMAIL}e
  RequestHeader set REMOTE_USER_FIRSTNAME %{REMOTE_USER_FIRSTNAME}e
  RequestHeader set REMOTE_USER_LASTNAME %{REMOTE_USER_LASTNAME}e
  RequestHeader set REMOTE_USER_GROUPS %{REMOTE_USER_GROUPS}e
</LocationMatch>