Kerberos SSO Broken After 2.2.2 Upgrade

After upgrading from 2.1.4 to 2.2.2, our Foreman instance is no longer able to authenticate Kerberos SSO users. There is no failure message on the web interface when it displays the standard login page, but production.log shows SSO failed after the attempt.

Expected outcome:
Users are able to automatically log in using an active Kerberos ticket.

Foreman and Proxy versions:

Distribution and version:
CentOS 7.9.2009

Other relevant data:

We’re using the standard Kerberos SSO config from Foreman :: Manual. This has been functional for many versions up until now.

In the past, errors with Kerberos SSO would show a small error message at the top left of the browser from the ErrorDocument 401 '<html><meta http-equiv="refresh" content="0; URL=/users/login"><body>Kerberos authentication did not pass.</body></html>' tag in /etc/httpd/conf.d/auth_kerb.conf. This no longer appears.

I tested after the upgrade before running foreman-installer and got a failed attempt. I also ran foreman-installer and reconfigured our TLS to no avail. Logging set to debug doesn’t provide much useful info. We’re willing to upgrade again to 2.3.1 if that’s the fix.


This is possibly related to the switch from Passenger to Puma behind reverse proxy. It seems similar to Bug #30535: When using Puma with Foreman 2.1 FreeIPA external authentication does not work - Foreman which should be fixed in 2.2.2 already.
Can you confirm if you are running with Puma or Passenger as the webserver? Is there anything in the apache error logs? Which Kerberos provider are you using?


Interesting - I’m not specifying --foreman-passenger true so I’m assuming our instance has moved to Puma. If there’s a way to verify, let me know and I can do that.

One thing I see in the Apache config file (/etc/httpd/conf.d/05-foreman-ssl.conf) is these REMOTE USER unset lines:

  RequestHeader unset REMOTE_USER
  RequestHeader unset REMOTE_USER_EMAIL
  RequestHeader unset REMOTE_USER_FIRSTNAME
  RequestHeader unset REMOTE_USER_LASTNAME
  RequestHeader unset REMOTE_USER_GROUPS

I don’t see any reference to HTTP_REMOTE_USER as discussed in the issue you linked. Perhaps that’s related?

We’re using AD for Kerberos auth. As for apache errors, there are some but none seem to be appearing as the failed login occurs. There are tons of these in foreman_error.log:

[Thu Jan 14 01:38:13.216559 2021] [proxy:error] [pid 1948] [client MY_IP:43926] AH00898: Error reading from remote server returned by /cgi-bin/forum_2.php
[Thu Jan 14 01:38:13.612230 2021] [proxy_http:error] [pid 5888] (20014)Internal error: [client MY_IP:44054] AH01102: error reading status line from remote server

There’s not much interesting in error_log.


Hi, the interesting log in this case would be /var/log/httpd/foreman-ssl_error_ssl.log as there should be logged the SSO failure.

1 Like

Thanks for the heads up. After changing the Apache LogLevel to debug and comparing to another instance where SSO is still working (2.1.4), this seems to be where things diverge:

2.2.2 (Broken):

[ssl:debug] [pid 29922] ssl_engine_kernel.c(225): [client CLIENT_IP:58902] AH02034: Initial (No.1) HTTPS request received for child 5 (server SERVER_FQDN:443)
[authz_core:debug] [pid 29922] mod_authz_core.c(835): [client CLIENT_IP:58902] AH01628: authorization result: granted (no directives)
[proxy:debug] [pid 29922] mod_proxy.c(1123): [client CLIENT_IP:58902] AH01143: Running scheme http handler (attempt 0)

2.1.4 (Working)

[ssl:debug] [pid 15197] ssl_engine_kernel.c(225): [client CLIENT_IP:54259] AH02034: Initial (No.1) HTTPS request received for child 2 (server SERVER_FQDN:443)
[authz_core:debug] [pid 15197] mod_authz_core.c(809): [client CLIENT_IP:54259] AH01626: authorization result of Require all granted: granted
[authz_core:debug] [pid 15197] mod_authz_core.c(809): [client CLIENT_IP:54259] AH01626: authorization result of <RequireAny>: granted
[deflate:debug] [pid 15197] mod_deflate.c(849): [client CLIENT_IP:54259] AH01384: Zlib: Compressed 16103 to 4093 : URL /

Specifically the authz_core:debug lines. Everything up until that point looks the same on both servers. Unfortunately this doesn’t mean much to me - any thoughts on what authorization result: granted (no directives) might refer to?

That means nothing to me :thinking:

First of all please try if the server has actually joined the realm by realm list.

If that is ok,
Could you compare the /etc/httpd/conf.d/05-foreman-ssl.d/ configs on both servers?
Maybe try the /var/log/httpd/foreman-ssl_access_ssl.log if there’s something interesting.

We aren’t using realmd in this config - just a keytab and an AD service account. That’s been working for ages on multiple Foreman instances, but I can try joining it to the domain if you think that would help.

As for 05-foreman-ssl.conf, this entire block is present on 2.2.2 and not present on 2.1.4:

  ## Request header rules
  ## as per
  RequestHeader set X_FORWARDED_PROTO "https"
  RequestHeader set SSL_CLIENT_S_DN "%{SSL_CLIENT_S_DN}s"
  RequestHeader set SSL_CLIENT_CERT "%{SSL_CLIENT_CERT}s"
  RequestHeader unset REMOTE_USER
  RequestHeader unset REMOTE_USER_EMAIL
  RequestHeader unset REMOTE_USER_FIRSTNAME
  RequestHeader unset REMOTE_USER_LASTNAME
  RequestHeader unset REMOTE_USER_GROUPS

  ## SSL Proxy directives
  SSLProxyEngine On

  ## Proxy rules
  ProxyRequests Off
  ProxyPreserveHost On
  ProxyAddHeaders On
  ProxyPass /pulp !
  ProxyPass /pulp2 !
  ProxyPass /streamer !
  ProxyPass /pub !
  ProxyPass /icons !
  ProxyPass / retry=0
  ProxyPassReverse /
  ## Rewrite rules
  RewriteEngine On

  #Upgrade Websocket connections
  RewriteCond %{HTTP:Upgrade} =websocket [NC]
  RewriteRule /(.*) ws://$1 [P,L]

From the issue linked in tbrisker’s reply, it seems like there should be some header set in this file.

I have fixed this by changing app/services/sso/apache.rb to use HTTP_REMOTE_USER instead of REMOTE_USER and setting HTTP_REMOTE_USER in apache. I have created pull requests to foreman and puppet-foreman projects in case this is acceptable.

I don’t see any set lines for HTTP_REMOTE_USER in the config file, though our apache.rb does have references to HTTP_REMOTE_USER.

For what it’s worth, neither of the below lines fixed the issue when added to the Apache config file:

  RequestHeader set REMOTE_USER "%{REMOTE_USER}s"