Kerberos SSO Broken After 2.2.2 Upgrade

Ours matches this exactly, though the file name is lookup_identity.conf.erb, not lookup_identity.conf

I had the opportunity the last two days to get Foreman out of production use and test it intensively. The following was tested on a Debian 10.10 with Apache 2.4.38 including mod_auth_kerb without FreeIPA in combination with Foreman 2.4, 2.4.1 and 2.5.1.

The file /etc/apache2/conf.d/auth_kerb.conf looks like this:

<Location /users/extlogin>
 AuthType Kerberos
 AuthName "Kerberos Login"
 KrbMethodNegotiate On
 KrbSaveCredentials off
 KrbMethodK5Passwd on
 KrbAuthRealms REALM.NAME
 Krb5KeyTab /.../krb5-apache.keytab
 KrbLocalUserMapping on
 require valid-user
 ErrorDocument 401 '<html><meta http-equiv="refresh" content="0; URL=/users/login"><body>Kerberos authentication did not pass.</body></html>'
</Location>

The lookup_identity.conf file is no longer in use, it is not relevant to our scenario.
I set the logging from apache as well as from foreman to debug.

Scenario 1 - Foreman with Puma:

Apache receives the Kerberos Ticket, acknowledges in the log that it is cutting off the realm, that it is a valid_user and forwards the information to the foreman.

kerb_authenticate_user entered with user (NULL) and auth_type Kerberos
Acquiring creds for HTTP@...
Verifying client data using KRB5 GSS-API
Client delegated us their credential
GSS-API token of length 22 bytes will be sent back
kerb_authenticate_a_name_to_local_name user@... -> user
AH01626: authorization result of Require valid-user : granted
AH01626: authorization result of <RequireAny>: granted
AH01143: Running scheme http handler (attempt 0)
AH00942: HTTP: has acquired connection for (foreman)
AH00944: connecting http://foreman/users/extlogin to foreman:80
AH02545: http: has determined UDS as /run/foreman.sock
AH00947: connected /users/extlogin to httpd-UDS:0
AH00943: http: has released connection for (foreman)
AH02001: Connection closed to child 141 with standard shutdown (server ...:443)

In the production log of the Foreman you can read only:

Started GET "/users/extlogin" for *.*.*.* at 2021-06-29 15:26:19 +0200
Processing by UsersController#extlogin as HTML
SSO failed
falling back to login form
Redirected to https://.../users/login
Filter chain halted as :require_login rendered or redirected

If I remove in /usr/share/foreman/app/controllers/application_controller.rb and /usr/share/foreman/app/services/sso/apache.rb all leading HTTP_ in front of REMOTE_USER, REMOTE_USER_EMAIL etc. then nothing changes.

I have also tried other solutions from the Internet, some of which scriner has already tried, as I noticed when I reread this topic, unfortunately without success. The only noticeable change came with:

  RequestHeader set REMOTE_USER "%{REMOTE_USER}s"
  RequestHeader set HTTP_REMOTE_USER "%{HTTP_REMOTE_USER}s"

After I added it to …/sites-enabled/05-foreman-ssl.conf above ``RequestHeader set X_FORWARDED_PROTO “https”``` and commented out the “unset” lines below it, the foreman seemed to accept Kerberos authentication, but displayed an error that the user’s name must not be empty. I wasn’t sure what name exactly was meant since it didn’t explicitly say “username”. Unfortunately I couldn’t find the exact wording, although I documented everything separately.

Scenario 2 - Foreman with Passenger:

From the existing environment, I run foreman-installer --foreman-passenger true. Now passenger is running instead of puma. Out of the box, in the files /usr/share/foreman/app/controllers/application_controller.rb and /usr/share/foreman/app/services/sso/apache.rb the variables still have HTTP_ in front of them and Kerberos authentication fails. But if I remove this HTTP_ in front of the variables, I can log in via Kerberos. Regardless of this, node.rb is no longer usable to query the hosts. The error is always that the request was made with a certificate without DN and is therefore not allowed. Strange, but it works with puma. I have run this process several times to be sure that I have not changed anything else. Unfortunately, I could not determine here where the error comes from exactly and why. Some files like …/sites-enabled/05-foreman-ssl.conf or the Foreman configurations under /etc/foreman/ I could compare with another working production system ( Foreman 1.24 ) and the difference was not worth mentioning. Also the puppetserver didn’t seem to be changed. At least I didn’t find any changed configuration. But I also didn’t used find to list all the last changed files, in retrospect I should have.

During my internet research I noticed that the program “zammad” might also have had this problem in connection with puma. Unfortunately, I have not found out exactly. Probably due to my inexperience in analyzing such problems I must have missed something, but my impression is that mod_auth_kerb passes the information in the HTTP header with REMOTE_USER and puma expects HTTP_REMOTE_USER and thus it fails.

I hope that my explanations can be useful in some way or that someone has found a solution in the meantime.

1 Like

Recently we switched from mod_auth_kerb to mod_auth_gssapi:
https://github.com/theforeman/puppet-foreman/commit/91850a2b4522406a28dbc98e2c3f215130d08dab

This will land in the next release. Perhaps this is something you can try? Foreman :: Manual is updated

Hey thanks for the feedback. I have tried this directly. I have included the Nightly Repsoitory for Debian and updated the Foreman to version 2.6.0-develop. Next installed auth_gssapi for apache2. My /etc/apache2/conf.d/05-foreman-ssl.d/auth_kerb.conf now looks like this

<Location /users/extlogin>
 AuthType GSSAPI
 AuthName "GSSAPI Single Sign On Login"
 GssapiCredStore keytab:/etc/apache2/krb5-apache.keytab
 GssapiSSLonly On
 GssapiLocalName On
 require valid-user
 ErrorDocument 401 '<html><meta http-equiv="refresh" content="0; URL=/users/login"><body>Kerberos authentication did not pass.</body></html>'
</Location>

Apache seems happy with it and forwards it to foreman

[authz_core:debug] [pid 76614:tid 139623145666304] mod_authz_core.c(846): [client x.x.x.x:50530] AH01628: authorization result: granted (no directives)
[auth_gssapi:debug] [pid 76614:tid 139623145666304] mod_auth_gssapi.c(727): [client x.x.x.x:50530] GSSapiImpersonate not On, skipping impersonation.
[proxy:debug] [pid 76614:tid 139623145666304] mod_proxy.c(1249): [client x.x.x.x:50530] AH01143: Running scheme http handler (attempt 0)
[proxy:debug] [pid 76614:tid 139623145666304] proxy_util.c(2316): AH00942: HTTP: has acquired connection for (foreman)
[proxy:debug] [pid 76614:tid 139623145666304] proxy_util.c(2369): [client x.x.x.x:50530] AH00944: connecting http://foreman/ to foreman:80
[ssl:debug] [pid 76614:tid 139623145666304] ssl_engine_kernel.c(2235): [client x.x.x.x:50530] AH02041: Protocol: TLSv1.3, Cipher: TLS_AES_256_GCM_SHA384 (256/256 bits)
[proxy:debug] [pid 76614:tid 139623145666304] proxy_util.c(2406): [client x.x.x.x:50530] AH02545: http: has determined UDS as /run/foreman.sock
[proxy:debug] [pid 76614:tid 139623145666304] proxy_util.c(2578): [client x.x.x.x:50530] AH00947: connected / to httpd-UDS:0
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(495): AH00831: socache_shmcb_store (0xa7 -> subcache 7)
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(849): AH00847: insert happened at idx=6, data=(8511:8543)
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(854): AH00848: finished insert, subcache: idx_pos/idx_used=0/7, data_pos/data_used=0/8742
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(516): AH00834: leaving socache_shmcb_store successfully
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(495): AH00831: socache_shmcb_store (0xd9 -> subcache 25)
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(849): AH00847: insert happened at idx=2, data=(3289:3321)
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(854): AH00848: finished insert, subcache: idx_pos/idx_used=0/3, data_pos/data_used=0/3520
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(516): AH00834: leaving socache_shmcb_store successfully
[proxy:debug] [ppid 76614:tid 139623145666304] proxy_util.c(2331): AH00943: http: has released connection for (foreman)
[ssl:debug] [pid 76614:tid 139623145666304] ssl_engine_kernel.c(383): [client x.x.x.x:50530] AH02034: Subsequent (No.2) HTTPS request received for child 72 (server xxx:443)
[authz_core:debug] [pid 76614:tid 139623145666304] mod_authz_core.c(820): [client x.x.x.x:50530] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[authz_core:debug] [pid 76614:tid 139623145666304] mod_authz_core.c(820): [client x.x.x.x:50530] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[auth_gssapi:debug] [pid 76614:tid 139623145666304] mod_auth_gssapi.c(895): [client x.x.x.x:50530] URI: /users/extlogin, no main, no prev
[auth_gssapi:info] [pid 76614:tid 139623145666304] [client x.x.x.x:50530] NO AUTH DATA Client did not send any authentication headers
..........
[auth_gssapi:debug] [pid 76614:tid 139623145666304] mod_auth_gssapi.c(895): [client x.x.x.x:50530] URI: /users/extlogin, no main, no prev
[authz_core:debug] [pid 76614:tid 139623145666304] mod_authz_core.c(820): [client x.x.x.x:50530] AH01626: authorization result of Require valid-user : granted
[authz_core:debug] [pid 76614:tid 139623145666304] mod_authz_core.c(820): [client x.x.x.x:50530] AH01626: authorization result of <RequireAny>: granted
..........
[deflate:debug] [pid 76614:tid 139623145666304] mod_deflate.c(856): [client x.x.x.x:50530] AH01384: Zlib: Compressed 2443 to 1266 : URL /users/login

In the Apache log, however, as you can see now more text has been added. From my point of view it is remarkable that it takes about 3 attempts until authz_core Require valid-user : granted reported.

Foreman acknowledges these attempts in his production log with

[I|app|69a71bed] Started GET "/users/extlogin" for x.x.x.x at 2021-07-22 14:17:10 +0200
[I|app|69a71bed] Processing by UsersController#extlogin as HTML
[W|app|69a71bed] SSO failed
[W|app|69a71bed] falling back to login form
[I|app|69a71bed] Redirected to https://xxxx/users/login
[I|app|69a71bed] Filter chain halted as :require_login rendered or redirected
[I|app|69a71bed] Completed 302 Found in 2ms (ActiveRecord: 0.0ms | Allocations: 441)

Therefore it does not work for me yet.
As the debug log of Foreman says little about how it comes to “SSO failed” ( and I’m actually sure that foreman is in debug log mode ) I can’t see well how it happens exactly.
I am also afraid that it is not very helpful what I contribute here. I’m thinking about setting up a new Foreman instance in the near future and testing it with a fresh install using sssd instead of ldap & krb5 for pam authentication. Because then I can use “require pam-account foreman-prod” instead of “require valid-user”. I don’t know how far that might make a difference.

Thanks! I updated our dev server to Nightly and configured GSSAPI as outlined in the manual. We are now seeing the “Kerberos authentication did not pass” screen briefly after hitting https://$FOREMAN_URL/users/extlogin, which is new.

There’s not much of interest in production.log, but I am seeing these lines in /var/log/httpd/foreman-ssl_error_ssl.log:

[Thu Jul 29 09:12:22.739170 2021] [authnz_pam:warn] [pid 4398] [client IP_ADDR:58962] PAM account validation failed for user USERNAME: Authentication failure
[Thu Jul 29 09:12:22.739214 2021] [authz_core:error] [pid 4398] [client IP_ADDR:58962] AH01631: user USERNAME: authorization failure for "/users/extlogin":
[Thu Jul 29 09:12:22.784175 2021] [auth_gssapi:error] [pid 4398] [client IP_ADDR:58962] INTERNAL ERROR Mechanism needs continuation but neither GssapiConnectionBound nor GssapiUseSessions are configured

I noticed the keytab file in the manual is http.keytab, and ours has always been krb5.keytab. If I update /etc/httpd/conf.d/auth_kerb.conf accordingly, this shouldn’t be an issue right?