That means nothing to me 
First of all please try if the server has actually joined the realm by realm list.
If that is ok,
Could you compare the /etc/httpd/conf.d/05-foreman-ssl.d/ configs on both servers?
Maybe try the /var/log/httpd/foreman-ssl_access_ssl.log if there’s something interesting.
We aren’t using realmd in this config - just a keytab and an AD service account. That’s been working for ages on multiple Foreman instances, but I can try joining it to the domain if you think that would help.
As for 05-foreman-ssl.conf, this entire block is present on 2.2.2 and not present on 2.1.4:
## Request header rules
## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader
RequestHeader set X_FORWARDED_PROTO "https"
RequestHeader set SSL_CLIENT_S_DN "%{SSL_CLIENT_S_DN}s"
RequestHeader set SSL_CLIENT_CERT "%{SSL_CLIENT_CERT}s"
RequestHeader set SSL_CLIENT_VERIFY "%{SSL_CLIENT_VERIFY}s"
RequestHeader unset REMOTE_USER
RequestHeader unset REMOTE_USER_EMAIL
RequestHeader unset REMOTE_USER_FIRSTNAME
RequestHeader unset REMOTE_USER_LASTNAME
RequestHeader unset REMOTE_USER_GROUPS
## SSL Proxy directives
SSLProxyEngine On
## Proxy rules
ProxyRequests Off
ProxyPreserveHost On
ProxyAddHeaders On
ProxyPass /pulp !
ProxyPass /pulp2 !
ProxyPass /streamer !
ProxyPass /pub !
ProxyPass /icons !
ProxyPass / http://127.0.0.1:3000/ retry=0
ProxyPassReverse / http://127.0.0.1:3000/
## Rewrite rules
RewriteEngine On
#Upgrade Websocket connections
RewriteCond %{HTTP:Upgrade} =websocket [NC]
RewriteRule /(.*) ws://127.0.0.1:3000/$1 [P,L]
From the issue linked in tbrisker’s reply, it seems like there should be some header set in this file.
I have fixed this by changing app/services/sso/apache.rb to use HTTP_REMOTE_USER instead of REMOTE_USER and setting HTTP_REMOTE_USER in apache. I have created pull requests to foreman and puppet-foreman projects in case this is acceptable.
I don’t see any set lines for HTTP_REMOTE_USER in the config file, though our apache.rb does have references to HTTP_REMOTE_USER.
For what it’s worth, neither of the below lines fixed the issue when added to the Apache config file:
RequestHeader set REMOTE_USER "%{REMOTE_USER}s"
RequestHeader set HTTP_REMOTE_USER "%{HTTP_REMOTE_USER}s"
It looks like this problem has not been solved yet. We also have the problem. I’ve also been watching this post since February and have tried since Foreman 2.2.2 through 2.3.3 to now 2.4.0. Unfortunately no success. We have been using the Kerberos config at …/apache2/conf.d/05-foreman-ssl.d/auth_kerb.conf for almost 6 years now and the switch to PUMA probably broke it.
So at the moment we are running Foreman 2.4 on Debian 10 without IPA or AD.
If I let apache be Verbose, I also see “authorization result: granted (no directives)”.
But I can’t log in via Kerberos. If I change auth_kerb.conf so that no proper authentication is possible, I see at least the error from this line "ErrorDocument 401 ‘Kerberos authentication did not pass.’
". The authentication itself works in my eyes, but not the transfer of this information.
Glad to hear it’s not just us - if you guys come across a solution, please share!
I somehow missed this.
As a workaround, you can switch back to Passenger (for now). If you used the installer, it’s --foreman-passenger true. However, we are planning to drop Passenger support so that’s a blocker to upgrade at some point.
Can you tell us how you upgraded? Did you use the installer?
Sadly I don’t have a Kerberos setup to replicate.
We use the debian repository. After the obligatory changes in the sources.list and the apt update && upgrade I run ‘foreman-installer’ without Parameter. No error appear in the process. I checked that the foreman-installer answers ipa is set to no, to exclude this.
I have not yet returned to passenger because I have the time to wait until Puma can do it. Rather I help with troubleshooting, sometime I have to use it anyway. 
Rolling back to Passenger didn’t fix this on our dev server. I’ve tried it both ways a few times. The --foreman-passenger true option on foreman-installer actually seems to leave a lot of the Puma config lines in the Apache config file; the RequestHeader lines, ProxyPass lines that refer to port 3000, etc. These lines aren’t present on our prod servers which are still using Passenger on 2.1.4.
We upgrade using the steps in the Upgrade section of the manual and foreman-installer.
I have found a workaround for a similar issue:
https://community.theforeman.org/t/actualize-sso-external-auth-from-2-1-to-2-4/22930
If you could confirm it works in your setting it would be possible to open a bugfix request backed up with some experimental data.
No luck here, but we are using Puma; it seems like you’re using Passenger? I’m hesitant to get comfortable with a Passenger-based config since it will eventually be going away, but I can give it a try if you’re looking for additional data points.
Yes, I’m still on Passenger (I think it’s because I come from a default Debian setup).
No, really it’s not worth: if you are on Puma the env variables’ names (actually the headers because there is a reverse proxy, but anyhow) are just fine.
Remind me if you don’t mind, what version of Foreman are you running and on what OS?
Sure - our dev server is on Version 2.3.3 running on CentOS 7.9.2009.
Same here. No luck with the above solution.
The Foreman Version ist 2.4.0 on Debian 10.
Could you share the contents of your lookup_identity.conf file please?
<LocationMatch ^/users/(ext)?login$>
LookupUserAttr email REMOTE_USER_EMAIL
LookupUserAttr firstname REMOTE_USER_FIRSTNAME
LookupUserAttr lastname REMOTE_USER_LASTNAME
LookupUserGroups REMOTE_USER_GROUPS :
LookupUserGroupsIter REMOTE_USER_GROUP
# Set headers for proxy requests
RequestHeader set REMOTE_USER %{REMOTE_USER}e
RequestHeader set REMOTE_USER_EMAIL %{REMOTE_USER_EMAIL}e
RequestHeader set REMOTE_USER_FIRSTNAME %{REMOTE_USER_FIRSTNAME}e
RequestHeader set REMOTE_USER_LASTNAME %{REMOTE_USER_LASTNAME}e
RequestHeader set REMOTE_USER_GROUPS %{REMOTE_USER_GROUPS}e
</LocationMatch>Ours matches this exactly, though the file name is lookup_identity.conf.erb, not lookup_identity.conf
I had the opportunity the last two days to get Foreman out of production use and test it intensively. The following was tested on a Debian 10.10 with Apache 2.4.38 including mod_auth_kerb without FreeIPA in combination with Foreman 2.4, 2.4.1 and 2.5.1.
The file /etc/apache2/conf.d/auth_kerb.conf looks like this:
<Location /users/extlogin>
AuthType Kerberos
AuthName "Kerberos Login"
KrbMethodNegotiate On
KrbSaveCredentials off
KrbMethodK5Passwd on
KrbAuthRealms REALM.NAME
Krb5KeyTab /.../krb5-apache.keytab
KrbLocalUserMapping on
require valid-user
ErrorDocument 401 '<html><meta http-equiv="refresh" content="0; URL=/users/login"><body>Kerberos authentication did not pass.</body></html>'
</Location>
The lookup_identity.conf file is no longer in use, it is not relevant to our scenario.
I set the logging from apache as well as from foreman to debug.
Scenario 1 - Foreman with Puma:
Apache receives the Kerberos Ticket, acknowledges in the log that it is cutting off the realm, that it is a valid_user and forwards the information to the foreman.
kerb_authenticate_user entered with user (NULL) and auth_type Kerberos
Acquiring creds for HTTP@...
Verifying client data using KRB5 GSS-API
Client delegated us their credential
GSS-API token of length 22 bytes will be sent back
kerb_authenticate_a_name_to_local_name user@... -> user
AH01626: authorization result of Require valid-user : granted
AH01626: authorization result of <RequireAny>: granted
AH01143: Running scheme http handler (attempt 0)
AH00942: HTTP: has acquired connection for (foreman)
AH00944: connecting http://foreman/users/extlogin to foreman:80
AH02545: http: has determined UDS as /run/foreman.sock
AH00947: connected /users/extlogin to httpd-UDS:0
AH00943: http: has released connection for (foreman)
AH02001: Connection closed to child 141 with standard shutdown (server ...:443)
In the production log of the Foreman you can read only:
Started GET "/users/extlogin" for *.*.*.* at 2021-06-29 15:26:19 +0200
Processing by UsersController#extlogin as HTML
SSO failed
falling back to login form
Redirected to https://.../users/login
Filter chain halted as :require_login rendered or redirected
If I remove in /usr/share/foreman/app/controllers/application_controller.rb and /usr/share/foreman/app/services/sso/apache.rb all leading HTTP_ in front of REMOTE_USER, REMOTE_USER_EMAIL etc. then nothing changes.
I have also tried other solutions from the Internet, some of which scriner has already tried, as I noticed when I reread this topic, unfortunately without success. The only noticeable change came with:
RequestHeader set REMOTE_USER "%{REMOTE_USER}s"
RequestHeader set HTTP_REMOTE_USER "%{HTTP_REMOTE_USER}s"
After I added it to …/sites-enabled/05-foreman-ssl.conf above ``RequestHeader set X_FORWARDED_PROTO “https”``` and commented out the “unset” lines below it, the foreman seemed to accept Kerberos authentication, but displayed an error that the user’s name must not be empty. I wasn’t sure what name exactly was meant since it didn’t explicitly say “username”. Unfortunately I couldn’t find the exact wording, although I documented everything separately.
Scenario 2 - Foreman with Passenger:
From the existing environment, I run foreman-installer --foreman-passenger true. Now passenger is running instead of puma. Out of the box, in the files /usr/share/foreman/app/controllers/application_controller.rb and /usr/share/foreman/app/services/sso/apache.rb the variables still have HTTP_ in front of them and Kerberos authentication fails. But if I remove this HTTP_ in front of the variables, I can log in via Kerberos. Regardless of this, node.rb is no longer usable to query the hosts. The error is always that the request was made with a certificate without DN and is therefore not allowed. Strange, but it works with puma. I have run this process several times to be sure that I have not changed anything else. Unfortunately, I could not determine here where the error comes from exactly and why. Some files like …/sites-enabled/05-foreman-ssl.conf or the Foreman configurations under /etc/foreman/ I could compare with another working production system ( Foreman 1.24 ) and the difference was not worth mentioning. Also the puppetserver didn’t seem to be changed. At least I didn’t find any changed configuration. But I also didn’t used find to list all the last changed files, in retrospect I should have.
During my internet research I noticed that the program “zammad” might also have had this problem in connection with puma. Unfortunately, I have not found out exactly. Probably due to my inexperience in analyzing such problems I must have missed something, but my impression is that mod_auth_kerb passes the information in the HTTP header with REMOTE_USER and puma expects HTTP_REMOTE_USER and thus it fails.
I hope that my explanations can be useful in some way or that someone has found a solution in the meantime.
Recently we switched from mod_auth_kerb to mod_auth_gssapi:
https://github.com/theforeman/puppet-foreman/commit/91850a2b4522406a28dbc98e2c3f215130d08dab
This will land in the next release. Perhaps this is something you can try? Foreman :: Manual is updated
Hey thanks for the feedback. I have tried this directly. I have included the Nightly Repsoitory for Debian and updated the Foreman to version 2.6.0-develop. Next installed auth_gssapi for apache2. My /etc/apache2/conf.d/05-foreman-ssl.d/auth_kerb.conf now looks like this
<Location /users/extlogin>
AuthType GSSAPI
AuthName "GSSAPI Single Sign On Login"
GssapiCredStore keytab:/etc/apache2/krb5-apache.keytab
GssapiSSLonly On
GssapiLocalName On
require valid-user
ErrorDocument 401 '<html><meta http-equiv="refresh" content="0; URL=/users/login"><body>Kerberos authentication did not pass.</body></html>'
</Location>
Apache seems happy with it and forwards it to foreman
[authz_core:debug] [pid 76614:tid 139623145666304] mod_authz_core.c(846): [client x.x.x.x:50530] AH01628: authorization result: granted (no directives)
[auth_gssapi:debug] [pid 76614:tid 139623145666304] mod_auth_gssapi.c(727): [client x.x.x.x:50530] GSSapiImpersonate not On, skipping impersonation.
[proxy:debug] [pid 76614:tid 139623145666304] mod_proxy.c(1249): [client x.x.x.x:50530] AH01143: Running scheme http handler (attempt 0)
[proxy:debug] [pid 76614:tid 139623145666304] proxy_util.c(2316): AH00942: HTTP: has acquired connection for (foreman)
[proxy:debug] [pid 76614:tid 139623145666304] proxy_util.c(2369): [client x.x.x.x:50530] AH00944: connecting http://foreman/ to foreman:80
[ssl:debug] [pid 76614:tid 139623145666304] ssl_engine_kernel.c(2235): [client x.x.x.x:50530] AH02041: Protocol: TLSv1.3, Cipher: TLS_AES_256_GCM_SHA384 (256/256 bits)
[proxy:debug] [pid 76614:tid 139623145666304] proxy_util.c(2406): [client x.x.x.x:50530] AH02545: http: has determined UDS as /run/foreman.sock
[proxy:debug] [pid 76614:tid 139623145666304] proxy_util.c(2578): [client x.x.x.x:50530] AH00947: connected / to httpd-UDS:0
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(495): AH00831: socache_shmcb_store (0xa7 -> subcache 7)
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(849): AH00847: insert happened at idx=6, data=(8511:8543)
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(854): AH00848: finished insert, subcache: idx_pos/idx_used=0/7, data_pos/data_used=0/8742
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(516): AH00834: leaving socache_shmcb_store successfully
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(495): AH00831: socache_shmcb_store (0xd9 -> subcache 25)
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(849): AH00847: insert happened at idx=2, data=(3289:3321)
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(854): AH00848: finished insert, subcache: idx_pos/idx_used=0/3, data_pos/data_used=0/3520
[socache_shmcb:debug] [pid 76614:tid 139623145666304] mod_socache_shmcb.c(516): AH00834: leaving socache_shmcb_store successfully
[proxy:debug] [ppid 76614:tid 139623145666304] proxy_util.c(2331): AH00943: http: has released connection for (foreman)
[ssl:debug] [pid 76614:tid 139623145666304] ssl_engine_kernel.c(383): [client x.x.x.x:50530] AH02034: Subsequent (No.2) HTTPS request received for child 72 (server xxx:443)
[authz_core:debug] [pid 76614:tid 139623145666304] mod_authz_core.c(820): [client x.x.x.x:50530] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[authz_core:debug] [pid 76614:tid 139623145666304] mod_authz_core.c(820): [client x.x.x.x:50530] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[auth_gssapi:debug] [pid 76614:tid 139623145666304] mod_auth_gssapi.c(895): [client x.x.x.x:50530] URI: /users/extlogin, no main, no prev
[auth_gssapi:info] [pid 76614:tid 139623145666304] [client x.x.x.x:50530] NO AUTH DATA Client did not send any authentication headers
..........
[auth_gssapi:debug] [pid 76614:tid 139623145666304] mod_auth_gssapi.c(895): [client x.x.x.x:50530] URI: /users/extlogin, no main, no prev
[authz_core:debug] [pid 76614:tid 139623145666304] mod_authz_core.c(820): [client x.x.x.x:50530] AH01626: authorization result of Require valid-user : granted
[authz_core:debug] [pid 76614:tid 139623145666304] mod_authz_core.c(820): [client x.x.x.x:50530] AH01626: authorization result of <RequireAny>: granted
..........
[deflate:debug] [pid 76614:tid 139623145666304] mod_deflate.c(856): [client x.x.x.x:50530] AH01384: Zlib: Compressed 2443 to 1266 : URL /users/login
In the Apache log, however, as you can see now more text has been added. From my point of view it is remarkable that it takes about 3 attempts until authz_core Require valid-user : granted reported.
Foreman acknowledges these attempts in his production log with
[I|app|69a71bed] Started GET "/users/extlogin" for x.x.x.x at 2021-07-22 14:17:10 +0200
[I|app|69a71bed] Processing by UsersController#extlogin as HTML
[W|app|69a71bed] SSO failed
[W|app|69a71bed] falling back to login form
[I|app|69a71bed] Redirected to https://xxxx/users/login
[I|app|69a71bed] Filter chain halted as :require_login rendered or redirected
[I|app|69a71bed] Completed 302 Found in 2ms (ActiveRecord: 0.0ms | Allocations: 441)
Therefore it does not work for me yet.
As the debug log of Foreman says little about how it comes to “SSO failed” ( and I’m actually sure that foreman is in debug log mode ) I can’t see well how it happens exactly.
I am also afraid that it is not very helpful what I contribute here. I’m thinking about setting up a new Foreman instance in the near future and testing it with a fresh install using sssd instead of ldap & krb5 for pam authentication. Because then I can use “require pam-account foreman-prod” instead of “require valid-user”. I don’t know how far that might make a difference.
Thanks! I updated our dev server to Nightly and configured GSSAPI as outlined in the manual. We are now seeing the “Kerberos authentication did not pass” screen briefly after hitting https://$FOREMAN_URL/users/extlogin, which is new.
There’s not much of interest in production.log, but I am seeing these lines in /var/log/httpd/foreman-ssl_error_ssl.log:
[Thu Jul 29 09:12:22.739170 2021] [authnz_pam:warn] [pid 4398] [client IP_ADDR:58962] PAM account validation failed for user USERNAME: Authentication failure
[Thu Jul 29 09:12:22.739214 2021] [authz_core:error] [pid 4398] [client IP_ADDR:58962] AH01631: user USERNAME: authorization failure for "/users/extlogin":
[Thu Jul 29 09:12:22.784175 2021] [auth_gssapi:error] [pid 4398] [client IP_ADDR:58962] INTERNAL ERROR Mechanism needs continuation but neither GssapiConnectionBound nor GssapiUseSessions are configured
I noticed the keytab file in the manual is http.keytab, and ours has always been krb5.keytab. If I update /etc/httpd/conf.d/auth_kerb.conf accordingly, this shouldn’t be an issue right?