Getting "SSO failed" after successfully logging into Keycloak

Problem:
I’m getting the following error in Foreman after successfully logging into Keycloak as part of the extlogin process

2021-08-18T20:46:33 [I|app|96c46a9a] Started GET "/users/extlogin" for 64.125.235.5 at 2021-08-18 20:46:33 +0000
2021-08-18T20:46:33 [I|app|96c46a9a] Processing by UsersController#extlogin as HTML
2021-08-18T20:46:33 [W|app|96c46a9a] SSO failed
2021-08-18T20:46:33 [W|app|96c46a9a] falling back to login form
2021-08-18T20:46:33 [I|app|96c46a9a] Redirected to https://puppet-workstations.<redacted>.com/users/login
2021-08-18T20:46:33 [I|app|96c46a9a] Filter chain halted as :require_login rendered or redirected
2021-08-18T20:46:33 [I|app|96c46a9a] Completed 302 Found in 21ms (ActiveRecord: 0.9ms | Allocations: 10912)

Expected outcome:
The user logging in through Keycloak should be created and should be logged in.

Foreman and Proxy versions: 2.5.2

Distribution and version: Ubuntu 18.04.5 LTS

Keycloak version: 15.0.0

Other relevant data:

My Foreman instance is at https://puppet-workstations..com
My Keycloak instance is at https://keycloak..com:8443

I’m running Ubuntu 18.04 so my Apache configuration is a little different. I was able to install the apache2-mod-auth-openidc through the apt repository, and I was able to get the keycloak tool through the github repo GitHub - jdennis/keycloak-httpd-client-install: Python support for Keycloak.

keycloak-httpd-client-install --app-name foreman-openidc --keycloak-server-url https://keycloak.<redacted>.com.:8443 --keycloak-admin-username admin --keycloak-realm master --keycloak-admin-realm master --keycloak-auth-role root-admin -t openidc -l /users/extlogin
enter admin password:
[Step  1] Assure HTTP config directory is present
[Step  2] Assure HTTP federation directory is present
[Step  3] Set up template environment
[Step  4] Build OIDC httpd config file
[Step  5] Build Keycloak OIDC clientRepresentation
[Step  6] Connect to Keycloak Server as admin
[Step  7] Query realms from Keycloak server
[Step  8] Use existing realm on Keycloak server
[Step  9] Query realm clients from Keycloak server
[Step 10] Force delete client on Keycloak server
[Step 11] Creating new client from native
[Step 12] Completed Successfully

After running the keycloak-httpd-client-install tool I was able to get the Apache config in /etc/httpd/conf.d and move it under /etc/apache2/conf.d , and it looks to be loading fine as I’m able to get redirected to Keycloak when I try to access Foreman.

These are the Authentication settings I have in Foreman:

Authorize login delegation	yes

Authorize login delegation 
auth source user autocreate	External

OIDC Algorithm			RS256
OIDC Audience			[puppet-workstations.<redacted>.com-foreman-openidc]
OIDC Issuer			https://keycloak.<redacted>.com:8443/auth/realms/ssl-realm
OIDC JWKs URL			https://keycloak.<redacted>.com:8443/auth/realms/ssl-realm/protocol/openid-connect/certs

And these are the Apache settings in the config created by the keycloak tool:

root@puppet-workstations:/etc/apache2/conf.d# cat foreman-openidc_oidc_keycloak_ssl-realm.conf
OIDCClientID puppet-workstations.<redacted>.com-foreman-openidc
OIDCProviderMetadataURL https://keycloak.<redacted>.com.:8443/auth/realms/ssl-realm/.well-known/openid-configuration
OIDCCryptoPassphrase c8d4ecdf527a
OIDCClientSecret 2a3844991756
OIDCRedirectURI https://puppet-workstations.<redacted>.com/users/extlogin/redirect_uri
OIDCRemoteUserClaim sub

<Location /users/extlogin>
    AuthType openid-connect
    Require valid-user
</Location>

I confirmed the Client ID and Secret match what’s in Keycloak. I created the “audience-mapper” and “group-mapper”. I also confirmed I have both “https://puppet-workstations..com/users/extlogin” and “https://puppet-workstations..com/users/extlogin/redirect_uri” listed in the Valid Redirect URIs in Keycloak.

Any help is greatly appreciated, and thank you in advance

I got on IRC and mcorr mentioned this doc: Administering Foreman

I tried going through the doc and I did notice some differences to this doc: Foreman :: Manual

I didnt have luck with either one though :frowning:

Obviously I’m affected by this too now.

Up to yesterday, I had a working foreman 1.24 setup on Debian 10 (buster) with a working OIDC-Integration to Keycloak.
Today, I upgraded to foreman 2.5 (1.24->2.0->2.1->2.2->2.3->2.4->2.5) and see the same problem, that the SSO with Keycloak isn’t working anymore. The browser still gets redirected to Keycloak, but foreman obiously doesn’t accept the session key and falls back to the regular login form.
I checked this with each version step and IIRC the SSO broke when going from 2.2 to 2.3 (if that helps anything).

It would help to understand the actual cause of the problem.
I already set logging to “debug”, but log is not showing any significant, other than “SSO failed”.

Very interesting @jdelker I also went through that same upgrade process before attempting the OIDC integration (1.24->2.0->2.1->2.2->2.3->2.4->2.5)

Interesting. I remember that there was an SSO issue with the switch from from Passenger to Puma, but this has been fixed.

I am looking through old threads for clues and saw this and wondering if it is worth a look: "SSO Failed" error while integrating with Keycloak - #8 by autorune

You’re saying it broken when upgrading from 2.2 to 2.3? That’s interesting. 2.3 introduced the following change:
https://github.com/theforeman/foreman/commit/05e0aaf0acfed4fe8be3c5f6fdd9ddc158fe7a72

Which was supposed to help with external authentication when using Puma.

Maybe still worth trying reverting this in your setup, or looking why your setup is not setting the headers correctly?

Another thing, do you have an Apache config like the following?

Hi @evgeni I dont seem to have that LocationMatch

root@puppet-workstations:/etc/apache2# grep -R "LocationMatch" .
./sites-available/05-foreman-ssl.conf:<LocationMatch "^/(assets|webpack)">
./sites-available/05-foreman-ssl.conf:</LocationMatch>
./sites-available/05-foreman.conf:<LocationMatch "^/(assets|webpack)">
./sites-available/05-foreman.conf:</LocationMatch>
./sites-enabled/05-foreman-ssl.conf:<LocationMatch "^/(assets|webpack)">
./sites-enabled/05-foreman-ssl.conf:</LocationMatch>
./sites-enabled/05-foreman.conf:<LocationMatch "^/(assets|webpack)">
./sites-enabled/05-foreman.conf:</LocationMatch>

I do have the following in my Apache config:

  ## Request header rules
  ## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader
  RequestHeader set X_FORWARDED_PROTO "https"
  RequestHeader set SSL_CLIENT_S_DN "%{SSL_CLIENT_S_DN}s"
  RequestHeader set SSL_CLIENT_CERT "%{SSL_CLIENT_CERT}s"
  RequestHeader set SSL_CLIENT_VERIFY "%{SSL_CLIENT_VERIFY}s"
  RequestHeader unset REMOTE_USER
  RequestHeader unset REMOTE_USER_EMAIL
  RequestHeader unset REMOTE_USER_FIRSTNAME
  RequestHeader unset REMOTE_USER_LASTNAME
  RequestHeader unset REMOTE_USER_GROUPS

I decided to just deploy a new instance of Foreman 2.5.2 and use my existing Keycloak instance (that I was using for my existing Foreman instance). I went through this doc that Melanie had sent me: Administering Foreman

After running,

# foreman-installer --foreman-keycloak true \
--foreman-keycloak-app-name "foreman-openidc" \
--foreman-keycloak-realm "Foreman_Realm"

I noticed there was a “foreman-openidc_oidc_keycloak_Foreman_Realm.conf” file under “/etc/apache2/conf.d/” but it was empty. Not sure if thats expected.

So I ended up moving the “foreman-openidc_oidc_keycloak_Foreman_Realm.conf” file from “/etc/httpd/conf.d/” and replacing the empty file under “/etc/apache2/conf.d/”

I kept going with the documentation until I got to the part:

13.8.4. Configuring Foreman Settings for Keycloak Authentication
....
9. Navigate to Administer > Authentication Sources and click External.
10. Click Create LDAP Authentication Source and select the Keycloak server.

I ended up skipping step 10 above, because I couldn’t “select Keycloak server” when trying to create a LDAP Authentication source.

I crossed my fingers and attempted to login using my Keycloak credentials, and it worked! I’m still not sure whats different between my existing Foreman instance (that I upgraded from 1.24 → 2.5.2, version after version) and my new 2.5.2 test instance.

I’m currently looking into this and trying to get things sorted. Some things still don’t fit together, and I need to figure out if the error is with me, or the docs/tools/etc.

@eleneshector: Is is possible, that you paste your content of “foreman-openidc_oidc_keycloak_Foreman_Realm.conf” (of course with all sensistive information masked)?

With foreman-installer and keycloak-httpd-client-install just doing some intransparent magic things, it becomes hard to follow correctness of configuration.

Well, as said, I’m not 100% confident it happened on that version step. And I don’t want to rollback everything, just to find out :wink:

I would love to figure out the cause - and wrong headers is certainly a hot candidate.
However, that would require two things:

  1. A way to figure out, what headers are passed from Apache to Foreman. Any hint on that?
  2. Some reference what correct headers would look like. Any info on that?

A sidenote:
For me, tools like foreman-installer and keycloak-httpd-client-install overcomplicate things, although they aim to do the exact opposite :wink:
The former is a true biest with all it’s CLI params and I’ve given up using it. I didn’t find a way to get along with it in a slightly more complex environment and it wasn’t able to handle the Keycloak integration before.
The latter isn’t available on Debian and needs to be fetched from Github, leaving some doubt, that it matches the Foreman configuration.
Now, I don’t want to bug on those tools. On the right environment, they are probably real timesavers. However, they don’t match my usecase and it would help tremendously if the docs show some reference configuration for the Keycloak integration.

hey @jdelker this is my “foreman-openidc_oidc_keycloak_Foreman_Realm.conf” file on my working test instance:

root@foreman-test:~# cat /etc/apache2/conf.d/foreman-openidc_oidc_keycloak_Foreman_Realm.conf
OIDCClientID foreman-test.<redacted>.com-foreman-openidc
OIDCProviderMetadataURL https://keycloak.<redacted>.com:8443/auth/realms/Foreman_Realm/.well-known/openid-configuration
OIDCCryptoPassphrase <redacted>
OIDCClientSecret <redacted>
OIDCRedirectURI https://foreman-test.<redacted>.com/users/extlogin/redirect_uri
OIDCRemoteUserClaim sub

<Location /users/extlogin>
    AuthType openid-connect
    Require valid-user
</Location>

+1 for this too.

@evgeni Is it possible to have the OIDC flow happen outside of Apache/Foreman and just pass the proper headers to Apache/Foreman? Judging by your LocationMatch Apache configuration question, it sounds possible? :crossed_fingers:

Sorry I missed this earlier. I saw this thread on my first Google search hehe
Thats what gave me the hint to move the apache oidc auth config to the debian-specific Apache2 etc directory. That got me closer to getting it to work, but was still giving me the “SSO failed” error :frowning:

1 Like

Some context, I’ve set up some sites with GitHub - vouch/vouch-proxy: an SSO and OAuth / OIDC login solution for Nginx using the auth_reques but that only has examples using Nginx so I dont think I wanna go down that rabbit hole.

I’m definitely sticking to Apache’s mod_auth_openidc module for getting this to work with Foreman

That is probably the fundamental question.
When reading through the docs, I simply don’t get a clear understanding on how Foreman’s Keycloak integration is suppose to work. The keycloak-httpd-client-install does (undocumented) things and obviously sets up the usual Apache OIDC integration directives. Despite of that, Foreman shall be configured also with OIDC parameters.
I’ve already integrated dozens of web applications with Keycloak, and it was always either a configuration in Apache (with mod_auth_oidc), passing all SSO-stuff via header variables or in the application itself, using their own native OpenID implementation. But never both.
It would help a lot, if someone could clarify this. Why has Foreman it’s own OIDC config, if this is already handled by Apache’s mod_auth_oidc?

Thanks @eleneshector. That basically matches what I have.

I think this is what was discussed in the following thread:

GitHub - jdennis/keycloak-httpd-client-install: Python support for Keycloak doesn’t know that on Debian the config is in /etc/apache2 vs /etc/httpd (like on CentOS), and miss-places it.

I think it’s the RequestHeader set REMOTE_USER %{REMOTE_USER}e and friends from the Apache config I posted above.

To elaborate a bit more on the architecture behind our setup and “how it should work”:

  • We used to deploy Foreman with Passenger as an Apache module, which meant that the Foreman application ran inside the Apache process and had access to environment variables Apache and its modules set.
  • This is also how most of the mod_auth_something modules work: you configure them to talk to some kind of authentication service (Kerberos, Keycloak, you name it) and after the authentication succeeded they set a few environment variables to let “the rest of the stack” (other modules, applications, etc) know who is logged in.
  • With Foreman 2.1 we moved the deployment to Puma, which is running as a standalone process and Apache is talking HTTP to it. That especially means that any application running inside Puma can’t access environment variables from Apache, thus breaking authentication.
  • With Foreman 2.3 we fixed that by forwarding the relevant variables as HTTP headers.

And I think the last bit was missing in your setup, and was now corrected by the installer.

1 Like

Yeah, sure. You should only need to set the REMOTE_USER* headers appropriately for the request from Apache to Puma/Foreman, then the user should be accepted as “externally authenticated”.

1 Like