Help with kerberos authentication for remote execution

Support
1

Problem:
I would like to use kerberos for authentication on remote execution jobs, but I am not sure I am installing and enabling it correctly.

I attempted to make the plugin available via foreman-installer:

foreman-installer -v --foreman-proxy-plugin-remote-execution-ssh-ssh-kerberos-auth

Then set :kerberos_auth: true in /etc/smart_proxy_dynflow_core/settings.d/remote_execution_ssh.yml

I am under the impression foreman should now attempt kerberos auth before falling back to key-based authenticaiton.

My question now may be about kerberos more than foreman – how do I ensure that foreman-proxy has a valid TGT? I use FreeIPA for identity management and kerberos, is it as simple as creating an account for foreman-proxy, or does it need a service principle?

Expected outcome:
Enabling kerberos authentication for remote execution and configuring an account with a kerberos TGT for running remote commands.

Foreman and Proxy versions:
Foreman 1.21.4 and Smart Proxy 1.21.4

Distribution and version:
CentOS 7.7

2

Hi,
I think just creating an account for foreman-proxy should be enough. There are two ways to get the TGT, either you have do it manually every now and then by running kinit as foreman-proxy user. The other one is more automated, but requires you to set up a keytab. If you’d be interested in the other one, it is described in Satellite 6 docs quite nicely.

1 Like
3

Thanks for your help and the link! I’ll give that a try.

4

Right, so I gave the instructions in the Satellite doc a go, but I’ve run into trouble getting the gssapi ruby gem to work. Based on the suggestions in the doc, I installed the gem needed for the plugin:

Dep-Install tfm-rubygem-ffi-1.4.0-8.el7.x86_64         @foreman-plugins
Dep-Install tfm-rubygem-gssapi-1.2.0-5.el7.noarch      @foreman-plugins
Install     tfm-rubygem-net-ssh-krb-0.4.0-3.el7.noarch @foreman-plugins

Then tried activating the plugin with foreman-installer:
foreman-installer --scenario foreman --foreman-proxy-plugin-remote-execution-ssh-ssh-kerberos-auth true

Then I restarted Foreman. Unfortunately, now my foreman website will not load. I am not yet allowed to attach files, so I will try to paste in /var/log/httpd/error.log which includes details from the RuntimeError. It appears the new ruby gems I installed are not playing nice. I’m assuming there is some version mismatch or dependency problem:

[ 2020-05-06 22:45:38.4760 1481/7f7b33f88700 Pool2/Implementation.cpp:287 ]: Could not spawn process for application /usr/share/foreman: An error occured while starting up the preloader.
Error ID: 8f4c8401
Error details saved to: /tmp/passenger-error-P7v8WU.html
Message from application: (RuntimeError)
/opt/theforeman/tfm/root/usr/share/gems/gems/ffi-1.4.0/lib/ffi/library.rb:253:in attach' /opt/theforeman/tfm/root/usr/share/gems/gems/ffi-1.4.0/lib/ffi/library.rb:253:in attach_function’
/opt/theforeman/tfm/root/usr/share/gems/gems/gssapi-1.2.0/lib/gssapi/lib_gssapi.rb:13:in <module:LibGSSAPI>' /opt/theforeman/tfm/root/usr/share/gems/gems/gssapi-1.2.0/lib/gssapi/lib_gssapi.rb:8:in module:GSSAPI’
/opt/theforeman/tfm/root/usr/share/gems/gems/gssapi-1.2.0/lib/gssapi/lib_gssapi.rb:7:in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require’
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require' /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in block in require' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in load_dependency’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in require' /opt/theforeman/tfm/root/usr/share/gems/gems/gssapi-1.2.0/lib/gssapi.rb:17:in <top (required)>’
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require' /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require’
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in block in require’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in load_dependency' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in require’
/opt/theforeman/tfm/root/usr/share/gems/gems/net-ssh-krb-0.4.0/lib/net/ssh/authentication/methods/gssapi_with_mic.rb:3:in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require’
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require' /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in block in require' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in load_dependency’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in require' /opt/theforeman/tfm/root/usr/share/gems/gems/net-ssh-krb-0.4.0/lib/net/ssh/kerberos.rb:7:in <top (required)>’
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require' /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require’
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in block in require’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in load_dependency' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in require’
/opt/theforeman/tfm/root/usr/share/gems/gems/net-ssh-krb-0.4.0/lib/net/ssh/krb.rb:1:in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in require’
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in rescue in require' /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:39:in require’
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in block in require’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in load_dependency' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in require’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution_core-1.1.5/lib/foreman_remote_execution_core/script_runner.rb:6:in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require’
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require' /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in block in require' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in load_dependency’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in require' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution_core-1.1.5/lib/foreman_remote_execution_core.rb:68:in module:ForemanRemoteExecutionCore’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution_core-1.1.5/lib/foreman_remote_execution_core.rb:3:in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require’
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require' /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in block in require' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in load_dependency’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in require' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.7.0/lib/foreman_remote_execution/engine.rb:1:in <top (required)>’
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require' /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require’
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in block in require’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in load_dependency' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in require’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.7.0/lib/foreman_remote_execution.rb:4:in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in require’
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in rescue in require' /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:39:in require’
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in block in require’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in load_dependency' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in require’
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:41:in block in system_require' /opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:37:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:37:in system_require' /opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:19:in block in system_require’
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in system_require’
/usr/share/foreman/config/application.rb:17:in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require’
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require' /usr/share/foreman/config/environment.rb:2:in <top (required)>’
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require' /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in require’
config.ru:5:in block in <main>' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/builder.rb:55:in instance_eval’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/builder.rb:55:in initialize' config.ru:1:in new’
config.ru:1:in <main>' /usr/share/passenger/helper-scripts/rack-preloader.rb:112:in eval’
/usr/share/passenger/helper-scripts/rack-preloader.rb:112:in preload_app' /usr/share/passenger/helper-scripts/rack-preloader.rb:158:in module:App’
/usr/share/passenger/helper-scripts/rack-preloader.rb:29:in <module:PhusionPassenger>' /usr/share/passenger/helper-scripts/rack-preloader.rb:28:in ’

[ 2020-05-06 22:45:38.4875 1481/7f7b31a04700 agents/HelperAgent/RequestHandler.h:2306 ]: [Client 20] Cannot checkout session because a spawning error occurred. The identifier of the error is 8f4c8401. Please see earlier logs for details about the error.

5

I must admit this is new to me. I think there were some issues around FFI and SELinux in the past, could you check if there are any related denials in the audit log?

Could you paste contents of /tmp/passenger-error-P7v8WU.html somewhere (if not here then pastebin or github gist or something like that)?

6

Unfortunately I don’t see anything in the audit logs around that time that appear to be related (no denials).

I can retry after setting SELinux to be permissive, if that would help diagnose.

Interestingly, I attempted to view /tmp/passenger-error-P7v8WU.html in my initial troubleshooting, but the file was not found, even very shortly after incurring the error, so I don’t think it was cleaned up – I’m not sure why it suggested I check it.

I’ll try again with SELinux temporarily disabled, and if that doesn’t help, see if I can get that html file after reproducing the error again.

7

I’ve done some digging and made a bit of progress. It’s not clear to me why the foreman page wasn’t loading before, but after re-running the yum install job for the plugin, then rebooting the server, I’m still able to load the page normally.

However, it does not appear that the proxy is attempting to use kerberos for ssh authentication. I don’t have a lot to go on, but any job I try to run fails.

The packages are installed:
Dep-Install tfm-rubygem-ffi-1.4.0-8.el7.x86_64 @foreman-plugins
Dep-Install tfm-rubygem-gssapi-1.2.0-5.el7.noarch @foreman-plugins
Install tfm-rubygem-net-ssh-krb-0.4.0-3.el7.noarch @foreman-plugins

And per the plugin manual, I have set the proxy to try kerberos authentication:

[root@foreman log]# grep kerberos /etc/smart_proxy_dynflow_core/settings.d/remote_execution_ssh.yml
:kerberos_auth: true

And foreman-installer, at least, confirms the plugin is installed:

[root@foreman log]# foreman-installer --help | grep kerberos --foreman-proxy-plugin-remote-execution-ssh-ssh-kerberos-auth Enable kerberos authentication for SSH (current: true)

However, I don’t think it’s attempting kerberos authentication because, well, it works otherwise if I try to ssh from the foreman server to my mail server as the foreman-proxy user. But when I run a remote execution job, I only see it attempt preauth and then the job immediately fails:

May 8 00:36:19 mail1 sshd[30941]: Connection closed by <foreman's IP address> port 41328 [preauth]

My suspicion is that the plugin is either not really installed, not really enabled, or kerberos auth isn’t really enabled. Between the plugin manual and the Satellite docs it’s possible I’ve missed a step, but I’ve gone back over them a few times and I’m not sure where.

Any help or suggestions on where to look for additional information in the logs is very much appreciated.

8

Well, the web page no longer loads again, but this time the tmp file exists, though the full contents are somewhat less than helpful:
warning: %posttrans(tfm-rubygem-foreman_remote_execution-1.7.0-1.fm1_21.el7.noarch) scriptlet failed, signal 2

9

The 500 error persisted in preventing the Foreman web page from loading even after uninstalling the kerberos remote execution packages, so I thought I had hose my instance entirely. On a lark, I tried disabling SELinux and now it loads again.

It appears the problem is almost certainly related to SELinux. I’ll check audit logs again and also see if I can get it working while set to permissive. Ideally I’ll be able to it working with SELinux on, but I bet it’s just a matter of generating the right policy.

10

OK, so I solved the SELinux issue the way I should have the first time. Looks like passenger was being denied:

allow passenger_t self:process execmem;

seems to have solved the 500 errors when --foreman-proxy-plugin-remote-execution-ssh-ssh-kerberos-auth is enabled.

However, SSH still fails on remote execution. I’m assuming I’ve done something wrong, but I’m not sure what. kinit foreman-proxy works, and I can subsequently SSH to another host on the domain, but remote execution fails without detail.

Sorry for the multiple posts. I think I’ve reached a dead end now.

11

Glad to hear you got the SELinux issues sorted out.

As what user did you run kinit foreman-proxy?

12

As my personal user account, just another IPA user.

13

As far as I remember you sohuld run it as foreman-proxy user so the smart proxy has access to the obtained tgt

14

Fantastic, you were absolutely right, running kinit foreman-proxy as the foreman-proxy user appears to have retrieved the TGT necessary, as I am now able to execute remote commands via Foreman.

Thanks for all of your help. I hope I have set up the kerberos keytab correctly so I don’t have to kinit before running commands, but I will test that out shortly.