Katello nightly pipeline failed:
https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/28/
Well, at least both tests are now failing with the same error, thanks @ehelms for fixing the cyclic dependency issue with bastion!
It appears that the remaining failure is caused by an selinux denial triggered by zest gem (api bindings for pulp 3) requiring typhoeus, which in turn requires ethon that causes the denial when trying to register some function:
[ 2019-04-02 11:45:24.8094 25345/7f1e68367700 Pool2/Implementation.cpp:287 ]: Could not spawn process for application /usr/share/foreman: An error occured while starting up the preloader.
Error ID: 003a694f
Error details saved to: /tmp/passenger-error-UvHGWu.html
Message from application: (RuntimeError)
/opt/theforeman/tfm/root/usr/share/gems/gems/ffi-1.4.0/lib/ffi/library.rb:253:in `attach'
/opt/theforeman/tfm/root/usr/share/gems/gems/ffi-1.4.0/lib/ffi/library.rb:253:in `attach_function'
/opt/theforeman/tfm/root/usr/share/gems/gems/ethon-0.12.0/lib/ethon/libc.rb:16:in `<module:Libc>'
/opt/theforeman/tfm/root/usr/share/gems/gems/ethon-0.12.0/lib/ethon/libc.rb:6:in `<module:Ethon>'
/opt/theforeman/tfm/root/usr/share/gems/gems/ethon-0.12.0/lib/ethon/libc.rb:1:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/ethon-0.12.0/lib/ethon.rb:14:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/typhoeus-1.3.1/lib/typhoeus.rb:2:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/zest-0.0.4/lib/zest/api_client.rb:17:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/zest-0.0.4/lib/zest.rb:14:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.pre.master/lib/katello.rb:14:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in `require'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in `rescue in require'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:39:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:41:in `block in system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:37:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:37:in `system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:19:in `block in system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in `system_require'
/usr/share/foreman/config/application.rb:17:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/usr/share/foreman/config/environment.rb:2:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
config.ru:5:in `block in <main>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/builder.rb:55:in `instance_eval'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/builder.rb:55:in `initialize'
config.ru:1:in `new'
config.ru:1:in `<main>'
/usr/share/passenger/helper-scripts/rack-preloader.rb:112:in `eval'
/usr/share/passenger/helper-scripts/rack-preloader.rb:112:in `preload_app'
/usr/share/passenger/helper-scripts/rack-preloader.rb:158:in `<module:App>'
/usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<module:PhusionPassenger>'
/usr/share/passenger/helper-scripts/rack-preloader.rb:28:in `<main>'
and audit.log shows:
type=AVC msg=audit(1554205524.546:3010): avc: denied { execmem } for pid=25601 comm="ruby" scontext=system_u:system_r:passenger_t:s0 tcontext=system_u:system_r:passenger_t:s0 tclass=process permissive=0
type=SYSCALL msg=audit(1554205524.546:3010): arch=c000003e syscall=10 success=no exit=-13 a0=7f0938ce8000 a1=1000 a2=5 a3=7ffe28868860 items=0 ppid=25600 pid=25601 auid=4294967295 uid=997 gid=993 euid=997 suid=997 fsuid=997 egid=993 sgid=993 fsgid=993 tty=(none) ses=4294967295 comm="ruby" exe="/opt/rh/rh-ruby25/root/usr/bin/ruby" subj=system_u:system_r:passenger_t:s0 key=(null)
The audit2allow output for the denial is:
#============= passenger_t ==============
allow passenger_t self:process execmem;
Which i think is prettty dangerous.
@lzap or anyone else - any idea on how to fix this?
Hello,
we had the “execmem” denial reported previously, however it turned out to actually be bug on our side - it appeared when dynflow or foreman attempted to recompile assets, the asset pipeline then had some library (I think v8) which was causing execmem. The fix was to change the behavior so everything is compiled during packaging time and we had to do nothing in our SELinux policy.
Do we see this denial during runtime? If so, do we understand why this is happening? From the backtrace it looks like FFI wants to register ethon function which appears to be just a libcurl wrapper. FFI is known to handle some function calls (callbacks specifically) in a way that it is actually executing mmapped memory which is prevented by SELinux (for a reason).
How to solve this.
-
We can allow execmem by a rule, howevever this is dumb solution. I suggest this only to buy time if we are blocked. This can be exploited by attackers to actually execute code in theory, it’s a good practice not to allow it.
-
Avoid ethon library. The most simple fix - do we actually need this library for Katello? The tree looks like to be katello - zest - typhoeus - ethon - ffi. If we can drop this, no problem at all.
-
Fix ethon library - it is possible to fix this, newer versions of FFI provides a SELinux friendly API for function callbacks, however this can be lengthy and I am not very familiar with FFI.
Edit: I do not understand why Pulp API needs to do HTTP calls using a FFI library, I would rather see us doing native Ruby HTTP calls rather than using FFI which can easily contain memory leaks or other serious issues which can lead to process kill.
1 Like
Okay after some further reading I propose to allow execmem for both Foreman core and Foreman proxy. Explanation in the ticket and in the PR:
- Feature #26520: Allow execmem for passenger due to Ruby FFI - SELinux - Foreman
- https://github.com/theforeman/foreman-selinux/pull/88
I think this is the only reasonable thing we can do, however I still think we should not be using FFI in the runtime, at least not on per-request basis. FFI code more likely contain fatal bugs.
This is a new library introduced by @Justin_Sherrill. It’s only used in Katello so would it make sense to only add it to katello-selinux rather than foreman-selinux?
I’d rather enable this globally for both core and proxy so it’s obvious and for all plugins rather than “sneaky” katello-only module. I would like to be very clear about this, this should go into Release Notes as well.
@tbrisker was thinking the other day - how about tracking Release Notes in a Discourse thread rather than doing PRs? If we convert OP to Wiki this could be pretty convenient for all.
Thanks for looking into this matter! I’m not too happy with allowing execmem (it might make sense if the context was limited to ffi itself, not all of passenger) but I guess that our other option atm is completely dropping the libraries using ffi. Is there some way of more granularly defining the permission somehow?
Not everyone reads the announcement on discourse, and I would prefer that we review notes before publishing them.
Not for this one, it’s black or white in this case unfortunately. Usually SELinux can be very granular in general.
Yeah, agreed. What I mean is that we would use Discourse to track WIP Release Notes and once it’s ready you take them and turn into documentation. The markdown is almost the same. My thinking is being more flexible - creating PRs is clunky, then we run into rebase issues etc. This could be a single document, easy to edit for everyone.
Rebases shouldn’t be an issue. We now have a nightly branch in the docs repository where we keep the release notes so once we agree on the wording we can simply merge it. It may not be as light weight as a wiki, but as a release manager I like to have some insight and process around it.
Other projects have started with drop in directories to gather changelog entries so the changelog is part of the original PR, but I don’t think we’re ready for that. It would help with cherry picking because the release notes automatically pick up the new fragment.
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.