Katello-nightly-rpm-pipeline 25 failed

Infra & CI
1

Katello nightly pipeline failed:

https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/25/

2

Katello test is failing with:

[ERROR 2019-04-02T11:46:48 verbose]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[pipeline-katello-nightly-centos7.n53.example.com]: Could not evaluate: Proxy pipeline-katello-nightly-centos7.n53.example.com cannot be retrieved: unknown error (response 500)

httpd error_log shows:

[ 2019-04-02 11:45:24.8094 25345/7f1e68367700 Pool2/Implementation.cpp:287 ]: Could not spawn process for application /usr/share/foreman: An error occured while starting up the preloader.
  Error ID: 003a694f
  Error details saved to: /tmp/passenger-error-UvHGWu.html
  Message from application:  (RuntimeError)
  /opt/theforeman/tfm/root/usr/share/gems/gems/ffi-1.4.0/lib/ffi/library.rb:253:in `attach'
  /opt/theforeman/tfm/root/usr/share/gems/gems/ffi-1.4.0/lib/ffi/library.rb:253:in `attach_function'
  /opt/theforeman/tfm/root/usr/share/gems/gems/ethon-0.12.0/lib/ethon/libc.rb:16:in `<module:Libc>'
  /opt/theforeman/tfm/root/usr/share/gems/gems/ethon-0.12.0/lib/ethon/libc.rb:6:in `<module:Ethon>'
  /opt/theforeman/tfm/root/usr/share/gems/gems/ethon-0.12.0/lib/ethon/libc.rb:1:in `<top (required)>'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/ethon-0.12.0/lib/ethon.rb:14:in `<top (required)>'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/typhoeus-1.3.1/lib/typhoeus.rb:2:in `<top (required)>'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/zest-0.0.4/lib/zest/api_client.rb:17:in `<top (required)>'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/zest-0.0.4/lib/zest.rb:14:in `<top (required)>'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.12.0.pre.master/lib/katello.rb:14:in `<top (required)>'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in `require'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:135:in `rescue in require'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:39:in `require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:41:in `block in system_require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:37:in `each'
  /opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:37:in `system_require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:19:in `block in system_require'
  /opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in `each'
  /opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in `system_require'
  /usr/share/foreman/config/application.rb:17:in `<top (required)>'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /usr/share/foreman/config/environment.rb:2:in `<top (required)>'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  config.ru:5:in `block in <main>'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/builder.rb:55:in `instance_eval'
  /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/builder.rb:55:in `initialize'
  config.ru:1:in `new'
  config.ru:1:in `<main>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:112:in `eval'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:112:in `preload_app'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:158:in `<module:App>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<module:PhusionPassenger>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:28:in `<main>'

audit.log shows possible denial:

type=AVC msg=audit(1554205524.546:3010): avc:  denied  { execmem } for  pid=25601 comm="ruby" scontext=system_u:system_r:passenger_t:s0 tcontext=system_u:system_r:passenger_t:s0 tclass=process permissive=0
type=SYSCALL msg=audit(1554205524.546:3010): arch=c000003e syscall=10 success=no exit=-13 a0=7f0938ce8000 a1=1000 a2=5 a3=7ffe28868860 items=0 ppid=25600 pid=25601 auid=4294967295 uid=997 gid=993 euid=997 suid=997 fsuid=997 egid=993 sgid=993 fsgid=993 tty=(none) ses=4294967295 comm="ruby" exe="/opt/rh/rh-ruby25/root/usr/bin/ruby" subj=system_u:system_r:passenger_t:s0 key=(null)

Upgrade test is also failing, with different error:

[1m[36mUpgrade Step: migrate_foreman...[0m
[[32m INFO[0m [34m2019-04-02T12:11:02[0m [36mverbose[0m] Upgrade Step: migrate_foreman...
[33mrake aborted![0m
[33mTSort::Cyclic: topological sort failed: [#<Rails::Initializable::Initializer:0x000000000372efe0 @name=:build_middleware_stack, @context=#<Foreman::Application:0x0000000003ccc8f8 @_all_autoload_paths=["/usr/share/foreman/lib", "/usr/share/foreman/app/controllers/concerns", "/usr/share/foreman/app/models/concerns", "/usr/share/foreman/app/services", "/usr/share/foreman/app/mailers", "/usr/share/foreman/app/models/auth_sources", "/usr/share/foreman/app/models/compute_resources", "/usr/share/foreman/app/models/fact_names", "/usr/share/foreman/app/models/lookup_keys", "/usr/share/foreman/app/models/host_status", "/usr/share/foreman/app/models/operatingsystems", "/usr/share/foreman/app/models/parameters", "/usr/share/foreman/app/models/trends", "/usr/share/foreman/app/models/taxonomies", "/usr/share/foreman/app/models/mail_notifications", "/usr/share/foreman/app/controllers", "/usr/share/foreman/app/graphql", "/usr/share/foreman/app/helpers", "/usr/share/foreman/app/jobs", "/usr/share/foreman/app/models", "/usr/share/foreman/app/registries", "/usr/share/foreman/app/validators"], @_all_load_paths=["/usr/share/foreman/lib", "/usr/share/foreman/app/controllers/concerns", "/usr/share/foreman/app/models/concerns", "/usr/share/foreman/app/services", "/usr/share/foreman/app/mailers", "/usr/share/foreman/app/models/auth_sources", "/usr/share/foreman/app/models/compute_resources", "/usr/share/foreman/app/models/fact_names", "/usr/share/foreman/app/models/lookup_keys", "/usr/share/foreman/app/models/host_status....
3

The selinux denials sound suspiciously similar to what is described in Passenger and EXECMEM denial from 5 years ago. Could it be that the bastion merge causes something to trigger this issue? @lzap @Justin_Sherrill any ideas?

4

can someone from @katello team please try looking into the failing upgrade test?

5

upgrade issue is fixed, continuing to work on install issue in katello-nightly-rpm-pipeline 28 failed

6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.