I have foreman/passenger/puppet installed on my server (running for several
months) and since this morning foreman/passenger are crashed and I'm not
able to start them.
The only thing I changed yesterday is the puppet-agent version (1.10.4 ->
1.10.5), I rollbacked it but it didn't solve my issue (I also tried to
reinstall foreman with the installer, same problem).
Here is my error when I restart httpd service :
[ 2017-10-02 14:17:40.6037 4341/7f75bf7fe700 agents/HelperAgent/RequestHandler.h:2306 ]: [Client 20] Cannot checkout session because a spawning error occurred. The identifier of the error is e6332702. Please see earlier logs for details about the error.
[ 2017-10-02 14:17:40.6044 4341/7f75bf7fe700 agents/HelperAgent/RequestHandler.h:2306 ]: [Client 23] Cannot checkout session because a spawning error occurred. The identifier of the error is e6332702. Please see earlier logs for details about the error.
App 4417 stdout:
App 4417 stderr: --> passenger_native_support was compiled for Ruby API version 2.0.0, but you're currently running a Ruby interpreter with API version 2.2.0.
App 4417 stderr: Refusing to load existing passenger_native_support.
App 4417 stderr: --> No passenger_native_support.so found for current Ruby interpreter.
App 4417 stderr: This library provides various optimized routines that make
App 4417 stderr: Phusion Passenger faster. Please run 'sudo yum install passenger-devel-4.0.53'
App 4417 stderr: so that Phusion Passenger can compile one on the next run.
App 4417 stderr: --> Continuing without passenger_native_support.so.
App 4417 stdout:
[ 2017-10-02 14:17:42.1249 4341/7f75ce4b4700 Pool2/Implementation.cpp:287 ]: Could not spawn process for application /usr/share/foreman: An error occured while starting up the preloader.
Error ID: f2bee560
Error details saved to: /tmp/passenger-error-tE9GFc.html
Message from application: cannot load such file – fcntl (LoadError)
/opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in require' /opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:inrequire'
/usr/share/passenger/phusion_passenger/request_handler.rb:26:in <top (required)>' /opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:inrequire'
/opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in require' /usr/share/passenger/phusion_passenger.rb:237:inrequire_passenger_lib'
/usr/share/passenger/helper-scripts/rack-preloader.rb:80:in init_passenger' /usr/share/passenger/helper-scripts/rack-preloader.rb:157:in<module:App>'
/usr/share/passenger/helper-scripts/rack-preloader.rb:29:in <module:PhusionPassenger>' /usr/share/passenger/helper-scripts/rack-preloader.rb:28:in<main>'
All my foreman/httpd/ruby/passenger installed packages :
[root@:~]# rpm -qa|grep httpd
httpd-2.4.6-40.el7.centos.x86_64
httpd-tools-2.4.6-40.el7.centos.x86_64
[root@:~]# rpm -qa|grep foreman
foreman-selinux-1.15.3-1.el7.noarch
foreman-cli-1.15.3-1.el7.noarch
foreman-release-scl-3-1.el7.noarch
foreman-proxy-1.15.3-1.el7.noarch
foreman-1.15.4-1.el7.noarch
foreman-release-1.15.3-1.el7.noarch
foreman-postgresql-1.15.4-1.el7.noarch
foreman-debug-1.15.3-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-0.10.2-1.el7.noarch
foreman-installer-1.15.3-1.el7.noarch
tfm-rubygem-foreman_setup-5.0.0-1.fm1_13.el7.noarch
[root@:~]# rpm -qa|grep passenger
tfm-rubygem-passenger-native-4.0.18-9.11.el7.x86_64
mod_passenger-4.0.53-4.el7.x86_64
tfm-rubygem-passenger-4.0.18-9.11.el7.x86_64
tfm-rubygem-passenger-native-libs-4.0.18-9.11.el7.x86_64
passenger-4.0.53-4.el7.x86_64
[root@:~]# rpm -qa|grep ruby
rh-ruby22-rubygem-io-console-0.4.3-16.el7.x86_64
sclo-ror42-rubygem-builder-3.2.2-2.el7.noarch
sclo-ror42-rubygem-thread_safe-0.3.5-2.el7.noarch
sclo-ror42-rubygem-loofah-2.0.2-3.el7.noarch
tfm-rubygem-http-cookie-1.0.2-1.el7.noarch
tfm-rubygem-addressable-2.3.6-4.el7.noarch
sclo-ror42-rubygem-arel-6.0.3-2.el7.noarch
sclo-ror42-rubygem-erubis-2.7.0-9.el7.noarch
tfm-rubygem-deep_cloneable-2.2.2-1.el7.noarch
tfm-rubygem-rabl-0.12.0-2.el7.noarch
sclo-ror42-rubygem-sprockets-rails-2.3.3-1.el7.noarch
sclo-ror42-rubygem-actionmailer-4.2.5.1-1.el7.noarch
rubygem-io-console-0.4.2-25.el7_1.x86_64
rubygem-rake-0.9.6-29.el7.noarch
tfm-rubygem-unicode-display_width-1.0.5-1.el7.noarch
rubygem-clamp-1.0.0-5.el7.noarch
rh-ruby22-ruby-irb-2.2.2-16.el7.noarch
sclo-ror42-rubygem-rack-1.6.2-3.el7.noarch
sclo-ror42-rubygem-mime-types-1.25.1-2.el7.noarch
tfm-rubygem-net-scp-1.1.0-6.el7.noarch
tfm-rubygem-unf_ext-0.0.6-6.el7.x86_64
tfm-rubygem-pg-0.15.1-3.el7.x86_64
tfm-rubygem-netrc-0.7.7-7.el7.noarch
sclo-ror42-rubygem-coffee-script-source-1.10.0-1.el7.noarch
sclo-ror42-rubygem-activesupport-4.2.5.1-1.el7.noarch
tfm-rubygem-audited-4.4.1-1.el7.noarch
sclo-ror42-rubygem-actionview-4.2.5.1-3.el7.noarch
sclo-ror42-rubygem-turbolinks-2.5.3-2.el7.noarch
rubygem-gssapi-1.1.2-3.el7.noarch
rubygem-tilt-2.0.7-1.el7.noarch
tfm-rubygem-apipie-bindings-0.2.0-1.el7.noarch
tfm-rubygem-passenger-native-4.0.18-9.11.el7.x86_64
tfm-rubygem-safemode-1.3.2-1.el7.noarch
rubygem-rdoc-4.0.0-29.el7.noarch
rubygem-kafo_wizards-0.0.1-2.el7.noarch
rubygem-little-plugger-1.1.3-21.el7.noarch
rubygem-kafo-2.0.0-1.el7.noarch
rh-ruby22-ruby-libs-2.2.2-16.el7.x86_64
rh-ruby22-ruby-2.2.2-16.el7.x86_64
rh-ruby22-rubygem-psych-2.0.8-16.el7.x86_64
tfm-rubygem-sexp_processor-4.4.4-3.el7.noarch
sclo-ror42-rubygem-mail-2.6.1-1.el7.noarch
rh-ruby22-rubygem-rake-10.4.2-16.el7.noarch
tfm-rubygem-ruby2ruby-2.1.3-4.el7.noarch
sclo-ror42-rubygem-rack-test-0.6.3-3.el7.noarch
tfm-rubygem-unf-0.1.3-5.el7.noarch
tfm-rubygem-excon-0.51.0-1.el7.noarch
tfm-rubygem-sshkey-1.9.0-1.el7.noarch
tfm-rubygem-roadie-3.2.1-1.el7.noarch
tfm-rubygem-rest-client-1.8.0-1.el7.noarch
tfm-rubygem-logging-1.8.2-4.el7.noarch
sclo-ror42-rubygem-execjs-2.2.0-2.el7.noarch
rh-ruby22-rubygem-bundler-1.7.8-2.sc1.el7.noarch
sclo-ror42-rubygem-activemodel-4.2.5.1-1.el7.noarch
tfm-rubygem-ancestry-2.2.1-1.el7.noarch
tfm-rubygem-friendly_id-5.1.0-3.el7.noarch
sclo-ror42-rubygem-activejob-4.2.5.1-1.el7.noarch
sclo-ror42-rubygem-actionpack-4.2.5.1-2.el7.noarch
tfm-rubygem-x-editable-rails-1.5.5-1.el7.noarch
tfm-rubygem-roadie-rails-1.1.1-1.el7.noarch
tfm-rubygem-apipie-rails-0.4.0-2.el7.noarch
rubygem-psych-2.0.0-25.el7_1.x86_64
rubygem-bigdecimal-1.2.0-25.el7_1.x86_64
rubygems-2.0.14.1-29.el7.noarch
rubygem-rb-inotify-0.9.7-2.el7.noarch
rubygem-thor-0.19.1-1.el7.noarch
rubygem-concurrent-ruby-1.0.3-1.el7.noarch
rubygem-rack-protection-1.5.3-3.el7.noarch
rubygem-highline-1.7.8-2.el7.noarch
rubygem-hashie-2.0.5-4.el7.noarch
rubygem-multi_json-1.10.1-3.el7.noarch
tfm-rubygem-clamp-1.0.0-5.el7.noarch
tfm-rubygem-highline-1.7.8-2.el7.noarch
rubygem-daemon_controller-1.1.2-2.el7.noarch
ruby-libs-2.0.0.648-29.el7.x86_64
rh-ruby22-rubygem-json-1.8.1-16.el7.x86_64
rh-ruby22-rubygem-bigdecimal-1.2.6-16.el7.x86_64
sclo-ror42-rubygem-nokogiri-1.6.7-0.4.rc3.el7.x86_64
tfm-rubygem-net-ldap-0.15.0-1.el7.noarch
sclo-ror42-rubygem-i18n-0.7.0-3.el7.noarch
sclo-ror42-rubygem-tzinfo-1.2.2-2.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-5.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-1.el7.noarch
tfm-rubygem-useragent-0.16.8-1.el7.noarch
tfm-rubygem-oauth-0.5.1-1.el7.noarch
tfm-rubygem-formatador-0.2.1-9.el7.noarch
tfm-rubygem-deacon-1.0.0-1.el7.noarch
sclo-ror42-rubygem-concurrent-ruby-1.0.0-2.el7.noarch
sclo-ror42-rubygem-coffee-script-2.4.1-1.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-1.el7.noarch
sclo-ror42-rubygem-activerecord-4.2.5.1-4.el7.noarch
tfm-rubygem-will_paginate-3.1.5-1.el7.noarch
tfm-rubygem-ldap_fluff-0.4.6-1.el7.noarch
sclo-ror42-rubygem-rails-deprecated_sanitizer-1.0.3-5.el7.noarch
sclo-ror42-rubygem-railties-4.2.5.1-2.el7.noarch
tfm-rubygem-responders-2.3.0-1.el7.noarch
tfm-rubygem-rails-i18n-4.0.9-1.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-1.el7.noarch
tfm-rubygem-facter-2.4.0-3.el7.x86_64
rubygem-ffi-1.9.10-2.el7.x86_64
rubygem-rkerberos-0.1.3-5.el7.x86_64
rubygem-bundler-1.7.8-3.el7.noarch
rubygem-sinatra-1.4.8-2.el7.noarch
tfm-rubygem-awesome_print-1.7.0-1.el7.noarch
tfm-rubygem-hammer_cli-0.10.2-1.el7.noarch
rubygem-ansi-1.4.3-2.el7.noarch
rubygem-kafo_parsers-0.1.6-1.el7.noarch
tfm-rubygem-passenger-4.0.18-9.11.el7.x86_64
tfm-rubygem-hammer_cli_foreman-0.10.2-1.el7.noarch
ruby-2.0.0.648-29.el7.x86_64
rh-ruby22-rubygem-rdoc-4.2.0-16.el7.noarch
tfm-rubygem-fast_gettext-1.1.0-1.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.2.1-3.el7.noarch
tfm-rubygem-little-plugger-1.1.3-21.el7.noarch
tfm-rubygem-secure_headers-3.4.1-1.el7.noarch
tfm-rubygem-fog-core-1.42.0-1.el7.noarch
sclo-ror42-rubygem-sprockets-3.5.2-1.el7.noarch
rh-ruby22-rubygem-minitest-5.4.3-16.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-3.el7.noarch
sclo-ror42-rubygem-rails-dom-testing-1.0.5-3.el7.noarch
sclo-ror42-rubygem-coffee-rails-4.1.0-3.el7.noarch
rubygem-rack-1.6.4-2.el7.noarch
rubygem-bundler_ext-0.4.1-1.el7.noarch
tfm-rubygem-passenger-native-libs-4.0.18-9.11.el7.x86_64
tfm-rubygem-net-ssh-4.0.1-2.el7.noarch
ruby-irb-2.0.0.648-29.el7.noarch
rubygem-logging-1.8.2-4.el7.noarch
rh-ruby22-runtime-2.0-6.sc1.el7.x86_64
rh-ruby22-rubygems-2.4.5-16.el7.x86_64
tfm-rubygem-ruby_parser-3.6.3-4.el7.noarch
rh-ruby22-rubygem-thor-0.19.1-1.el7.noarch
sclo-ror42-rubygem-rails-html-sanitizer-1.0.3-1.el7.noarch
tfm-rubygem-net-ping-2.0.1-1.el7.noarch
tfm-rubygem-css_parser-1.4.7-1.el7.noarch
sclo-ror42-rubygem-multi_json-1.10.1-3.el7.noarch
rh-ruby22-rubygem-net-http-persistent-2.9.4-4.el7.noarch
tfm-rubygem-scoped_search-4.1.0-1.el7.noarch
sclo-ror42-rubygem-globalid-0.3.3-3.el7.noarch
tfm-rubygem-activerecord-session_store-0.1.2-2.el7.noarch
sclo-ror42-rubygem-rails-4.2.5.1-1.el7.noarch
rubygem-json-1.7.7-25.el7_1.x86_64
rubygem-net-http-persistent-2.8-5.el7.noarch
rubygem-rubyipmi-0.10.0-2.el7.noarch
tfm-rubygem-locale-2.0.9-11.el7.noarch
tfm-rubygem-foreman_setup-5.0.0-1.fm1_13.el7.noarch
rubygem-powerbar-1.0.17-1.el7.noarch
Here are more informations about my system :
Error ID
54ed08b1
Application root
/usr/share/foreman
Environment (value of RAILS_ENV, RACK_ENV, WSGI_ENV, NODE_ENV and PASSENGER_APP_ENV)
production
Passenger/Ruby versions
mod_passenger-4.0.53-4.el7.x86_64
passenger-4.0.53-4.el7.x86_64
tfm-rubygem-passenger-4.0.18-9.11.el7.x86_64
tfm-rubygem-passenger-native-4.0.18-9.11.el7.x86_64
tfm-rubygem-passenger-native-libs-4.0.18-9.11.el7.x86_64
Ruby interpreter command
/usr/bin/tfm-ruby
User and groups
uid=995(foreman) gid=494(foreman) groups=494(foreman),497(puppet) context=system_u:system_r:passenger_t:s0
Environment variables
MANPATH = /opt/theforeman/tfm/root/usr/share/man:/opt/rh/rh-ruby22/root/usr/share/man:/opt/rh/sclo-ror42/root/usr/share/man:
SHELL = /bin/false
LIBRARY_PATH = /opt/theforeman/tfm/root/usr/lib64dd
PYTHONUNBUFFERED = 1
PASSENGER_DEBUG_DIR = /tmp/passenger.spawn-debug.XXXXxdj6Sq
SERVER_PORT = 80
X_SCLS = rh-ruby22 sclo-ror42 tfm
USER = foreman
DOCUMENT_ROOT = /usr/share/foreman/public
LD_LIBRARY_PATH = /opt/theforeman/tfm/root/usr/lib64:/opt/rh/rh-ruby22/root/usr/lib64:/opt/rh/sclo-ror42/root/usr/lib64
REQUEST_URI = /
CPATH = /opt/theforeman/tfm/root/usr/include
IN_PASSENGER = 1
RACK_ENV = production
REMOTE_PORT = 49178
PATH = /opt/theforeman/tfm/root/usr/bin:/opt/rh/rh-ruby22/root/usr/bin:/opt/rh/sclo-ror42/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
WSGI_ENV = production
_ = /opt/rh/rh-ruby22/root/usr/bin/ruby
PWD = /usr/share/foreman
SERVER_ADMIN = [no address given]
LANG = C
NODE_PATH = /usr/share/passenger/node
NODE_ENV = production
NOTIFY_SOCKET = /run/systemd/notify
REMOTE_ADDR = 127.0.0.1
HOME = /usr/share/foreman
SHLVL = 4
RAILS_ENV = production
LOGNAME = foreman
SERVER_SOFTWARE = Apache
QUERY_STRING =
SERVER_ADDR = 127.0.0.1
GEM_PATH = /opt/theforeman/tfm/root/usr/share/gems:/opt/rh/sclo-ror42/root/usr/share/gems:/usr/share/foreman/.gem/ruby:/opt/rh/rh-ruby22/root/usr/share/gems:/opt/rh/rh-ruby22/root/usr/local/share/gems/opt/rh/sclo-ror42/root/usr/share/gems:/usr/share/foreman/.gem/ruby:/opt/rh/rh-ruby22/root/usr/share/gems:/opt/rh/rh-ruby22/root/usr/local/share/gems
XDG_DATA_DIRS = /opt/rh/rh-ruby22/root/usr/share
PKG_CONFIG_PATH = /opt/theforeman/tfm/root/usr/lib64/pkgconfig:/opt/rh/rh-ruby22/root/usr/lib64/pkgconfig:/opt/rh/sclo-ror42/root/usr/lib64/pkgconfig
PASSENGER_APP_ENV = production
SERVER_PROTOCOL = HTTP/1.1
REQUEST_METHOD = HEAD
Thanks for the help!