Weird, as this is not what foreman-installer showed me. I checked it out; and, at first, it appeared as though the --verbose -l DEBUG flags only apply to stdout, and not katello.log. That said, upon further inspection, it’s now clear that foreman-installer omits this line from the log by design; which, I think, unintentionally provides a false narrative:
EXPECTATION
Foreman 2.2 Manual > 3.2.1 Installation > Running the installer:
After it completes, the installer will print some details about where to find Foreman and the Smart Proxy. Output should be similar to this:* Foreman is running at https://theforeman.example.com Initial credentials are admin / 3ekw5xtyXCoXxS29 * Foreman Proxy is running at https://theforeman.example.com:8443 * The full log is at /var/log/foreman-installer/foreman-installer.log
NARRATIVE
/var/log/foreman-installer/katello.log (2.5 MB):
[DEBUG 2020-12-03T20:15:31 main] /File[/opt/puppetlabs/puppet/cache/locales]: Adding autorequire relationship with File[/opt/puppetlabs/puppet/cache] [DEBUG 2020-12-03T20:15:31 main] Finishing transaction 53036520 [DEBUG 2020-12-03T20:15:31 main] Received report to process from satellite.gtkcentral.net [ INFO 2020-12-03T20:15:33 main] Puppet has finished, bye! [ INFO 2020-12-03T20:15:33 main] Executing hooks in group post [DEBUG 2020-12-03T20:15:33 main] Hook /usr/share/foreman-installer/hooks/post/30-upgrade.rb returned nil [DEBUG 2020-12-03T20:15:33 main] Hook /usr/share/foreman-installer/hooks/post/99-post_install_message.rb returned nil [DEBUG 2020-12-03T20:15:33 main] cdn_ssl_version already migrated, skipping [DEBUG 2020-12-03T20:15:33 main] Hook /usr/share/foreman-installer/katello/hooks/post/31-cdn_setting.rb returned [#<Logging::Logger:0x000000 ... @caller_tracing=false>] [DEBUG 2020-12-03T20:15:33 main] Hook /usr/share/foreman-installer/katello/hooks/post/99-version_locking.rb returned nil [ INFO 2020-12-03T20:15:33 main] All hooks in group post finished [DEBUG 2020-12-03T20:15:33 main] Exit with status code: 6 (signal was 6) [DEBUG 2020-12-03T20:15:33 main] Cleaning /tmp/kafo_installation20201203-5806-1yc6h7s [DEBUG 2020-12-03T20:15:33 main] Cleaning /tmp/kafo_installation20201203-5806-1cjg6v0 [DEBUG 2020-12-03T20:15:33 main] Cleaning /tmp/default_values.yaml [ INFO 2020-12-03T20:15:33 main] Installer finished in 329.333699211 seconds
REALITY
STDOUT(truncated for clarity):[ INFO 2020-12-03T20:15:33 verbose] Puppet has finished, bye! [ INFO 2020-12-03T20:15:33 verbose] Executing hooks in group post [DEBUG 2020-12-03T20:15:33 verbose] Hook /usr/share/foreman-installer/hooks/post/30-upgrade.rb returned nil Something went wrong! Check the log for ERROR-level output The full log is at /var/log/foreman-installer/katello.log [DEBUG 2020-12-03T20:15:33 verbose] Hook /usr/share/foreman-installer/hooks/post/99-post_install_message.rb returned nil [DEBUG 2020-12-03T20:15:33 verbose] cdn_ssl_version already migrated, skipping ... [ INFO 2020-12-03T20:15:33 verbose] Installer finished in 329.333699211 seconds
That’s pretty weird. I’ve never really had to check that stuff, so I was unaware (and a bit befuddled) that the, “something went wrong” message would be omitted from it.
Either way, in my experience, the installer will always say it, finished in ###.##### seconds, whether the install was successful or not. I’ve never been given the impression that it is indicative of success, just completion. It’s like a race, one can finish arace; but a race’s completion is not indicative of one’s success in said race.
I think so. /usr/lib/systemd/system/rh-redis5-redis.service states the conf file as, /etc/opt/rh/rh-redis5/redis.conf. That conf file is installed with the following path properties (numbered for clarity):
20 pidfile /var/opt/rh/rh-redis5/run/redis_6379.pid
64 unixsocket /var/run/redis/redis.sock
96 logfile /var/log/redis/redis.log
180 dir /var/opt/rh/rh-redis5/lib/redis
I wasn’t sure about this one, to be honest. At one point, My troubles lead me to an issue similar to this one. OP received no further response; so, I made a point to bring it up in case it was accidentally lost in time.
In addition, it is normally assumed that the user in question owns it’s own $HOME folder; so, you could say it just subconsciously raised a red-flag for me, as someone on the outside looking in.
I never have a system ruby installed prior to the fresh install; nor do I use ruby version managers. Though in the distant, past of my Foreman journey (journeyman? 
), I dabbled with rvm, and learned early on, to just avoid them all together*). Whatever version of ruby is installed as a result of foreman/katello’s dependencies is what my ruby environment ends up being.
That said, all versions of ruby are detailed in the archive I uploaded; and from the katello.log:
16363 [ WARN 2020-12-04T11:13:29 main] /Stage[main]/Foreman::Database/Foreman_config_entry[db_pending_seed]: Skipping because of failed dependencies
16366 [ WARN 2020-12-04T11:13:29 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Skipping because of failed dependencies
16372 [ WARN 2020-12-04T11:13:29 main] /Service[httpd]: Skipping because of failed dependencies
16377 [ WARN 2020-12-04T11:13:29 main] /Stage[main]/Katello::Application/Foreman_config_entry[pulp_client_cert]: Skipping because of failed dependencies
16379 [ WARN 2020-12-04T11:13:29 main] /Stage[main]/Katello::Application/Foreman_config_entry[pulp_client_key]: Skipping because of failed dependencies
16381 [ WARN 2020-12-04T11:13:29 main] /Stage[main]/Foreman/Foreman::Rake[apipie:cache:index]/Exec[foreman-rake-apipie:cache:index]: Skipping because of failed dependencies
16384 [ WARN 2020-12-04T11:13:29 main] /Service[dynflow-sidekiq@worker-hosts-queue]: Skipping because of failed dependencies
16432 [ WARN 2020-12-04T11:13:36 main] /Service[foreman]: Skipping because of failed dependencies
16437 [ WARN 2020-12-04T11:13:36 main] /Stage[main]/Foreman::Service/Foreman::Dynflow::Worker[orchestrator]/File[/etc/foreman/dynflow/orchestrator.yml]: Skipping because of failed dependencies
16439 [ WARN 2020-12-04T11:13:36 main] /Service[dynflow-sidekiq@orchestrator]: Skipping because of failed dependencies
16442 [ WARN 2020-12-04T11:13:36 main] /Stage[main]/Foreman::Service/Foreman::Dynflow::Worker[worker]/File[/etc/foreman/dynflow/worker.yml]: Skipping because of failed dependencies
16444 [ WARN 2020-12-04T11:13:36 main] /Service[dynflow-sidekiq@worker]: Skipping because of failed dependencies
16457 [ WARN 2020-12-04T11:13:37 main] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[satellite.gtkcentral.net]: Skipping because of failed dependencies
16461 [ WARN 2020-12-04T11:13:37 main] /Service[puppet]: Skipping because of failed dependencies
16464 [ WARN 2020-12-04T11:13:37 main] /Service[puppet-run.timer]: Skipping because of failed dependencies
16466 [ WARN 2020-12-04T11:13:37 main] /Stage[main]/Puppet::Agent::Service::Systemd/File[/etc/systemd/system/puppet-run.timer]: Skipping because of failed dependencies
16468 [ WARN 2020-12-04T11:13:37 main] /Stage[main]/Puppet::Agent::Service::Systemd/File[/etc/systemd/system/puppet-run.service]: Skipping because of failed dependencies
16470 [ WARN 2020-12-04T11:13:37 main] /Stage[main]/Puppet::Agent::Service::Systemd/Exec[systemctl-daemon-reload-puppet]: Skipping because of failed dependencies
16473 [ WARN 2020-12-04T11:13:37 main] /Stage[main]/Puppet::Agent::Service::Cron/Cron[puppet]: Skipping because of failed dependencies
I wasn’t sure which of these pertain to ruby dependencies or yum dependencies; though, I would guess the former, given the appropriate yum repos were installed as guided by the Official Docs.
But this is why I’m here; because, I only know what I know. For all I know, this is normal behaviour; and “dependency,” in this case, could mean, “no --enable-****** option was set for this stage,”–I don’t really know. I set the install to --trace to get a clearer picture; but, some portions remain fairly scarce in detail either way.
So yeah, this is a result of my attempts to get to the root of:
[DEBUG 2020-12-03T20:14:18 main] Exec[foreman-rake-db:migrate](provider=posix): Executing check '/usr/sbin/foreman-rake db:abort_if_pending_migrations'
[DEBUG 2020-12-03T20:14:18 main] Executing with uid=foreman: '/usr/sbin/foreman-rake db:abort_if_pending_migrations'
[DEBUG 2020-12-03T20:14:19 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: rake aborted!
[DEBUG 2020-12-03T20:14:19 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: LoadError: cannot load such file -- apipie/middleware/checksum_in_headers
[DEBUG 2020-12-03T20:14:19 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: /usr/share/foreman/config/application.rb:5:in `<top (required)>'
[DEBUG 2020-12-03T20:14:19 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: /usr/share/foreman/Rakefile:1:in `<top (required)>'
[DEBUG 2020-12-03T20:14:19 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
[DEBUG 2020-12-03T20:14:19 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless: (See full trace by running task with --trace)
[DEBUG 2020-12-03T20:14:19 main] Exec[foreman-rake-db:migrate](provider=posix): Executing '/usr/sbin/foreman-rake db:migrate'
[DEBUG 2020-12-03T20:14:19 main] Executing with uid=foreman: '/usr/sbin/foreman-rake db:migrate'
[ WARN 2020-12-03T20:14:19 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: rake aborted!
[ WARN 2020-12-03T20:14:19 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: LoadError: cannot load such file -- apipie/middleware/checksum_in_headers
[ WARN 2020-12-03T20:14:19 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/foreman/config/application.rb:5:in `<top (required)>'
[ WARN 2020-12-03T20:14:19 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /usr/share/foreman/Rakefile:1:in `<top (required)>'
[ WARN 2020-12-03T20:14:19 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
[ WARN 2020-12-03T20:14:19 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: (See full trace by running task with --trace)
So I found the manifests that were calling, db:abort_if_pending_migrations, and db:migrate, respectively, during this phase:
- I created a quick script,
/usr/sbin/foreman-env:#!/bin/bash declare STMP=`/usr/bin/date +'%y%m%d%H%M%S%N'`; { # ... declare ECO='/usr/bin/echo'; declare WCH='/usr/bin/which'; ${ECO} -e "USR: `$($WCH id)`"; ${ECO} -e "PWD: `$($WCH pwd)`"; ${ECO} -e "VER: Ruby | `$($WCH ruby) -v`"; ${ECO} -e "VER: Gem | `$($WCH gem) -v`"; ${ECO} -e "\nENV: Gem"; `$WCH gem` env; ${ECO} -e "\nENV: Linux"; `$WCH env` | /usr/bin/sort; ${ECO} -e "\nENV: Procs"; `$WCH ps` -f --forest; } >/tmp/foreman-inst.${STMP}.env - I edited
line 25ofdatabase.ppto:foreman::rake { 'db:migrate': unless => '/usr/sbin/foreman-env ; /usr/sbin/foreman-rake db:abort_if_pending_migrations', } - I edited
line 24ofrake.ppto:exec { "foreman-rake-${title}": command => "/usr/sbin/foreman-env ; /usr/sbin/foreman-rake ${title}", user => $user, environment => sort(join_keys_to_values(merge({'HOME' => $app_root}, $environment), '=')), - Then I ran the installer (keep in mind, this was a previous attempt–not related to the attempt my uploaded archive relates to).
- What I found was:
USR: uid=985(foreman) gid=982(foreman) groups=982(foreman) PWD: /root VER: Ruby | ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux] VER: Gem | 2.7.6.2 ENV: Gem RubyGems Environment: - RUBYGEMS VERSION: 2.7.6.2 - RUBY VERSION: 2.5.5 (2019-03-15 patchlevel 157) [x86_64-linux] - INSTALLATION DIRECTORY: /opt/rh/rh-ruby25/root/usr/share/gems - USER INSTALLATION DIRECTORY: /usr/share/foreman/.gem/ruby - RUBY EXECUTABLE: /opt/rh/rh-ruby25/root/usr/bin/ruby - EXECUTABLE DIRECTORY: /opt/rh/rh-ruby25/root/usr/bin - SPEC CACHE DIRECTORY: /usr/share/foreman/.gem/specs - SYSTEM CONFIGURATION DIRECTORY: /etc/opt/rh/rh-ruby25 - RUBYGEMS PLATFORMS: - ruby - x86_64-linux - GEM PATHS: - /opt/rh/rh-ruby25/root/usr/share/gems - /usr/share/foreman/.gem/ruby - /opt/rh/rh-ruby25/root/usr/local/share/gems - GEM CONFIGURATION: - :update_sources => true - :verbose => true - :backtrace => false - :bulk_threshold => 1000 - "gem" => "--user-install --bindir /usr/share/foreman/bin" - REMOTE SOURCES: - https://rubygems.org/ - SHELL PATH: - /opt/rh/rh-ruby25/root/usr/local/bin - /opt/rh/rh-ruby25/root/usr/bin - /opt/theforeman/tfm/root/usr/bin - /usr/local/sbin - /usr/local/bin - /usr/sbin - /usr/bin - /opt/puppetlabs/bin - /usr/share/foreman/bin - /root/bin - /sbin ENV: Linux APT_LISTBUGS_FRONTEND=none APT_LISTCHANGES_FRONTEND=none CONFIGURE_ARGS=with-pg-config=/opt/rh/rh-postgresql12/root/usr/bin/pg_config CPATH=/opt/theforeman/tfm/root/usr/include DEBIAN_FRONTEND=noninteractive ... FOREMAN_PROXY=satellite.gtkcentral.net FQDN=satellite.gtkcentral.net ... HOME=/usr/share/foreman HOSTNAME=satellite.gtkcentral.net ... LD_LIBRARY_PATH=/opt/rh/rh-ruby25/root/usr/local/lib64:/opt/rh/rh-ruby25/root/usr/lib64 ... LIBRARY_PATH=/opt/theforeman/tfm/root/usr/lib64dd ... MANPATH=/opt/rh/rh-ruby25/root/usr/local/share/man:/opt/rh/rh-ruby25/root/usr/share/man:/opt/theforeman/tfm/root/usr/share/man::/opt/puppetlabs/puppet/share/man PATH=/opt/rh/rh-ruby25/root/usr/local/bin:/opt/rh/rh-ruby25/root/usr/bin:/opt/theforeman/tfm/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/puppetlabs/bin:/usr/share/foreman/bin:/root/bin:/sbin PKG_CONFIG_PATH=/opt/rh/rh-ruby25/root/usr/local/lib64/pkgconfig:/opt/rh/rh-ruby25/root/usr/lib64/pkgconfig:/opt/theforeman/tfm/root/usr/lib64/pkgconfig ... PWD=/root SATELLITE_DIR=/etc/satellite SHELL=/bin/bash SHLVL=4 ... SSLD=/etc/ssl/certs ... XDG_DATA_DIRS=/opt/rh/rh-ruby25/root/usr/local/share:/opt/rh/rh-ruby25/root/usr/share:/usr/local/share:/usr/share XDG_RUNTIME_DIR=/run/user/0 XDG_SESSION_ID=254 X_SCLS=rh-ruby25 tfm ENV: Procs UID PID PPID C STIME TTY TIME CMD foreman 16919 15719 0 11:13 ? 00:00:00 sh -c /usr/sbin/foreman-env ; /usr/sbin/foreman-rake db:abort_if_pending_migrations foreman 16922 16919 0 11:13 ? 00:00:00 \_ /bin/bash /usr/sbin/foreman-env foreman 16959 16922 0 11:13 ? 00:00:00 \_ /usr/bin/ps -f --forest - Which honestly looked good to me. Looks like this phase of the install has everything in place to successfully run those
rakes. So I decided to try it myself:[root@satellite ~]# su -s /bin/bash foreman bash-4.2$ scl enable rh-ruby25 'scl enable tfm bash' bash-4.2$ /usr/sbin/foreman-rake db:abort_if_pending_migrations --trace --backtrace ** Invoke db:abort_if_pending_migrations (first_time) ** Invoke db:load_config (first_time) ** Invoke environment (first_time) ** Execute environment ** Execute db:load_config ** Execute db:abort_if_pending_migrations ** Invoke dynflow:abort_if_pending_migrations (first_time) ** Invoke environment ** Execute dynflow:abort_if_pending_migrations bash-4.2$ /usr/sbin/foreman-rake db:migrate --trace --backtrace ** Invoke db:migrate (first_time) ** Invoke db:load_config (first_time) ** Invoke environment (first_time) ** Execute environment ** Execute db:load_config ** Invoke plugin:refresh_migrations (first_time) ** Invoke environment ** Execute plugin:refresh_migrations ** Execute db:migrate ** Invoke db:_dump (first_time) ** Execute db:_dump ** Invoke dynflow:migrate (first_time) ** Invoke environment ** Execute dynflow:migrate
And it worked! Though not in the context of this post, that is, foreman-installer’s ability to run successfully; hence, it is broken.
Like a rear-view mirror (foreman-rake) whose become detached from it’s vehicle (foreman-installer). Sure, the mirror still works if I hold it in front of myself; but, it’s still broken in the context of providing me vantage while I drive. And, yeah, I know foreman-rake is more than a rear-view mirror; but, for analogy’s sake.