Failed to start Foreman Proxy - cannot load such file -- smart_proxy_main (LoadError)

Support
,
1

Problem:

The Foreman-proxy service stopped working after Foreman was updated following the official tutorial, see the journalctl errors:

2024-03-13 15:51:42 [DEBUG ] [configure] Running journalctl command to get logs for systemd start failure: journalctl -n 50 --since '5 minutes ago' -u foreman-proxy --no-pager
2024-03-13 15:51:42 [DEBUG ] [configure] Executing: 'journalctl -n 50 --since '5 minutes ago' -u foreman-proxy --no-pager'
2024-03-13 15:51:42 [ERROR ] [configure] Systemd start for foreman-proxy failed!
2024-03-13 15:51:42 [ERROR ] [configure] journalctl log for foreman-proxy:
2024-03-13 15:51:42 [ERROR ] [configure] -- Logs begin at Wed 2024-03-13 15:05:07 -03, end at Wed 2024-03-13 15:51:42 -03. --
2024-03-13 15:51:42 [ERROR ] [configure] Mar 13 15:49:49 sv-cmdb-p05.tjgo.ldc systemd[1]: Starting Foreman Proxy...
2024-03-13 15:51:42 [ERROR ] [configure] Mar 13 15:49:49 sv-cmdb-p05.tjgo.ldc smart-proxy[12123]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- smart_proxy_main (LoadError)
2024-03-13 15:51:42 [ERROR ] [configure] Mar 13 15:49:49 sv-cmdb-p05.tjgo.ldc smart-proxy[12123]:         from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
2024-03-13 15:51:42 [ERROR ] [configure] Mar 13 15:49:49 sv-cmdb-p05.tjgo.ldc smart-proxy[12123]:         from /usr/share/foreman-proxy/bin/smart-proxy:5:in `<main>'
2024-03-13 15:51:42 [ERROR ] [configure] Mar 13 15:49:49 sv-cmdb-p05.tjgo.ldc systemd[1]: foreman-proxy.service: Main process exited, code=exited, status=1/FAILURE
2024-03-13 15:51:42 [ERROR ] [configure] Mar 13 15:49:49 sv-cmdb-p05.tjgo.ldc systemd[1]: foreman-proxy.service: Failed with result 'exit-code'.
2024-03-13 15:51:42 [ERROR ] [configure] Mar 13 15:49:49 sv-cmdb-p05.tjgo.ldc systemd[1]: Failed to start Foreman Proxy.

Expected outcome:

After the update, all Foreman services were restored to their normal working state.

Foreman and Proxy versions:

Installed Packages
ansible-collection-theforeman-foreman-4.0.0-2.el8.noarch
ansiblerole-foreman_scap_client-0.2.0-2.el8.noarch
candlepin-4.3.12-1.el8.noarch
candlepin-selinux-4.3.12-1.el8.noarch
foreman-3.11.0-0.2.develop.20240312180849git4abf925.el8.noarch
foreman-cli-3.11.0-0.2.develop.20240312180849git4abf925.el8.noarch
foreman-dynflow-sidekiq-3.11.0-0.2.develop.20240312180849git4abf925.el8.noarch
foreman-installer-3.11.0-0.1.develop.20240312031240git3f846ff.el8.noarch
foreman-installer-katello-3.11.0-0.1.develop.20240312031240git3f846ff.el8.noarch
foreman-postgresql-3.11.0-0.2.develop.20240312180849git4abf925.el8.noarch
foreman-proxy-3.11.0-0.1.develop.20240226141913gite8fe59f.el8.noarch
foreman-redis-3.11.0-0.2.develop.20240312180849git4abf925.el8.noarch
foreman-release-3.11.0-0.1.develop.el8.noarch
foreman-selinux-3.11.0-0.1.develop.20240226091422gita8664f9.el8.noarch
foreman-service-3.11.0-0.2.develop.20240312180849git4abf925.el8.noarch
foreman-vmware-3.11.0-0.2.develop.20240312180849git4abf925.el8.noarch
katello-4.13.0-0.1.master.el8.noarch
katello-certs-tools-2.9.0-2.el8.noarch
katello-client-bootstrap-1.7.9-2.el8.noarch
katello-common-4.13.0-0.1.master.el8.noarch
katello-repos-4.13-0.1.nightly.el8.noarch
katello-selinux-5.0.2-1.el8.noarch
pulpcore-selinux-2.0.1-1.el8.x86_64
python3.11-pulp-ansible-0.20.2-3.el8.noarch
python3.11-pulp-certguard-1.7.1-2.el8.noarch
python3.11-pulp-cli-0.21.2-5.el8.noarch
python3.11-pulp-container-2.16.4-1.el8.noarch
python3.11-pulp-deb-3.0.1-1.el8.noarch
python3.11-pulp-file-1.15.1-2.el8.noarch
python3.11-pulp-glue-0.21.2-3.el8.noarch
python3.11-pulp-python-3.10.0-3.el8.noarch
python3.11-pulp-rpm-3.23.3-1.el8.noarch
python3.11-pulpcore-3.39.11-1.el8.noarch
rubygem-foreman-tasks-9.1.0-1.fm3_10.el8.noarch
rubygem-foreman_ansible-13.0.3-2.fm3_10.el8.noarch
rubygem-foreman_bootdisk-21.2.2-1.fm3_11.el8.noarch
rubygem-foreman_discovery-24.0.1-1.fm3_10.el8.noarch
rubygem-foreman_hooks-0.3.17-3.fm3_9.el8.noarch
rubygem-foreman_maintain-1.5.1-1.el8.noarch
rubygem-foreman_openscap-7.1.1-2.fm3_10.el8.noarch
rubygem-foreman_puppet-6.2.0-1.fm3_10.el8.noarch
rubygem-foreman_remote_execution-12.0.5-3.fm3_10.el8.noarch
rubygem-hammer_cli-3.11.0-0.1.pre.develop.el8.noarch
rubygem-hammer_cli_foreman-3.11.0-0.1.pre.develop.el8.noarch
rubygem-hammer_cli_foreman_puppet-0.0.7-1.fm3_10.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.3.0-1.fm3_10.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.20-1.fm3_10.el8.noarch
rubygem-hammer_cli_katello-1.12.0-0.1.pre.master.20240122184453git9a6ae05.el8.noarch
rubygem-katello-4.13.0-0.1.pre.master.20240308204117git8bedda9.el8.noarch
rubygem-pulp_ansible_client-0.20.3-1.el8.noarch
rubygem-pulp_certguard_client-1.6.5-1.el8.noarch
rubygem-pulp_container_client-2.16.4-1.el8.noarch
rubygem-pulp_deb_client-3.0.1-1.el8.noarch
rubygem-pulp_file_client-1.15.1-1.el8.noarch
rubygem-pulp_ostree_client-2.1.3-1.el8.noarch
rubygem-pulp_python_client-3.10.0-1.el8.noarch
rubygem-pulp_rpm_client-3.23.2-1.el8.noarch
rubygem-pulpcore_client-3.39.9-1.el8.noarch
rubygem-smart_proxy_pulp-3.3.0-1.fm3_10.el8.noarch

Foreman and Proxy plugin versions:

Installed Packages
ansible-collection-theforeman-foreman-4.0.0-2.el8.noarch
ansiblerole-foreman_scap_client-0.2.0-2.el8.noarch
candlepin-4.3.12-1.el8.noarch
candlepin-selinux-4.3.12-1.el8.noarch
foreman-3.11.0-0.2.develop.20240312180849git4abf925.el8.noarch
foreman-cli-3.11.0-0.2.develop.20240312180849git4abf925.el8.noarch
foreman-dynflow-sidekiq-3.11.0-0.2.develop.20240312180849git4abf925.el8.noarch
foreman-installer-3.11.0-0.1.develop.20240312031240git3f846ff.el8.noarch
foreman-installer-katello-3.11.0-0.1.develop.20240312031240git3f846ff.el8.noarch
foreman-postgresql-3.11.0-0.2.develop.20240312180849git4abf925.el8.noarch
foreman-proxy-3.11.0-0.1.develop.20240226141913gite8fe59f.el8.noarch
foreman-redis-3.11.0-0.2.develop.20240312180849git4abf925.el8.noarch
foreman-release-3.11.0-0.1.develop.el8.noarch
foreman-selinux-3.11.0-0.1.develop.20240226091422gita8664f9.el8.noarch
foreman-service-3.11.0-0.2.develop.20240312180849git4abf925.el8.noarch
foreman-vmware-3.11.0-0.2.develop.20240312180849git4abf925.el8.noarch
katello-4.13.0-0.1.master.el8.noarch
katello-certs-tools-2.9.0-2.el8.noarch
katello-client-bootstrap-1.7.9-2.el8.noarch
katello-common-4.13.0-0.1.master.el8.noarch
katello-repos-4.13-0.1.nightly.el8.noarch
katello-selinux-5.0.2-1.el8.noarch
pulpcore-selinux-2.0.1-1.el8.x86_64
python3.11-pulp-ansible-0.20.2-3.el8.noarch
python3.11-pulp-certguard-1.7.1-2.el8.noarch
python3.11-pulp-cli-0.21.2-5.el8.noarch
python3.11-pulp-container-2.16.4-1.el8.noarch
python3.11-pulp-deb-3.0.1-1.el8.noarch
python3.11-pulp-file-1.15.1-2.el8.noarch
python3.11-pulp-glue-0.21.2-3.el8.noarch
python3.11-pulp-python-3.10.0-3.el8.noarch
python3.11-pulp-rpm-3.23.3-1.el8.noarch
python3.11-pulpcore-3.39.11-1.el8.noarch
rubygem-foreman-tasks-9.1.0-1.fm3_10.el8.noarch
rubygem-foreman_ansible-13.0.3-2.fm3_10.el8.noarch
rubygem-foreman_bootdisk-21.2.2-1.fm3_11.el8.noarch
rubygem-foreman_discovery-24.0.1-1.fm3_10.el8.noarch
rubygem-foreman_hooks-0.3.17-3.fm3_9.el8.noarch
rubygem-foreman_maintain-1.5.1-1.el8.noarch
rubygem-foreman_openscap-7.1.1-2.fm3_10.el8.noarch
rubygem-foreman_puppet-6.2.0-1.fm3_10.el8.noarch
rubygem-foreman_remote_execution-12.0.5-3.fm3_10.el8.noarch
rubygem-hammer_cli-3.11.0-0.1.pre.develop.el8.noarch
rubygem-hammer_cli_foreman-3.11.0-0.1.pre.develop.el8.noarch
rubygem-hammer_cli_foreman_puppet-0.0.7-1.fm3_10.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.3.0-1.fm3_10.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.20-1.fm3_10.el8.noarch
rubygem-hammer_cli_katello-1.12.0-0.1.pre.master.20240122184453git9a6ae05.el8.noarch
rubygem-katello-4.13.0-0.1.pre.master.20240308204117git8bedda9.el8.noarch
rubygem-pulp_ansible_client-0.20.3-1.el8.noarch
rubygem-pulp_certguard_client-1.6.5-1.el8.noarch
rubygem-pulp_container_client-2.16.4-1.el8.noarch
rubygem-pulp_deb_client-3.0.1-1.el8.noarch
rubygem-pulp_file_client-1.15.1-1.el8.noarch
rubygem-pulp_ostree_client-2.1.3-1.el8.noarch
rubygem-pulp_python_client-3.10.0-1.el8.noarch
rubygem-pulp_rpm_client-3.23.2-1.el8.noarch
rubygem-pulpcore_client-3.39.9-1.el8.noarch
rubygem-smart_proxy_pulp-3.3.0-1.fm3_10.el8.noarch

Distribution and version:

NAME="Rocky Linux"
VERSION="8.9 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.9"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.9 (Green Obsidian)"

Other relevant data:

The foreman update followed the command sequence below on today’s date.

Here we move from Foreman 3.10.0-rc1 to 3.11.0-develop and Katello from 4.12.0-rc1 to katello-4.13.0-0.1.master

sudo foreman-rake katello:upgrade_check
sudo dnf update -y
sudo dnf update -y https://yum.theforeman.org/releases/3.10/el8/x86_64/foreman-release.rpm \
https://yum.theforeman.org/katello/4.12/katello/el8/x86_64/katello-repos-latest.rpm
sudo dnf module disable pulpcore
sudo dnf module enable katello:el8
sudo foreman-maintain service stop
sudo dnf update -y
sudo foreman-installer
sudo dnf needs-restarting --reboothint
sudo reboot

sudo dnf update -y

$ sudo dnf update -y
Last metadata expiration check: 2:18:32 ago on Wed 13 Mar 2024 12:36:16 PM -03.
Dependencies resolved.
=====================================================================================================================================================================================================================================
 Package                                                   Architecture                          Version                                                                              Repository                                Size
=====================================================================================================================================================================================================================================
Upgrading:
 foreman                                                   noarch                                3.11.0-0.2.develop.20240312180849git4abf925.el8                                      foreman                                   34 M
 foreman-cli                                               noarch                                3.11.0-0.2.develop.20240312180849git4abf925.el8                                      foreman                                   34 k
 foreman-dynflow-sidekiq                                   noarch                                3.11.0-0.2.develop.20240312180849git4abf925.el8                                      foreman                                   37 k
 foreman-installer                                         noarch                                1:3.11.0-0.1.develop.20240312031240git3f846ff.el8                                    foreman                                  1.6 M
 foreman-installer-katello                                 noarch                                1:3.11.0-0.1.develop.20240312031240git3f846ff.el8                                    foreman                                   91 k
 foreman-postgresql                                        noarch                                3.11.0-0.2.develop.20240312180849git4abf925.el8                                      foreman                                   35 k
 foreman-proxy                                             noarch                                3.11.0-0.1.develop.20240226141913gite8fe59f.el8                                      foreman                                  155 k
 foreman-redis                                             noarch                                3.11.0-0.2.develop.20240312180849git4abf925.el8                                      foreman                                   35 k
 foreman-selinux                                           noarch                                3.11.0-0.1.develop.20240226091422gita8664f9.el8                                      foreman                                   55 k
 foreman-service                                           noarch                                3.11.0-0.2.develop.20240312180849git4abf925.el8                                      foreman                                   37 k
 foreman-vmware                                            noarch                                3.11.0-0.2.develop.20240312180849git4abf925.el8                                      foreman                                   35 k
 katello                                                   noarch                                4.13.0-0.1.master.el8                                                                katello                                   17 k
 katello-common                                            noarch                                4.13.0-0.1.master.el8                                                                katello                                   25 k
 postgresql                                                x86_64                                13.14-1.module+el8.9.0+1736+ef8fa13a                                                 appstream                                1.5 M
 rubygem-apipie-dsl                                        noarch                                2.6.2-1.el8                                                                          foreman                                  107 k
 rubygem-dynflow                                           noarch                                1.8.3-1.el8                                                                          foreman                                  381 k
 rubygem-fast_gettext                                      noarch                                2.3.0-1.el8                                                                          foreman                                   35 k
 rubygem-fog-vsphere                                       noarch                                3.6.4-1.el8                                                                          foreman                                   80 k
 rubygem-hammer_cli                                        noarch                                3.11.0-0.1.pre.develop.el8                                                           foreman                                  135 k
 rubygem-hammer_cli_foreman                                noarch                                3.11.0-0.1.pre.develop.el8                                                           foreman                                  234 k
 rubygem-i18n                                              noarch                                1.14.4-1.el8                                                                         foreman                                   53 k
 rubygem-katello                                           noarch                                4.13.0-0.1.pre.master.20240308204117git8bedda9.el8                                   katello                                   25 M
 rubygem-marcel                                            noarch                                1.0.4-1.el8                                                                          foreman                                   46 k
 rubygem-mime-types-data                                   noarch                                3.2024.0206-1.el8                                                                    foreman                                  148 k
 rubygem-pg                                                x86_64                                1.5.6-1.el8                                                                          foreman                                  117 k
 rubygem-rack                                              noarch                                2.2.8.1-1.el8                                                                        foreman                                  131 k
 rubygem-rubyipmi                                          noarch                                0.11.1-2.el8                                                                         foreman                                   35 k
 rubygem-sequel                                            noarch                                5.78.0-1.el8                                                                         foreman                                  575 k
Installing dependencies:
 ipmitool                                                  x86_64                                1.8.18-19.el8                                                                        appstream                                394 k
2

It was necessary to disable SELinux using the command below:

sudo setenforce 0

3

Setting SELinux to non-enforcing shouldn’t be needed. Did you check which denials you got? We may have missed something in the policy.

4

I don’t have much experience with SELinux, could you tell me how I can obtain this information? Thank you very much in advance.

5

The RHEL docs on identifying denials is useful and should also work on systems that run in permissive mode.

6

We found some entries in the SELinux logs, perhaps they can help with the analysis.

----
time->Wed Mar 20 15:25:54 2024
type=PROCTITLE msg=audit(1710959154.595:2236): proctitle=2F7573722F62696E2F72756279002F7573722F73686172652F666F72656D616E2D70726F78792F62696E2F736D6172742D70726F7879
type=PATH msg=audit(1710959154.595:2236): item=0 name="/usr/share/foreman-proxy/config/settings.yml" inode=134527656 dev=fd:00 mode=0100640 ouid=0 ogid=485600057 rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1710959154.595:2236): cwd="/"
type=SYSCALL msg=audit(1710959154.595:2236): arch=c000003e syscall=257 success=yes exit=7 a0=ffffff9c a1=5616934b95b0 a2=80000 a3=0 items=1 ppid=1 pid=5717 auid=4294967295 uid=485600057 gid=485600057 euid=485600057 suid=485600057 fsuid=485600057 egid=485600057 sgid=485600057 fsgid=485600057 tty=(none) ses=4294967295 comm="smart-proxy" exe="/usr/bin/ruby" subj=system_u:system_r:init_t:s0 key=(null)
type=AVC msg=audit(1710959154.595:2236): avc:  denied  { read } for  pid=5717 comm="smart-proxy" name="settings.yml" dev="dm-0" ino=68630131 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=lnk_file permissive=1
----
time->Wed Mar 20 15:25:54 2024
type=PROCTITLE msg=audit(1710959154.786:2237): proctitle=2F7573722F62696E2F72756279002F7573722F73686172652F666F72656D616E2D70726F78792F62696E2F736D6172742D70726F7879
type=SYSCALL msg=audit(1710959154.786:2237): arch=c000003e syscall=10 success=yes exit=0 a0=7f13bee7a000 a1=1000 a2=5 a3=22 items=0 ppid=1 pid=5717 auid=4294967295 uid=485600057 gid=485600057 euid=485600057 suid=485600057 fsuid=485600057 egid=485600057 sgid=485600057 fsgid=485600057 tty=(none) ses=4294967295 comm="smart-proxy" exe="/usr/bin/ruby" subj=system_u:system_r:init_t:s0 key=(null)
type=AVC msg=audit(1710959154.786:2237): avc:  denied  { execmem } for  pid=5717 comm="smart-proxy" scontext=system_u:system_r:init_t:s0 tcontext=system_u:system_r:init_t:s0 tclass=process permissive=1
----
time->Wed Mar 20 15:25:56 2024
type=PROCTITLE msg=audit(1710959156.741:2244): proctitle=2F7573722F62696E2F72756279002F7573722F62696E2F736964656B6971002D650070726F64756374696F6E002D72002F7573722F73686172652F666F72656D616E2F6578747261732F64796E666C6F772D736964656B69712E7262002D43002F6574632F666F72656D616E2F64796E666C6F772F776F726B65722D312E796D
type=SYSCALL msg=audit(1710959156.741:2244): arch=c000003e syscall=257 success=no exit=-2 a0=ffffff9c a1=5655224a7d80 a2=0 a3=0 items=0 ppid=1 pid=5709 auid=4294967295 uid=992 gid=485600058 euid=992 suid=992 fsuid=992 egid=485600058 sgid=485600058 fsgid=485600058 tty=(none) ses=4294967295 comm="sidekiq" exe="/usr/bin/ruby" subj=system_u:system_r:foreman_rails_t:s0 key=(null)
type=AVC msg=audit(1710959156.741:2244): avc:  denied  { search } for  pid=5709 comm="sidekiq" name="krb5" dev="dm-3" ino=1073741965 scontext=system_u:system_r:foreman_rails_t:s0 tcontext=system_u:object_r:krb5_keytab_t:s0 tclass=dir permissive=1
----
time->Wed Mar 20 15:26:12 2024
type=PROCTITLE msg=audit(1710959172.212:2265): proctitle=2F7573722F62696E2F72756279002F7573722F62696E2F736964656B6971002D650070726F64756374696F6E002D72002F7573722F73686172652F666F72656D616E2F6578747261732F64796E666C6F772D736964656B69712E7262002D43002F6574632F666F72656D616E2F64796E666C6F772F776F726B65722D312E796D
type=SYSCALL msg=audit(1710959172.212:2265): arch=c000003e syscall=42 success=yes exit=0 a0=a a1=7ffd1f9b45a0 a2=6e a3=0 items=0 ppid=1 pid=5767 auid=4294967295 uid=992 gid=485600058 euid=992 suid=992 fsuid=992 egid=485600058 sgid=485600058 fsgid=485600058 tty=(none) ses=4294967295 comm="sidekiq" exe="/usr/bin/ruby" subj=system_u:system_r:foreman_rails_t:s0 key=(null)
type=AVC msg=audit(1710959172.212:2265): avc:  denied  { write } for  pid=5767 comm="sidekiq" name=".heim_org.h5l.kcm-socket" dev="tmpfs" ino=24644 scontext=system_u:system_r:foreman_rails_t:s0 tcontext=system_u:object_r:sssd_var_run_t:s0 tclass=sock_file permissive=1
7

It looks like you have a policy that enforces foreman-proxy to be running confined, but we don’t support this. Do you happen to have foreman-proxy-selinux installed? We used to ship it, but didn’t properly obsolete it.

8

I ran the commands. Could these be?

foreman-selinux-enable
foreman-selinux-relabel
9

Please check if you have foreman-proxy-selinux installed: rpm -qv foreman-proxy-selinux.

10

I believe it has not been installed. This is a new server where we are preparing the Foreman application to manage and provision our instances.

$ sudo rpm -qv foreman-proxy-selinux
package foreman-proxy-selinux is not installed