Problem:
Sometime after updating certificates I executed a remote job (via ssh) from the foreman server. Although the jobs executed successfully on the hosts, the jobs never returned a status to the foreman server and eventually timed out.
Looking at the logs, I came across this message:
/var/log/foreman-proxy/proxy.log:
2023-07-21T15:40:21 8abd0f6f [I] Finished GET /dynflow/tasks/d313f2d3-b519-4ac7-91dd-f00a70cd45d7/status with 200 (10.21 ms)
2023-07-21T15:40:21 9224a45b [E] <RuntimeError> Failed performing callback to Foreman server: 403 {
"error": {"message":"Access denied","details":"Missing one of the required permissions: ","missing_permissions":[]}
I have search google and tried many things, all of which have failed to resolve the issue. I believe the most significant thing I found was around the client ca cert:
related to: SSL Errors with proxy and installer - #12 by madmahdi
[root@foreman ~]# openssl verify -CAfile /etc/foreman/proxy_ca.pem /etc/foreman/client_cert.pem
C = US, ST = North Carolina, O = FOREMAN, OU = PUPPET, CN = foreman.ivenix.net
error 20 at 0 depth lookup: unable to get local issuer certificate
error /etc/foreman/client_cert.pem: verification failed
[root@foreman ~]# openssl verify -CAfile /etc/pki/katello/certs/katello-default-ca.crt /etc/foreman/client_cert.pem
/etc/foreman/client_cert.pem: OK
I thought updating the client ca cert to katello-default-ca.crt:
foreman-installer --scenario katello --foreman-client-ssl-ca /etc/pki/katello/certs/katello-default-ca.crt
would resolve the issue, but only resulted in:
RuntimeError: The only applicable proxy foreman.ivenix.net is down.
Been trying to get this working for the past week, but I have hit a wall on trying things. I do have a snaphot to get me back to the original issue ("Missing one of the required permissions).
Expected outcome:
Remote job returns success status, foreman displays success status on web page.
Foreman and Proxy versions:
3.6.1
Foreman and Proxy plugin versions:
nstalled Packages
- candlepin-4.2.13-1.el8.noarch
- candlepin-selinux-4.2.13-1.el8.noarch
- foreman-3.6.1-1.el8.noarch
- foreman-cli-3.6.1-1.el8.noarch
- foreman-debug-3.6.1-1.el8.noarch
- foreman-dynflow-sidekiq-3.6.1-1.el8.noarch
- foreman-installer-3.6.1-1.el8.noarch
- foreman-installer-katello-3.6.1-1.el8.noarch
- foreman-postgresql-3.6.1-1.el8.noarch
- foreman-proxy-3.6.1-1.el8.noarch
- foreman-release-3.6.1-1.el8.noarch
- foreman-selinux-3.6.1-1.el8.noarch
- foreman-service-3.6.1-1.el8.noarch
- katello-4.8.3-1.el8.noarch
- katello-ca-consumer-foreman.ivenix.net-1.0-2.noarch
- katello-certs-tools-2.9.0-1.el8.noarch
- katello-client-bootstrap-1.7.9-1.el8.noarch
- katello-common-4.8.3-1.el8.noarch
- katello-debug-4.8.3-1.el8.noarch
- katello-repos-4.8.3-1.el8.noarch
- katello-selinux-4.0.2-3.el8.noarch
- pulpcore-selinux-1.3.2-1.el8.x86_64
- python39-pulp-ansible-0.16.0-1.el8.noarch
- python39-pulp-certguard-1.5.6-1.el8.noarch
- python39-pulp-cli-0.14.0-4.el8.noarch
- python39-pulp-container-2.14.6-1.el8.noarch
- python39-pulp-deb-2.20.2-1.el8.noarch
- python39-pulp-file-1.12.0-1.el8.noarch
- python39-pulp-python-3.8.0-1.el8.noarch
- python39-pulp-rpm-3.19.8-1.el8.noarch
- python39-pulpcore-3.22.7-1.el8.noarch
- qpid-proton-c-0.37.0-1.el8.x86_64
- rubygem-foreman-tasks-7.2.1-2.fm3_6.el8.noarch
- rubygem-foreman_maintain-1.3.0-1.el8.noarch
- rubygem-foreman_remote_execution-9.1.0-1.fm3_6.el8.noarch
- rubygem-hammer_cli-3.6.0-1.el8.noarch
- rubygem-hammer_cli_foreman-3.6.0-1.el8.noarch
- rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
- rubygem-hammer_cli_foreman_tasks-0.0.18-1.fm3_5.el8.noarch
- rubygem-hammer_cli_katello-1.8.3-1.el8.noarch
- rubygem-katello-4.8.3-1.el8.noarch
- rubygem-pulp_ansible_client-0.16.0-1.el8.noarch
- rubygem-pulp_certguard_client-1.5.7-1.el8.noarch
- rubygem-pulp_container_client-2.14.3-1.el8.noarch
- rubygem-pulp_deb_client-2.20.2-1.el8.noarch
- rubygem-pulp_file_client-1.12.0-1.el8.noarch
- rubygem-pulp_ostree_client-2.0.0-1.el8.noarch
- rubygem-pulp_python_client-3.8.0-1.el8.noarch
- rubygem-pulp_rpm_client-3.19.0-1.el8.noarch
- rubygem-pulpcore_client-3.22.2-1.el8.noarch
- rubygem-qpid_proton-0.37.0-1.el8.x86_64
- rubygem-smart_proxy_pulp-3.2.0-3.fm3_3.el8.noarch
Distribution and version:
Other relevant data: