Foreman RPM nightly pipeline failed:
https://ci.theforeman.org/job/foreman-nightly-rpm-pipeline/601/
foreman-nightly-centos7-test (failed)
foreman-nightly-centos8-test (failed)
foreman-nightly-centos7-upgrade-test (failed)
install test still failing on httpd config:
[ INFO 2020-06-07T22:09:25 verbose] Class[Apache::Service]: Scheduling refresh of Service[httpd]
[ERROR 2020-06-07T22:09:25 verbose] Systemd start for httpd failed!
[ERROR 2020-06-07T22:09:25 verbose] journalctl log for httpd:
[ERROR 2020-06-07T22:09:25 verbose] -- Logs begin at Sun 2020-06-07 21:58:00 UTC, end at Sun 2020-06-07 22:09:25 UTC. --
[ERROR 2020-06-07T22:09:25 verbose] Jun 07 22:09:25 pipeline-foreman-server-nightly-centos7.n18.example.com systemd[1]: Starting The Apache HTTP Server...
[ERROR 2020-06-07T22:09:25 verbose] Jun 07 22:09:25 pipeline-foreman-server-nightly-centos7.n18.example.com httpd[2545]: AH00526: Syntax error on line 58 of /etc/httpd/conf.d/05-foreman-ssl.conf:
[ERROR 2020-06-07T22:09:25 verbose] Jun 07 22:09:25 pipeline-foreman-server-nightly-centos7.n18.example.com httpd[2545]: SSLCertificateFile: file '/etc/puppetlabs/puppet/ssl/certs/pipeline-foreman-server-nightly-centos7.n18.example.com.pem' does not exist or is empty
[ERROR 2020-06-07T22:09:25 verbose] Jun 07 22:09:25 pipeline-foreman-server-nightly-centos7.n18.example.com systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
[ERROR 2020-06-07T22:09:25 verbose] Jun 07 22:09:25 pipeline-foreman-server-nightly-centos7.n18.example.com systemd[1]: Failed to start The Apache HTTP Server.
[ERROR 2020-06-07T22:09:25 verbose] Jun 07 22:09:25 pipeline-foreman-server-nightly-centos7.n18.example.com systemd[1]: Unit httpd.service entered failed state.
[ERROR 2020-06-07T22:09:25 verbose] Jun 07 22:09:25 pipeline-foreman-server-nightly-centos7.n18.example.com systemd[1]: httpd.service failed.
upgrade test fails on proxy refresh:
[ INFO 2020-06-07T22:25:58 verbose] Class[Foreman_proxy::Service]: Scheduling refresh of Class[Foreman_proxy::Register]
[ INFO 2020-06-07T22:25:58 verbose] Class[Foreman_proxy::Register]: Scheduling refresh of Foreman_smartproxy[pipeline-up-foreman-nightly-centos7.n17.example.com]
[ERROR 2020-06-07T22:26:58 verbose] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[pipeline-up-foreman-nightly-centos7.n17.example.com]: Could not evaluate: Proxy pipeline-up-foreman-nightly-centos7.n17.example.com cannot be retrieved: unknown error (response 502)
[ERROR 2020-06-07T22:26:58 verbose] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:7:in `proxy'
[ERROR 2020-06-07T22:26:58 verbose] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2020-06-07T22:26:58 verbose] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'The only change I found so far that touches this area is https://github.com/theforeman/puppet-foreman/commit/2dc5e7d802e2a11e35bd1e34d28cce05195040ce which was merged about a week before the tests started failing, so i doubt it is related. The changes for the first failing run don’t seem to include anything suspicious:
Changes
- Update rubygem-foreman_maintain to 0.6.5 (details / githubweb)
- Add Requires on libmodulemd (details / githubweb)
- Adding pulp_certguard_client (details / githubweb)
- regeneate hammer_cli_katello (details / githubweb)
- regenerate hammer_cli (details / githubweb)
- regenerate hammer_cli_foreman (details / githubweb)
- bump hammer-cli-katello to 0.23 (details / githubweb)
I am seeing an SELinux denial:
type=AVC msg=audit(1591629222.402:2243): avc: denied { getattr } for pid=6981 comm="httpd" path="/etc/puppetlabs/puppet/ssl/certs/pipeline-foreman-server-nightly-centos7.war.example.com.pem" dev="vda1" ino=2491205 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=file
The timing of failures looks to align to this change (https://github.com/theforeman/foreman-selinux/pull/100) and beware it’s been CP’d into 2.1 as well.
@lzap could you prioritize taking a look? If this is a SELinux policy issue or an installer issue with these changes.
1 Like
I’ve untagged the nightly foreman-selinux build to unblock pipelines while this gets investigated:
$ koji untag-pkg foreman-nightly-nonscl-rhel7 foreman-selinux-2.2.0-0.2.develop.20200603081628git3087a3f.el7
$ koji untag-pkg foreman-nightly-el8 foreman-selinux-2.2.0-0.2.develop.20200603081628git3087a3f.el8
This might be a related error on the Katello side:
[ WARN 2020-06-08T17:16:13 verbose] /File[/etc/pulp/server.conf]/seltype: seltype changed 'httpd_sys_content_t' to 'etc_t'
[ERROR 2020-06-08T17:16:13 verbose] Could not set 'link' on ensure: No such file or directory @ dir_chdir - /etc/pki/pulp/content (file: /usr/share/foreman-installer/modules/pulp/manifests/config.pp, line: 20)
Note: untagging selinux package updates did fix things:
https://ci.theforeman.org/job/foreman-nightly-rpm-pipeline/603/console
1 Like
You should have pinged me on IRC, I saw it in the morning. Here is the patch:
Apache httpd is configured to read puppet certificate files which are tagged as puppet_etc_t. Can you explain me why we did not see this during testing? Is this some regression? Oh maybe I know, I did not restart httpd during testing. Let me check that.
I have found the issue, I was only testing this on Katello/Satellite setup where Puppet certificates are not in use. Anyway, I did more testing and I have found some minor issues with logrotation and cronjobs, added few more rules.
1 Like
Looks like the Pulp issue is not selinux but Pulp 3 RPM related.
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.