foreman-nightly-rpm-pipeline 601 failed

Foreman RPM nightly pipeline failed:

foreman-nightly-centos7-test (failed)
foreman-nightly-centos8-test (failed)
foreman-nightly-centos7-upgrade-test (failed)

install test still failing on httpd config:

[ INFO 2020-06-07T22:09:25 verbose]  Class[Apache::Service]: Scheduling refresh of Service[httpd]
[ERROR 2020-06-07T22:09:25 verbose]  Systemd start for httpd failed!
[ERROR 2020-06-07T22:09:25 verbose] journalctl log for httpd:
[ERROR 2020-06-07T22:09:25 verbose] -- Logs begin at Sun 2020-06-07 21:58:00 UTC, end at Sun 2020-06-07 22:09:25 UTC. --
[ERROR 2020-06-07T22:09:25 verbose] Jun 07 22:09:25 systemd[1]: Starting The Apache HTTP Server...
[ERROR 2020-06-07T22:09:25 verbose] Jun 07 22:09:25 httpd[2545]: AH00526: Syntax error on line 58 of /etc/httpd/conf.d/05-foreman-ssl.conf:
[ERROR 2020-06-07T22:09:25 verbose] Jun 07 22:09:25 httpd[2545]: SSLCertificateFile: file '/etc/puppetlabs/puppet/ssl/certs/' does not exist or is empty
[ERROR 2020-06-07T22:09:25 verbose] Jun 07 22:09:25 systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
[ERROR 2020-06-07T22:09:25 verbose] Jun 07 22:09:25 systemd[1]: Failed to start The Apache HTTP Server.
[ERROR 2020-06-07T22:09:25 verbose] Jun 07 22:09:25 systemd[1]: Unit httpd.service entered failed state.
[ERROR 2020-06-07T22:09:25 verbose] Jun 07 22:09:25 systemd[1]: httpd.service failed.

upgrade test fails on proxy refresh:

[ INFO 2020-06-07T22:25:58 verbose]  Class[Foreman_proxy::Service]: Scheduling refresh of Class[Foreman_proxy::Register]
[ INFO 2020-06-07T22:25:58 verbose]  Class[Foreman_proxy::Register]: Scheduling refresh of Foreman_smartproxy[]
[ERROR 2020-06-07T22:26:58 verbose]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[]: Could not evaluate: Proxy cannot be retrieved: unknown error (response 502)
[ERROR 2020-06-07T22:26:58 verbose] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:7:in `proxy'
[ERROR 2020-06-07T22:26:58 verbose] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2020-06-07T22:26:58 verbose] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'

The only change I found so far that touches this area is which was merged about a week before the tests started failing, so i doubt it is related. The changes for the first failing run don’t seem to include anything suspicious:

  1. Update rubygem-foreman_maintain to 0.6.5 (details / githubweb)
  2. Add Requires on libmodulemd (details / githubweb)
  3. Adding pulp_certguard_client (details / githubweb)
  4. regeneate hammer_cli_katello (details / githubweb)
  5. regenerate hammer_cli (details / githubweb)
  6. regenerate hammer_cli_foreman (details / githubweb)
  7. bump hammer-cli-katello to 0.23 (details / githubweb)

I am seeing an SELinux denial:

type=AVC msg=audit(1591629222.402:2243): avc:  denied  { getattr } for  pid=6981 comm="httpd" path="/etc/puppetlabs/puppet/ssl/certs/" dev="vda1" ino=2491205 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=file

The timing of failures looks to align to this change ( and beware it’s been CP’d into 2.1 as well.

@lzap could you prioritize taking a look? If this is a SELinux policy issue or an installer issue with these changes.

1 Like

I’ve untagged the nightly foreman-selinux build to unblock pipelines while this gets investigated:

$ koji untag-pkg foreman-nightly-nonscl-rhel7 foreman-selinux-2.2.0-0.2.develop.20200603081628git3087a3f.el7
$ koji untag-pkg foreman-nightly-el8 foreman-selinux-2.2.0-0.2.develop.20200603081628git3087a3f.el8

This might be a related error on the Katello side:

    [ WARN 2020-06-08T17:16:13 verbose]  /File[/etc/pulp/server.conf]/seltype: seltype changed 'httpd_sys_content_t' to 'etc_t'
    [ERROR 2020-06-08T17:16:13 verbose]  Could not set 'link' on ensure: No such file or directory @ dir_chdir - /etc/pki/pulp/content (file: /usr/share/foreman-installer/modules/pulp/manifests/config.pp, line: 20)

Note: untagging selinux package updates did fix things:

1 Like

You should have pinged me on IRC, I saw it in the morning. Here is the patch:

Apache httpd is configured to read puppet certificate files which are tagged as puppet_etc_t. Can you explain me why we did not see this during testing? Is this some regression? Oh maybe I know, I did not restart httpd during testing. Let me check that.

I have found the issue, I was only testing this on Katello/Satellite setup where Puppet certificates are not in use. Anyway, I did more testing and I have found some minor issues with logrotation and cronjobs, added few more rules.

The patch is ready for review by @aruzicka, I will file backport PR once it’s merged @tbrisker

1 Like

Merged, CP PR:

Looks like the Pulp issue is not selinux but Pulp 3 RPM related.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.