Katello night failing with SELinux

Suddenly I am seeing the following on Katello nightly:

[root@pipeline-katello-server-nightly-centos7 vagrant]# rpm -q foreman
foreman-2.2.0-0.27.develop.20200701110638git180c7fa.el7.noarch
[root@pipeline-katello-server-nightly-centos7 vagrant]# grep -r den /var/log/audit/
/var/log/audit/audit.log:type=AVC msg=audit(1593656210.748:3041): avc:  denied  { name_bind } for  pid=25652 comm="httpd" src=5000 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:commplex_main_port_t:s0 tclass=tcp_socket
/var/log/audit/audit.log:type=AVC msg=audit(1593656210.748:3042): avc:  denied  { name_bind } for  pid=25652 comm="httpd" src=5000 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:commplex_main_port_t:s0 tclass=tcp_socket
/var/log/audit/audit.log:type=AVC msg=audit(1593656314.782:3045): avc:  denied  { write } for  pid=16852 comm="sidekiq" name="home" dev="vda1" ino=1838178 scontext=system_u:system_r:foreman_rails_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir
/var/log/audit/audit.log:type=AVC msg=audit(1593656314.782:3046): avc:  denied  { write } for  pid=16852 comm="sidekiq" name="home" dev="vda1" ino=1838178 scontext=system_u:system_r:foreman_rails_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir
/var/log/audit/audit.log:type=AVC msg=audit(1593656483.378:3103): avc:  denied  { name_bind } for  pid=17807 comm="httpd" src=5000 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:commplex_main_port_t:s0 tclass=tcp_socket
/var/log/audit/audit.log:type=AVC msg=audit(1593656483.378:3104): avc:  denied  { name_bind } for  pid=17807 comm="httpd" src=5000 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:commplex_main_port_t:s0 tclass=tcp_socket
Jul 02 02:21:23 pipeline-katello-server-nightly-centos7.war.example.com systemd[1]: Starting The Apache HTTP Server...
-- Subject: Unit httpd.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit httpd.service has begun starting up.
Jul 02 02:21:23 pipeline-katello-server-nightly-centos7.war.example.com httpd[17807]: (13)Permission denied: AH00072: make_sock: could not bind to address [::]:5000
Jul 02 02:21:23 pipeline-katello-server-nightly-centos7.war.example.com httpd[17807]: (13)Permission denied: AH00072: make_sock: could not bind to address 0.0.0.0:5000
Jul 02 02:21:23 pipeline-katello-server-nightly-centos7.war.example.com httpd[17807]: no listening sockets available, shutting down
Jul 02 02:21:23 pipeline-katello-server-nightly-centos7.war.example.com httpd[17807]: AH00015: Unable to open logs
Jul 02 02:21:23 pipeline-katello-server-nightly-centos7.war.example.com systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Jul 02 02:21:23 pipeline-katello-server-nightly-centos7.war.example.com systemd[1]: Failed to start The Apache HTTP Server.

Please check if the port 5000 is still assigned to commplex type:

# semanage port -l | grep 5000
commplex_main_port_t           tcp      5000
commplex_main_port_t           udp      5000

And if our policy still has the rule:

# sesearch --allow -s httpd_t -t commplex_main_port_t
Found 8 semantic av rules:
   allow httpd_t port_type : tcp_socket { recv_msg send_msg } ; 
   allow httpd_t port_type : udp_socket { recv_msg send_msg } ; 
   allow httpd_t commplex_main_port_t : tcp_socket name_bind ; 
   allow nsswitch_domain port_type : tcp_socket { recv_msg send_msg } ; 
   allow httpd_t port_type : tcp_socket name_connect ; 
   allow nsswitch_domain port_type : udp_socket recv_msg ; 
   allow nsswitch_domain port_type : udp_socket send_msg ; 
   allow httpd_t commplex_main_port_t : tcp_socket name_bind ; 

Here are the results:

[root@pipeline-katello-server-nightly-centos7 vagrant]# semanage port -l | grep 5000
cluster_port_t                 tcp      5149, 40040, 50006-50008
cluster_port_t                 udp      5149, 50006-50008
commplex_main_port_t           tcp      5000
commplex_main_port_t           udp      5000
hplip_port_t                   tcp      1782, 2207, 2208, 8290, 8292, 9100, 9101, 9102, 9220, 9221, 9222, 9280, 9281, 9282, 9290, 9291, 50000, 50002
[root@pipeline-katello-server-nightly-centos7 vagrant]# sesearch --allow -s httpd_t -t commplex_main_port_t
Found 7 semantic av rules:
   allow httpd_t port_type : tcp_socket { recv_msg send_msg } ; 
   allow httpd_t port_type : udp_socket { recv_msg send_msg } ; 
   allow nsswitch_domain port_type : tcp_socket { recv_msg send_msg } ; 
   allow httpd_t port_type : tcp_socket name_connect ; 
   allow nsswitch_domain port_type : udp_socket recv_msg ; 
   allow nsswitch_domain port_type : udp_socket send_msg ; 
   allow httpd_t commplex_main_port_t : tcp_socket name_bind ; 

Those rules are there, denials should not be there.

Unless http comes up before the policy is loaded, can you check that?

From system log:

Jul  2 13:00:17 pipeline-katello-server-nightly-centos7 yum[17077]: Installed: katello-selinux-3.3.0-1.el7.noarch

Jul  2 13:07:46 pipeline-katello-server-nightly-centos7 systemd: Starting The Apache HTTP Server...

So looks like katello-selinux is installed 7 minutes before Apache is first started.

Further logs:

Thu Jul  2 12:57:43 UTC 2020 /usr/sbin/foreman-selinux-enable
port -a -t foreman_container_port_t -p tcp 2375
port -a -t foreman_container_port_t -p tcp 2376
port -a -t websm_port_t -p tcp 19090
boolean -m --on httpd_setrlimit
Thu Jul  2 13:00:17 UTC 2020 /usr/sbin/katello-selinux-enable

This is the most selinux related change that is recent I can find: https://github.com/Katello/katello-selinux/commit/85e372a3e34319331a5f9ec0fa3e46d10a3853d0

I can try backing out that release and see if the error goes away?

I am booting up nightly to see myself.

I am unable to reproduce this, am I doing this right?

[lzap@box forklift]$ vagrant destroy pipeline-up-foreman-nightly-centos7

[lzap@box forklift]$ vagrant up pipeline-up-foreman-nightly-centos7

[lzap@box forklift]$ vagrant ssh pipeline-up-foreman-nightly-centos7
[vagrant@pipeline-up-foreman-nightly-centos7 ~]$ sudo ausearch -m AVC
<no matches>

You are spinning up just foreman, sub in katello for foreman.

In nightly, I backed out the most recent foreman-selinux build so as to test reverting (Fixes #30279 - allow http(s) proxy ports by default · theforeman/foreman-selinux@11ceb13 · GitHub) to see if that was cuasing the Apache binding to 5000 issue. This revert of the build did allow nightly Katello to pass.

The change (Fixes #30279 - allow http(s) proxy ports by default · theforeman/foreman-selinux@11ceb13 · GitHub) is present in foreman-selinux 2.1 GA. However, Katello 3.16 is not suffering from this issue. The difference with 2.1/3.16 is that it has katello-selinux-3.12 which is missing this commit:

Is there perhaps a conflict in the policies that is causing Katello’s not to get loaded or some other type of issue?

So I started this:

ansible-playbook pipelines/install_pipeline.yml -e pipeline_type=katello -e pipeline_os=centos7 -e pipeline_version=nightly

No denials in:

  • pipeline-katello-server-nightly-centos7
  • pipeline-katello-proxy-nightly-centos7
  • pipeline-katello-smoker-nightly-centos7

boxes.

However I have noticed this:

[root@pipeline-katello-server-nightly-centos7 vagrant]# rpm -q katello-selinux
package katello-selinux is not installed
[root@pipeline-katello-server-nightly-centos7 vagrant]# rpm -q foreman-selinux
foreman-selinux-2.2.0-0.2.develop.20200618110456git7b1c804.el7.noarch

There were some errors reported by ansible, I think the scenario did not even finish:

TASK [foreman_proxy_content : Detect certs generate] ******************************************************************************************************************************************
fatal: [pipeline-katello-proxy-nightly-centos7 -> 192.168.122.252]: FAILED! => changed=true 
  cmd:
  - which
  - foreman-proxy-certs-generate
  delta: '0:00:00.003123'
  end: '2020-07-14 08:31:28.148143'
  msg: non-zero return code
  rc: 1
  start: '2020-07-14 08:31:28.145020'
  stderr: 'which: no foreman-proxy-certs-generate in (/sbin:/bin:/usr/sbin:/usr/bin)'
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
...ignoring
 [started TASK: foreman_proxy_content : Generate Certs on pipeline-katello-proxy-nightly-centos7]
 [started TASK: foreman_proxy_content : Generate Certs on pipeline-katello-proxy-nightly-centos7]

TASK [foreman_proxy_content : Generate Certs] *************************************************************************************************************************************************
fatal: [pipeline-katello-proxy-nightly-centos7 -> 192.168.122.252]: FAILED! => changed=false 
  cmd: capsule-certs-generate --capsule-fqdn pipeline-katello-proxy-nightly-centos7.box.example.com --certs-tar /root/pipeline-katello-proxy-nightly-centos7.box.example.com.tar.gz
  msg: '[Errno 2] Adresář nebo soubor neexistuje'
  rc: 2

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=5    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
pipeline-katello-proxy-nightly-centos7 : ok=47   changed=21   unreachable=0    failed=1    skipped=31   rescued=0    ignored=2   
pipeline-katello-server-nightly-centos7 : ok=47   changed=16   unreachable=0    failed=0    skipped=35   rescued=0    ignored=1   

That’s weird because foreman-installer-katello should depend on it. Is foreman-installer-katello installed?

Thanks @lzap! https://github.com/Katello/katello-selinux/commit/220aad50c8565a49b25be9a3d4152db6e8b820af fixed the nightly breakage and has been released in a 3.3.1 of katello-selinux.

do we need to backport anything to stable branches for 2.1/3.16?

I’ve reviewed both repos:

We should release katello-selinux 3.3.1 which lives in the Foreman repository (due to the installer) for the Katello 3.16 release.

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.