Pulp 3 SELinux issues on CentOS 8 Content Proxy

Now that we have Pulp 3 deploying on content proxies in nightly, I started testing an EL8 content proxy and I get 503s from Pulp on the proxy. Investigating, these appear to be SELinux related. I do not see the same issues on a Katello nightly server on EL8 with Pulp 3 installed.

I am hoping someone can spot the difference in the installer or find holes that we missed in SELinux policy.

@ekohl @evgeni @lzap

Here is what audit2allow shows:

[root@pipe-katello-proxy-nightly-centos8 vagrant]# audit2allow -e -a


#============= httpd_t ==============
# audit(1611877594.770:3287):
#  scontext="system_u:system_r:httpd_t:s0" tcontext="system_u:object_r:pulpcore_var_run_t:s0"
#  class="sock_file" perms="write"
#  comm="httpd" exe="" path=""
#  message="type=AVC msg=audit(1611877594.770:3287): avc:  denied  { write } for
#   pid=48382 comm="httpd" name="pulpcore-api.sock" dev="tmpfs" ino=121904
#   scontext=system_u:system_r:httpd_t:s0
#   tcontext=system_u:object_r:pulpcore_var_run_t:s0 tclass=sock_file
#   permissive=0"
# audit(1611877594.844:3288):
#  scontext="system_u:system_r:httpd_t:s0" tcontext="system_u:object_r:pulpcore_var_run_t:s0"
#  class="sock_file" perms="write"
#  comm="httpd" exe="" path=""
#  message="type=AVC msg=audit(1611877594.844:3288): avc:  denied  { write } for
#   pid=48379 comm="httpd" name="pulpcore-api.sock" dev="tmpfs" ino=121904
#   scontext=system_u:system_r:httpd_t:s0
#   tcontext=system_u:object_r:pulpcore_var_run_t:s0 tclass=sock_file
#   permissive=0"
# audit(1611877594.903:3289):
#  scontext="system_u:system_r:httpd_t:s0" tcontext="system_u:object_r:pulpcore_var_run_t:s0"
#  class="sock_file" perms="write"
#  comm="httpd" exe="" path=""
#  message="type=AVC msg=audit(1611877594.903:3289): avc:  denied  { write } for
#   pid=48383 comm="httpd" name="pulpcore-api.sock" dev="tmpfs" ino=121904
#   scontext=system_u:system_r:httpd_t:s0
#   tcontext=system_u:object_r:pulpcore_var_run_t:s0 tclass=sock_file
#   permissive=0"
allow httpd_t pulpcore_var_run_t:sock_file write;

#============= pulpcore_server_t ==============
# audit(1611877581.261:3274):
#  scontext="system_u:system_r:pulpcore_server_t:s0" tcontext="system_u:object_r:httpd_config_t:s0"
#  class="dir" perms="search"
#  comm="gunicorn" exe="" path=""
#  message="type=AVC msg=audit(1611877581.261:3274): avc:  denied  { search } for
#   pid=48022 comm="gunicorn" name="httpd" dev="vda1" ino=109150726
#   scontext=system_u:system_r:pulpcore_server_t:s0
#   tcontext=system_u:object_r:httpd_config_t:s0 tclass=dir permissive=1"
allow pulpcore_server_t httpd_config_t:dir search;
# audit(1611877579.610:3272):
#  scontext="system_u:system_r:pulpcore_server_t:s0" tcontext="system_u:system_r:kernel_t:s0"
#  class="unix_dgram_socket" perms="sendto"
#  comm="gunicorn" exe="" path=""
#  message="type=AVC msg=audit(1611877579.610:3272): avc:  denied  { sendto } for
#   pid=48016 comm="gunicorn" path="/run/systemd/notify"
#   scontext=system_u:system_r:pulpcore_server_t:s0
#   tcontext=system_u:system_r:kernel_t:s0 tclass=unix_dgram_socket permissive=1"
allow pulpcore_server_t kernel_t:unix_dgram_socket sendto;
# audit(1611877579.610:3271):
#  scontext="system_u:system_r:pulpcore_server_t:s0" tcontext="system_u:system_r:pulpcore_server_t:s0"
#  class="unix_dgram_socket" perms="create"
#  comm="gunicorn" exe="" path=""
#  message="type=AVC msg=audit(1611877579.610:3271): avc:  denied  { create } for
#   pid=48016 comm="gunicorn" scontext=system_u:system_r:pulpcore_server_t:s0
#   tcontext=system_u:system_r:pulpcore_server_t:s0 tclass=unix_dgram_socket
#   permissive=1"
# audit(1611877579.610:3272):
#  scontext="system_u:system_r:pulpcore_server_t:s0" tcontext="system_u:system_r:pulpcore_server_t:s0"
#  class="unix_dgram_socket" perms="connect"
#  comm="gunicorn" exe="" path=""
#  message="type=AVC msg=audit(1611877579.610:3272): avc:  denied  { connect }
#   for  pid=48016 comm="gunicorn"
#   scontext=system_u:system_r:pulpcore_server_t:s0
#   tcontext=system_u:system_r:pulpcore_server_t:s0 tclass=unix_dgram_socket
#   permissive=1"
allow pulpcore_server_t self:unix_dgram_socket { connect create };

#============= pulpcore_t ==============
# audit(1611877578.334:3270):
#  scontext="system_u:system_r:pulpcore_t:s0" tcontext="system_u:object_r:httpd_config_t:s0"
#  class="dir" perms="search"
#  comm="rq" exe="" path=""
#  message="type=AVC msg=audit(1611877578.334:3270): avc:  denied  { search } for
#   pid=47771 comm="rq" name="httpd" dev="vda1" ino=109150726
#   scontext=system_u:system_r:pulpcore_t:s0
#   tcontext=system_u:object_r:httpd_config_t:s0 tclass=dir permissive=1"
# audit(1611877586.193:3277):
#  scontext="system_u:system_r:pulpcore_t:s0" tcontext="system_u:object_r:httpd_config_t:s0"
#  class="dir" perms="search"
#  comm="rq" exe="" path=""
#  message="type=AVC msg=audit(1611877586.193:3277): avc:  denied  { search } for
#   pid=48269 comm="rq" name="httpd" dev="vda1" ino=109150726
#   scontext=system_u:system_r:pulpcore_t:s0
#   tcontext=system_u:object_r:httpd_config_t:s0 tclass=dir permissive=1"
allow pulpcore_t httpd_config_t:dir search;

Installed selinux packages:

libselinux.x86_64                                                                   2.9-4.el8_3                                      @anaconda            
libselinux-utils.x86_64                                                             2.9-4.el8_3                                      @anaconda            
pulpcore-selinux.x86_64                                                             1.2.3-2.el8                                      @pulpcore-repository 
python3-libselinux.x86_64                                                           2.9-4.el8_3                                      @anaconda            
rpm-plugin-selinux.x86_64                                                           4.14.3-4.el8                                     @anaconda            
selinux-policy.noarch                                                               3.14.3-54.el8                                    @anaconda            
selinux-policy-targeted.noarch                                                      3.14.3-54.el8                                    @anaconda  

I can confirm it was installed in the hooks:

/var/log/foreman-installer/foreman-proxy-content.log:2021-01-29 00:24:12 [INFO  ] [pre] Ensuring pulpcore-selinux to package state installed

Can be spun up with having Enable content proxy testing on EL8 Katello pipelines by ehelms · Pull Request #1285 · theforeman/forklift · GitHub

ansible-playbook pipelines/install_pipeline.yml -e pipeline_version=nightly -e pipeline_os=centos8 -e pipeline_type=katello

This looks odd. I’d expect them to be labeled pulpcore_server_var_run_t. Can you check the output of ls -Z /run/pulpcore*?

I really wonder which directory this is. Is /etc/pulp mislabeled by any chance?

This is probably due to:

Previously we didn’t really use systemd’s notification. I’m surprised we didn’t see this failure elsewhere though.

Pretty sure this is also due to the systemd socket activation feature.

This one also surprises me and I wonder which file this is hitting. Probably the same thing as above with httpd_config_t but with a different service (notice pulpcore_server_t vs pulpcore_t).

       system_u:object_r:pulpcore_var_run_t:s0 /run/pulpcore-api.sock
       system_u:object_r:pulpcore_var_run_t:s0 /run/pulpcore-content.sock

/run/pulpcore-api:

/run/pulpcore-content:

/run/pulpcore-resource-manager:

/run/pulpcore-worker-1:

/run/pulpcore-worker-2:
[root@pipe-katello-proxy-nightly-centos8 vagrant]# ls -Z /var/lib/pulp*
 system_u:object_r:pulpcore_var_lib_t:s0 assets
 system_u:object_r:pulpcore_var_lib_t:s0 media
system_u:object_r:httpd_sys_content_t:s0 pulpcore_static
          system_u:object_r:var_lib_t:s0 sync_imports
 system_u:object_r:pulpcore_var_lib_t:s0 tmp

[root@pipe-katello-proxy-nightly-centos8 vagrant]# ls -Z /var/lib
 system_u:object_r:var_lib_t:s0 pulp

Note to whoever will dig into this: compare sesearch -A -s <SOURCE_CTX> -t <TARGET_CTX> on Katello and Proxy.

My expectation would have been, this is identical on both Katello and Proxy deployments (and broken on both, if broken), but you say it’s not, which is… weird.

You can also add -w /etc/shadow -p w to /etc/audit/rules.d/audit.rules and restart auditd, so you get the paths in the audit.log. A trick I have from Why doen't SELinux give me the full path in an error message? - Dan Walsh's Blog — LiveJournal

This could help with debugging here.

One interesting thing maybe, I do see denialals on EL7 too, and audit2allow yields

#============= pulpcore_server_t ==============
allow pulpcore_server_t httpd_sys_rw_content_t:dir { add_name getattr open read write };
allow pulpcore_server_t httpd_sys_rw_content_t:file { create getattr ioctl open read write };

#!!!! The file '/run/systemd/notify' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /run/systemd/notify
allow pulpcore_server_t kernel_t:unix_dgram_socket sendto;
allow pulpcore_server_t self:unix_dgram_socket { connect create };

#============= pulpcore_t ==============
allow pulpcore_t httpd_sys_rw_content_t:dir { getattr open read };

not sure about the notify part, it seems to be labeled correctly at the time I look at it.

Could this explain why we see File Not Found every now and then? Something being blocked by SELinux?

Funny…

[root@pipe-katello-server-nightly-centos8 ~]# sesearch -A -s httpd_t -t pulpcore_var_run_t -c sock_file
allow domain pidfile:sock_file { append getattr open write };
[root@pipe-katello-proxy-nightly-centos8 ~]# sesearch -A -s httpd_t -t pulpcore_var_run_t -c sock_file
<empty>

the policies are indeed different…

Possible, yes.

I started testing variations between server and proxy. The first one solved the problem. And that is, on the server we have container-selinux installed, on a content proxy we do not. After installing container-selinux, everything worked.

I also noted the following from the audit logs after running fb-katello-proxy.bats:

type=AVC msg=audit(1611927928.467:3637): avc:  denied  { write } for  pid=55613 comm="gunicorn" name="pulp" dev="vda1" ino=117514024 scontext=system_u:system_r:pulpcore_server_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1611927928.467:3637): avc:  denied  { add_name } for  pid=55613 comm="gunicorn" name="tmpej98d4jh" scontext=system_u:system_r:pulpcore_server_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1611927928.467:3637): avc:  denied  { create } for  pid=55613 comm="gunicorn" name="tmpej98d4jh" scontext=system_u:system_r:pulpcore_server_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
type=AVC msg=audit(1611927928.467:3637): avc:  denied  { read write open } for  pid=55613 comm="gunicorn" path="/var/lib/pulp/tmpej98d4jh" dev="vda1" ino=117502657 scontext=system_u:system_r:pulpcore_server_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
type=SYSCALL msg=audit(1611927928.467:3637): arch=c000003e syscall=257 success=yes exit=18 a0=ffffff9c a1=7f1a02993150 a2=a00c2 a3=180 items=4 ppid=55602 pid=55613 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295 comm="gunicorn" exe="/usr/libexec/platform-python3.6" subj=system_u:system_r:pulpcore_server_t:s0 key=(null)ARCH=x86_64 SYSCALL=openat AUID="unset" UID="pulp" GID="pulp" EUID="pulp" SUID="pulp" FSUID="pulp" EGID="pulp" SGID="pulp" FSGID="pulp"
type=PATH msg=audit(1611927928.467:3637): item=0 name=(null) inode=117514024 dev=fd:01 mode=040775 ouid=1001 ogid=1001 rdev=00:00 obj=system_u:object_r:var_lib_t:s0 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="pulp" OGID="pulp"
type=PATH msg=audit(1611927928.467:3637): item=1 name=(null) nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1611927928.467:3637): item=2 name=(null) inode=117514024 dev=fd:01 mode=040775 ouid=1001 ogid=1001 rdev=00:00 obj=system_u:object_r:var_lib_t:s0 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="pulp" OGID="pulp"
type=PATH msg=audit(1611927928.467:3637): item=3 name=(null) inode=117502657 dev=fd:01 mode=0100600 ouid=1001 ogid=1001 rdev=00:00 obj=system_u:object_r:var_lib_t:s0 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="pulp" OGID="pulp"
type=PROCTITLE msg=audit(1611927928.467:3637): proctitle=2F7573722F6C6962657865632F706C6174666F726D2D707974686F6E002F7573722F62696E2F67756E69636F726E0070756C70636F72652E636F6E74656E743A736572766572002D2D776F726B65722D636C6173730061696F687474702E47756E69636F726E576562576F726B6572002D770032002D2D6163636573732D6C6F
type=AVC msg=audit(1611927928.468:3638): avc:  denied  { getattr } for  pid=55613 comm="gunicorn" path="/var/lib/pulp/tmpej98d4jh" dev="vda1" ino=117502657 scontext=system_u:system_r:pulpcore_server_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
type=SYSCALL msg=audit(1611927928.468:3638): arch=c000003e syscall=5 success=yes exit=0 a0=12 a1=7fffd6eb47d0 a2=7fffd6eb47d0 a3=7f1a02989fea items=0 ppid=55602 pid=55613 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295 comm="gunicorn" exe="/usr/libexec/platform-python3.6" subj=system_u:system_r:pulpcore_server_t:s0 key=(null)ARCH=x86_64 SYSCALL=fstat AUID="unset" UID="pulp" GID="pulp" EUID="pulp" SUID="pulp" FSUID="pulp" EGID="pulp" SGID="pulp" FSGID="pulp"
type=PROCTITLE msg=audit(1611927928.468:3638): proctitle=2F7573722F6C6962657865632F706C6174666F726D2D707974686F6E002F7573722F62696E2F67756E69636F726E0070756C70636F72652E636F6E74656E743A736572766572002D2D776F726B65722D636C6173730061696F687474702E47756E69636F726E576562576F726B6572002D770032002D2D6163636573732D6C6F
type=AVC msg=audit(1611927928.468:3639): avc:  denied  { ioctl } for  pid=55613 comm="gunicorn" path="/var/lib/pulp/tmpej98d4jh" dev="vda1" ino=117502657 ioctlcmd=0x5401 scontext=system_u:system_r:pulpcore_server_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
type=SYSCALL msg=audit(1611927928.468:3639): arch=c000003e syscall=16 success=no exit=-25 a0=12 a1=5401 a2=7fffd6eb4790 a3=7f1a037493d8 items=0 ppid=55602 pid=55613 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295 comm="gunicorn" exe="/usr/libexec/platform-python3.6" subj=system_u:system_r:pulpcore_server_t:s0 key=(null)ARCH=x86_64 SYSCALL=ioctl AUID="unset" UID="pulp" GID="pulp" EUID="pulp" SUID="pulp" FSUID="pulp" EGID="pulp" SGID="pulp" FSGID="pulp"
type=PROCTITLE msg=audit(1611927928.468:3639): proctitle=2F7573722F6C6962657865632F706C6174666F726D2D707974686F6E002F7573722F62696E2F67756E69636F726E0070756C70636F72652E636F6E74656E743A736572766572002D2D776F726B65722D636C6173730061696F687474702E47756E69636F726E576562576F726B6572002D770032002D2D6163636573732D6C6F

For that section of audit log:

#============= pulpcore_server_t ==============
allow pulpcore_server_t var_lib_t:dir { add_name write };
allow pulpcore_server_t var_lib_t:file { create getattr ioctl open read write };

This feels like a bug in Pulp itself. It shouldn’t write temporary files in /var/lib/pulp. We do have /var/lib/pulp/tmp and I would have expected it to write there. Otherwise you can clutter up /var/lib/pulp.

I looked at the el7 server and while i do see similar audit messages
such as:

type=AVC msg=audit(1611846737.577:2914): avc: denied { read } for
pid=13077 comm=“gunicorn” name=“pulp” dev=“vda1” ino=101257974
scontext=system_u:system_r:pulpcore_server_t:s0
tcontext=system_u:object_r:httpd_sys_rw_content_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1611846737.577:2914): avc: denied { open } for
pid=13077 comm=“gunicorn” path="/var/lib/pulp" dev=“vda1” ino=101257974
scontext=system_u:system_r:pulpcore_server_t:s0
tcontext=system_u:object_r:httpd_sys_rw_content_t:s0 tclass=dir permissive=1
type=SYSCALL msg=audit(1611846737.577:2914): arch=c000003e syscall=257
success=yes exit=4 a0=ffffffffffffff9c a1=7f3544930710 a2=90800 a3=0
items=0 ppid=1 pid=13077 auid=4294967295 uid=1001 gid=1001 euid=1001
suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none)
ses=4294967295 comm=“gunicorn” exe="/usr/bin/python3.6"
subj=system_u:system_r:pulpcore_server_t:s0 key=(null)
type=AVC msg=audit(1611846737.624:2915): avc: denied { create } for
pid=13077 comm=“gunicorn”
scontext=system_u:system_r:pulpcore_server_t:s0
tcontext=system_u:system_r:pulpcore_server_t:s0 tclass=unix_dgram_socket
permissive=1
type=SYSCALL msg=audit(1611846737.624:2915): arch=c000003e syscall=41
success=yes exit=3 a0=1 a1=80002 a2=0 a3=0 items=0 ppid=1 pid=13077
auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001
egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295 comm=“gunicorn”
exe="/usr/bin/python3.6" subj=system_u:system_r:pulpcore_server_t:s0
key=(null)
type=AVC msg=audit(1611846737.624:2916): avc: denied { connect } for
pid=13077 comm=“gunicorn”
scontext=system_u:system_r:pulpcore_server_t:s0
tcontext=system_u:system_r:pulpcore_server_t:s0 tclass=unix_dgram_socket
permissive=1
type=AVC msg=audit(1611846737.624:2916): avc: denied { sendto } for
pid=13077 comm=“gunicorn” path="/run/systemd/notify"
scontext=system_u:system_r:pulpcore_server_t:s0
tcontext=system_u:system_r:kernel_t:s0 tclass=unix_dgram_socket permissive=1
type=SYSCALL msg=audit(1611846737.624:2916): arch=c000003e syscall=42
success=yes exit=0 a0=3 a1=7ffcd39d0200 a2=15 a3=f0 items=0 ppid=1
pid=13077 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001
fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295
comm=“gunicorn” exe="/usr/bin/python3.6"
subj=system_u:system_r:pulpcore_server_t:s0 key=(null)

when mapping the timestamp to the apache logs, it appears all of these
are happening at startup/install time not at the time when the 502’s
we’ve been seeing are happening.

1611846737 = Thursday, January 28, 2021 15:12:17

192.168.121.46 - - [28/Jan/2021:15:41:36 +0000] “GET
/pulp/repos/Test_Organization/Library/Composite/custom/Test_Product/rpm-deps/repodata/c0cace4d6bd977ad29096c46467682b30b7ef65aff60fe7f7238096f9003d898-updateinfo.xml.gz
HTTP/1.1” 502 713 “-” “pulpcore/3.9.1 (cpython 3.6.8-final0, Linux
x86_64) (aiohttp 3.7.2)”

So what probably happens is that systemd starts the service. Then the service tries to open the notification socket. That’s denied by SELinux. Systemd is never notified and thinks the service failed to start up and kills it after some time. Then Apache can’t talk to the content backend. That gives a HTTP 502. Untested and didn’t look at the logs myself, but based on past experiences this is how I’d expect them to interact.

On EL7 katello-common depends on pulp-selinux which contains the label for /var/lib/pulp. It doesn’t on EL8.

That means /var/lib/pulp is probably labeled var_lib_t while on EL7 it’s labeled httpd_sys_rw_content_t. I only installed EL7 now, haven’t checked an EL8 install.

And before someone suggests shipping a label for /var/lib/pulp in pulpcore-selinux too: then those will clash if both policies are installed…

Not probably:

[root@pipe-katello-proxy-nightly-centos8 vagrant]# ls -Z /var/lib
 system_u:object_r:var_lib_t:s0 pulp

The answer then is, getting Fixes #31692 - Send and receive katello-agent messages by jturel · Pull Request #9088 · Katello/katello · GitHub settled. This is my number 1 priority. However, we would still need the label in pulpcore-selinux eventually.

On the EL8 side, should we install container-selinux as a stop gap for right now?

pulp-selinux and pulpcore-selinux need to remain co-installable untill we have migrated all usecases to p3 only, but yeah, then we can add that label there.

Sounds like quite a hammer (:sweat_smile:) given the issue we’re hittin’ is not even remotely container related… On the other hand, if it helps, it helps. So OK if that’s a temporary workaround.