katello-nightly-rpm-pipeline 1033 failed

Katello nightly pipeline failed:

https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/1033/

foreman-pipeline-katello-nightly-centos7-install (failed) (remote job)
foreman-pipeline-katello-nightly-centos8-stream-install (failed) (remote job)
foreman-pipeline-katello-nightly-centos8-install (failed) (remote job)
foreman-pipeline-katello-nightly-centos7-upgrade (failed) (remote job)
foreman-pipeline-katello-nightly-centos8-upgrade (passed) (remote job)

this seems to be broken rather “nicely”, almost like Katello wouldn’t work at all.

the very first error you see in production log is

2021-07-15T23:50:13 [E|app|942eccf6] Error occurred while starting Katello::CandlepinEventListener
2021-07-15T23:50:13 [E|app|942eccf6] Client failed to start in 2 seconds

but there is a success some time later:

2021-07-15T23:50:29 [I|app|942eccf6] Subscribed to katello.candlepin.candlepin_events

but the rest still looks like it couldn’t process anything.

@katello yo, this is continuing to fail, until today.

https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/1036/

interestingly, upgrades seem to pass :confused:

One (unrelated) observation is:

type=AVC msg=audit(1626394111.747:3480): avc:  denied  { write } for  pid=8039 comm="pulpcore-worker" name="exports" dev="vda1" ino=6029750 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1626394111.747:3480): avc:  denied  { add_name } for  pid=8039 comm="pulpcore-worker" name="Test_Organization" scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1626394111.747:3480): avc:  denied  { create } for  pid=8039 comm="pulpcore-worker" name="Test_Organization" scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1626394111.748:3481): avc:  denied  { create } for  pid=8039 comm="pulpcore-worker" name="export-76a2de76-edaa-45dc-8fba-ccc2da4771e2-20210716_0008.tar.gz" scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
type=AVC msg=audit(1626394111.748:3481): avc:  denied  { write open } for  pid=8039 comm="pulpcore-worker" path="/var/lib/pulp/exports/Test_Organization/Test_CV/1.0/2021-07-16T00-08-31-00-00/export-76a2de76-edaa-45dc-8fba-ccc2da4771e2-20210716_0008.tar.gz" dev="vda1" ino=6411379 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
type=AVC msg=audit(1626394111.748:3482): avc:  denied  { getattr } for  pid=8039 comm="pulpcore-worker" path="/var/lib/pulp/exports/Test_Organization/Test_CV/1.0/2021-07-16T00-08-31-00-00/export-76a2de76-edaa-45dc-8fba-ccc2da4771e2-20210716_0008.tar.gz" dev="vda1" ino=6411379 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
type=AVC msg=audit(1626394111.748:3483): avc:  denied  { ioctl } for  pid=8039 comm="pulpcore-worker" path="/var/lib/pulp/exports/Test_Organization/Test_CV/1.0/2021-07-16T00-08-31-00-00/export-76a2de76-edaa-45dc-8fba-ccc2da4771e2-20210716_0008.tar.gz" dev="vda1" ino=6411379 ioctlcmd=5401 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
type=AVC msg=audit(1626394111.785:3484): avc:  denied  { read } for  pid=8039 comm="pulpcore-worker" name="export-76a2de76-edaa-45dc-8fba-ccc2da4771e2-20210716_0008.tar.gz" dev="vda1" ino=6411379 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1

seems /var/lib/pulp/exports is miss-labeled as var_lib_t, but that happens later than the first error

The first error in the bats log is

[2021-07-18T23:51:04.813Z]     #   Error: Option '--content-type': Value must be one of ..

Which sounds a bit like something coming from Fixes #31616 - Remove content_types requirement in katello.yaml and u… · Katello/katello@304de97 · GitHub?

I think the fact that upgrades are working is actually a “bug”, as hammer is still using the old API doc and thus not failing