Katello 3.16rc5 and pulp-admin wrt CentOS Errata

Problem:
In 3.15 I was able to use pulp-admin to push CentOS errata into the appropriate repositories. It appears that is not possible in 3.16 - I presume because of the move to pulp 3.

If I hit https://katello.server/katello/api/v2/repositories I get back the list of my repositories, however the equivalent pulp-admin rpm repo list returns a blank list. I looked through the pulp-list and pulp-dev mailing lists and didn’t see anything obvious about a change to pulp-admin or the rpm plugin only working on v2 repos, so I thought I would at least raise the issue here in case other people are using this method for managing Errata. I am continuing to investigate the pulp_rpm and other APIs to see if there is a method for replicating what pulp-admin rpm repo uploads errata does.

Expected outcome:
pulp-admin rpm works on v3 repos or some alternative for pushing Errata is documented

Foreman and Proxy versions:
Foreman 2.1

Foreman and Proxy plugin versions:
Katello 3.16.0rc5

Distribution and version:
CentOS 7

Other relevant data:
I have been using the errata import script from https://github.com/nicolas-r/katello-centos-errata-import, but I don’t know that it’s directly relevant since in the end it’s just running a subprocess.call(pulp_cmd) where the pulp_cmd is just the pulp-admin rpm repo errata ... with appropriate arguments.

Thanks!
Matt

1 Like

I think you’re seeing a side-effect of the 3.16 selinux issue outlined in another thread [1]. I’ll update here once the correct katello-selinux version is in the 3.16 repos. In the mean time you can probably confirm it is selinux related by tailing /var/log/audit/audit.log while running the pulp-admin command.

[1] SELinux issues while running `hammer repository export'

I’m currently on 3.16RC5, but I had the same results on RC4 as well.

I’m using the user-cert.pem method and other pulp-admin commands succeed (e.g. pulp-admin status). I know things are in flux a bit with the transition to pulp 3, but this is kind of a big deal for those of us that have to respond to security audits and the like on CentOS systems.

I’m in the process of trying to hit the pulp API directly (localhost:24817/pulp/api/v3/) but I can’t get the auth working correctly with httpie yet. It appears that basic auth is what is configured in /etc/pulp/settings.py, however I can’t get past a 403 Forbidden no matter what I’ve tried. If anyone has any tips there, I’d be grateful.

Fully understood, and we appreciate you testing our 3.16 release candidate. You’ve given us the ability to fix the problem before it affects the GA release and a much broader base of users.

2 Likes

Hmm, I’m not seeing any new denials in audit.log when running pulp-admin rpm repo list. audit2allow does show:

============= foreman_rails_t ==============

allow foreman_rails_t rpm_exec_t:file getattr;

allow foreman_rails_t tftpdir_rw_t:dir search;

#============= tomcat_t ==============

allow tomcat_t var_log_t:file { open read };

But it looks like those are largely related to sidekiq and were from yesterday to boot. Thanks for pointing it out, though, something to definitely keep an eye on! I did update the policies from nightly when I was running RC4 to allow candlepin workers to function, so I’m not a stranger to selinux issues.

Thanks,
Matt

Yeah failed as you suspected, at the exact point it called pulp-admin.

I’m not sure what you’ve got going on - seems like your /etc/pulp/admin.conf file is misconfigured. Mine works it just doesn’t return anything:

[root@satellite ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

[root@satellite ~]# pulp-admin repo list
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

[root@satellite ~]# 

I configured mine to use the certificate for pulp-admin:
cat /etc/pki/katello/certs/pulp-client.crt /etc/pki/katello/private/pulp-client.key > /root/.pulp/user-cert.pem

Okay.
I am sorry for mentioning it as it now detracts from your question.

No worries - like you said, it is a bigger issue than just “CentOS Errata” as there are recommendations in the documentation to use pulp-admin for troubleshooting, but given the state of things that doesn’t even seem possible.

I belive Katello 3.16 moved the yum repo backend to Pulp v3, as mentioned here: Foreman :: Katello - Saying goodbye to Pulp 2

Pulp v3 is using PostgreSQL as a backend to store the yum repo metadata but pulp-admin is a part of Pulp v2 and using the old MongoDB backend.

According to https://pulpproject.org/about-pulp-3/ there is currently no CLI available for Pulp v3.

Sorry, I accidentally deleted my last comment. I am using an old method from a book which had dust on it, to create a centos repo with errata.
It is a single repo comprised of the latest base/updates/extras. The number of errata seems quite low.
image

Does it look right?

I have some more details to share here:

In new Katello 3.16 installations Pulp 3 will be used by default for yum content. In order to continue using the katello-centos-errata-import tooling you can switch it back to Pulp 2 by running the installer with this option: --katello-use-pulp-2-for-yum=true. If you’ve set up a new 3.16 installation with some repositories, it’s recommended to rebuild the instance or remove those repositories before re-running the installer to switch back to Pulp 2.

Once Pulp 3 becomes the only option, the katello-centos-errata-import won’t work as-is. Since it’s a third-party tool it’s not on our roadmap to update it or provide a new utility to support CentOS errata. We’re hoping that the Katello community will fill that need as it sees fit.

Thanks for the update, Jonathan. As time allows, I’m looking at the pulp3 api to see if I can modify/write a replacement for the import tools for the CentOS Errata, but no timelines on if/when that will find its way onto my schedule. I’ll certainly share it back out to the community if I get something working.

Matt

2 Likes

I’m gonna be watching this carefully. I’ve been hoping to not have to write Yet Another CentOS Errata Importer (YACEI?)

Is there an API call for importing errata (or one erratum) on the roadmap though? Because that’s all that is needed to be able to update CentOS errata import scripts.

Hey, issue I had. I’m using errata from private repo updates http://updateinfo-origin.cefs.steve-meier.de/7-sync/updates/x86_64/repodata/. Updateinfo/errata file is hidden and password protected.

Since Katello 3.16, since yum content was migrated to pulp3, I was seeing a lot more errata applicable on main server Content host page but when running yum update-minimal --security --bugfix I would see no packages marked for minimal update.

After digging deeper here Package/errata info disappears after a while on Content hosts page/Foreman 2.1.2 we’ve found that since pulp3 is ruuning on main server it is showing correct data, while pulp2 which is running on smart proxy is filtering out any errata that refrecened packages are not part of updates.

Workaround which works is mirroring locally official centos repos os and extras, modifying repodata to include errata in each and then syncing. Pulp3 knows this is duplicate so it merges them into one. But when pulp2 which is running on smart proxies sync it, it keeps all errata, but spread across os, extras, updates repos.

So basically this way you can import errata through repos without the need for any script or anything custom inside Foreman. All you need is a server to mirror official centos repos, modified to include updateinfo/errata.

1 Like

I started looking at this a while back, but didn’t make much progress. It looks to me like https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/content_rpm_advisories_create is the way to get errata into pulp3 for RPMs. The description of the fields leaves something to be desired, but it could just be that I’m not terribly familiar with it.

The workaround @matemikulic suggested is nice, but requires duplicating all of the repos required. It would be great if we could get the right shim between the XML files provided by Steve Meier and the pulp3 backend similar to what we had for pulp2. It’s still something I’d like to get to someday, but with all the “do more with less” we have going on, I don’t currently have the free cycles.