Can't sync RHEL 8 AppStream RPMs with Katello 3.18

Support
1

I’m having trouble syncing Red Hat Enterprise Linux 8 for x86_64 - AppStream RPMs 8 with Katello 3.18.

I believe this has to do with /var/lib/pulp/media and /var/lib/pulp/docroot directory change but I’m not sure how best to fix the issue. I’ve got lots of content in both directories.

[root@puppetmaster-prod-01 pulp]# du -sh media/
148G media/
[root@puppetmaster-prod-01 pulp]# du -sh docroot/
114G docroot/

[Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/c6/ad96d3b545d1d24026446a88eb660b31d49827d5d0b66dc33b412ce7a3da54’Error message: the server returns an error
HTTP status code: 400
Response headers: {“date”=>“Tue, 09 Feb 2021 22:31:53 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“GET, PUT, PATCH, DELETE, HEAD, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“62”, “via”=>“1.1 puppetmaster-prod-01”, “connection”=>“close”}
Response body: {“publication”:[“Invalid hyperlink - Object does not exist.”]}[Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/c6/ad96d3b545d1d24026446a88eb660b31d49827d5d0b66dc33b412ce7a3da54’Error message: the server returns an error
HTTP status code: 400
Response headers: {“date”=>“Tue, 09 Feb 2021 22:35:23 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“GET, PUT, PATCH, DELETE, HEAD, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“62”, “via”=>“1.1 puppetmaster-prod-01”, “connection”=>“close”}
Response body: {“publication”:[“Invalid hyperlink - Object does not exist.”]}

Foreman and Proxy versions:

foreman-2.3.0-0.7.rc1
katello 3.18-rc1
pulp-server-2.21.4
python3-pulpcore-3.7.3

Distribution and version:
RHEL 7.9

What’s the best way to resolve this? Right now I can’t download RHEL8 repos or publish content views containing RHEL8 repos.

2

Is this only happening for this one repo or other repos as well?

1 Like
3

I’m having a similar issue. Only happens with one repo that I’ve been able to tell so far - unfortunately it’s the CentOS 7 OS repo that underpins most of my CVs. I suspect it may have started when I ran into a problem removing some CV versions that had no different packages in them - I stumbled onto Fixes #31359 - do not delete repo version 0 · Katello/katello@dc5b581 · GitHub and applied that change to my copy of repository.rb which allowed the deletes to happen. But it seems I’ve put myself into that much more of a pickle. Regenerating the repo, validating checksums, syncs, etc. all fail with

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Mon, 15 Feb 2021 20:11:26 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"62", "via"=>"1.1 katello.domain", "connection"=>"close"}
Response body: {"publication":["Invalid hyperlink - Object does not exist."]}

Similarly if I click the Published At: link on the repository page I get a 404: Not Found message on the page.

Any help on getting that repo to regenerate itself would be great - I would hate to have to delete all of those CVs and recreate them!

4

Same here on Katello 3.18.1, for multiple content views (4 out of ~30).

5

I’m having this issue all of a sudden too. I can’t publish a new version in any of my repos. This is not good.

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Tue, 20 Apr 2021 13:13:47 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"62", "via"=>"1.1 hq-1pforeman.internal.ieeeglobalspec.com", "connection"=>"close"}
Response body: {"publication":["Invalid hyperlink - Object does not exist."]}Error message: the server returns an error
HTTP status code: 400

Seeing this in the pulp logs:
hq-1pforeman pulp: pulp.server.webservices.middleware.exception:INFO: Missing resource(s): repository=ad5e7200-ec7f-4148-8d02-fb87b9ac6b0f
Apr 20 09:07:40 hq-1pforeman pulp: django.request:WARNING: Not Found: /pulp/api/v2/repositories/ad5e7200-ec7f-4148-8d02-fb87b9ac6b0f/

6

Looking in /var/lib/pulp/published/yum/http or http and the folders are empty.

My Deb/Ubuntu repos are fine though. So I’m guessing it’s a pulp3 issue, since I’m still on pulp2 for debian.

7

can you use foreman-tail to reproduce the issue and upload the logs?

Also provide your /etc/foreman/plugins/katello.yaml file, that would be helpful. The error looks “WARNING: Not Found: /pulp/api/v2/repositories/ad5e7200-ec7f-4148-8d02-fb87b9ac6b0f/” looks like its pulp2, but you’re syaing you’ve migrated your yum content over to pulp3?

8

Hi Justin,

Yes, I should be running Pulp 3. I did the migration for RPM’s, but not my Ubuntu repos since there is no migration path and I didn’t want to completely delete and recreate the deb repos. Everything was fine - I last published a couple weeks ago. Went to publish a new version today to push Errata and ran into errors. Nothing should have changed with the server.

Here is the log and the katello.yaml.

The forum still won’t let me upload files - super frustrating when you’re trying to get support.

9

Thanks!

It looks like something went wrong (and we need to figure that out), but there’s a pr upstream that should allow katello to ‘heal’ itself: Fixes #31322 - repo update fails immediately after repo create by ianballou · Pull Request #9193 · Katello/katello · GitHub

This will be in 4.0, but we should probably backport it to 3.18. I opened a pr here: Fixes #31322 - repo update fails immediately after repo create by jlsherrill · Pull Request #9301 · Katello/katello · GitHub

Would you be able to apply the patch (https://github.com/Katello/katello/pull/9301.patch) to your local katello install?

You’d need to restart the services afterwards:

foreman-maintain service restart

I can provide more detailed instructions for patching if you need. Let me know!. After patching, restarting services, try a resync.

10

Can you provide more info on how to patch this? No idea where the files are that need to be patched here. I installed via the rpm from the repo.

11

I found this directory that starts to match the patch file, but there is no katello folder in it.
/usr/share/foreman/app/services

12

np, it should be as simple as:

cd /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1
wget https://github.com/Katello/katello/pull/9301.patch
patch -p1 < 9301.patch
foreman-maintain service restart
1 Like
13

Perfect! This completely resolved the issue. Everything is publishing again.

Thank you!

14

I take that back. I published a few new versions without issue, now I’m getting the error again.

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Wed, 21 Apr 2021 17:52:41 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"62", "via"=>"1.1 hq-1pforeman.internal.ieeeglobalspec.com", "connection"=>"close"}
Response body: {"publication":["Invalid hyperlink - Object does not exist."]}Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Wed, 21 Apr 2021 17:52:41 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"62", "via"=>"1.1 hq-1pforeman.internal.ieeeglobalspec.com", "connection"=>"close"}
Response body: {"publication":["Invalid hyperlink - Object does not exist."]}
15

It could be that there are issues with the underlying repos having warnings on their last sync that I can’t get to clear.

16

Looks like there are issues with a few underlying repos having warnings on their last sync that I can’t get to clear. Not sure why I can’t get these repos to sync successfully anymore. I removed the failing repos from the content view and republished it successfully though.

Things like this - from the official Puppet repo.

Package matching query does not exist.Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Wed, 21 Apr 2021 16:44:24 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"62", "via"=>"1.1 hq-1pforeman.internal.ieeeglobalspec.com", "connection"=>"close"}
Response body: {"publication":["Invalid hyperlink - Object does not exist."]}

Or this one from the Docker repo:

PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Wed, 21 Apr 2021 18:10:44 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"62", "via"=>"1.1 hq-1pforeman.internal.ieeeglobalspec.com", "connection"=>"close"}
Response body: {"publication":["Invalid hyperlink - Object does not exist."]}