Docker hub Repository is not syncing after update

Problem:
I am unable to subscribe to grafana repository in dockerhub. I just now upgraded from katello 2 to katello 3.17.
Expected outcome:
Repository should sync
Foreman and Proxy versions:
2.2.3
Foreman and Proxy plugin versions:

Distribution and version:
RHEL 7
Other relevant data:

This is the error

Error message: the server returns an error
HTTP status code: 403
Response headers: {"date"=>"Tue, 21 Sep 2021 13:39:20 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"63", "via"=>"1.1 ingbtcpic6vl232.code1.emi.com", "connection"=>"close"}
Response body: {"detail":"You do not have permission to perform this action."}undefined method `each' for nil:NilClass

This is output of dynflow

Any help would be appreciated

My hunch is that my server time is in IST and in the error i can see in GMT. Dont know if it has any relation

Hi @almond,

Dockerhub introduced rate limiting some months ago, so it’s possible that you’re experiencing that. Did the error occur immediately, or have you already been syncing some Docker content?

If that is the case, it seems you’re not the only person hitting it with Grafana. I found an older thread: https://github.com/grafana/grafana/issues/27677

By the way, I would recommend that you go up to Katello 3.18. Katello 3.18 is the last release to support Pulp 2, so it received quite a bit more attention than Katello 3.17.

3 Likes

Can you provide me the link for upgrading to Katello 3.18? Also, Its happening to all the repositories. Not only grafana @iballou

Here are the instructions: Foreman :: Plugin Manuals
Note that you can ignore the migration steps until you want to upgrade to Pulp 3 (unless you already did on your upgrade up to 3.17).

Can you show me what you are entering in for the upstream url and repository names? Also, on your primary smart proxy’s Services tab, can you paste me everything that’s under “Content”? That would be the smart proxy with the same hostname as your Katello server, in case you have multiple.

Upstream url and repo names:

Service tab:

I no longer think it’s related to rate limiting. Pulp 3 (which you’re on for Docker + File) works around it. I think you might be hitting this same issue: [ContentMigration] foreman-maintain content prepare failed - #5 by touchardv

To get around this, first try:

cd /tmp
sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py'  /usr/bin/pulpcore-manager reset-admin-password

Set the password to anything and try syncing again.

If that doesn’t work, try the workaround in the comment I linked above.

You can get access to the Pulpcore database by:

sudo -i -u postgres
psql
\c pulpcore

@iballou Password setting failed.

I tried the next step and after that I tried to create a product and I am getting this error

@almond,

Try this: systemctl restart pulpcore* --all
If the pulpcore services aren’t all running successfully, there should be an error in /var/log/messages. I’d like to see that if so.

@iballou
Hey so I restarted pulp services and after adding a product, now I’m seeing a different error

Ah ok, so that is actually the rate limiting now. Can I see your Pulp versions? sudo pip3 list | grep pulp

@iballou

[root@ingbtcpic6vl232 ~]# sudo pip3 list | grep pulp
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pulp-2to3-migration (0.11.4)
pulp-certguard (1.0.3)
pulp-container (2.1.2)
pulp-file (1.3.0)
pulp-rpm (3.11.2)
pulpcore (3.7.8)

Do you need all tags for the Grafana repo? Can you try only syncing the latest tag? You can do so by setting the repository’s “Limit Sync Tags” to “latest” (or whatever tags you need). That should reduce the amount of downloads.

That was working before also. Let me try once more. I want to add more tags. @iballou

@iballou This time I tried with alpine docker image with a specific tag and still I’m getting same error

The tip I received from the Pulp team was that Pulpcore 3.14 carried improvements for retrying downloads that should help. Not to mention that the Pulpcore from Katello 3.17 has the old tasking system which didn’t work so well.

Since you’ve already migrated partially to Pulp 3, I would recommend to continue upgrading up to Katello 4.1 so that you receive the best version of Pulp 3.

There’s an issue though. At the moment, upgrades to Katello 4.0 and 4.1 are broken due to the qpid-proton-c.x86_64 0:0.35.0-1.el7 package being in EPEL. More info: Katello installs for 4.0 and 4.1 are broken due to qpid-proton update in EPEL on EL7 - #45 by ekohl

I was able to work around it by removing the RPM, temporarily disabling EPEL, and installing the RPM from our koji build system: qpid-proton-0.34.0-2.el7 | Build Info | koji

So what do you prefer? Should I want to do an upgrade to 4.1 knowing there’s an issue on that version? @iballou

First, it might be good to bring up a small test Katello 4.1 system to double check that it does indeed fix your syncing issue. I was able to sync Alpine myself, but I’m not sure if you were trying to sync a bunch of Docker repos at once. If you haven’t done this before, Forklift makes it easy. It’ll create a small VM for you and install Katello 4.1 on it.

If you are really stuck on the syncing issue and 4.1 definitely fixes it, feel free to upgrade keeping in mind the EPEL workaround from above.

If you prefer the absolute safest route with no workarounds, then I would wait for Katello 4.0.3 and 4.1.4 to be released before upgrading.

The upgrade to 4.1 should be smooth now, if you still want to try that and haven’t already.

1 Like