Suddenly package installation on clients stopped working

Support
,
1

Problem:
Suddenly, clients are not able to download yum packages.

Expected outcome:
Packages should be available and downloadable

Foreman and Proxy versions:
2.3.3

Foreman and Proxy plugin versions:
Katello 3.18.2.1, foreman_remote_execution 4.2.2., foreman-tasks 3.0.5

Distribution and version:
RHEL 7

Other relevant data:
@iballou (per Dirk suggestion),

Suddenly my systems got this error when trying to check for yum content:

[root@XXXXXXXXX ~]# yum reinstall vim
Updating Subscription Management repositories.
BaseOS (RPMs)                                                                                                             48 kB/s | 2.6 kB     00:00
AppStream (RPMs)                                                                                                         2.0 kB/s | 611  B     00:00
Errors during downloading metadata for repository 'AppStream':
  - Status code: 502 for https://foremanserver/pulp/repos/MyOrg/Prod/Rhel_8_Clients/custom/Appstream/AppStream_RPMs/repodata/repomd.xml (IP: xxxxxxxxxx)

Notice the difference on metadata size

This error seems to be correlated when requesting the resource, with this error on the messages log:

[Errno 2] No such file or directory: '/var/lib/pulp/media/artifact ***********

I already deleted the repo, from content view. I tried to add the repo from fresh, by deleting the repo, from product and adding it again, with no avail (still same no such files error, although path varies).
I even created a new product only with that repo, with no avail also.
I tried checksum checks, republish metadata, advanced sync, with no avail.
Sometimes I get the error of too many open files, which I can workaround increasing nofiles on the kernel settings.

It did finish a sync without errors once, but although the repo shows on the content view of the clients, no packages are able to be downloaded (I tried with a simple vim reinstall and is not able to find packages that should be there).

Any ideas would be appreciated.
Thank you very much!!

2

Hi @maramorim,

Is there an artifact folder anywhere under /var/lib/pulp? I see you’re on 3.18, so you must’ve done the Pulp 2 to Pulp 3 migration at some point?

I’ll just chime in that Katello 3.18 is pretty old at this point, with numerous Pulp 3 issues (since it’s the first release to fully support Pulp 3). I’d suggest upgrading up to a “supported” Katello version as soon as possible (4.3 or higher, soon to be 4.4 or higher starting in June).

1 Like
3

Hi @iballou ,
thank you very much for the reply (kinda feeling alone here =) )

1)Yes there is an aritifact folder under /var/lib/pulp .

2)I did not migration, it is a fresh install, with the katello scenario on foreman installer. Just followed the manual available at that time, I guess.
This was my bible:

https://theforeman.org/plugins/katello/3.18/index.html

3)So overall you recommend upgrading katello, which I think it is also the only solution, as the system just keeps pumping errors like " FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pa/artifact/57/",

  1. Is there any scenario for RHEL8? Noticed all the manuals, go till CentOS8. So it would be a rhel7 with

  2. Could you explain, any way to do this upgrade for a noob? as it is an upgrade I imagine I will retain, the systems, products, etcetera.

Thank you a lot mate

4

Hi @maramorim,

I gotcha, I forgot that 3.18 fresh installs got Pulp 3 by default.

For RHEL 8 you should be able to follow the same instructions as CentOS8. It’ll be the same repositories. Katello 4.0 is the first version that supported EL8.

When you upgrade you will retain all your old data. The basic upgrade workflow is to update the repositories to the next Foreman + Katello version, yum update, and then run the installer. It’s a similar idea for any smart proxies you have. Note that you cannot skip Katello versions when doing upgrades, so you’ll have to upgrade to 4.0, then 4.1, 4.2, etc. Note that Katello 4.5 will likely be the last release to support EL7.

The docs here should help you out (Upgrading and Updating Foreman). You can skip the section titled “Preparing to Migrate Content to Pulp 3” because you’re already on Pulp 3. Foreman 2.5 pairs up with Katello 4.0, Foreman 3.0 pairs up with Katello 4.1, etc. The repositories noted in the instructions are a good reminder of which versions of the two projects you need to pair up.

If you’re still seeing the artifact issue after you upgrade, we can debug it here. It should be easier to debug because you’ll be on a newer version of Pulp. The Katello 3.18 Pulp is super old though, so I have a good feeling that upgrading will help.

1 Like
5

Gotcha!!!
Just in doubt if remaking all in a rhel8 then, or go for the upgrde on the rhel7 .
I guess that as an exercise, and to see if we can recover this somehow, and debug I will try the upgrade of the foreman+katello on rhe rhel 7.

I will read the manuals, but that sounds that I have to update only once foreman, and 3 times katello?

thanks a lot for your assistance, great community

1 Like
6

So the upgrade is still just ‘n’ versions up for both Foreman and Katello. Their versions just don’t match up exactly.

Let us know if you hit any issues!

7

I have to say that Ian provided top notch support, we were able to get rid of the artifact issue by upgrading to a recent version of foreman katello and doing a reset of the environment keeping the hosts (repo, content views, were deleted and orphaned content deleted manually). To be able to delete content view, using hammer and bash all hosts were moved in a batch to the default content view.
After this the repo was recreated and the hosts assigned back to the original content view, with the same batch script.

Cheers

2 Likes