Updating Foreman 2.2.1 to 2.3.1 and Katello 3.17.1 to 3.18 issue

Support
1

Problem:
Running yum update to get the latest version of the packages before doing the upgrade throws me two errors:
Error: katello-agent conflicts with pulp-consumer-client-2.21.4-2.el7.noarch
You could try using --skip-broken to work around the problem
I have neither Katello-agent nor katello-host-tools installed on the Foreman server.

And second when using --skip-broken to avoid the “upgrade” of the katello-agent:
Package tfm-rubygem-foreman_remote_execution-4.2.1-1.fm2_3.el7.noarch.rpm is not signed

I have both Foreman 2.3, Foreman Plugins 2.3 and Katello 3.18 mirrored to the local Foreman server and the repository points to it. I was trying to use the Foreman 2.3 signing key for my first try and now am using Foreman 2.2 signing key. Same result.

Expected outcome:
The package upgrade to
not complain about katello-agent
and all packages to be signed.

Foreman and Proxy versions:
Foreman 2.2.1
Katello 3.17.1

Foreman and Proxy plugin versions:

Distribution and version:
CentOS 7.9

Other relevant data:

2

You simply should remote katello-agent which is deprecated anyway, as mentioned in the Katello release notes.
If the second problem persists, I remember similar problems in the past. You should be able to find solutions via a search.

3

I understand, but as I mentioned in my initial post, I don’t have the agent installed. It is somehow included in the yum update process, dunno why. Message is “Katello-agent is marked as obsoleted” or something along those lines.

I did a search, but couldn’t find much. I will try to dig deeper.

Thank you.

4

That was my mistake, I forgot that the plugin rpms are not signed yet. Thank you for pointing this out.

5

For the first problem: I haven’t tried the upgrade myself yet. Perhaps there are some conflicting repositories. Can you show the output of “yum repolist”?

6

Sure:
Ansible
CentOS 7
CentOS 7 Extras
CentOS 7 Updates
EPEL
Foreman 2.3 Client
Puppet 6
Zabbix Client 5.2
CentOS Software Collections SIG (RH)
Ansible Runner
Candlepin
Foreman 2.3
Foreman 2.3 Plugins
Katello 3.18
Pulp Community Release
Pulpcore

Please remember, they are from the Forman server.

7

Probably the main difference to my setup are the Pulp repos. Do you absolutely need them? They are for Pulp 3 only I think. I preferred to stay with Pulp 2 for now. You could temporary disable them and see if it works then.

8

I will try that. From what I read in the docs (my setup was initially Katello 3.16), it automatically uses Pulp 3, so I am not sure I have a chance to disable them.

9

I think the option “–foreman-proxy-plugin-pulp-pulpcore-enabled false” in the Foreman installer will be sufficient to exclude Pulp 3.

10

Nice call, as soon as I disable the “Pulp Community Release” repo, the error goes away. I will go ahead with the upgrade an hopefully will not run into any more issues :slight_smile:
Thanks!

11

All right. Seems like the upgrade really went wrong.

No errors during the migration (after temporarily disabling the pulp community release repo), restarted the system.
I can log in just fine, no pending tasks etc. Version has been updated to 2.3.1 and 3.18.0.

However, yum on a registered client can no longer access any files:

[root@srv1 ~]# yum update
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager, tracer_upload
Loading mirror speeds from cached hostfile
https://foreman.local/pulp/repos/org/Production/Foreman_Katello/custom/CentOS_7/Ansible/repodata/repomd.xml: [Errno 14] HTTPS Error 502 - Bad Gateway

On the Foreman server I see this in /v/l/m:

Dec 21 14:16:43 foreman pulpcore-content: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/31/685cbb7f2e8a3b4c53527e9d9180e80dd73fcf49ee4aed418be11aa732771e'
Dec 21 14:17:57 foreman pulpcore-content: [2020-12-21 13:17:57 +0000] [2281] [ERROR] Unhandled exception
Dec 21 14:17:57 foreman pulpcore-content: Traceback (most recent call last):
Dec 21 14:17:57 foreman pulpcore-content: File "/usr/lib64/python3.6/site-packages/aiohttp/web_protocol.py", line 461, in start
Dec 21 14:17:57 foreman pulpcore-content: await prepare_meth(request)
Dec 21 14:17:57 foreman pulpcore-content: File "/usr/lib64/python3.6/site-packages/aiohttp/web_fileresponse.py", line 223, in prepare
Dec 21 14:17:57 foreman pulpcore-content: st = await loop.run_in_executor(None, filepath.stat)
Dec 21 14:17:57 foreman pulpcore-content: File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
Dec 21 14:17:57 foreman pulpcore-content: result = self.fn(*self.args, **self.kwargs)
Dec 21 14:17:57 foreman pulpcore-content: File "/usr/lib64/python3.6/pathlib.py", line 1158, in stat
Dec 21 14:17:57 foreman pulpcore-content: return self._accessor.stat(self)
Dec 21 14:17:57 foreman pulpcore-content: File "/usr/lib64/python3.6/pathlib.py", line 387, in wrapped
Dec 21 14:17:57 foreman pulpcore-content: return strfunc(str(pathobj), *args)

So now I have a mixup in my setup? I am not sure. I can access the files just fine via the repository published URL, but via pulp it doesn’t seem to work. The published files seem to be under /var/lib/pulp/docroot/artifacts, not in /var/lib/pulp/media/artifacts …

What can I do to rectify this?

12

Yes, it could be a mixup between Pulp 2 and 3. Of course I cannot tell you what version you should use, that’s up to you. But if you want to go with Pulp 2 make sure that no components of Pulp 3 are running and vice versa.

For me, it was using “–foreman-proxy-plugin-pulp-pulpcore-enabled false” and “–foreman-proxy-plugin-pulp-enabled true” and making sure that no services of pulpcore were running (stopping and disabling them).

13

To be honest, I didn’t make any choice. The doc said, that starting with a certain Katello release (for new installations) they use Pulp 3 anyway, so I assume that’s what I am on. Hopefully someone from the @katello team can chime in and assist here.

Thank you!

14

To see what Pulp you’re currently using, go in the web UI to Infrastructure > Smart Proxies > (your foreman server name) > Services tab.

You’re using Pulp 2 for the repo types it lists under “Pulp”, and Pulp 3 for the repo types listed under “Pulpcore.”

15

Hi Jeremy,

thanks for chiming in. This is how it looks like:

Screen Shot 2020-12-21 at 17.11.21
Screen Shot 2020-12-21 at 17.11.21850×1696 74.1 KB

16

So to me that looks like yum repos are served via Pulp 3.

But why is the pulpcore-content then trying to serve it from a Pulp 2 directory?

17

I think things may have gotten really off track here. A couple of notes:

  • do not disable the pulp3 repository, even if you’re still on pulp2, upgrades will not be happy and will likely leave you in a broken state
  • You hit a dependency issue with ‘pulp-consumer-client’. I’m not sure why this package was even installed, it conflicts with katello-agent and should not be used or installed
  • –foreman-proxy-plugin-pulp-pulpcore-enabled false should NOT be run unless you really know what you’re doing
  • what makes you think that pulp3 is trying to serve content out of the pulp2 directory? /var/lib/pulp/media/artifacts is a pulp3 directory

It looks to me like there is some content in pulp3 that went missing for some reason. Can you try to ‘repair’ the repository?

Go to Content > Products > click a product > click the repository > in the top right click the drop down > click ‘Verify Content Checksum’

see if that helps at all

18

Hi Justin,

thank you for your assistance here. To clarify a few of your points:

  • During the original install and all updates since then I always had all repositories as shown enabled.
  • I do not know why the package pulp-consumer-client was installed, I only installed the additional plugin for VMware, everything else was handled by installing Katello in the first place (I think it was 3.16). Katello-Agent was never installed.
  • I did not use the switch during install or update.
  • The reason I am saying that is the error in /v/l/m, but I am really not sure on how to say that differently. The content seems to live in /var/lib/pulp/docroot/…, while the error shows me /var/lib/pulp/media/…

Lets get a bit more into detail with the last point, as this the situation I am faced with right now. Before the upgrade I had no errors using the Katello instance. I have about 30+ hosts subscribed and all got updates.
After the upgrade the previously shown error started to appear for all hosts. The directory /var/lib/pulp/media was completely empty. That was yesterday, interestingly enough, today I see a few packages inside this directory. So I assume the daily repo sync schedule started downloading new packages there and no longer into /var/lib/pulp/docroot. I cannot explain why that would happen.

I was under the assumption that Katello 3.16 will default to pulp 3 for a new install, so I never really give a second thought until that error started to appear. I will try the content repair and get back to you.

19

I can verify that the directories underneath /var/lib/pulp/media/artifact have all been created this morning during the sync schedule, they did not exist before.
docroot has 137G
media has 2.1G

I repaired the CentOS 7 base repository, here is the output of the task:

Total steps: 8362/8362
--------------------------------
Identify corrupted units: 4181/4181
Repair corrupted units: 4181/4181

and its still running. I will do that same with CentOS 7 updates after to see a change and give more feedback.

20

BTW: during the repair the size of the media directory has already increased to 6.2G …