Updating Foreman 2.2.1 to 2.3.1 and Katello 3.17.1 to 3.18 issue

Running yum update to get the latest version of the packages before doing the upgrade throws me two errors:
Error: katello-agent conflicts with pulp-consumer-client-2.21.4-2.el7.noarch
You could try using --skip-broken to work around the problem
I have neither Katello-agent nor katello-host-tools installed on the Foreman server.

And second when using --skip-broken to avoid the “upgrade” of the katello-agent:
Package tfm-rubygem-foreman_remote_execution-4.2.1-1.fm2_3.el7.noarch.rpm is not signed

I have both Foreman 2.3, Foreman Plugins 2.3 and Katello 3.18 mirrored to the local Foreman server and the repository points to it. I was trying to use the Foreman 2.3 signing key for my first try and now am using Foreman 2.2 signing key. Same result.

Expected outcome:
The package upgrade to
not complain about katello-agent
and all packages to be signed.

Foreman and Proxy versions:
Foreman 2.2.1
Katello 3.17.1

Foreman and Proxy plugin versions:

Distribution and version:
CentOS 7.9

Other relevant data:

You simply should remote katello-agent which is deprecated anyway, as mentioned in the Katello release notes.
If the second problem persists, I remember similar problems in the past. You should be able to find solutions via a search.

I understand, but as I mentioned in my initial post, I don’t have the agent installed. It is somehow included in the yum update process, dunno why. Message is “Katello-agent is marked as obsoleted” or something along those lines.

I did a search, but couldn’t find much. I will try to dig deeper.

Thank you.

That was my mistake, I forgot that the plugin rpms are not signed yet. Thank you for pointing this out.

For the first problem: I haven’t tried the upgrade myself yet. Perhaps there are some conflicting repositories. Can you show the output of “yum repolist”?

CentOS 7
CentOS 7 Extras
CentOS 7 Updates
Foreman 2.3 Client
Puppet 6
Zabbix Client 5.2
CentOS Software Collections SIG (RH)
Ansible Runner
Foreman 2.3
Foreman 2.3 Plugins
Katello 3.18
Pulp Community Release

Please remember, they are from the Forman server.

Probably the main difference to my setup are the Pulp repos. Do you absolutely need them? They are for Pulp 3 only I think. I preferred to stay with Pulp 2 for now. You could temporary disable them and see if it works then.

I will try that. From what I read in the docs (my setup was initially Katello 3.16), it automatically uses Pulp 3, so I am not sure I have a chance to disable them.

I think the option “–foreman-proxy-plugin-pulp-pulpcore-enabled false” in the Foreman installer will be sufficient to exclude Pulp 3.

Nice call, as soon as I disable the “Pulp Community Release” repo, the error goes away. I will go ahead with the upgrade an hopefully will not run into any more issues :slight_smile:

All right. Seems like the upgrade really went wrong.

No errors during the migration (after temporarily disabling the pulp community release repo), restarted the system.
I can log in just fine, no pending tasks etc. Version has been updated to 2.3.1 and 3.18.0.

However, yum on a registered client can no longer access any files:

[root@srv1 ~]# yum update
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager, tracer_upload
Loading mirror speeds from cached hostfile
https://foreman.local/pulp/repos/org/Production/Foreman_Katello/custom/CentOS_7/Ansible/repodata/repomd.xml: [Errno 14] HTTPS Error 502 - Bad Gateway

On the Foreman server I see this in /v/l/m:

Dec 21 14:16:43 foreman pulpcore-content: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/31/685cbb7f2e8a3b4c53527e9d9180e80dd73fcf49ee4aed418be11aa732771e'
Dec 21 14:17:57 foreman pulpcore-content: [2020-12-21 13:17:57 +0000] [2281] [ERROR] Unhandled exception
Dec 21 14:17:57 foreman pulpcore-content: Traceback (most recent call last):
Dec 21 14:17:57 foreman pulpcore-content: File "/usr/lib64/python3.6/site-packages/aiohttp/web_protocol.py", line 461, in start
Dec 21 14:17:57 foreman pulpcore-content: await prepare_meth(request)
Dec 21 14:17:57 foreman pulpcore-content: File "/usr/lib64/python3.6/site-packages/aiohttp/web_fileresponse.py", line 223, in prepare
Dec 21 14:17:57 foreman pulpcore-content: st = await loop.run_in_executor(None, filepath.stat)
Dec 21 14:17:57 foreman pulpcore-content: File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
Dec 21 14:17:57 foreman pulpcore-content: result = self.fn(*self.args, **self.kwargs)
Dec 21 14:17:57 foreman pulpcore-content: File "/usr/lib64/python3.6/pathlib.py", line 1158, in stat
Dec 21 14:17:57 foreman pulpcore-content: return self._accessor.stat(self)
Dec 21 14:17:57 foreman pulpcore-content: File "/usr/lib64/python3.6/pathlib.py", line 387, in wrapped
Dec 21 14:17:57 foreman pulpcore-content: return strfunc(str(pathobj), *args)

So now I have a mixup in my setup? I am not sure. I can access the files just fine via the repository published URL, but via pulp it doesn’t seem to work. The published files seem to be under /var/lib/pulp/docroot/artifacts, not in /var/lib/pulp/media/artifacts …

What can I do to rectify this?

Yes, it could be a mixup between Pulp 2 and 3. Of course I cannot tell you what version you should use, that’s up to you. But if you want to go with Pulp 2 make sure that no components of Pulp 3 are running and vice versa.

For me, it was using “–foreman-proxy-plugin-pulp-pulpcore-enabled false” and “–foreman-proxy-plugin-pulp-enabled true” and making sure that no services of pulpcore were running (stopping and disabling them).

To be honest, I didn’t make any choice. The doc said, that starting with a certain Katello release (for new installations) they use Pulp 3 anyway, so I assume that’s what I am on. Hopefully someone from the @katello team can chime in and assist here.

Thank you!

To see what Pulp you’re currently using, go in the web UI to Infrastructure > Smart Proxies > (your foreman server name) > Services tab.

You’re using Pulp 2 for the repo types it lists under “Pulp”, and Pulp 3 for the repo types listed under “Pulpcore.”

Hi Jeremy,

thanks for chiming in. This is how it looks like:

So to me that looks like yum repos are served via Pulp 3.

But why is the pulpcore-content then trying to serve it from a Pulp 2 directory?

I think things may have gotten really off track here. A couple of notes:

  • do not disable the pulp3 repository, even if you’re still on pulp2, upgrades will not be happy and will likely leave you in a broken state
  • You hit a dependency issue with ‘pulp-consumer-client’. I’m not sure why this package was even installed, it conflicts with katello-agent and should not be used or installed
  • –foreman-proxy-plugin-pulp-pulpcore-enabled false should NOT be run unless you really know what you’re doing
  • what makes you think that pulp3 is trying to serve content out of the pulp2 directory? /var/lib/pulp/media/artifacts is a pulp3 directory

It looks to me like there is some content in pulp3 that went missing for some reason. Can you try to ‘repair’ the repository?

Go to Content > Products > click a product > click the repository > in the top right click the drop down > click ‘Verify Content Checksum’

see if that helps at all

Hi Justin,

thank you for your assistance here. To clarify a few of your points:

  • During the original install and all updates since then I always had all repositories as shown enabled.
  • I do not know why the package pulp-consumer-client was installed, I only installed the additional plugin for VMware, everything else was handled by installing Katello in the first place (I think it was 3.16). Katello-Agent was never installed.
  • I did not use the switch during install or update.
  • The reason I am saying that is the error in /v/l/m, but I am really not sure on how to say that differently. The content seems to live in /var/lib/pulp/docroot/…, while the error shows me /var/lib/pulp/media/…

Lets get a bit more into detail with the last point, as this the situation I am faced with right now. Before the upgrade I had no errors using the Katello instance. I have about 30+ hosts subscribed and all got updates.
After the upgrade the previously shown error started to appear for all hosts. The directory /var/lib/pulp/media was completely empty. That was yesterday, interestingly enough, today I see a few packages inside this directory. So I assume the daily repo sync schedule started downloading new packages there and no longer into /var/lib/pulp/docroot. I cannot explain why that would happen.

I was under the assumption that Katello 3.16 will default to pulp 3 for a new install, so I never really give a second thought until that error started to appear. I will try the content repair and get back to you.

I can verify that the directories underneath /var/lib/pulp/media/artifact have all been created this morning during the sync schedule, they did not exist before.
docroot has 137G
media has 2.1G

I repaired the CentOS 7 base repository, here is the output of the task:

Total steps: 8362/8362
Identify corrupted units: 4181/4181
Repair corrupted units: 4181/4181

and its still running. I will do that same with CentOS 7 updates after to see a change and give more feedback.

BTW: during the repair the size of the media directory has already increased to 6.2G …