Findings during and after the Upgrade to Foreman 3.2 / Katello 4.4

Hi everybody!

I already have some versions of Foreman / Katello behind me now, but only getting to a point where I would say that I know enough to really report back.

So this was the upgrade from 4.3 to 4.4 on my side, and I would like to point out some things I found before, during and after the upgrade process.

Upgrade Documentation:

https://docs.theforeman.org/3.2/Upgrading_and_Updating/index-katello.html

  • I noticed that the upgrade docs don’t reference anything about Enterprise Linux 8 (Rocky Linux 8 in my case), which might be getting the main version after a long time now. I’m actually already used to it not finding it there, but I think it would be time to either add or replace OS version in the upgrade documentation now. (just noticed that the Smart-Proxy docs actually already have it in it)
  • The guide mentions the use of foreman-installer without any parameters right after the package update, but shouldn’t it be foreman-installer --scenario katello for Katello incl. installations?
  • I also noticed that in the upgrade docs screen is referenced, this is deprecated since RHEL 8, so it might be better to reference tmux (as mentioned in the knowledgebase) instead
  • (also noticed on checking for new ports, that the install guides mentioned CentOS 8 as well :+1:t2:)

Issues Pre-Upgrade:

  • I actually had this issue as well (but spoiler alert, can confirm that’s fixed now!):
  • I also had the issue that I wasn’t able to install multiple Erratas from the host page (with the admin user), also fixed now :tada:
  • I further had an issue creating incremental CVs (which also looks to be fixed in 4.4 :partying_face:):
  • One other thing is, that on the Errata page, if Installable is selected, it still shows only applicable Erratas (that’s still happening) (even looks like there is no difference between only Applicable and Installable) (still happening):

Upgrade:

The upgrade itself ran through flawlessly!

Post-Upgrade:

In the release announcement, there is an issue mentioned about Migration of encrypted fields..., this actually should have hit me, but I didn’t notice anything about this issue before the upgrade, nevertheless I wanted to check if the repair dry-run would show me any needed repairs, but it just crashes:

# pulpcore-manager datarepair-2327 --dry-run
Traceback (most recent call last):
  File "/usr/bin/pulpcore-manager", line 11, in <module>
    load_entry_point('pulpcore==3.16.6', 'console_scripts', 'pulpcore-manager')()
  File "/usr/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 363, in execute
    settings.INSTALLED_APPS
  File "/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/lib/python3.8/site-packages/pulpcore/app/settings.py", line 301, in <module>
    settings = dynaconf.DjangoDynaconf(
  File "/usr/lib/python3.8/site-packages/dynaconf/contrib/django_dynaconf_v2.py", line 79, in load
    lazy_settings.populate_obj(django_settings_module)
  File "/usr/lib/python3.8/site-packages/dynaconf/base.py", line 113, in __getattr__
    self._setup()
  File "/usr/lib/python3.8/site-packages/dynaconf/base.py", line 163, in _setup
    self._wrapped = Settings(
  File "/usr/lib/python3.8/site-packages/dynaconf/base.py", line 233, in __init__
    self.execute_loaders()
  File "/usr/lib/python3.8/site-packages/dynaconf/base.py", line 967, in execute_loaders
    self.pre_load(env, silent=silent, key=key)
  File "/usr/lib/python3.8/site-packages/dynaconf/base.py", line 986, in pre_load
    self.load_file(path=preloads, env=env, silent=silent, key=key)
  File "/usr/lib/python3.8/site-packages/dynaconf/base.py", line 1013, in load_file
    if py_loader.try_to_load_from_py_module_name(
  File "/usr/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py", line 66, in try_to_load_from_py_module_name
    mod = importlib.import_module(str(name))
  File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/lib/python3.8/site-packages/pulp_ansible/app/settings.py", line 24, in <module>
    ANSIBLE_CONTENT_HOSTNAME = settings.CONTENT_ORIGIN + "/pulp/content"
  File "/usr/lib/python3.8/site-packages/dynaconf/base.py", line 136, in __getattr__
    value = getattr(self._wrapped, name)
  File "/usr/lib/python3.8/site-packages/dynaconf/base.py", line 277, in __getattribute__
    return super().__getattribute__(name)
AttributeError: 'Settings' object has no attribute 'CONTENT_ORIGIN'

And after everything finished I noticed a few more things:

  • After the successful creation of a incremental CV (which took forever, but that could just be the mass of packages/repos which are in that one) I noticed that the description got filled with a immense load of package names, this looks to bug out in the CV version overview (the picture shows the full screen height):
  • There are some pages, where the package names push the page border in the visible overflow, which creates a horizontal scrollbar (/job_invocations, /template_invocations when the command is visible, as well as the preview tab on the same page and /foreman_tasks/tasks on the raw tab)
  • Nearly missed that one, if any /job_invocations page gets opened there is a error message in the production.log:
2022-03-29T00:44:00 [I|app|b473f28f] Started GET "/javascripts/charts.js" for 10.0.0.10 at 2022-03-29 00:44:00 +0200
2022-03-29T00:44:00 [F|app|b473f28f]
 b473f28f | ActionController::RoutingError (No route matches [GET] "/javascripts/charts.js"):
 b473f28f |
 b473f28f | lib/foreman/middleware/logging_context_request.rb:11:in `call'
 b473f28f | katello (4.4.0) lib/katello/prevent_json_parsing.rb:12:in `call'
2022-03-29T00:44:00 [I|app|15de4e87] Started GET "/javascripts/charts.js" for 10.0.0.10 at 2022-03-29 00:44:00 +0200
2022-03-29T00:44:00 [F|app|15de4e87]
 15de4e87 | ActionController::RoutingError (No route matches [GET] "/javascripts/charts.js"):
 15de4e87 |
 15de4e87 | lib/foreman/middleware/logging_context_request.rb:11:in `call'
 15de4e87 | katello (4.4.0) lib/katello/prevent_json_parsing.rb:12:in `call'

Other notable things:

  • The new documenation design is really great (not talking about the info itself which also was so helpful over the time)! Only one little thing about usability, it is not possible to click beside the dropdowns to close them again.
  • I once tested how far the Ubuntu support already has come in Katello and looked at the Content Host docs for that part. It took me a while to understand where the GPG key thumbprints came from in point 3 of 5.5:
    https://docs.theforeman.org/3.2/Content_Management_Guide/index-katello.html#Extracting_GPG_Public_Key_Fingerprints_from_a_Release_Files_content-management
  • And the hammer command in 5.23 on the same page for importing GPG keys looks to be different now (hammer content-credentials create)
  • Btw, I’m really looking forward to the finished deb Errata functionality, because that is kind of the milestone, where it is easier to argue about using Katello over other solutions
  • One last notable project I’m trying to figure out is, how to run automatic Errata installs for all hosts via REX or Ansible initiated from the server side, I saw that the hammer errata commands are only for the katello-agent and that I would need to somehow do the parsing of the necessary packages myself and then push it to the hammer job-invocation command, but I didn’t come any further so far on that (maybe somebody has built something or written a blog post already I missed)

Oh… as this got quite a long and dense wall of information, I totally can split out some parts to where they belong if needed!

As always, thanks for this great project!

Foreman and Proxy plugin versions:
foreman-release 3.2.0-1.el8
katello-repos 4.4.0-1.el8

Used plug-ins:
VMware provider
Tasks
Ansible
Bootdisk
Puppet
Remote Execution
Snapshot Management
Statistics
Templates

2 Likes

No, once you have run the installer with a --scenario flag, it did write that down and you don’t need to pass that flag again (you can, but it won’t do anything different).

1 Like

Hi @lumarel,
Thanks for your feedback!

I will create a PR to improve this.

I will have a look at this as well.

I would like to refer you to the Report Templates Guide in the orcharhino documentation. It’s on my todo to upstream this in the future.

2 Likes

Please run the command like this (and make sure you do a “cd /tmp” beforehand):
sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager datarepair-2327 --dry-run

2 Likes

Okay thank you for the insight, didn’t know that up to now!

Hey @maximilian!

Thank you these fixes then :slightly_smiling_face:
And I’m not sure if I fully understand that now, can I use the reports for the automatic application of the Errata then?
Because if I go to Content → Errata I already see all the missing/Installable Errata already, am also able to install them in bulk there but not via hammer or in another automated way? :thinking:

Thank you for the hint!
I will test that as soon as I have access to this system again :+1:t2:

At least as far as I think I understand it, I should be able to do nearly the same thing which is possible through the UI in hammer (or even with another automation block, recurring task, recurring ansible job)
If I saw it correctly, in the UI it is possible either through Content → Errata, or via Hosts → Host-Groups → Errata Installation, right? :slight_smile:

Yep, that worked, thank you @rbremer!

# sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' pulpcore-manager datarepair-2327 --dry-run
System check identified some issues:

WARNINGS:
?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: `AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend')`.
Remotes with un-encrypted fields: 0
Remotes encrypted multiple times: 0
1 Like

Hi @maximilian,

sorry for bothering again (don’t know if you already saw my previous message),
as I already have written most of the things here, might be good to continue my story here:

On further trying to get a similar experience for Ubuntu as for the EL distributions, I noticed that the missing updates list doesn’t automatically update in Katello, if the updates are installed via REX (or locally), looks like I had to install katello-upload-profile (which depends on python3-subscription-manager). Though the latter one is the one mentioned in the guides, does this mean I’m doing something wrong or is the upload reactor just not a essential feature?

Also noticed that if katello-host-tools-tracer is installed, the Katello UI still shows that the package is installable/not installed:

I actually got a bit used to the processes of Uyuni over the last year, and what should I say, the workflow with Katello and rhsm just seems a lot more clean!
Looking forward to your answer :slight_smile:

You might have been swamped in work as much as me lately, hope you can have another look some time :slight_smile:

Hi @lumarel
sorry for the late reply.

I am unsure what the exact issue is.

The katello-host-tools are used to send the updated package status to Foreman Server after each apt action. Without the host-tools, Foreman cannot detect if a package was updated, installed, or uninstalled afterwards. Those packages are available on apt.atix.de.
The python3-katello-host-tools-tracer is used to determine if the installation or update of a package requires a reboot of the server, for example after a Linux Kernel update.

I finally found the solution to that!
It’s actually so easy :confused:

You just have to create a continuously scheduled job, which runs the “Install Errata by search query - Katello SSH Default” template against all hosts, with a asterisk in the query:

I think this should be mentioned really prominently in the documentation!

Thank you for taking the time again @maximilian!

So to say it in a different way, normally the subscription manager is already present or will be installed by the bootstrap.py script, which joins the machine to the Katello server.
As this is not possible for Katello with Ubuntu systems, I followed the mentioned way presented on the apt.atix.de page, to just install subscription-manager.
But this is not enough to use it for Katello, the docs over there are not specifically for Katello, which maybe was my missing point. (another point for the downstream product orcharhino, which has that in the bootstrap.py, if I saw it correctly in the docs :slight_smile: )
So to use the subscription-manager from the apt.atix.de repo with Katello you have actually to install katello-upload-profile rather than subscription-manager (which then is a dependency of that package) or both.
Thank you for helping me to understand that!

About katello-host-tools-tracer, yeah I know for what these tools are used, and they work so great for what they are intended to be used! But Katello doesn’t recognize that it is installed, even if it is (it is also showing up in the Katello package list for that system), as you can see in the picture:

I also noticed then later now, that the deb packages don’t show up in the new UI up to now, but that is okay as the old interface is in place anyway up to now :+1:

And now after reading what behind the job template is, I somehow also understand what maximilian meant with reading the report template docs :+1:
It’s just a lot easier to already have something available, and not build such a generic task yourself
(this excludes Ubuntu/Debian systems up to now, as they are not treated in the mentioned job template)

Hey @lumarel

Thanks for the feedback, I will extend the instructions on apt.atix.de.

OK, here I cannot help you. cc @m-bucher

This is a known issues as far as I know and my colleagues are working on it.

1 Like

We are working on it: Add needrestart for tracing of services by louisrokitta · Pull Request #124 · Katello/katello-host-tools · GitHub

Good to see the progress!

Thank’s to both of you for the help and answers :slight_smile: