Hello community,
As of Katello 4.16.0, bootc (image mode) hosts now have some level of support. You can register them, see some bootc information about them, and remotely execute bootc jobs.
During the design of this initial implementation, we decided that bootc hosts should be able to have patching actions run against them. In the current implementation, unless you run bootc usr-overlay, the commands will fail. This decision was a good one because DNF has added support for a new persistence option, so installing packages at runtime is going to be a generally-accepted bootc machine workflow.
From this, we know that Katello should support transiently installing packages as well. Supporting this for normal remote execution (REX) shell commands should be easy because we just need to add that transient flag depending on if the host is a bootc host or not.
The story is a bit more complicated for Ansible REX jobs, however. I recently created a request for Ansible to provide support for the transient option in the package module. So far, it looks like the request will be difficult to backport and requires more information from the DNF team. With the current implementation, DNF blindly uses the usr-overlay, so there’s an argument that Katello should just run bootc usr-overlay before installing packages.
With this all in consideration, my proposal is to use the transient flag wherever possible since it is the most official way to install packages on bootc machines. If the transient installation becomes more sophisticated in the future, and Katello is already using it, we will immediately benefit. This would mean, however, that Ansible REX templates for installing packages would need to use plain shell commands rather than the built in package module.
The alternative is to enable the usr-overlay ourselves, but that would need to be a shell command in Ansible anyway. Whatever we do, it would be best to be consistent between Ansible and normal REX jobs.
There are some cases where we do not have control of the Ansible scripts. SCAP and Insights both provide remediation scripts that will fail on machines that are not using the usr-overlay. For SCAP, we use snippets, so we can enable the overlay before running the remediation playbook. I believe we do the same for Insights (foreman_rh_cloud plugin), so hopefully the same logic applies there (@Partha_Aji do you concur there?)
Transient packages will cause bootc clients to drift from their container images. Users should know about this since a reboot will wipe out all of those new packages.
To start, we are planning on adding warnings to the UI that packages will be installed transiently:
- For host details, in the content → packages/errata tab (2 banners total)
- For all hosts, in the install package/errata wizard (2 banners total)
- For registration, in the Hosts > Register host UI
- For openscap remediations, in the remediation wizard
However, eventually, it would be best for Katello to also report on which packages were installed transiently. On the UI this would likely be easy, since we would just need to add a new icon or column on the table that shows a host’s packages on the host details page. Getting this information is difficult, however. At the moment, the only history of packages being installed transiently is via the commands saved by dnf history, which is unreliable because it doesn’t include packages installed via the normal bootc usr-overlay without dnf --transient.
If there is eventually a documented way to query transient package info from libdnf, RHSM could potentially add that to the package profile so Katello can consume it. I would be curious to hear if anyone has any other ideas for tracking which packages were installed transiently.
Once we know what packages are transiently installed, we can give users reports on transient packages per host and potentially instruct them on how to build these packages into their next container image. Or, we could give them a command to uninstall the transient images in case they’d like to return to the base state. In the very least, however, users should be able easily see what packages will disappear at the next reboot.
– Ian