Setup Foreman+Kartello for bootc container provisioning on bare metal during hackathon (EU OS)

Dear all,

I am the project lead of EU OS. EU OS has as a goal to build a proof of concept on how public sector organisations could leverage modern FOSS software to migrate to Linux on the Desktop. This is a hot topic in public administrations world wide, because the end of life of Microsoft Windows 10 and Office 2016 is in October 2025. The Microsoft successor products increase the dependency on the US cloud services, which poses for certain use cases issues: think of data protection authorities or competition authorities working on cases linked to Microsoft. Some courts also reconsider their dependency on the US cloud (ICC example).

31 March to 4 June, I meet with volunteers in Paris with the aim to have on Wednesday a setup to

  • build custom OCI containers (with bootc and blue build),
  • provision them on some test devices (read: Thinkpads),
  • monitor them,
  • update them, as well as
  • document everything in upstream documentation and blog posts and on the website https://eu-os.eu

Current event planning:

I believe that theforeman+kartello is key and I have zero experience with it. @iballou sent me already some pointers. Thanks for this.

Call for participation (on-site in Paris or online): Please let me know if you are interested to help us setup foreman+kartello for a bootc-based desktop setup.

Tutorial: It would be tremendously helpful if the foreman community could support our endeavour and offer a tutorial with granular steps that we can follow during the event days. Such a tutorial could contain some recommendations:

  1. which version of theforman+kartello should we use for our PoC? 3.14?
  2. would you recommend to setup theforeman using the quick install or using podman quadlets
  3. there are several ways on how to provision bare metal computers with kartello. What is the best way for such a PoC demo? I guess we have some university wifi, can bring a router and use some cloud VMs.

Follow this project: Please subscribe already to this topic if you can help and/or are curious/interested.

4 Likes

One more topic popped up:

How would we do configuration management (think of printing/cups)? Of course, this could be packaged as part of the OCI container, but maybe other options offer more flexibility?

Unfortunately I am on vacation during this time, so I can not directly support you. But at least I have some links and tips.

One additional source for knowledge I can recommend is our publicly available training material.

For the version I would always use the latest, stable release, so Foreman 3.14 / Katello 4.16 would be fine.

For reproducibility I recommend Ansible, theforeman-operations will help with the setup even if you need to construct the installer parameters and theforeman-foreman does the setup normally done in the UI. So this would be the quick install in automated fashion. Quadlet is still the future, so keep this in mind, but I would not use it yet.

Finding the best way for provisioning is the most difficult part of projects. Typically I recommend for bare metal the discovery plugin. If you only have wifi it could be complicated to get the installer to the system as (from my experience at least) systems can not boot from wifi. So booting from the discovery image via usb and then accessing the installer, answer files and everything else from Foreman via http could work. I prefer some wires and classic PXE installation if possible as it is the most simple solution.

I already recommended Ansible for the setup and I like it also as configuration management, but I am not sure about the combination with OCI containers. When using it for clients which are not always up, you will face the problem when, how often, … to run it so you are sure every system got configured. When using Ansible and Foreman’s Remote Execution you will at least see which ones fail and why with the option to re-run.

1 Like

Thank you @Dirk for the hints! Indeed, for desktop laptop fleets one cannot assume that hosts are always on. I tried to see if ansible/salt/puppet would have an advantage over the other for configuration management using pulling instead of pushing.

I understand that with pull-mqtt and yggdrasil, foreman supports pulling of jobs. So it should work?!

https://docs.theforeman.org/3.14/Managing_Configurations_Ansible/index-foreman-el.html#transport-modes-for-remote-execution_ansible

Currently it works only for scripts, not for ansible. @aruzicka showed some hacks and current state of development to support also ansible in pull mode. If pulling is required he can give you perhaps an update on this.

1 Like

Hello,

there are several options where Foreman can be helpful. I have to say first, this software is built for managing servers installed from traditional repositories, so everything is centered around that. That does not mean you cannot achieve laptop bootc installations tho.

First off, Foreman does not have any native support for building OS images yet. As you stated, you plan to use some other software to do that. My understanding is that the target OS image you want to deploy will be bootable container aka bootc. That is great because Red Hat Anaconda directly supports bootc deployments. While Foreman does not come with a kickstart template for that, this can change as it is really matter of updating a kickstart and providing the necessary ostreecontainer option.

Second, the content capabilities of Katello plugin can be helpful in order to synchronize OCI repositories. This is how Foreman/Katello/Pulp can be used to host OCI container images and push out updates in a controlled manner. Then Ansible or SSH Remote Execution can be used to trigger bootc updates.

When it comes to provisioning, there are many options. Practically any workflow that works with Fedora/CentOS/RHEL will work too. You can use PXE to boot into Anaconda, you can boot a bootable image via bootdisk. As long as the kickstart template is correct, that should work just fine.

Alternatively, you can also use bootc-image-builder to turn OCI repository into a self-contained ISO file that can be written to USB drive where unattended installation can be performed from. This could be a good fit for laptops which are sometimes incapable of network booting over WiFi, or it is not practical to do so.

2 Likes

I have found an internal document which I created more than year ago, I cleaned it up and published here. That might be quite helpful:

3 Likes