Next for Foreman Discovery: immutable OS idea

Development
,
1

Although my priorities for the upcoming months will be mostly PQC, I would like to continue researching something that I started a while ago. In this thread, I would like to present an idea of where to take Foreman Discovery next. Please share your opinions, because this is a radical refactoring of how FDI works today.

The original FDI was created by Greg and it was small: based on Tiny Core Linux. It was simply one script, facter, smart proxy. Then I took over. I had to prepare it for downstream. We chose CentOS/RHEL as the OS, and FDI lived for quite some time through big changes like NetworkManager and systemd. Over the years, the current FDI has accumulated a complex mix of shell scripts, systemd units, Ruby code, and various features.

Edit: This is a 2nd, vastly improved, version of the design.

The main goal

The objective is to develop a next-generation FDI that is easy to build, customize, and configure. Key aspects:

  • Minimum Code in Host OS: Because making a new release of FDI is such a painful workflow, the main motivation is to keep the host OS as minimal as possible. The system will contain only the kernel, base tools, and podman. All the code logic (gathering facts, check-in, rebooting) will be shipped as regular containers.

  • Simple Configuration: Instead of relying on numerous kernel command-line options, the host will download an Ignition configuration from Foreman at boot (see Workflow). Quadlet container definitions are set in that Ignition; check-in responses supply runtime actions and environment variables.

  • Secure by Default: By default, only containers signed by Red Hat or Foreman keys will be executed. This restriction can be disabled via a kernel command-line flag. The communication is pull-based with CA validation and mTLS.

  • More frequent updates: By moving all of the code logic into containers, fixing a bug or bringing a new feature is as easy as resyncing OCI containers from Foreman repos and rebooting.

While a Fedora/CentOS/RHEL bootable container would be sufficient because the base image already contains everything that is needed (namely systemd and podman), the operating system of choice is Fedora CoreOS or RHCOS because it not only provides a stable platform with systemd and podman, but also comes with Ignition bootstrap support. This YAML/JSON language will be used to configure the system (networking, hostname, CA certs) as well as the discovery logic (containers to be downloaded and executed). It comes with tooling that allows embedding the configuration into the image (both PXE and ISO), which can then be overridden by users if needed (networks with VLANs/bonds, etc.).

There will be no TUI interface. Users can configure the network more flexibly via Ignition and have the option to either configure it from Foreman templates or use the CoreOS customize command to embed the configuration into the image.

Container provided by Foreman

The Foreman team will build and maintain one container named foreman-discovery that will host several commands (aka “actions”). There is no inbound API on the discovery node; discovery-checkin calls the Smart Proxy over HTTPS and uses D-Bus to trigger local systemd oneshot services.

Previously, discovery ran an API and had to generate a TLS certificate on the fly. The new design is pull-based, which is not only simpler but also more secure. The following (container) binaries will be configured:

Action: checkin

Binary: /usr/local/bin/discovery-checkin

Mode: privileged

Runs: timer (interval from Foreman setting, rendered into Ignition)

Logic: Collects basic system facts (CPU, memory, disks, network cards/MACs/IPs) and computes a stable node identity string (see New Discovery check-in API). Reports identity and facts to the Smart Proxy (Capsule) via mTLS over HTTPS. Uses a new check-in API that not only collects facts but also responds with an action to take. When an action is returned, dispatches it on the host via the contract described in Host action dispatch. All executables from /usr/local/libexec/facts will also be mounted, executed, and reported as facts.

Action: reboot

Binary: /usr/local/bin/discovery-reboot

Mode: privileged

Runs: oneshot

Logic: Performs immediate reboot

Host action dispatch

discovery-checkin runs in a privileged container but must start oneshot units in the host’s systemd (PID 1). The interface between the container and the host is fixed and documented so it can be reviewed, tested, and extended without ad hoc systemctl calls or open-ended D-Bus access. Bus will be at unix:path=/run/dbus/system_bus_socket (bind-mounted into the check-in container).

Before calling StartUnit, check-in writes the response key/value pairs to a host path (bind-mounted into the container) at /run/foreman-discovery/action.env

Dispatch sequence:

  1. Complete mTLS check-in with the Smart Proxy.

  2. If action is empty, stop.

  3. If action is not in the whitelist, log and stop.

  4. Acquire an exclusive lock on /run/foreman-discovery/dispatch.lock (e.g. flock).

  5. Write /run/foreman-discovery/action.env.

  6. Call StartUnit for discovery-<action>.service with mode replace.

  7. Release the lock.

  8. If the unit is already active (e.g. reboot in progress), log and do not start a second instance.

Required container mounts for discovery-checkin:

  • /run/dbus/system_bus_socket — system D-Bus

  • /run/foreman-discovery/ — environment file and lock (host tmpfs, created by Ignition)

  • /sys/class/dmi/id/ — DMI identifiers for node identity (read-only)

  • /usr/local/libexec/facts/ — custom fact scripts (read-only)

Privileged mode and host network are required for fact collection and the mounts above, not for unrestricted host access.

While this design allows for various functionality to be built, for example running kexec instead of reboot, or calling bootc install to install a complete bootc OS from a custom container, this is out of scope for the initial version.

New Discovery check-in API (Smart Proxy)

A new check-in API will be implemented on the Smart Proxy (Discovery Proxy plugin required). Discovered nodes talk only to the proxy; the plugin forwards to Foreman on the backend. This will vastly simplify the configuration matrix we used to have (Foreman vs Proxy, different ports, callbacks).

Since we are migrating off Facter, some auto-discovery rules will cease working and rules will need to be updated manually. We will provide far fewer facts than Facter. Users will be able to provide their own facts easily via custom scripts in /usr/local/libexec/facts.

Authentication is done via shared client certificate (embedded in Ignition, validated by the proxy via authorize_with_ssl_client). The client cert is maintained and rotated by the installer.

Request:

  • Identity: base62-encoded CRC-64/ECMA of canonical hardware string (see Node identity)

  • Booted MAC address (if BOOTIF was present), also included in facts

  • List of facts and values

Response:

  • Action to run, or an empty string for a no-op.

  • Optional list of key-value options as ENV variables for the command.

Changes will be made to the Foreman API as well, both APIs are expected to be major version bump.

Node identity

Primary key for discovered hosts: base62(CRC-64/ECMA(canonical)) — 64 bits, typically ~11 characters, alphanumeric ([0-9A-Za-z]), shorter than a 12-character MAC (AABBCCDDEEFF). Stable across reboot and re-PXE on the same machine.

Canonical string: DMI fields from /sys/class/dmi/id/ in fixed order (product_uuid, product_serial, board_serial, chassis_serial; skip empty), then all non-loopback MACs sorted lexicographically (lowercase, colon-separated). Exact formatting is fixed in implementation and must not change across releases without a version bump.

Installation

  • The installer downloads the official Fedora CoreOS PXE image (and kernel and initramdisk)

  • The installer uses coreos-installer pxe customize to create a customized PXE image with Foreman/Katello CA cert as --ignition-ca (trust anchor for Ignition fetched at boot; see Workflow)

  • Foreman contains a setting with an image reference prefix for discovery, which defaults to quay.io/foreman/foreman-discovery

  • Foreman contains a discovery_checkin_interval setting (seconds between check-ins; default 60) rendered by the provisioning template into the check-in systemd timer

  • A provisioning template uses the setting to construct the final Ignition YAML/JSON

  • The same image can be used to provision against any Foreman Proxy (Satellite Capsule)

The template will be common for all hosts. Foreman already supports Ignition transpiling (YAML > JSON) and this functionality will be used. Users are advised to change that setting to a local mirror or install Katello, which will do this automatically.

Configuration

The Ignition configuration will contain the following:

  • Hostname

  • Remote Execution SSH key (if installed) for root (optional)

  • Network configuration: DHCP by default, but users can change it

  • Foreman/Katello CA file that goes into system bundle

  • Client cert and key for Smart Proxy mTLS (/etc/foreman/client_{cert,key}.pem)

  • Foreman GPG release key for podman

  • Systemd quadlet configuration for all containers (including check-in timer interval from discovery_checkin_interval)

  • Custom facts as executable scripts (/usr/local/libexec/facts)

  • Host state directory /run/foreman-discovery/ (tmpfs) for action dispatch (see Host action dispatch)

Workflow

  • Host boots via PXE/HTTP/HTTPS or via removable media (ISO)

  • The PXE image contains the CA cert (--ignition-ca); therefore, a kernel command-line option that points to the Foreman provisioning template is safe to use

  • Host fetches Ignition from the template URL over HTTPS; Ignition configures the live root image in RAM

  • Systemd (Quadlets) downloads the container image(s) from the ref provided in Ignition (Foreman template / Foreman setting)

  • The check-in service is scheduled to run for the first time, reports facts and gets the result.

  • With auto-provisioning, result can be immediate reboot.

  • The check-in service repeats until “reboot” command is issued.

Security

Bootstrap (Ignition download) is HTTPS with CA validation. Once the system has booted, all communication is pull-based mTLS over HTTPS.

By default, SSH is enabled and listening, but only SSH key authorization is allowed for root. SSH can be disabled.

SELinux is turned on and enforcing, policy is modified if needed so containers still run in individual domains.

Hardware Limitations

Because the host OS runs entirely from memory, the discovery process will require some extra memory. This is no different from the current FDI.

  • CoreOS takes about 1 GB of RAM for the squashfs image.

  • The foreman-discovery container, based on Fedora Minimal, takes approximately 400 MB.

  • Sufficient memory must remain available for standard OS operations.

Expect a minimum of approximately 3 GiB of RAM.

Deprecated features

  • TUI. Use Ignition for more detailed network setup; configuration can be embedded into the image. Alternatively, set a root password, log in, and use nmcli to configure the network.

  • TUI custom facts editor. Use custom fact scripts in /usr/local/libexec/facts instead (see Configuration).

  • ZIP extensions. Not needed; configure a custom container instead.

  • Kexec. This is no longer a supported feature. Users can build their own containers with kexec if they need to.

Links

1 Like
2

Wildly incomplete, but this part jumped out to me.

Would it make sense to rely on systemd socket activation? That actually works with quadlets: systemd will open the socket and start the service as needed. In foremanctl we set this up for both Foreman (Puma) and Pulp (gunicorn). Since systemd can do this, it can enforce permissions. For foremanctl this means the unix socket is owned by Apache on the host side. The container only gets a file descriptor.

3

I think FDI is the best component to start the idea of immutable hosts to run smart-proxy related tasks. @ekohl and I have talked about the idea of exploring the container delivery with bootc for quite some time now, not using quadlet for FDI make a lot sense as well unless we get the chain of services really tight based on how services/unit should start to avoid racing conditions that I remember seeing with Openshift Immutable nodes in the past.

4

Only the “api” action actually opens a HTTPS port, all the other actions are trigger only. It could be done, but I wonder how this works when image needs to be downloaded first. The initial connection will stall until the container is fully downloaded and started.

One of the best features of the new design is that all the “actions” are fully configurable. Users can create their own actions, including those which will be auto-started. This means these must be dynamically created - I explicitly do not want to bake any unnecessary logic into the OS bootc container. Thus I would like to avoid calling systemctl daemon-reload during OS boot - I wonder if this could mess up the booting process.

This could be solved by waiting until OS is fully booted and then performing the bootstrap (configuration of systemd services). If we go this route, we might as well build the whole thing on quadlets.

Do not get me wrong, I really like how foremanctl works and how quadlets are utilized, the only concern is messing around with booting system. If we can figure that one out, I think deploying quadlets is superior design to ad-hoc starting via systemd-run.

Perhaps waiting until multi-user.target is a good option. What other benefits we can get from using quadlets?

5

One thing to consider is SecureBoot which requires a different FDI depending on the OS to provision as switching to an Ubuntu from a RHEL based FDI via kexec will not work if my understanding is correct. A full boot cycle should work which should work if we have an environment where we have full control over DHCP, but from my experience this is not always the case.
@Jan has very likely the best knowledge about this when I remember past topics.

I really like the idea as a way to modernize this part!

6

This sounds really similar to the Fedora CoreOS Live ISO, which I would link to but I can’t because my forum trust level isn’t high enough to post links.

It is already a system built from a bootc-style container image.

I think it’s been really elegant and working well for CoreOS derivatives to have a clean story where the “live ISO” Is the same as the installed operating system by default - both can be configured via Ignition. And note for the “install to disk” case it’s interesting because the “first Ignition” typically helps bootstrap the “installed system Ignition”.

7

Interesting:

I will research this further, thanks for the info.

My only concern is downstream - I am not aware of any Red Hat base CoreOS image I could use for Satellite. I am not too familiar with CoreOS, but I think it is used for some layered products (RHOS), so I am a bit confused on how this would work. We do not want to be building base image from scratch I guess.

What I really like:

  • Ability to configure network with ignition (this is exactly what some users would like to do)
  • Configuring quadlets via ignition (AFAIK ignition runs before systemd starts booting the system so it solves our problem)
8

This means the discovery depends on Katello right? I’m not against at all, just a new requirement.

What keys are used for this? Are the public keys added during the build time? Is this a single Foreman instance or Foreman organization key in case of Foreman keys?

How does the mTLS work if this is self-signed and generated live? How will Foreman trust this? Perhaps a sequence diagram would be useful here.

shouldn’t we drop this already? This was always problematic for our use case.

Since this will replace the old discovery/FDI, what features of the old solution would be lost besides TUI? I guess patching with zip is one. It would be interesting to see some list of features and their “destiny” like implemented, implementable, dropped.

Thanks @lzap, improvements and modernization in this area are always welcome!

9

Well this was just my chain of thought in the Satellite world, this can be done without Katello, however, defaulting to downloading from the internet every single time a discovered node boots is something we should definitely not do (we might put unnecessary stress on quay.io specifically our organization).

The podman on the host will be configured to deny by default policy where only containers signed by RH GPG and Foreman GPG are accepted. If I am not mistaken, we can use our regular RPM GPG key to sign our containers during release process.

Nothing is generated live, except the configuration JSON. The initial request is not mTLS in the current design. This is the typical chicken and egg problem, while users can modify the container more easily now, the default deployment will simply use what is shipped with Foreman (and signed by Foreman team). Thus we cannot embedd any client TLS certificate there.

The attack surface shrinks down tho, an attacker can craft a JSON configuration with malicious image refs, but the podman will only execute signed containers. Once the configuration is deployed, all the communication is mTLS (because the client cert is part of it).

The assumption is that in a EFI HTTPS SecureBoot environment, the whole host image is secure. A possible client cert could be embedded in the ISO if we are going to implement my other proposal about fully secure HTTPS provisioning. I am open for other ideas on how to further secure this.

That is my proposal, yeah. We can probably build a container with kexec upstream for users to manually configure this if they want, but it whould be not in the default configuration.

These are:

  • ZIPs (not needed use own containers)
  • SSH + root password (customize the image if needed, we will support build ENV variable)
  • TUI (we can build a simple CLI or base the image on CoreOS which has a rich networking configurability)
  • DNS SRV discovery (we can hardcode the functionality into the base image tho)
  • VLAN (configuring image with VLAN tagging - if we use CoreOS that should be covered)
  • various irrelevant hacky options (cachefacts, ipwait, nmwait)
1 Like
10

One idea just came to my mind tho. If we build this on top of CoreOS, then there are tools that can embed ignition easily into the image. It supports embedding everything, or just the CA cert: Customizing installation | CoreOS Installer

This is very interesting, our installer could simply generate such image based on one ignition configuration (downloaded from Foreman). We would use the official Fedora CoreOS image and all the functionality would live as containers, including the main “dispatcher” that starts/stops containers on-demand (this is doable via privileged process + exposing systemd D-Bus).

This is a slight departure from my initial idea, it is just better version:

  • Minimum Code in Host OS: yes, in fact, zero code running on the host. Everything is in containers.
  • Simple configuration: yes, instead a custom JSON, ignition is used. It is flexible, it has a parser, validator and it is widely used with many features (network configuration, file customizations). Also it runs early so no problems dropping systemd units and quadlets.
  • Security by Default: yes, embedding ignition even solves chicken and egg problem. The system boots without the need of doing insecure communication - mTLS is used from the beginning.
  • More frequent updates: yes, no change here. Everything is a container.

By building this on CoreOS, we get to choose from many image types for various platforms: ISO, PXE, AWS, Azure etc. Both ISO and PXE images are also small: just 1 GiB.

For downstream (Satellite), we would use Red Hat CoreOS which is I think just slightly modified Fedora CoreOS. I mean, the image has the same size so I guess all we need is there (podman, systemd, ignition): https://mirror.openshift.com/pub/openshift-v5/x86_64/dependencies/rhcos/latest/

I will explore this further. I will likely rewrite my proposal to use CoreOS.

1 Like
11

I have updated the design to build it solely on Fedora CoreOS (or RHCOS downstream). Everything clicks together. I further simplified the communication pattern - everything is now pull-based. I tightened security by embedding Foreman CA and Foreman client key/cert in the image so the communication is fully mTLS.

1 Like