Foreman and bootable containers

Community
1

Tutorial: Install CentOS bootable container using Foreman over PXE

Environment: Katello nightly (1/2024)

This is a short tutorial from January 2024 briefly describing how to install bootable containers with Foreman.

Content setup

RPM repository

Created new product and new RPM repository from Fedora ELN BaseOS upstream. Synchronized the repository.

1220×1600 182 KB

Problem: Pulp synchronizes the files correctly, however, Katello fails to perform post-processing tasks because “ELN” cannot be stored in the Operating System database field Version (integer).

Workaround: I manually created an Operating System entry and associated all templates.

Problem: With On Demand policy, PXE files get corrupted the next day in case of compose repositories which are regenerated on a daily basis. This can be either caused by Foreman Proxy (it performs re-download of existing file via “continue” wget operation), or by Pulp.

Workaround: Changed Installation media URL to upstream instead of Pulp. Could be probably solved by changing to Immediate policy, but I did not want to sync the whole repo just for this research as the demo system is shared across the whole team.

Container repository

Created a new “docker” repository within the product with upstream set to Fedora ELN bootable container.

1218×1212 125 KB

Problem: Pulp is not configured to publish container repositories publicly, authentication is needed in order to consume the content. Not necessarily a problem, but in provisioning, usernames/passwords/tokens are passed to provisioning systems via clear text and RPM installations do also use public repositories published via HTTP.

Workaround: A katello username/password will be used to authenticate to Pulp.

Provisioning setup

Kickstart template

Created a new kickstart template of “provisioning” kind and associated it with the Operating System created below (you need to return to this screen later). The full template text.

1600×706 133 KB

Problem: I was not sure about the syntax of the ostreecontainer kickstart command, according to docs it should support the URL format of ostree native containers, but I haven’t tested that.

Solution: I used skopeo to login and copy the container locally to tmpfs overlay file system.

Installation media and OS

Note: This step can be skipped once the mentioned problems with Katello kickstart tree version (“ELN”) is resolved.

Created the following Installation Media with URL to upstream not Pulp because of the PXE files corruption issue.

1600×558 89.6 KB

Created Operating System entry. Note the Major Version must be higher than 32 for Fedora or 9 for RHEL, otherwise incorrect (and unsupported) kernel command line options will be rendered (e.g. “ks” instead “inst.ks”):

1600×823 87.7 KB

Selected the installation media created above:

1600×749 72.1 KB

Provisioning

I used libvirt Compute Resource VM with 5 GB RAM. Make sure to use at least 2GB + size of the bootable container. I created a new host with the specified Operating System, Installation Media and Provisioning Template.

Important step is to create a Host Parameter named “kickstart_kernel_custom_options” with “inst.stage2=URL” with URL pointing to the kickstart repository. This ensures that the anaconda will be able to locate the stage2 (install.img) file even with kickstart which does not contain a repository URL.

The system boots via PXELinux:

1486×850 9.19 KB

The system took a while to boot because stage2 is downloaded from upstream URL (Brno to US), this is only because of my workaround, when using Pulp it will be faster as mirrored content will be picked:

1600×1195 399 KB

Anaconda executes the pre-installation step where skopeo copies the whole container into memory:

1600×1189 135 KB

And the installation proceeds then:

1600×1205 220 KB

Anaconda installs the bootloader (this was a BIOS system) and reboots:

1600×1198 163 KB

The system comes back running from a bootable container:

1600×1207 196 KB

Conclusion

The workaround is possible, few problems need to be resolved in order to make this easier for customers.

  • Provisioning from a container image via PXE works directly and no disk image is required.
  • This will apply to virt and bare metal.
  • The majority of issues encountered were with Fedora ELN and not bootable containers itself.
  • We will need some (minor) provisioning template updates to support provisioning from a bootable container.

10 MIN DEMO: https://youtu.be/a4I50sLunPg

Note: When I say “bifrost” on the record what I really mean is “bootable container”.

9 Likes
2

@lzap thanks for this tutorial it helped me a lot.

I’m trying to install a rhel based system and ran in the problem that the system fails on bootloader install because it can’t find the rpmdb.
After some investigation i found that /mnt/sysroot/var/lib/rpm links to /mnt/sysroot/usr/share/rpm which was missing.
Did you have similar issues? Do you have an idea how to solve this?

3

Ahh as a little info I try to instal an EFI system. I resolved the first problem when I created the missing dir in the %pre-install section of my kickstart file.

4

I found the issue. We partition /var and other filesystems, which result in a issue because / and /home are mounted on /mnt/sysimage and /var when partitioned is mounted on /mnt/sysroot.
This leads to a confusion for the bootloader installation.

5

Hi @ochnerd ,

for Provisioning of EU OS | EU OS I also want to explore this topic. Would you be able to share your kickstart file?

@lzap , you wrote:

Can you please elaborate a bit what you mean? Have you created already an issue to track this task?

In our project, we have been using this kickstart file:

https://gitlab.com/eu-os/workspace-images/eu-os-base-demo/-/blob/hackdays2025/config.toml?ref_type=heads

Excerpt:

# OSTree container setup
ostreecontainer --transport="oci" --url="/run/install/repo/container"

/run/install/repo/container is a standard location of the tool to create the iso. However, I understood any remote OCI registry would work here.

Best,
Robert

6

Hey, there is no issue I think yet. What I meant was to be able to use Foreman’s default templates, these would need changes. So use the one I provided in this tutorial in the meantime and update it as needed.

In order to use some of Foreman’s capabilities (e.g. remote execution) there are snippets that must be inserted into the template. But it is totally fine to use the one you have if you do not need any of these features.

@ochnerd that was a question for bootc people rather than Foreman community.

Apologies for the late reply. Cheers.

1 Like