Tutorial: Install CentOS bootable container using Foreman over PXE
Environment: Katello nightly (1/2024)
This is a short tutorial from January 2024 briefly describing how to install bootable containers with Foreman.
Content setup
RPM repository
Created new product and new RPM repository from Fedora ELN BaseOS upstream. Synchronized the repository.
Problem: Pulp synchronizes the files correctly, however, Katello fails to perform post-processing tasks because “ELN” cannot be stored in the Operating System database field Version (integer).
Workaround: I manually created an Operating System entry and associated all templates.
Problem: With On Demand policy, PXE files get corrupted the next day in case of compose repositories which are regenerated on a daily basis. This can be either caused by Foreman Proxy (it performs re-download of existing file via “continue” wget operation), or by Pulp.
Workaround: Changed Installation media URL to upstream instead of Pulp. Could be probably solved by changing to Immediate policy, but I did not want to sync the whole repo just for this research as the demo system is shared across the whole team.
Container repository
Created a new “docker” repository within the product with upstream set to Fedora ELN bootable container.
Problem: Pulp is not configured to publish container repositories publicly, authentication is needed in order to consume the content. Not necessarily a problem, but in provisioning, usernames/passwords/tokens are passed to provisioning systems via clear text and RPM installations do also use public repositories published via HTTP.
Workaround: A katello username/password will be used to authenticate to Pulp.
Provisioning setup
Kickstart template
Created a new kickstart template of “provisioning” kind and associated it with the Operating System created below (you need to return to this screen later). The full template text.
Problem: I was not sure about the syntax of the ostreecontainer kickstart command, according to docs it should support the URL format of ostree native containers, but I haven’t tested that.
Solution: I used skopeo to login and copy the container locally to tmpfs overlay file system.
Installation media and OS
Note: This step can be skipped once the mentioned problems with Katello kickstart tree version (“ELN”) is resolved.
Created the following Installation Media with URL to upstream not Pulp because of the PXE files corruption issue.
Created Operating System entry. Note the Major Version must be higher than 32 for Fedora or 9 for RHEL, otherwise incorrect (and unsupported) kernel command line options will be rendered (e.g. “ks” instead “inst.ks”):
Selected the installation media created above:
Provisioning
I used libvirt Compute Resource VM with 5 GB RAM. Make sure to use at least 2GB + size of the bootable container. I created a new host with the specified Operating System, Installation Media and Provisioning Template.
Important step is to create a Host Parameter named “kickstart_kernel_custom_options” with “inst.stage2=URL” with URL pointing to the kickstart repository. This ensures that the anaconda will be able to locate the stage2 (install.img) file even with kickstart which does not contain a repository URL.
The system boots via PXELinux:
The system took a while to boot because stage2 is downloaded from upstream URL (Brno to US), this is only because of my workaround, when using Pulp it will be faster as mirrored content will be picked:
Anaconda executes the pre-installation step where skopeo copies the whole container into memory:
And the installation proceeds then:
Anaconda installs the bootloader (this was a BIOS system) and reboots:
The system comes back running from a bootable container:
Conclusion
The workaround is possible, few problems need to be resolved in order to make this easier for customers.
- Provisioning from a container image via PXE works directly and no disk image is required.
- This will apply to virt and bare metal.
- The majority of issues encountered were with Fedora ELN and not bootable containers itself.
- We will need some (minor) provisioning template updates to support provisioning from a bootable container.
10 MIN DEMO: https://youtu.be/a4I50sLunPg
Note: When I say “bifrost” on the record what I really mean is “bootable container”.