Hello,
I am being hammered with questions from our users about Ubuntu Focal (20.04) and later. If you don’t follow this, Canonical made a breaking change in this release: debian-installer is no longer used for installations and LiveCD must be booted instead with cloud-init used to do the provisioning.
This alone would not be a huge issue for Foreman, but there is a change that comes with it. Boot files which are required to boot such LiveCD (kernel/initramdisk) are not published via HTTP(S) protocol directly, instead, users are asked to download the whole LiveCD ISO (1.2GB), mount it and extract it into the TFTP/HTTP directories.
This does not work well with our current design when Foreman “understands” directory structures of various Linux distributions and then hands over URLs to smart proxy when OS needs to be installed. Smart proxy then downloads both files into the TFTP/HTTPBoot directory.
My initial solution to the Ubuntu problem was to implement the same behavior but for ISO file, it could be downloaded, extracted (we cannot use mount tho as smart proxy runs rootless) and technically this could be possible. However, Foreman performs this download on every host creation (or build operation). Smart proxy actually spawns curl (previously wget) and gives it parameters in a way that it should skip download if file is not newer than existing one, but the mirror the file is being downloaded from need to support some specific HTTP headers and this is not guaranteed. In the worst case, 1.2GB would be installed on every host entering a build mode. Also, I don’t like spawning many commands or extracting big tarballs on a API request.
Foreman PXE files downloading has always been problematic in its design. There are several flaws:
- Download process is asynchronous and we see very often systems booting into yet downloading kernel/initramdisk.
- Re-download of PXE files is initiated on every new build/host. This is not needed, in reality these PXE files almost never change. Typically they only change on every major RHEL release (e.g. 8.1, 8.2). Granted CentOS Stream has changed it - this OS actually rebuilds the PXE files quite often.
- When Katello is used and kickstart files are being promoted, the filenames which are derived from the source URLs are changing all the time creating many unnecessary files in the PXE/HTTPBoot directory. Those files are almost always the same and Foreman or Smart Proxy never cleans them.
For these reasons, I propose to radically change how PXE files are being deployed to the TFTP/HTTPBoot smart proxies. I believe that Foreman should NOT download them at the time when a host is created, but when an Operating System is created. Here is what I propose:
- Operating System model will have a new Template kind associated: Boot files (find a better name).
- The new template will use Foreman ERB system and will render to a shell script.
- This shell script will contain necessary commands to setup PXE/HTTPBoot environment (typically
wgetorcurlthe boot files from upstream repo) - Since ERB can be used, information provided by Foreman (Installation Media / Katello repository, OS Name, Release, Version…) can be used to construct the proper URL.
- When OS is created or updated (or deleted even), template is rendered and passed to a new API Smart Proxy call.
- Smart Proxy executes the script and sets up the environment.
- All TFTP orchestration code can be now dropped from the host creation as files should be in-place before any host is even created.
- The call could support both synchronous and async modes. When triggered via OS edit, it would do this on the background. When triggered via a button “Re-download boot files” the UI could actually show STDOUT/STDERR for easier debugging.
We could also leverage Remote Execution or Ansible for this, but I think it is an overkill - 90% of all cases will only need two curl commands.
Alternatively, we could hardcode this into Foreman or Smart Proxy or both. A new field for Ubuntu OS would be probably required in this case: LiveCD URL. That’s because Ubuntu LiveCD URL does not look very stable, they appear to delete old minor versions from the site:
Currently it’s https://releases.ubuntu.com/20.04/ubuntu-20.04.3-live-server-amd64.iso and .2 version is no longer available. I think this will break very often and that’s the reason why I want the new solution to be open and flexible and easy to change.