Hi there,
Thanks for your input and your discussions @lstejska, @lzap and @Jan. I thought about all that was said and rethought the whole principle of how we wanted to add SecureBoot support for various OSes with a new PXE loader. And yes, reinventing the wheel is not what we want. And that again made me think about how to better integrate what we want within what we already have.
I just pushed new commits onto the Foreman and Smart Proxy PRs following a different approach to achieve what we want in a way that better integrates with our long term goals. Please have a look at the PRs in Foreman and Smart Proxy.
To my new approach
In Foreman I now reuse the existing Grub2 PXE loaders and extended them to load the bootloader from a MAC address specific directory. To not break anything the Smart Proxyβs capabilities are queried and in case it doesnβt support the new approach (i.e. itβs version compared to Foreman is X.[Y-1]) the MAC specific part is cut out and everything works as expected.
On the Smart Proxy side I extended the Pxegrub2 class in the Proxy::TFTP module accordingly to copy OS specific grubx64.efi and shimx64.efi files from the bootloader_universe in case it is activated via foreman-installer and the files are present in it. If that is not the case, it defaults to copying the βgenericβ files from the grub2 directory in /var/lib/tftpboot to the MAC specific directory.
With this new approach I was able to remove a lot of complexity and minimize the requested changes.
Also itβs good preparation for moving other bootloader files to the MAC specific directory.
What do you think?
As Jan already mentioned we want to solve this with good documentation for now. The documentation PR is linked above and referenced in the Foreman PR.
@lzap Regarding the version specific directories inside bootloader_universe like centos-9.0. I like that idea and it should be easy to integrate. I will have a look and talk with @Jan.
I will let LeoΕ‘ and others to do line by line review, however, all I want to say is that part of Foreman desperately needs more love and I am not against bigger refactorings. As long as the new code is clean and covered with tests
Great, whatever you come up with, make sure to document it so it is easier to add more OS names, versions and architectures.
Totally welcome. Perhaps I have expressed myself somewhat misleadingly, just wanted to bring discussions back on this concrete technical enhancement and to avoid a bit opening up too many more construction sites (which might also be valid of course).
Yeah, same for SUSE (https://news.opensuse.org/2023/12/20/systemd-fde/). Not quite sure how this should work with PXE boot. From my understanding it supports only βlocal bootβ by reading its loaders from (local) ESP. No option to retrieve remote config/loader via network. For provisioning we still would need GRUB2.
But then we would also need to change boot order in EFI (via efibootmgr to systemd-boot) during OS installation because chainloading does not work with enabled SB (https://www.gnu.org/software/grub/manual/grub/grub.html#UEFI-secure-boot-and-shim). And this would mean that we would lose automated start of new provisioning. For our current GRUB2 implementation we can instead use some sort of chainloader (hd0,gpt1)/efi/dist/grub.cfg (configured in Foremanβs GRUB2 local boot template) to boot from local disk.
We should definitely try to be as flexible and open as possible here to easily adapt to new bootloaders.
At least for EFI we could do that but in first iteration I would like to have it still separated to keep shim binary out of the game at all for SB disabled systems because it loads GRUB2 binary and behaves differently from distribution to distribution. I donβt want to end up with broken network boot because of shim on SB disabled systems.
This is true if we directly boot GRUB2 binary (what we will maybe remove later on to reduce list of PXE loaders). Problem is shim binary here because - depending on distribution - it looks for hardcoded <prefix>/grub<arch>.efi (e.g. for Alma/Rocky/Oracle).
SUSE/SLESβ shim binary wants <prefix>/grub.efi. To be compatible to all distributions (at least we tested manually) we would need to have something like (here for x86_64, filename=00-11-22-33-44-55/grub2/shim.efi):
Analogues for e.g. ARM. I donβt think we want to have special conditions for SLES in Foreman code.
Thatβs btw. another problem that every distributorβs shim/GRUB2 binaries behave slightly different! And that we canβt modify/create them because of signature.
Thatβs part of the documentation PR as stated already by @goarsna. But current version is missing architecture subdirectory, we will rework this of course (good point @lzap, we totally focused on x86_64 only )
Instead of providing shim/GRUB2 binaries for every majon.minor plus architecture we thought about having a default directory with βlatestβ shim/GRUB2 binaries. Keys are mostly βdownward compatibleβ (if not revoked for reasons), there is no strict separation of used keys for e.g. every major version.
But user should be able to optionally provide a major.minor directory for a certain distribution to still be able to provide correct shim/GRUB2 binary if nessecary. Example:
With this we could e.g. also provide proper binaries for RHEL 8.0, 8.1, 8.2 which are then preferably copied to 00-11-22-33-44-55/grub2/.
Good question. We should definitely test this regularly and in an automated way but how/who? I mean you would need some test environment with a Foreman and VMs with SB enabled AND disabled booting up in order to check if it works for all the different distributions.
We agreed that the latest proposal works the best, it does not introduce any new PXE loaders but it modifies the current UEFI ones to create both the legacy and MAC-based configurations.
The longest discussion was about unmanaged DHCP setups, we made sure that it will work with it.
Either symlinking or hardlinking or even plain copy is fine for bootloaders.
The bootloaders structure is sufficient.
In order to fix βentrypointsβ which can differ from one distribution to other, we will consider using symlinks for each version directory. For example, boot-sb.efi could point to shim.efi while boot.efi could point to grubXXX.efi or other bootloader depending on the distro. Then Foreman would need to resolve these βentrypointβ symlink in order to do hardlink (or another symlink). Other option is to keep some sort of metadata (in a JSON or something) which would tell which file should be used for EFI and which for SB-EFI. In any case, this will allow to have much more simple DHCP orchestration code where filename option can be simply MAC/boot-sb.efi and TFTP orchestration does not need to figure out which file is which - just one to one copy should be enough.
The implementation will be done in a way that it will not affect current EL-CentOS-Fedora users while it will vastly improve things for other distributions. Specifically, all must work fine even when there is no bootloader-universe available.
After all PRs are merged, we need to do retesting of all supported workflows with managed and unmanaged DHCP setups.
I am working on the OCI netboot distribution prototype, I will update you soon. This, however, will not affect this work only improve it later.
Thanks for the summary, @lzap (I havenβt managed it the last day).
Correct. At minimum, a bootloader-universe/<dist>/default/<arch>/ directory must exist with latest EFI binaries. If any major+minor combination needs alternative EFI binaries (e.g. older binaries containing revoked keys for older kernel), additional directories with pinned version can be provided which will be used instead for copying to MAC directory:
bootloader-universe/ubuntu/20.04.2/x86_64
βββ boot.efi -> grubx64.efi
βββ boot-sb.efi -> shimx64.efi
βββ grubx64.efi # alternative version of Grub2
βββ shimx64.efi # alternative version of shim
bootloader-universe/ubuntu/default/x86_64
βββ boot.efi -> grubx64.efi
βββ boot-sb.efi -> shimx64.efi
βββ grubx64.efi
βββ shimx64.efi
If multiple versions share same alternative binaries, we can also work with symlinks:
The corresponding bootloader-universe directory (distribution+version+arch) should take care of all files/symlinks needed for PXE boot (see above). Foreman code copies only the entire content to corresponding MAC directory. Having this, Foreman does not need to know anything about specific files.
For now we will describe exact bootloader-universe file structure in documentation, later this can be provided by e.g. OCI netboot distribution.
Correct. Current approach only adds additional MAC directories with distribution specific EFI binaries and GRUB2 configuration files (same as in tftpboot/grub2/ currently). Content of tftpboot/grub2/ β namely grubx64.efi, shimx64.efi, grub.cfg-00:*, grub.cfg-01-* β remains.
This guarantees that provisioning works as it did in the past:
if DHCP is unmanaged or
if Smart Proxy has no arbitrary distribution SB support at all or
if corresponding distribution is not available in bootloader-universe directory or
All GRUB2 configuration files have same content here.
Question:
@goarsna and I thought about using dedicated bootloader-universe directory in TFTP root directory (e.g. /var/lib/tftpboot/booloader-universe). Having this, we could then use relative symlinks for EFI binaries when relative symlinks are supported in Foremanβs Ruby version. EFI binaries are about ~3MB per host at the moment (10000 hosts β ~30G). Do you think itβs worth saving this disk space?
Is this disk space only used during provisioning of the hosts or stay the file even after successful provisioning? If the files stay, yes but I do not think it needs to be the highest priority. If the first it will not add up to such high numbers, so even a lower priority. But in general being conservative and reducing resource usage is always good even if it is not the highest priority.
Files stay until host gets removed/deleted from Foreman, not only during build/provisioning procedure.
As the PR is still work-in-progress (same for documentation) I would rather do it now than later.
I vote for putting it to dedicated /var/lib/tftpboot/bootloader-universe directory.
We can skip setting an arbitrary directory path via foreman-installer, keep PXE files logically together (TFTP root) and can use relatives symlinks in future for a) better overview on filesystem level and b) to save disk space.
Hi there and thanks @lzap and @Jan for your summaries!
I want to add / correct some parts:
In the TFTP root directory the MAC address based directory will be put at first place. I implemented it that way to be future proof in case we want to provide other bootloader files (for example for PXElinux) through the MAC address based directories
If we want to copy the entire directory content (what I absolutely agree that we should do), I would prepend the PXE loader as first level in the directory structure inside the bootloader_universe so that for PXEGrub2 it will look like bootloader_universe/pxegrub2/{os}/{version}/{arch}. By this we will again stay future proof in case we want to use the bootloader_universe directory for other PXE loaders like PXE linux.
I appreciate that idea. This only enforces an extra check for the X.Y to X.(Y-1) compatibility between Foreman and Smart Proxy as (in case DHCP is managed) the new file names boot.efi and boot-sb.efi can only be set as DHCP file name in case the Smart Proxy supports them.
After all a resulting example TFTP root directory for an Ubuntu and a SLES host would look as follows:
Regarding the location of the bootloader_universe I agree with Jan. We should use the advantages of putting the bootloader_universe inside the TFTP root directory.
The boot files do not need to stay for the whole lifetime of a host. When a host exits build mode, TFTP orchestration can be updated to delete these files completely. Today, it only rewrites the config files, it can do additional actions.
Consider using hard links instead of symlinks if symlinks do not work for any reason. I think keeping the βsourceβ files in /usr is better fit than in /var. If Ruby is unable to do relative symlinks, then relative symlinks can be simply copied and they will still work.
Consider exploring using metadata instead of boot*.efi symlinks. This was just one idea, if that is clunky for any reason, we could still create some kind of metadata file in each directory describing those βentrypointsβ in other way. For example, it could be bootentry.json or a config file or anything that is accessible and easily readable.
Do you mean for linking files from the bootloader_universe to the MAC address based directories inside the TFTP root?
As the major reason for this suggestion was to save disk space, we can stick to the current approach with the configurable bootloader_universe if we delete the bootloader files from the MAC address based directories inside the TFTP root once a host exits build mode.
BTW: We could add a suggestion to the docs to use /usr/local/share/bootloader_universe.
I would prefer to not use any metadata for the structure inside the bootloader_universe. As for now Users will have to take care of the bootloader_universe themselves, this will unnecessarily complicate things.
Hey @lstejska, @lzap,and @nofaralfasi,
I wanted to ask for an ACK regarding the location of the MAC address based directory for storing the bootloader files.
I wanted to put these directories directly inside the TFTP root directory to be prepared for furture adjustments when / if we decide to provide other bootloader files like for PXElinux via the MAC address based directories. By doing so I thought we can have one central place to look at for all MAC address based bootloader directories. But this requires changing ownership of the TFTP root directory from root to foreman-proxy* so that the Smart Proxy is able to create the MAC address based directories in it. If we decide so (and that is what I would prefer), I would open a PR against the TFTP puppet module for the required changes.
Another possibility would be to create a new directory, lets name it host_config, inside the TFTP root directory for storing all the MAC address based directories. This could be implemented within my already existing PR for adding the bootloader_universe setting against Foreman Proxy puppet module.
What do you think?
*) I wonder if there is a specific reason this directory is not owned by foreman-proxy.
/var/lib/tftpboot/ is provided by the package tftp-server, so not connected to Foreman at all. We have learned the lesson the hard way to not change the ownership of such files not owned by Foreman with the dhcp integration. So if you need permissions for user foreman-proxy, please use ACLs for this, otherwise an update without running the foreman-installer afterwards will break the system.
Thanks for that information, @Dirk. Then I tend to create a sub directory host_config inside the TFTP root directory. so we donβt have to mess with the TFTP root directory at all.
Hi folks,
I have implemented everything we discussed (and I rebased and squashed my commits). Hopefully I didnβt miss anything.
I did a lot of re-tests (up to now) with AlmaLinux, Oracle Linux, SLES, and Ubuntu and on my side everything worked as expected.
I would be happy if anybody of you, @lstejska, @nofaralfasi, or @lzap, could have a look soonish at my implementation although I know the tests are currently broken, as we would appreciate to get the Secure Boot support into a stable (and cherry-pickable ) state soon. So please ignore all the test files for now.
A few remarks to my changes:
The MAC address based directories are now created in a host_config sub folder inside the TFTP root directory. This folder is created by the Foreman Smart Proxy Puppet module (see my separate PR in puppet-forman_proxy, which is also part of the change set).
OS version is now supported in the bootloader universe. As mentioned above, the paths inside the bootloader universe to place the bootloader files at are ./{os}/{version}/{arch}/ and as fallback ./{os}/default/{arch}/ where version is {major}.{minor} as defined in the operating system.
What doesnβtt work is hard linking bootloader files. We have to copy them as hard linking fails due to missing permissions (the source files are owned by root:root). But IMO as we want to delete the bootloader files anyways as soon as a host exits build mode, we can stick to copying them.
What is missing up to now:
The above mentioned broken tests.
Deleting the bootloader files as soon as a host exits build mode.
Iβll hopefully be able to fix the remaining points soon.
Please let me know if anything is missing or if I got something wrong.
As we still keep/generate host specific GRUB2 configurations in tftpboot/grub2/grub.cfg-<MAC> we do not need to extend/adapt global GRUB2 global default template, or rather the corresponding snippet: pxegrub2_mac.erb
I just used boot and boot-alt symlinks for entrypoints without extension as I believe this will be useful also for non-EFI systems. Let me know what you think.
We discussed this a bit but not sure whether that will help with these symlinks.
ATM some boot and boot-alt symlinks only make somehow sense in this particular case for GRUB2 UEFI with or whithout SecureBoot.
If we think about adding more bootloaders into an image: configuring a default bootloader (via ./nboci push ... --entrypoint shimx64.efi) during image creation wonβt work.
If we want to provide fix βentrypointsβ and want to serve multiple bootloaders we would need something like:
Although I mentioned it once, after our discussion I thought we ruled managing BIOS bootloader out. I mean, it has been pretty stable for decades, there is no added benefit in distributing those. However, you make a point, for consistency it makes sense.
Tho I would like to keep the tool (and the specification) generic and relevant for non-intel platforms where BIOS/EFI/SB does not alway apply. I think I can introduce three symlink βannotationsβ then:
entry point (EFI SB)
alternate entry point (EFI)
legacy entry point (BIOS)
These terms are generic enough so they can be used for other purposes if needed. Would that work?
I want to integrate signing into the nboci tool itself, no need to package oras. That was actually my main goal - to have just single binary for all operations so no additional dependencies would be necessary.
Hey all, thanks for the work on this feature! Is this merged in to the main Foreman branch yet? I might be able to help test if you need it as I am also looking for this functionality.
)