thanks for commenting but I need to clarify some things.
I don’t see where we are going to chainboot (chainload?) grub2 twice. The planned changes do exactly what you proposed: point the “filename” in DHCP for a particular host to shim binary path of the distribution which is going to be installed.
We decided to separate everything in subdirectories (dir name is mac) for a host because of followup load of GRUB2 binary and it’s config file(s). This depends a bit from distribution to distribution. However, a shim referenced with grub2/foobar/shimx64.efi will try to load GRUB2 binary grub2/foobar/grubx64.efi (something hardcoded like basename + /grubx64.efi)
Using “filename” grub2/shimx64.efi-00:11:22:33:44:55 and put grub2/grubx64.efi-00:11:22:33:44:55 next to it won’t work.
True. We actively decided to skip the automation for now and to provide a handy description how to do this manually. I mean the whole topic is quite experimental right now and doing this once by hand would be OK for now. But that’s definitely something we can work on if first SecureBoot support got accepted upstream.
Your “ideal case” is already implemented (besides some minors like deletion and the automatic acquisition of the binaries).
Regarding IPv6 I have to admit that I’ve no experience at all ATM. Is Redfish still alive? I thought this was a stillborn (fun fact: ~10y ago I did some proof-of-concept regarding Redfish + libvirt) . Even if major HW vendors implement this, we would also need this on all supported hypervisors.
I will definitely have a look at this forester.
Anyway, what’s left to discuss? According to my understanding of your post we are on a good track (besides this IPv6 stuff where I need to catch up).
Ah, just a misunderstanding. The filename option on DHCP is a good way indeed.
This is pretty big change, but with the automatic acquisition it would all make sense. It would help to solve one big problem - currently foreman installs grub from the underlaying OS which is sometimes incompatible with Red Hat kernel being deployed. This new feature would help to mitigate this.
Well, yeah, DHCPv6 is a problem. There is no way to create host reservation via MAC address, only via DUID which is random. There are some extensions for MAC-based uids but I never got them working.
Alive indeed, problem is that most vendors support minimum relevant features. Most of the fancy stuff, like “boot from HTTP just once” or “boot this ISO file” are only implemented by few. As much as I would love to, Redfish (and EFI) does not really help to solve problems we are facing in datacenters. It is somewhat similar to IPv6. Things are utterly complex.
Well, we haven’t touched how PXE workflow works since the very beginning. If you want to pursue this idea, which I believe is on good track. It is important to make sure this works 100% on all major distributions including Fedora/RH, Debian/Ubuntu and Suse to name few. A good amount of tests would be necessary.
There is one thing, when host is created, smart proxy downloads PXE files via curl. Everytime those PXE files change (e.g. when you use Fedora Rawhide) curl corrupts those files. Since bootloader will now be not shared anymore (unique per MAC), it would make a lot of sence to refactor also PXE files downloading in a way that it is downloaded per host. To do this efficiently, there must be some kind of cache so files are only downloaded once and then reused. This would be useful for bootloader files as well.
One thing I just realised is BIOS - it would make sense to make this workflow universal for both BIOS and EFI. See, what I would like to see us is making provisioning more simple. Just adding another PXE loader option on top what we have is exactly the opposite.
So the downloader would need to download PXE linux binary from the upstream repository, that is possible for sure in case of Red Hat compatible system:
Thing is, PXE linux is getting removed from Fedora ISO files and what I expect is that it could be also dropped completely from there. Fedora devs state grub2 is the replacement going forward. Problem is, grub2 cannot be easily extracted from the upstream repo:
It needs to be built via grub2-mknetdir or grub2-mkimg, that cannot be done via different version which will be installed on TFTP Smart Proxy. In that case, it would be better to simply keep using pxelinux.0 or grub.0 respectively and copy/hardlink/symlink those files into the MAC-based subdirectories.
At least, the MAC-based directory should be “complete” and work for both BIOS and EFI systems if possible.
For completeness, here is the grub and shim for EFI nodes:
1 Link-layer address plus time
2 Vendor-assigned unique ID based on Enterprise Number
With focus on VMs (which can be created & provisioned) I would expect the first type. Looks like I need to start investigating a bit how this works with Proxmox/libvirt (Qemu) and VMware. In the best case you can even set the DUID via API when the VM gets created.
And regarding baremetal: as of today you need to find out the MAC anyway before you can add any host specific boot instructions. This would be same for DUID.
I see. So not dead, but also not really alive Back then it was traded as the successor to IPMI. But looks like IPMI is still today the defacto standard for remote control of baremetal (at least basic implementation by most vendors).
With PXE files you mean kernel/initrd? Aren’t they already stored in a distinct way in /var/lib/tftpboot/boot/? Don’t see any benefit to change this at the moment. These files are still referenced and used, independently of the actual bootloader.
Yeah, current list of PXE loaders is quite huge. I personally would be total fine with supporting only GRUB2 for EFI (with SecureBoot support) and BIOS. And maybe plus their HTTPboot versions.
We focus on EFI at the moment because there is no SecureBoot for BIOS. But yes, we should make it as uniform as possible. And as easy at possible.
Important: The SecureBoot enabled boot path also works on systems with disabled SecureBoot!
Means, if this is working reliable we can remove “normal” GRUB2/EFI PXE loaders anyway. At least for all distributions providing signed GRUB2 binary with network boot support.
I mean, having a working solution for SecureBoot now might help us in reaching our long term goals:
Independent of the PXE loader (GRUB2, pxelinux, ipxe) and firmware (BIOS, EFI), we should rework provisioning process in a way that host specific PXE files are used in every case using DHCP filename option and e.g. MAC separated directories.
We should implement automatic acquiring of all required PXE files directly from the target OS’ packages. Optimally only doing it once and reusing them wherever it’s needed (e.g. by using symlinks). Cleanup would also be nice after host deletion.
The last time I tried it over PXE there were no LL-DUIDs supported by the PXE firmware, I am curious how this actually works today.
Well, PXE provisioning should always be designed for bare-metal, VM provisioning over PXE is just a bonus.
Yeah, but then the DHCP server must be able to identify the client. I am just little bit worried that DHCPv6 implementations in PXE stacks are poor and do not generally support LL-DUIDs. Once some decent bootloader is loaded, things get usually better, but that is too late as we want to actually decide on the bootloader (shim) beforehand.
But this needs to be tested, I just briefly tried it years ago.
Also there is one hack but that is just misery: https://dhcpy6d.de/ (this special DHCP server uses MAC adress tables to find the client address, will not work over routed networks). I don’t like that at all.
Yeah it is unrelated to the feature you want, yet, I believe this part is worth some refactoring. I believe both files should be downloaded at the same time as the shim/grub pair.
Yeah, thanks for sharing that. We will discuss this within the core team this week.
There are a lot of discussions going on in the last time regarding provisioning processes and I love to see that. But we need to work on these step by step and try to separate them into several parts, without breaking existing functionality.
Given that, we would like to focus again on the current approach we (ATIX) have been working on for some time and we would like to see our changes being accepted upstream.
I don’t want to pre-empt the discussions in the actual PRs but @goarsna and I have agreed already on that we would implement a few more details in order to make the current PR complete.***
At first glance, this adds again an additional PXELoader to Foreman and we all agree on that we actually would like to have less. But with these changes we would:
a) have a first support for provisioning Linux on SecureBoot enabled hosts and
b) we are paving the way for a generally more flexible design of PXE boot for individual hosts.
Having this working reliable, we can then think about (order isn’t fix):
reducing list of PXELoader*
refactoring/automating the way of providing all required PXE boot files**
extend the host-specific NBPs (=DHCP filename) to other PXELoader as well
start looking at IPv6 PXE boot
Please speak up @lzap@ekohl@lstejska if this works for you guys or if we are wrong here.
*) e.g. having only one “Grub2 UEFI” PXELoader which corresponds to the new one
**) e.g. RFC: Distribution of netboot files via OCI registry
***) clean up MAC-directories after host deletion, check for new PXELoader support utilizing SP capabilities, manual testing the other distributions RHEL/Oracle/SLES
Given that, we would like to focus again on the current approach we (ATIX) have been working on for some time and we would like to see our changes being accepted upstream.
Yes, that’s our plan, right now we just have discussions and trying to think about all the cases and problems, but for sure we don’t want to reinvent wheel.
I agree with the design and direction you are taking, tho, I am not the one to decide as I am no longer involved in active development or testing.
One thing that I would like to propose is the PXE loader name tho, the proposed “Grub2 UEFI SecureBoot (target OS)” assumes that the target OS uses grub. This might be true for current Red Hat and Debian based systems, however, Fedora developers are already experimenting with unikernels based on systemd-boot. While the target for today are local booting in cloud environments, it might be possible that in the future there will be no grub.
Edit: Note for The Register - I am not saying anyone is actively working on removing grub2 from the distribution, this is just my intuition that in the far future a different bootloader might be in use.
Additionally, I don’t think “SecureBoot” needs to be in the name directly, it might repel users who do not want to use SecureBoot while this workflow will work perfectly fine without SecureBoot. And in the future with BIOS too. If you want to stress out that it works in SB environments, let’s add a help or UI element to tell it.
Therefore, let’s be bold and come up with a more generic name. Something like:
Target OS EFI - loads AA-BB-CC-DD-EE-FF/boot.efi
Target OS BIOS - loads AA-BB-CC-DD-EE-FF/boot.0
Both files will be just (relative) symlink/hardlink to the correct files which are, this is important, architecture dependent (different link destination for intel system than for arm64). Since every MAC-based directory is host-unique, there is no need to introduce boot-architecture.efi naming patters because architecture is known already.
What this patchset does not explain or solve is distribution of the boot files, the (/usr/local/share/bootloader-universe). This is open question but I think it is a good POC, manual copying is fine, maybe foreman-installer could populate the directory at least for the most popular distributions. What I am still missing is full specification of the filesystem layout in that directory, perhaps a README with that would be great - specifically what are the expected architecture suffixes (is that aarch64 or arm64 or just arm, and world-famous x86_64 vs x64 vs amd64). Stuff like that.
Oh one more thing, I am excited about this work because it solves one particular problem we faced several times before: not using the shim/bootloader from the target OS. In the examples, you show that the bootloader universe contain one bootloader per OS:
Only with this naming convention, it will solve the painful problem of shim from 8.3 not being able to load RHEL 8.0-8.2 because keys were revoked because of a CVE. So I would suggest this is part of the initial version from the day one.
Hi there,
Thanks for your input and your discussions @lstejska, @lzap and @Jan. I thought about all that was said and rethought the whole principle of how we wanted to add SecureBoot support for various OSes with a new PXE loader. And yes, reinventing the wheel is not what we want. And that again made me think about how to better integrate what we want within what we already have.
I just pushed new commits onto the Foreman and Smart Proxy PRs following a different approach to achieve what we want in a way that better integrates with our long term goals. Please have a look at the PRs in Foreman and Smart Proxy.
To my new approach
In Foreman I now reuse the existing Grub2 PXE loaders and extended them to load the bootloader from a MAC address specific directory. To not break anything the Smart Proxy’s capabilities are queried and in case it doesn’t support the new approach (i.e. it’s version compared to Foreman is X.[Y-1]) the MAC specific part is cut out and everything works as expected.
On the Smart Proxy side I extended the Pxegrub2 class in the Proxy::TFTP module accordingly to copy OS specific grubx64.efi and shimx64.efi files from the bootloader_universe in case it is activated via foreman-installer and the files are present in it. If that is not the case, it defaults to copying the “generic” files from the grub2 directory in /var/lib/tftpboot to the MAC specific directory.
With this new approach I was able to remove a lot of complexity and minimize the requested changes.
Also it’s good preparation for moving other bootloader files to the MAC specific directory.
What do you think?
As Jan already mentioned we want to solve this with good documentation for now. The documentation PR is linked above and referenced in the Foreman PR.
@lzap Regarding the version specific directories inside bootloader_universe like centos-9.0. I like that idea and it should be easy to integrate. I will have a look and talk with @Jan.
I will let Leoš and others to do line by line review, however, all I want to say is that part of Foreman desperately needs more love and I am not against bigger refactorings. As long as the new code is clean and covered with tests
Great, whatever you come up with, make sure to document it so it is easier to add more OS names, versions and architectures.
Totally welcome. Perhaps I have expressed myself somewhat misleadingly, just wanted to bring discussions back on this concrete technical enhancement and to avoid a bit opening up too many more construction sites (which might also be valid of course).
Yeah, same for SUSE (https://news.opensuse.org/2023/12/20/systemd-fde/). Not quite sure how this should work with PXE boot. From my understanding it supports only “local boot” by reading its loaders from (local) ESP. No option to retrieve remote config/loader via network. For provisioning we still would need GRUB2.
But then we would also need to change boot order in EFI (via efibootmgr to systemd-boot) during OS installation because chainloading does not work with enabled SB (https://www.gnu.org/software/grub/manual/grub/grub.html#UEFI-secure-boot-and-shim). And this would mean that we would lose automated start of new provisioning. For our current GRUB2 implementation we can instead use some sort of chainloader (hd0,gpt1)/efi/dist/grub.cfg (configured in Foreman’s GRUB2 local boot template) to boot from local disk.
We should definitely try to be as flexible and open as possible here to easily adapt to new bootloaders.
At least for EFI we could do that but in first iteration I would like to have it still separated to keep shim binary out of the game at all for SB disabled systems because it loads GRUB2 binary and behaves differently from distribution to distribution. I don’t want to end up with broken network boot because of shim on SB disabled systems.
This is true if we directly boot GRUB2 binary (what we will maybe remove later on to reduce list of PXE loaders). Problem is shim binary here because - depending on distribution - it looks for hardcoded <prefix>/grub<arch>.efi (e.g. for Alma/Rocky/Oracle).
SUSE/SLES’ shim binary wants <prefix>/grub.efi. To be compatible to all distributions (at least we tested manually) we would need to have something like (here for x86_64, filename=00-11-22-33-44-55/grub2/shim.efi):
Analogues for e.g. ARM. I don’t think we want to have special conditions for SLES in Foreman code.
That’s btw. another problem that every distributor’s shim/GRUB2 binaries behave slightly different! And that we can’t modify/create them because of signature.
That’s part of the documentation PR as stated already by @goarsna. But current version is missing architecture subdirectory, we will rework this of course (good point @lzap, we totally focused on x86_64 only )
Instead of providing shim/GRUB2 binaries for every majon.minor plus architecture we thought about having a default directory with “latest” shim/GRUB2 binaries. Keys are mostly “downward compatible” (if not revoked for reasons), there is no strict separation of used keys for e.g. every major version.
But user should be able to optionally provide a major.minor directory for a certain distribution to still be able to provide correct shim/GRUB2 binary if nessecary. Example:
With this we could e.g. also provide proper binaries for RHEL 8.0, 8.1, 8.2 which are then preferably copied to 00-11-22-33-44-55/grub2/.
Good question. We should definitely test this regularly and in an automated way but how/who? I mean you would need some test environment with a Foreman and VMs with SB enabled AND disabled booting up in order to check if it works for all the different distributions.
We agreed that the latest proposal works the best, it does not introduce any new PXE loaders but it modifies the current UEFI ones to create both the legacy and MAC-based configurations.
The longest discussion was about unmanaged DHCP setups, we made sure that it will work with it.
Either symlinking or hardlinking or even plain copy is fine for bootloaders.
The bootloaders structure is sufficient.
In order to fix “entrypoints” which can differ from one distribution to other, we will consider using symlinks for each version directory. For example, boot-sb.efi could point to shim.efi while boot.efi could point to grubXXX.efi or other bootloader depending on the distro. Then Foreman would need to resolve these “entrypoint” symlink in order to do hardlink (or another symlink). Other option is to keep some sort of metadata (in a JSON or something) which would tell which file should be used for EFI and which for SB-EFI. In any case, this will allow to have much more simple DHCP orchestration code where filename option can be simply MAC/boot-sb.efi and TFTP orchestration does not need to figure out which file is which - just one to one copy should be enough.
The implementation will be done in a way that it will not affect current EL-CentOS-Fedora users while it will vastly improve things for other distributions. Specifically, all must work fine even when there is no bootloader-universe available.
After all PRs are merged, we need to do retesting of all supported workflows with managed and unmanaged DHCP setups.
I am working on the OCI netboot distribution prototype, I will update you soon. This, however, will not affect this work only improve it later.
Thanks for the summary, @lzap (I haven’t managed it the last day).
Correct. At minimum, a bootloader-universe/<dist>/default/<arch>/ directory must exist with latest EFI binaries. If any major+minor combination needs alternative EFI binaries (e.g. older binaries containing revoked keys for older kernel), additional directories with pinned version can be provided which will be used instead for copying to MAC directory:
bootloader-universe/ubuntu/20.04.2/x86_64
├── boot.efi -> grubx64.efi
├── boot-sb.efi -> shimx64.efi
├── grubx64.efi # alternative version of Grub2
└── shimx64.efi # alternative version of shim
bootloader-universe/ubuntu/default/x86_64
├── boot.efi -> grubx64.efi
├── boot-sb.efi -> shimx64.efi
├── grubx64.efi
└── shimx64.efi
If multiple versions share same alternative binaries, we can also work with symlinks:
The corresponding bootloader-universe directory (distribution+version+arch) should take care of all files/symlinks needed for PXE boot (see above). Foreman code copies only the entire content to corresponding MAC directory. Having this, Foreman does not need to know anything about specific files.
For now we will describe exact bootloader-universe file structure in documentation, later this can be provided by e.g. OCI netboot distribution.
Correct. Current approach only adds additional MAC directories with distribution specific EFI binaries and GRUB2 configuration files (same as in tftpboot/grub2/ currently). Content of tftpboot/grub2/ – namely grubx64.efi, shimx64.efi, grub.cfg-00:*, grub.cfg-01-* – remains.
This guarantees that provisioning works as it did in the past:
if DHCP is unmanaged or
if Smart Proxy has no arbitrary distribution SB support at all or
if corresponding distribution is not available in bootloader-universe directory or
All GRUB2 configuration files have same content here.
Question:
@goarsna and I thought about using dedicated bootloader-universe directory in TFTP root directory (e.g. /var/lib/tftpboot/booloader-universe). Having this, we could then use relative symlinks for EFI binaries when relative symlinks are supported in Foreman’s Ruby version. EFI binaries are about ~3MB per host at the moment (10000 hosts → ~30G). Do you think it’s worth saving this disk space?
Is this disk space only used during provisioning of the hosts or stay the file even after successful provisioning? If the files stay, yes but I do not think it needs to be the highest priority. If the first it will not add up to such high numbers, so even a lower priority. But in general being conservative and reducing resource usage is always good even if it is not the highest priority.
Files stay until host gets removed/deleted from Foreman, not only during build/provisioning procedure.
As the PR is still work-in-progress (same for documentation) I would rather do it now than later.
I vote for putting it to dedicated /var/lib/tftpboot/bootloader-universe directory.
We can skip setting an arbitrary directory path via foreman-installer, keep PXE files logically together (TFTP root) and can use relatives symlinks in future for a) better overview on filesystem level and b) to save disk space.
Hi there and thanks @lzap and @Jan for your summaries!
I want to add / correct some parts:
In the TFTP root directory the MAC address based directory will be put at first place. I implemented it that way to be future proof in case we want to provide other bootloader files (for example for PXElinux) through the MAC address based directories
If we want to copy the entire directory content (what I absolutely agree that we should do), I would prepend the PXE loader as first level in the directory structure inside the bootloader_universe so that for PXEGrub2 it will look like bootloader_universe/pxegrub2/{os}/{version}/{arch}. By this we will again stay future proof in case we want to use the bootloader_universe directory for other PXE loaders like PXE linux.
I appreciate that idea. This only enforces an extra check for the X.Y to X.(Y-1) compatibility between Foreman and Smart Proxy as (in case DHCP is managed) the new file names boot.efi and boot-sb.efi can only be set as DHCP file name in case the Smart Proxy supports them.
After all a resulting example TFTP root directory for an Ubuntu and a SLES host would look as follows:
Regarding the location of the bootloader_universe I agree with Jan. We should use the advantages of putting the bootloader_universe inside the TFTP root directory.
The boot files do not need to stay for the whole lifetime of a host. When a host exits build mode, TFTP orchestration can be updated to delete these files completely. Today, it only rewrites the config files, it can do additional actions.
Consider using hard links instead of symlinks if symlinks do not work for any reason. I think keeping the “source” files in /usr is better fit than in /var. If Ruby is unable to do relative symlinks, then relative symlinks can be simply copied and they will still work.
Consider exploring using metadata instead of boot*.efi symlinks. This was just one idea, if that is clunky for any reason, we could still create some kind of metadata file in each directory describing those “entrypoints” in other way. For example, it could be bootentry.json or a config file or anything that is accessible and easily readable.
Do you mean for linking files from the bootloader_universe to the MAC address based directories inside the TFTP root?
As the major reason for this suggestion was to save disk space, we can stick to the current approach with the configurable bootloader_universe if we delete the bootloader files from the MAC address based directories inside the TFTP root once a host exits build mode.
BTW: We could add a suggestion to the docs to use /usr/local/share/bootloader_universe.
I would prefer to not use any metadata for the structure inside the bootloader_universe. As for now Users will have to take care of the bootloader_universe themselves, this will unnecessarily complicate things.
Back then it was traded as the successor to IPMI. But looks like IPMI is still today the defacto standard for remote control of baremetal (at least basic implementation by most vendors).
)