over past couple of days, I was looking into dracut, a generic initramdisk generator for Linux used by many distributions, and Anaconda, the Red Hat installer. It was for a specific customer problem, but while I was working on this, I got an idea.
Discovery could have been part of the Anaconda installer itself. We would not be building and maintaining our own image, it’s a lot of work I can tell you. The idea is simple - Anaconda is booted from network or an official DVD (with our own kickstart included) and in the
%pre section it gathers few facts about the system, uploads them into Foreman and awaits instructions in a loop. Foreman could respond with either “here is your kickstart, continue” or - for different operating systems - with simply “all set, reboot”.
Anaconda includes pretty solid Linux environment, there’s full bash and python available. I think this could be a good time to maybe get rid of puppet during the discovery process, we really don’t need that amount of facts - all that matters is network configuration and this can be easily parsed and sent via our own format. Or we could simulate puppet facter via uFacter or even install the original facter at cost of increased size if we come to conclusion that’s the right thing - Anaconda image can be overlayed with
product.img - that’s the file Anaconda always attempts to download from the tree (and it ends up with 404).
Here is a pseudo-example which “downloads” a kickstart from HTTP and includes it. I had to encode the kickstart content itself because it breaks parsing, normally we would use
curl or something to get it.
%pre # 1. gather some facts about the system via shell or python # 2. upload it to the discovery endpoint via curl # 3. wait until the endpoint responds with a HTTP code # 4. the response can simply tell to "reboot" or # 4b it can carry rendered kickstart itself which can be included: # (I had to encode it because it contains %pre which breaks the syntax) base32 -d - >/tmp/discovered.ks <<'EOF' ORSXQ5AKNZSXI53POJVSALJNMRSXM2LDMU6WK3TQGFZTAIBNFVQWG5DJOZQXIZJAFUWWE33PORYH E33UN46WI2DDOAQC2LLON5UXA5RWEAWS22DPON2G4YLNMUQGW4ZNORSXG5AKMZUXEZLXMFWGYIBN FVSW4YLCNRSWICTVOJWCALJNOVZGYPJCNB2HI4B2F4XTCOJSFYYTMOBOGEZDELRRF5RWK3TUN5ZS 6OBNON2HEZLBNUXUEYLTMVHVGL3YHA3F6NRUF5XXGLZCBJZG633UOB3SALJNOBWGC2LOORSXQ5BA OJSWI2DBOQFGWZLZMJXWC4TEEAWS26DMMF4W65LUOM6XK4ZAFUWXMY3LMV4W2YLQHV2XGCTMMFXG OIDFNZPVKUZOKVKEMLJYBJZWK3DJNZ2XQIBNFVSW4ZTPOJRWS3THBJWG6Z3HNFXGOIBNFVWGK5TF NQ6WS3TGN4FHI2LNMV5G63TFEBKVGL2FMFZXIZLSNYFHEZLCN5XXICTCN5XXI3DPMFSGK4RAFUWW Y33DMF2GS33OHVWWE4QKMNWGKYLSOBQXE5BAFUWWC3DMEAWS22LONF2GYYLCMVWAU4TFOFYGC4TU BJYGC4TUEAXSALJNMZZXI6LQMU6SEZLYOQ2CEIBNFVZWS6TFHU2DAMBQBISXAYLDNNQWOZLTBJAG G33SMUFCKZLOMQFA==== EOF %end # 5. once %pre block is finished, the kickstart is included # 6. and anaconda continues with provisioning %include /tmp/discovered.ks
Another, more flexible possibility, is to install our own software into
product.img overlay, the discovery services would start before Anaconda (during dracut) and decide if to continue or restart. PXE-less wofklow could be possible as long as
kexec is available in the environment (Anaconda wants to remove it but we can add it back in the
Building on top of Anaconda has some pros and cons. First, all hardware certified for RHEL/CentOS will simply work. If there is a problem with drivers, we can reach out to Red Hat teams to fix this. Second, Red Hat users don’t need to actually reboot to start provisioning - this would work even in PXE-less environment. On the other hand, Anaconda initramdisk is 45MB and
install.img is about 600MB, that’s a slight increase for non-Red Hat users who would be rebooting into the final OS installer. Also, it’s a bit tricky to get to interactive stuff in
%pre, actually I haven’t found a way.
Therefore, I am thinking about a second solution, to build dracut module that would do the discovery process. This is slightly more challenging for implementation, dracut environment is more limited and complex (ton of shell + systemd) but users would be simply able to generate discovery image with dracut as long as we provided a dracut module (named base-foreman in the example):
dracut discovery-$(uname -r).img -m "base-foreman network terminfo debug udev-rules ssh-server"
There is even a dracut ssh server module and Anaconda also supports ssh server. Currently, Foreman Discovery is implemented via Request in, Request back and I think we should either leverage Remote Execution or at least change this model to polling.
This would work an most linux distributions, resulting image would be around 20-70MB depending on amount of modules and drivers included. Dracut is extremely flexible and fast to generate, it’s possible to include own scripts, modify behavior with hooks - we would be giving really powerful tool to foreman users so they can experiment, easily add their own facts, scripts, drivers, memory tests or install any kind of software that is available for their OS and call it via SSH if we choose to go this way.
Dracut option would give us great PXE-less possibilities - we could actually “merge” PXE-less discovery and bootdisk workflows into one. There would be only one (small) image that can be booted from network or from CD/DVD. Dracut have many options and bintools can be replaced with
busybox and most unwanted firmware can be easily removed to bring the image size down to few megabytes if needed but I think we would distribute the standard sized 40MB image for best possible hardware support out of box. The new discovery could be actually generated per host, per subnet or as a generic image effectively replacing bootdisk.
Another big advantage for both approaches (Anaconda uses Dracut so it’s the same) is well tested and documented networking possibilities (those
ip= options) which includes bonds, bridges etc. More info in
dracut.cmdline (and other) man pages - dracut is well documented.
And who knows, maybe
kexec could be more reliable also for PXE-less workflows since it’s early stage of boot process and not all drivers were loaded yet. In any way, we would be ditching most of the discovery work, including the TUI and we’d need to build something more simple, based on shell. I’d like to rather simplify things (maybe no facter + just ssh no REST API) than trying to be backward compatible.
If there was a way to write a module that would not require to replace “base” dracut module and that would hook into dracut in the correct way, we could even build Anaconda (or any OS installation) PXE/DVD images which are based on dracut and this would work the generic way covering both PXE and non-PXE environments (assuming there is a way to override kernel command line arguments for the installer):
- boot dracut with foreman module
- foreman code would pause the booting
- it would perform discovery
- foreman would trigger provisioning via reponse or ssh
- OS installer (Debian, Red Hat) would start installing (without restart)
I know it’s a lot, if you did read it through please give me a like so I know there’s anyone reading those. And drop your thoughts. If you have some experience with writing dracut modules, l am interested. Or maybe there’s someone already doing this, I’d not be surprised.