Foreman 3.4 change for subiquity breaks generic bootdisk / preseed iPXE install of Ubuntu 20.04 and above

damonmaria · February 14, 2023, 10:01pm

Problem:

Using a generic bootdisk image trying to install Ubuntu 20.04 it correctly gets the iPXE script but that script fails to download the kernel since the Ubuntu mirror URL is uses 404s.

The URL http://archive.ubuntu.com/ubuntu/casper/vmlinuz does not exist.

Expected outcome:

It should construct the correct URL: http://archive.ubuntu.com/ubuntu/dists/focal/main/installer-amd64/current/legacy-images/netboot/ubuntu-installer/amd64/linux

Foreman and Proxy versions:

Foreman and proxies all 3.5.1

Foreman and Proxy plugin versions:

foreman_bootdisk: 21.0.3
foreman_puppet: 5.0.0
foreman_setup: 8.0.1

Distribution and version:

Ubuntu 20.04

Other relevant data:

This worked in earlier versions of Foreman. This PR which merged for v3.4 is the source of the problem: Fixes #34942 - Adapt installation media search path for Ubuntu 20.04.3+ by bastian-src · Pull Request #9229 · theforeman/foreman · GitHub

maximilian · February 15, 2023, 7:51am

Hi @damonmaria

Is this related to the setup of your installation medium? See Creating an Operating System for Ubuntu

cc @bastian-src who has been working on support for Ubuntu Autoinstall for Ubuntu 20.04.4+ and Ubuntu 22.04+

damonmaria · February 15, 2023, 7:23pm

I have my Ubuntu 20.04 iPXE template set to Preseed default iPXE as per the instructions for the bootdisk plugin.

My Ubuntu mirror path is also the default (I think): http://archive.ubuntu.com/ubuntu

This was all working for me for 5+ years up until recently.

The problem is the change to the Debian Operatingsystem class returning the pxedir as 'casper'. Casper does not exist under http://archive.ubuntu.com/ubuntu.

To be clear here’s the rendered iPXE template showing the kernel and initrd URLs that don’t exist (the code on line 8 produces the correct URL):

#!gpxe
echo Trying to ping Gateway: ${netX/gateway}
ping --count 1 ${netX/gateway} || echo Ping to Gateway failed or ping command not available.
echo Trying to ping DNS: ${netX/dns}
ping --count 1 ${netX/dns} || echo Ping to DNS failed or ping command not available.


kernel http://archive.ubuntu.com/ubuntu/casper/vmlinuz initrd=initrd.img interface=auto url=********* ramdisk_size=10800 root=/dev/rd/0 rw auto BOOTIF=01-${netX/mac:hexhyp} console-setup/ask_detect=false console-setup/layout=USA console-setup/variant=USA keyboard-configuration/layoutcode=us localechooser/translation/warn-light=true localechooser/translation/warn-severe=true locale=en_US locale=en_US hostname=melba-radin.localdomain domain=localdomain

initrd http://archive.ubuntu.com/ubuntu/casper/initrd

imgstat
sleep 2
boot

bastian-src · February 16, 2023, 10:52am

Hey @damonmaria, thanks for your comment!

As @maximilian mentioned already, Canonical introduced a major change in Ubuntu 20.04.3+. Therefore, the piece of code you linked is necessary to apply the new boot file location (casper) for such Ubuntu versions. All prior versions (which do not use the new Subiquity Autoinstall mechanism) do still use the legacy path you mentioned before.

As you can see in the code, the function is_subiquity? checks whether a user wants to deploy a version higher than 20.04.2 (the last version supporting the legacy debian installer). I assume you want to deploy such a version with the legacy debian installer, right?
Nevertheless, it seems like Foreman decides to use the “new” path for your operating system. Could you please check in Foreman your “Ubuntu 20.04” Operating System configuration. The “major” and “minor” version field must be configured properly. In case you want to deploy Ubuntu 20.04.2 for example, the major field must be “20.04” and the minor field “2” (if you have a look at the documentation link by @maximilian’s post before, this differentiation is explicitly mentioned there).
So, keep in mind 20.04 is a major version and all bug fix/later releases are the minor. As you can see here, the latest Ubuntu 20.04 release is 20.04.5:

Let me know if this helps!

damonmaria · February 16, 2023, 6:52pm

Thanks for the detailed response @bastian-src.

This was actually a rebuild of a machine so I presume it’s picked up the point release number from Puppet facts applied to that host. So I can work around the issue by essentially setting the minor field to something below 2.

But what is the solution for more recent Ubuntu versions? The mirror URL generated by combining the standard Ubuntu mirror URL and pxedir from the new code is http://archive.ubuntu.com/ubuntu/casper/… which does not exist.

bastian-src · February 17, 2023, 7:55am

I’m glad it works now!

For more recent versions of Ubuntu, there are some manual steps needed. Ubuntu does not provide the boot files on their mirror like it’s been the case in the past - therefore, we must extract the Ubuntu Server ISO image in order to get the boot files. BUT, the new Ubuntu Subiquity Autoinstall mechanism does also need the ISO image iteself.
So, we must provide a) the extracted image files (to set up our installation media in Foreman and retrieve the kernel/initrd file) and also b) the image itself (for Ubuntu’s installation process). If you run Foreman with Katello, you can easily provide all of these files by putting them in your /var/www/html/pub/ directory. For a detailed description on how to set it up, have a look here:

If you run Foreman without Katello, you must provide these files in some other file repo. For the templates using the Autoinstall mechanism, it is just important that the ISO image is placed directly next to the extracted image (which is given as the installation media in Foreman) and has the same naming scheme. You can see what I mean in the GRUB2 Autoinstall template for example:

github.com

theforeman/foreman/blob/8cdb7ff1066519b518470b81475ce5574076b489/app/views/unattended/provisioning_templates/PXEGrub2/preseed_default_pxegrub2_autoinstall.erb#L23


      
          # Supported host/hostgroup parameters:
          #
          # blacklist = module1, module2
          #   Blacklisted kernel modules
          #
          # lang = en_US
          #   System locale
          #
          <%
            efi_suffix = 'efi'
            image_path = @preseed_path.sub(/\/?$/, '.iso')
            options = []
            options << 'ramdisk_size=1500000'
            options << 'root=/dev/rd/0 rw auto'
            options << "url=http://#{@preseed_server}#{image_path}"
            options << 'cloud-config-url=/dev/null'
            options << 'autoinstall'
          -%>
          
          set default=0
          set timeout=<%= host_param('loader_timeout') || 10 %>

damonmaria · February 18, 2023, 3:10am

Oh yowser. We obviously have a bit of reading to do and changes to make to our current install process.

Thanks for the deep dive into that.

We do all our installs through generic bootdisk ISOs made through the bootdisk plugin. I note that the bootdisk plugin says for preseed to use the Preseed default iPXE template. But there is no autoinstall iPXE template in Foreman (yet?). Do you know @bastian-src if it’s even going to be possible to do autoinstall with generic bootdisk?

damonmaria · February 18, 2023, 3:14am

Forgot to mention. No, we don’t run Katello.

bastian-src · February 24, 2023, 10:26am

I’ve gotta admit, I’m not familiar with bootdisk deployment - I usually use the PXELinux and PXEGrub2 templates. I think there aren’t any plans at the moment regarding an upstream iPXE Autoinstall template. I would be happy to assist creating one, so more people can benefit from it!

I think @LeperMessiah adapted the iPXE template to run with Ubuntu Autoinstall. Maybe you can provide some guidance here?

I just have this piece from a previous conversation:

kernel http://<ftp-server-with-extracted-image>/ubuntu/casper/vmlinuz

initrd http://<ftp-server-with-extracted-image>/ubuntu/casper/initrd 
imgargs vmlinuz initrd=initrd boot=casper maybe-ubiquity verbose url=http:<ftp-server-with-image>/ubuntu.iso ip=<IP configuration in case of static deployment>

boot

LeperMessiah · February 24, 2023, 11:24am

@bastian-src I used the following template.
I don’t know if that is the correct way to do it but it works.
(Now I can see that the Ubuntu version and the hostname of the foreman host are hard coded that should be fixed. If I remember correctly the variable of the foreman host used https but something did not work properly… )

#!gpxe
<%#
kind: iPXE
name: Preseed default iPXE
model: ProvisioningTemplate
oses:
- Debian
- Ubuntu
%>
<% if @host.operatingsystem.name == 'Debian' -%>
  <%- keyboard_params = "auto=true domain=#{@host.domain}" -%>
<% else -%>
  <%- keyboard_params = 'console-setup/ask_detect=false console-setup/layout=USA console-setup/variant=USA keyboard-configuration/layoutcode=us' -%>
<% end -%>
<% subnet = @host.subnet -%>
<% if subnet.nil? || subnet.dhcp_boot_mode? -%>
  <%- provision_url_suffix = '' -%>
  <%- netcfg_args = '' -%>
<% else -%>
  <%- provision_url_suffix = (@host.token.nil? ? '?' : '&') + 'static=yes' -%>
  <%- netcfg_args = 'netcfg/disable_dhcp=true netcfg/get_ipaddress=${netX/ip} netcfg/get_netmask=${netX/netmask} netcfg/get_gateway=${netX/gateway} netcfg/get_nameservers=${dns} netcfg/confirm_static=true' -%>
<% end -%>
<% boot_files_uris = @host.operatingsystem.boot_files_uri(medium_provider) -%>
<% kernel = boot_files_uris[0] -%>
<% initrd = boot_files_uris[1] -%>

kernel http://PUPPETMASTER/pub/installation_media/ubuntu/22.04-x86_64/casper/vmlinuz

initrd http://PUPPETMASTER/pub/installation_media/ubuntu/22.04-x86_64/casper/initrd 

imgargs vmlinuz ramdisk_size=10800 initrd=initrd boot=casper maybe-ubiquity verbose url=http://<%= foreman_request_addr %>/pub/installation_media/ubuntu/22.04-x86_64.iso ip=<%= @host.ip %>::<%= @host.interfaces.first.subnet.gateway %>:<%= @host.interfaces.first.subnet.mask %>:<%= @host.name %>:<%= @host.interfaces.first.identifier %>:none:<%= @host.interfaces.first.subnet.dns_primary %>:<%= @host.interfaces.first.subnet.dns_secondary %> cloud-config-url=/dev/null autoinstall ds=nocloud-net;s=http://<%= foreman_request_addr %>/userdata/

boot

damonmaria · February 27, 2023, 5:20am

Thanks all for the above advice and ideas. I think I’m getting there.

My situation is a bit different in that I’m using the bootdisk plugin, with DHCP (where I don’t control the DHCP server) and behind NAT. So this is shaking out some other issues that I’m happy to solve and post a PR for.

@lzap I note in this thread you seem to have been managing the provisioning template changes for autoinstall. I think there’s a bug with dhcp in the netplan template. All the checks calculating if an interface is DHCP like ….subnet.nil? ? false : … should use true if there is no subnet?

The other place I had trouble was getting @host to load for http://foreman-server/userdata/. Since I am behind NAT the remote IP doesn’t work. After finding the routing code I managed to get it to work by adding the primary interface MAC to that /userdata/ URL. But @lzap: In this comment you talk about adding token support which would be nicer but I can’t see how that works. My experiments with adding the build token as a query parma did not work. Is it possible to access /userdata/ by build token?

damonmaria · February 28, 2023, 8:15am

Reporting on progress. I have got this working with all of my requirements.

A couple other issues were caused by my hosts not having subnets assigned to them.

Turns out if no ip= kernel arg is added in the iPXE script then networking would not come up for me, so if it’s not static I always set ip=dhcp.

If there are no static IPv4 or IPv6 addresses then the addresses section of nameservers in the netplan needs to not be output, otherwise it ends up null which netplan does not like.

The outstanding issue I have is I’m still not sure how to handle my host’s interface not having subnets (we use non-Foreman-managed DHCP) in the netplan template. In other templates (like the Preseed default finish) no subnet implies dhcp. But the netplan template is setting up both dhcpv4 and dhcpv6. So how do we determine the difference between no IPv6 and dhcpv6 on an unknown subnet?

If someone can help me with that last point I’ll submit a PR with these changes.

damonmaria · February 28, 2023, 7:57pm

Also FYI @bastian-src. Trying to do the version number like this does not work with Puppet. The facts from Puppet treat the version number as, for example, 22.04 => major = 22, minor = 04. If I attempt to set the OS like you describe then on the next Puppet run a new OS will be made with major = 22, minor = 04 and the host OS will be changed to point to that.

damonmaria · March 8, 2023, 2:37am

FYI I have created an issue and PR of this:

github.com/theforeman/foreman

Template for Ubuntu autoinstall through iPXE

theforeman:develop ← damonmaria:ipxe-autoinstall

opened 02:05AM - 08 Mar 23 UTC

damonmaria

+92 -32

fixes #36180 - iPXE provisioning template for Ubuntu autoinstall Related discussion: https://community.theforeman.org/t/foreman-3-4-change-for-subiquity-breaks-generic-bootdisk-preseed-ipxe-install-of-ubuntu-20-04-and-above/32310 This template works with the new Ubuntu autoinstall mechanism but for iPXE, similarly to other autoinstall provisioning templates. I have tested it with the bootdisk plugin using DHCP. I have not tested it through iPXE network boot or with static addresses but the static address code was taken from a working example in the above community discussion.

I note the nameserver issue in the netplan template is already fixed.

For the dhcp4 issue in the netplan template I have made the following:

github.com/theforeman/foreman

pressed netplan provisioning template should treat no subnet consistently with finished template

theforeman:develop ← damonmaria:netplan-dhcp-when-no-subnet

opened 02:36AM - 08 Mar 23 UTC

damonmaria

+4 -6

fixes #36181 - pressed netplan provisioning template should treat no subnet cons…istently with finished template This changes the netplan setup to output `dhcp4: true` if there is no ipv4 subnet. This matches how the preseed finished template does it and is IMHO correct. I have not changed how `:dhcp6` is calculated. If the same change was applied there then the netplan would for have `dhcp6: true` on networks that probably don't want to use ipv6. And anyway, I do not have an ipv6 network on which to test and therefore don't want to change how that works. NOTE: I suspect there is another bug in the `preseed_netplan_setup.erb` template in that all the checks for the `:dhcp6` parameter look like the following: ``` :dhcp6 => bond.subnet6.nil? ? false : bond.subnet.dhcp_boot_mode? ``` I think the second reference to `.subnet` should be `.subnet6` like the first.

bastian-src · March 9, 2023, 11:07am

Thanks a lot for your PRs! I’m currently having a look at them. There are some things missing (generate snapshots of the template for testing and use a proper commit naming scheme according to Foreman’s requirements).

You can have a look at this PR which introduced new Autoinstall templates before:

If you want, I can also take your PR as a base and add the things missing in a separate PR(since generating snapshots can be a bit more complicated).

damonmaria · March 20, 2023, 2:36am

Apologies for the delay. I’ve now got a Foreman dev environment going and generated the templates, and pushed conforming commit messages. Details in the PR comment.

damonmaria · April 8, 2023, 5:04am

@bastian-src Giving this a bump. Is there anything I can do to the PR to help process this?

damonmaria · April 24, 2023, 12:43am

Hi @bastian-src. I’ve got some spare time this week to progress this if there’s anything you need me to do?

bastian-src · April 25, 2023, 7:30am

Hey @damonmaria, sry for the delay. I’ve been on vacation for a while.
I think your PRs look fine from your side - I can also have a closer look and review them, so we can discuss further steps on GitHub

damonmaria · April 25, 2023, 9:08am

Technically so am I, which is why I have some downtime while on long train journeys to work on this Please, enjoy your time off.