Foreman 3.4 change for subiquity breaks generic bootdisk / preseed iPXE install of Ubuntu 20.04 and above

Problem:

Using a generic bootdisk image trying to install Ubuntu 20.04 it correctly gets the iPXE script but that script fails to download the kernel since the Ubuntu mirror URL is uses 404s.

The URL http://archive.ubuntu.com/ubuntu/casper/vmlinuz does not exist.

Expected outcome:

It should construct the correct URL: http://archive.ubuntu.com/ubuntu/dists/focal/main/installer-amd64/current/legacy-images/netboot/ubuntu-installer/amd64/linux

Foreman and Proxy versions:

Foreman and proxies all 3.5.1

Foreman and Proxy plugin versions:

foreman_bootdisk: 21.0.3
foreman_puppet: 5.0.0
foreman_setup: 8.0.1

Distribution and version:

Ubuntu 20.04

Other relevant data:

This worked in earlier versions of Foreman. This PR which merged for v3.4 is the source of the problem: Fixes #34942 - Adapt installation media search path for Ubuntu 20.04.3+ by bastian-src Ā· Pull Request #9229 Ā· theforeman/foreman Ā· GitHub

Hi @damonmaria

Is this related to the setup of your installation medium? See Creating an Operating System for Ubuntu

cc @bastian-src who has been working on support for Ubuntu Autoinstall for Ubuntu 20.04.4+ and Ubuntu 22.04+

1 Like

I have my Ubuntu 20.04 iPXE template set to Preseed default iPXE as per the instructions for the bootdisk plugin.

My Ubuntu mirror path is also the default (I think): http://archive.ubuntu.com/ubuntu

This was all working for me for 5+ years up until recently.

The problem is the change to the Debian Operatingsystem class returning the pxedir as 'casper'. Casper does not exist under http://archive.ubuntu.com/ubuntu.

To be clear hereā€™s the rendered iPXE template showing the kernel and initrd URLs that donā€™t exist (the code on line 8 produces the correct URL):

#!gpxe
echo Trying to ping Gateway: ${netX/gateway}
ping --count 1 ${netX/gateway} || echo Ping to Gateway failed or ping command not available.
echo Trying to ping DNS: ${netX/dns}
ping --count 1 ${netX/dns} || echo Ping to DNS failed or ping command not available.


kernel http://archive.ubuntu.com/ubuntu/casper/vmlinuz initrd=initrd.img interface=auto url=********* ramdisk_size=10800 root=/dev/rd/0 rw auto BOOTIF=01-${netX/mac:hexhyp} console-setup/ask_detect=false console-setup/layout=USA console-setup/variant=USA keyboard-configuration/layoutcode=us localechooser/translation/warn-light=true localechooser/translation/warn-severe=true locale=en_US locale=en_US hostname=melba-radin.localdomain domain=localdomain

initrd http://archive.ubuntu.com/ubuntu/casper/initrd

imgstat
sleep 2
boot

Hey @damonmaria, thanks for your comment!

As @maximilian mentioned already, Canonical introduced a major change in Ubuntu 20.04.3+. Therefore, the piece of code you linked is necessary to apply the new boot file location (casper) for such Ubuntu versions. All prior versions (which do not use the new Subiquity Autoinstall mechanism) do still use the legacy path you mentioned before.

As you can see in the code, the function is_subiquity? checks whether a user wants to deploy a version higher than 20.04.2 (the last version supporting the legacy debian installer). I assume you want to deploy such a version with the legacy debian installer, right?
Nevertheless, it seems like Foreman decides to use the ā€œnewā€ path for your operating system. Could you please check in Foreman your ā€œUbuntu 20.04ā€ Operating System configuration. The ā€œmajorā€ and ā€œminorā€ version field must be configured properly. In case you want to deploy Ubuntu 20.04.2 for example, the major field must be ā€œ20.04ā€ and the minor field ā€œ2ā€ (if you have a look at the documentation link by @maximilianā€™s post before, this differentiation is explicitly mentioned there).
So, keep in mind 20.04 is a major version and all bug fix/later releases are the minor. As you can see here, the latest Ubuntu 20.04 release is 20.04.5:

Let me know if this helps!

Thanks for the detailed response @bastian-src.

This was actually a rebuild of a machine so I presume itā€™s picked up the point release number from Puppet facts applied to that host. So I can work around the issue by essentially setting the minor field to something below 2.

But what is the solution for more recent Ubuntu versions? The mirror URL generated by combining the standard Ubuntu mirror URL and pxedir from the new code is http://archive.ubuntu.com/ubuntu/casper/ā€¦ which does not exist.

Iā€™m glad it works now!

For more recent versions of Ubuntu, there are some manual steps needed. Ubuntu does not provide the boot files on their mirror like itā€™s been the case in the past - therefore, we must extract the Ubuntu Server ISO image in order to get the boot files. BUT, the new Ubuntu Subiquity Autoinstall mechanism does also need the ISO image iteself.
So, we must provide a) the extracted image files (to set up our installation media in Foreman and retrieve the kernel/initrd file) and also b) the image itself (for Ubuntuā€™s installation process). If you run Foreman with Katello, you can easily provide all of these files by putting them in your /var/www/html/pub/ directory. For a detailed description on how to set it up, have a look here:

If you run Foreman without Katello, you must provide these files in some other file repo. For the templates using the Autoinstall mechanism, it is just important that the ISO image is placed directly next to the extracted image (which is given as the installation media in Foreman) and has the same naming scheme. You can see what I mean in the GRUB2 Autoinstall template for example:

Oh yowser. We obviously have a bit of reading to do and changes to make to our current install process.

Thanks for the deep dive into that.

We do all our installs through generic bootdisk ISOs made through the bootdisk plugin. I note that the bootdisk plugin says for preseed to use the Preseed default iPXE template. But there is no autoinstall iPXE template in Foreman (yet?). Do you know @bastian-src if itā€™s even going to be possible to do autoinstall with generic bootdisk?

Forgot to mention. No, we donā€™t run Katello.

Iā€™ve gotta admit, Iā€™m not familiar with bootdisk deployment - I usually use the PXELinux and PXEGrub2 templates. I think there arenā€™t any plans at the moment regarding an upstream iPXE Autoinstall template. I would be happy to assist creating one, so more people can benefit from it!

I think @LeperMessiah adapted the iPXE template to run with Ubuntu Autoinstall. Maybe you can provide some guidance here?

I just have this piece from a previous conversation:

kernel http://<ftp-server-with-extracted-image>/ubuntu/casper/vmlinuz

initrd http://<ftp-server-with-extracted-image>/ubuntu/casper/initrd 
imgargs vmlinuz initrd=initrd boot=casper maybe-ubiquity verbose url=http:<ftp-server-with-image>/ubuntu.iso ip=<IP configuration in case of static deployment>

boot

@bastian-src I used the following template.
I donā€™t know if that is the correct way to do it but it works.
(Now I can see that the Ubuntu version and the hostname of the foreman host are hard coded that should be fixed. If I remember correctly the variable of the foreman host used https but something did not work properlyā€¦ )

#!gpxe
<%#
kind: iPXE
name: Preseed default iPXE
model: ProvisioningTemplate
oses:
- Debian
- Ubuntu
%>
<% if @host.operatingsystem.name == 'Debian' -%>
  <%- keyboard_params = "auto=true domain=#{@host.domain}" -%>
<% else -%>
  <%- keyboard_params = 'console-setup/ask_detect=false console-setup/layout=USA console-setup/variant=USA keyboard-configuration/layoutcode=us' -%>
<% end -%>
<% subnet = @host.subnet -%>
<% if subnet.nil? || subnet.dhcp_boot_mode? -%>
  <%- provision_url_suffix = '' -%>
  <%- netcfg_args = '' -%>
<% else -%>
  <%- provision_url_suffix = (@host.token.nil? ? '?' : '&') + 'static=yes' -%>
  <%- netcfg_args = 'netcfg/disable_dhcp=true netcfg/get_ipaddress=${netX/ip} netcfg/get_netmask=${netX/netmask} netcfg/get_gateway=${netX/gateway} netcfg/get_nameservers=${dns} netcfg/confirm_static=true' -%>
<% end -%>
<% boot_files_uris = @host.operatingsystem.boot_files_uri(medium_provider) -%>
<% kernel = boot_files_uris[0] -%>
<% initrd = boot_files_uris[1] -%>

kernel http://PUPPETMASTER/pub/installation_media/ubuntu/22.04-x86_64/casper/vmlinuz

initrd http://PUPPETMASTER/pub/installation_media/ubuntu/22.04-x86_64/casper/initrd 

imgargs vmlinuz ramdisk_size=10800 initrd=initrd boot=casper maybe-ubiquity verbose url=http://<%= foreman_request_addr %>/pub/installation_media/ubuntu/22.04-x86_64.iso ip=<%= @host.ip %>::<%= @host.interfaces.first.subnet.gateway %>:<%= @host.interfaces.first.subnet.mask %>:<%= @host.name %>:<%= @host.interfaces.first.identifier %>:none:<%= @host.interfaces.first.subnet.dns_primary %>:<%= @host.interfaces.first.subnet.dns_secondary %> cloud-config-url=/dev/null autoinstall ds=nocloud-net;s=http://<%= foreman_request_addr %>/userdata/

boot
1 Like

Thanks all for the above advice and ideas. I think Iā€™m getting there.

My situation is a bit different in that Iā€™m using the bootdisk plugin, with DHCP (where I donā€™t control the DHCP server) and behind NAT. So this is shaking out some other issues that Iā€™m happy to solve and post a PR for.

@lzap I note in this thread you seem to have been managing the provisioning template changes for autoinstall. I think thereā€™s a bug with dhcp in the netplan template. All the checks calculating if an interface is DHCP like ā€¦.subnet.nil? ? false : ā€¦ should use true if there is no subnet?

The other place I had trouble was getting @host to load for http://foreman-server/userdata/. Since I am behind NAT the remote IP doesnā€™t work. After finding the routing code I managed to get it to work by adding the primary interface MAC to that /userdata/ URL. But @lzap: In this comment you talk about adding token support which would be nicer but I canā€™t see how that works. My experiments with adding the build token as a query parma did not work. Is it possible to access /userdata/ by build token?

Reporting on progress. I have got this working with all of my requirements.

A couple other issues were caused by my hosts not having subnets assigned to them.

Turns out if no ip= kernel arg is added in the iPXE script then networking would not come up for me, so if itā€™s not static I always set ip=dhcp.

If there are no static IPv4 or IPv6 addresses then the addresses section of nameservers in the netplan needs to not be output, otherwise it ends up null which netplan does not like.

The outstanding issue I have is Iā€™m still not sure how to handle my hostā€™s interface not having subnets (we use non-Foreman-managed DHCP) in the netplan template. In other templates (like the Preseed default finish) no subnet implies dhcp. But the netplan template is setting up both dhcpv4 and dhcpv6. So how do we determine the difference between no IPv6 and dhcpv6 on an unknown subnet?

If someone can help me with that last point Iā€™ll submit a PR with these changes.

Also FYI @bastian-src. Trying to do the version number like this does not work with Puppet. The facts from Puppet treat the version number as, for example, 22.04 => major = 22, minor = 04. If I attempt to set the OS like you describe then on the next Puppet run a new OS will be made with major = 22, minor = 04 and the host OS will be changed to point to that.

FYI I have created an issue and PR of this:

I note the nameserver issue in the netplan template is already fixed.

For the dhcp4 issue in the netplan template I have made the following:

1 Like

Thanks a lot for your PRs! Iā€™m currently having a look at them. There are some things missing (generate snapshots of the template for testing and use a proper commit naming scheme according to Foremanā€™s requirements).

You can have a look at this PR which introduced new Autoinstall templates before:

If you want, I can also take your PR as a base and add the things missing in a separate PR(since generating snapshots can be a bit more complicated).

Apologies for the delay. Iā€™ve now got a Foreman dev environment going and generated the templates, and pushed conforming commit messages. Details in the PR comment.

1 Like

@bastian-src Giving this a bump. Is there anything I can do to the PR to help process this?

Hi @bastian-src. Iā€™ve got some spare time this week to progress this if thereā€™s anything you need me to do?

Hey @damonmaria, sry for the delay. Iā€™ve been on vacation for a while.
I think your PRs look fine from your side - I can also have a closer look and review them, so we can discuss further steps on GitHub :+1:

Technically so am I, which is why I have some downtime while on long train journeys to work on this :slight_smile: Please, enjoy your time off.

1 Like