Ideas for IPv6 netboot provisioning

Development
1

On of the main Foreman’s strength is PXE management (bare-metal provisioning). It’s built around management of the TFTP directory boot files and DHCP reservations. When a host is created in Foreman, DHCP configuration for particular client is generated and the server identifies the client using it’s unique key (MAC address), bootloader is downloaded from TFTP and it reads either generic or unique configuration leveraging again the unique key.

However in IPv6 world it’s different. DHCPv6 was designed in a way that unique identifier is always different, SLAAC protocol is, if I understand it correctly, unmanaged. There are issues with dual-stack networks, ISC DHCP server only support IPv4 and brand-new software which is not supported by Foreman yet (Kea) must be used for DHCPv6. I’ve seen several new implementations of DHCPv6 which aims to solve the problem of unique identifiers. It’s a mess.

So what I would like to hear from you guys is:

  • If you netboot IPv6 clients, how do you do it?
  • How did you solve the issues mentioned above?
  • Which workflow would you like to see in Foreman?

Thanks!

2

Adding some of my my rough ideas:

Option 1: Chainbooting iPXE over unmanaged DHCPv6/TFTP

This should work with both BIOS and UEFI bare-metal. For VMs with iPXE embedded the chainbooting can be simply skipped. The workflow:

  • a node initiates PXE boot requesting IPv6 address
  • a DHCPv6 server replies with filename “http://foreman/ipxe.efi” for UEFI or ipxe.lkrn for BIOS
  • iPXE initializes network and performs another IPv6 request
  • a DHCPv6 server replies with a different filename “http://foreman/script.ipxe”
  • The bootstrap configuration file contains statement to load “http://foreman/script.ipxe?mac=AA:BB:CC:DD:EE:FF”
  • The final configuration file contains kernel/initramdisk statements to boot Anaconda

Changes required in Foreman:

  • none if bootdisk plugin is installed
  • add MAC parameter for unattended controller otherwise (bootdisk provides this feature)

This is the easiest workflow I think, the only thing to do is to setup unmanaged DHCPv6 (ISC can be used), TFTP server with iPXE files, configuring it for chainbooting and that should be it. Foreman ships with required templates and bootstrapping has been recently added by Timo (intermediate iPXE template). One slight disadvantage is that tokens cannot be used, MAC-address spoofing is possible. On the other hand, this has been always possible when bootdisk is installed.

Option 2: HTTP UEFI Boot over unmanaged DHCPv6

This is a similar workflow but with HTTP UEFI bootstrapping procedure (no PXE/TFTP). Both iPXE or grub2 can be used. Since iPXE workflow is almost the same as the one above (except there is no TFTP bootstrap) I describe grub2 case:

This is a nice alternative for hardware where iPXE is not compatible. Again, there are no changes required in Foreman as we already have support for HTTP UEFI boot (the last PR is pending merge). I have tested HTTP UEFI Boot over IPv4, the question is if it works over IPv6 as well.

Option 3: PXE over managed IPv6

DHCPv6 does not advertise MAC address in the request so it is not possible to create host reservations and in PXE environment DUID (unique client identifier) changes every boot so the MAC address is the only reasonable way of identifing clients I think. To workaround that, the workflow would require DHCPv6 relay server with Client Link-Layer Address Option in DHCPv6 feature. In short:

Currently, the DHCPv6 specification [RFC3315] does not define a way
to communicate the client link-layer address to the DHCP server in
cases where the DHCP server is not connected to the same network link
as the DHCP client. The DHCPv6 specification mandates that all
clients prepare and send a DHCP Unique Identifier (DUID) as the
client identifier option in all the DHCPv6 message exchanges.
However, none of these methods provide a simple way to extract a
client’s link-layer address. This presents a problem to an operator
who is using an existing DHCPv4 system with the client link-layer
address as the customer identifier and who desires to correlate
DHCPv6 assignments using the same identifier. [RFC4361] describes a
mechanism for using the same DUID in both DHCPv4 and DHCPv6.
Unfortunately, this specification requires modification of existing
DHCPv4 clients, and has not seen broad adoption in the industry
(indeed, we are not aware of any commercial implementations).

Providing an option in DHCPv6 Relay-Forward messages to carry the
client link-layer address explicitly will help the above mentioned
scenarios. For example, it can be used along with other identifiers
to associate DHCPv4 and DHCPv6 messages from a dual-stack client.
Further, having the client link-layer address in DHCPv6 will help by
providing additional information for event debugging and logging
related to the client at the relay agent and the server. The
proposed option may be used in a wide range of networks; two notable
deployment models are service provider and enterprise network
environments.

So the workflow would be the typical PXE workflow as we have it today (new host or discovery) with the following:

  • DHCPv6 relay with MAC address feature
  • DHCPv6 server that can be managed via Foreman and supports identifying hosts by MAC
  • changes in orchestration and smart-proxy API to support IPv6 provisioning interface

The only DHCPv6 server that can be managed today and fulfills the requirements is dnsmasq. From its man page:

A single --dhcp-host may contain an IPv4 address or an IPv6 address, or both. IPv6 addresses must be brack‐eted by square brackets thus: --dhcp-host=laptop,[1234::56] IPv6 addresses may contain only the host-identifier part: --dhcp-host=laptop,[::56] in which case they act as wildcards in constructed dhcp ranges, with the appropriate network part inserted. Note that in IPv6 DHCP, the hardware address may not be available, though it normally is for direct-connected clients, or clients using DHCP relays which support RFC 6939.

The question is how to approach dual-stack nodes, the current idea treats DHCPv4 and DHCPv6 as two completely separate entities. There are advantages in managing the DHCP addresses in some combined way.

3

I just wanted to mention something, as it didn’t seem clear in your description.

From my understanding typically SLAAC is used for address assignment, and DHCPv6 is used for everything else (known as Stateless DHCP). So SLAAC assigns the address, and DHCPv6 provides everything else (DNS, Boot Info, etc). The Host communicates with the router to determine a IPv6 address, and the other info comes from the DHCP server.

Is this a model we are planning to support?

4

Note entirely true. RFC 8106 IPv6 Router Advertisement Options for DNS Configuration means SLAAC can provide the DNS info and don’t need DHCPv6 to be present.

5

Sure, but for our purposes (Network Boot) DHCPv6 needs to be present.

6

I haven’t looked at all the options so I can’t say for sure, but I was trying to say that many networks don’t need DHCPv6 so often it won’t be present yet.

7

What you mean “typically”? On my network, I don’t want to use SLAAC at all. It’s probably good for coffeeshop but not for datacenters where I prefer all IP addresses to be managed by Foreman.

SLAAC really is “stateless” and I see it as complementary that is not needed for DHCPv6 provisioning, you can assign addresses via DHCPv6 as well. So I’d not focus on SLAAC at all in our workflow no (3) if we want to implement it (that’s the question it’s the most challenging one out of the three).

For (1) or (2) it is not relevant how IPv6 addresses are assigned as Foreman does not manage addresses in these workflows. So users could use SLAAC, DHCPv6 or combination of both.

8

Reading through this thread, it seems nobody has actually tried using DHCPv6 for deployment yet.
I have tried it (with dnsmasq) and a roadblock you will hit is that the Client DUID will differ between the PXE environment, the installer and the final operating system. While this should be derived from the machine_id, many DHCP clients act differently (for example, systemd-networkd has different fallbacks for broken machine_id than dhcpcd, and PXE firmware will act differently again).

dnsmasq (conforming to RFC) will not hand out the same address to different Client DUIDs, even if the MAC is matching. So by default, automated deployment in pure v6 will fail.

There has been recent activity with RedHat involvement for OpenStack Ironic, which needs to solve the same issue of course, here:
http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2020q1/013771.html
and this will likely be part of the next dnsmasq release.

In short, the idea is to assign addresses from a free region for the PXE boot and installation environment, and only assign the final address when booting the operating system after deployment.

There’s no dnsmasq release with these patches out yet, so I believe currently, DHCPv6 automated deployment in a pure v6 environment is not possible without hacks.

1 Like
9

Hello, thanks for sharing. So to sum up, the problem is that a host with IPv6 reservation made through foreman would eventually not get the allocated IPv6. Maybe PXE or dracut DHCP client, but not the final DHCP request made by OS.

There is some discussion around this with a proposal to actually use a custom DHCP server to handle those requests. Unfortunately, the project (netboot) is no longer in active development. We need to explore ISC Kea too if that’s an option too.

Honestly, I don’t actually get “only assign the IP after deployment”. You mean static assignment? From what I understand, dnsmasq (with the patch) returns one of multiple IPv6 addresses. This can’t go well with DNS Foreman management.

What we have for now is “unmanaged” DHCPv6 provisioning:

10

It depends on how exactly you set it up. Coincidentally, I asked exactly your question in the thread:
http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2020q1/013744.html
The idea would be to have dnsmasq assign an address within a range used for “deployment only” for the PXE boot and Dracut phase, and assign the address registered with Foreman / DNS once the final system boots up. Tag support in dnsmasq should allow for this differentiation as outlined in the linked post.

The main issue here is that the RFCs are not really consistent with the requirements for the netboot case. I did not look at ISC Kea yet, but if they conform to RFCs strictly, the same issues will be present there.

dnsmasq history has actually had a different patch floating on the list:
http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2018q4/012707.html
which just assigned the same IP address to the same MAC even if the Client DUID changed. This was not accepted upstream (since it violates RFCs).

1 Like
11

I talked to Harald, what he recommends is to explicitly set dhclient userclass and match it in dnsmasq:

dhclient.conf:
options dhcp6.userclass code 15 = string;
# len: 00:08, "dhclient"
send dhcp6.userclass 00:08:64:68:63:6c:69:65:6e:74;

dnsmasq.conf:
dhcp-userclass=set:dhclient,dhclient
dhcp-host=52:54:00:3f:5c:c0,[fd12:3456:789a:1::aa04/126],host1
dhcp-host=tag:dhclient,52:54:00:3f:5c:c0,[fd12:3456:789a:1::aa01],host1

What I think is also feasible is to explicitly set DUID which is maybe little cleaner, after all installed OS will set it randomly or semi-randomly:

/etc/dhcp/dhclient6.conf
# DUID generated randomly by Foreman
interface "eth0" {
   send dhcp6.client-id 00:00:00:00:00:00:00:00:00:01;
}

/etc/dnsmasq.d/host1:
dhcp-host=52:54:00:3f:5c:c0,[fd12:3456:789a:1::aa04/126],host1
#alternative syntax:
#dhcp-host=52:54:00:3f:5c:c0,[fd12:3456:789a:1::aa02],[fd12:3456:789a:1::aa03],[fd12:3456:789a:1::aa04],host1
dhcp-host=id:00:00:00:00:00:00:00:00:00:01,[fd12:3456:789a:1::aa01],host1

Some systems like Red Hat (Anaconda) do support provisioning tokens and they will work even when all leases are random - the host will be matched using MAC address via Grub2/PXELinux/iPXE, Kickstart will be matched via the token and then the installed OS can take reservation thanks to the generated DUID. However, this will not work for all OSes we support where REMOTE IP (IPv6) is the only way to match hosts sometimes.

This should not be too bad to implement, we already have a dnsmasq DHCP and DNS providers. It needs to be changed to match to write these IPv6 hosts in this form and then Foreman unattended URL handling code must be aware that a host don’t have a single DHCPv6 address but 8 of them. And next free IPv6 address would return “gaps” between free addresses (reserved for network boot clients).

Now the question: Is it safe to assume that all Foreman users are okay with the fact that if they want to do IPv6 netboot provisioning, a single host with address 2001:db8::1 also allocate 2001:db8::2 - 2001:db8::7 for network boot purposes so the next available address would be 2001:db8::8.

12

Nice idea! That looks reasonable. Of course, some distributions may not use dhclient, but rather dhcpcd or systemd-networkd, but it can be done for all of course.

I’m not sure (and I’m not the right one to ask, since the DHCP server we must use at our site has a WEB GUI only, we are still fighting for API access :frowning: ). However, our plan already was to give at least 2 bits to each node (i.e. 1-3), so 1-7 would also be fine, so I would agree with this even though we can not use the DHCP feature yet (but that’s not your fault).

On the other hand, some users may want to rather have IPs consecutive, but I don’t see a clean solution there (unless you have provisioning tokens and use a dedicated, separate “install-only” IP range).

13

For the record, Red Hat seem to work on backporting those patches into RHEL 7 and 8:

14

After even more chat with Harald, I think you are actually able to implement dynamic DHCP range IPv6 provisioning with dnsmasq now:

/etc/dhcp/dhclient6.conf
# DUID generated randomly by Foreman
interface "eth0" {
   send dhcp6.client-id 00:00:00:00:00:00:00:00:00:01;
}

/etc/dnsmasq.d/host1:
dhcp-range=fd12:3456:789a:1::aaaa,fd12:3456:789a:1::afff,64,10m
dhcp-host=id:00:00:00:00:00:00:00:00:00:01,[fd12:3456:789a:1::aa01],host1

This would work as provisioning tokens and/or Anaconda MAC address in HTTP header will do the job of associating clients with data. Additional 7 IPv6 addresses can be reserved for future use (the new patch with IP ranges).

So the key change to your configuration is really deploying DUID to your clients and making use of the DUID in your dnsmasq configuration. If you need assistance, I can help. Would actually love to, we will be working on this feature probably in near future.

We do have dnsmasq DHCP/DNS plugins already, it probably needs a patch. DHCP API needs new field DUID but that’s easy thing to do. Other than that we are ready!

15

I’ve created a tracker issue for IPv6 DHCP feature.