AlmaLinux 9 Kickstart with bonded network interface

Support
1

Problem:
I am not sure whether this is a foreman or anaconda/kickstart issue: I have configured a new server in foreman for provisioning with AlmaLinux 9.7. The server has two ethernet interfaces which are supposed to be bonded.

I configure everything in Foreman: both ethernet interface as managed interfaces with their mac address and the interface name. In addition, the bond interface with mac address of one of the ethernet, name “bond0”, hostname, domain, ipv4 and ipv6 subnets and ipv4 and ipv6 addresses. The bond interface has all functions checked: managed, primary, provision and remote execution.

Mode 802.3ad, attached devices are the interface names of those two ethernet interfaces, bond options as well.

The installation works when I start PXE/UEFI boot on the server:

  1. The server gets pointed to the foreman server via dhcp.
  2. The foreman server sends the grub menu information to boot the kernel for kickstart, setting up the bond with the static ipv4 address (not IPv6) etc. (template Kickstart default PXEGrub2)
  3. The kernel starts the installer and it downloads the kickstart file which contains the network configuration line for both ipv4 and ipv6 with gateway, dns, etc. (template Kickstart default)
  4. The installer sets up the current network connection bonded, with working ipv4 and ipv6 address and runs the installation.
  5. The installation finishes and the server reboots

So far, all good. However, when the server boots it doesn’t have a working network connection. The lacp bond is set up, but the bond0 interface doesn’t have an ip address. The file /etc/NetworkManager/system-connections/bond0.nmconnection contains:

[connection]
id=bond0
uuid=baf65ccb-e4e6-42c9-9c0b-ea7a842f9a29
type=bond
autoconnect=false
autoconnect-priority=-100
autoconnect-retries=1
interface-name=bond0
multi-connect=1
timestamp=1778258229

[ethernet]
mtu=9000

[bond]
miimon=100
mode=802.3ad
xmit_hash_policy=layer2+3

[ipv4]
method=disabled

[ipv6]
addr-gen-mode=eui64
method=ignore

[proxy]

[user]
org.freedesktop.NetworkManager.origin=nm-initrd-generator

So ipv4 is disabled. ipv6 is ignored.

I have checked in a shell during the kickstart installation: it seems the installer writes this during installation.

I am seriously confused to why the installer or nm-initrd-generator as it seems would write this with ipv4 disabled. I don’t really understand if or how I could direct the installer to write the networkmanager configuration instead of relying on nm-initrd-generator which derives it from the kernel options. From the anaconda docs I get the impression that it should be possible to have anaconda configure networkmanager.

What is even more confusing is that if I run /usr/libexec/nm-initrd-generator -s -- ip=... with all the options which are passed to the kernel through grub (e.g. taken from the resolved Kickstart default PXEGrub2 template), it gives me a bond0 interface configuration with the IPv4 address, gateway and dns in place. (ipv6 is disabled, though, as it is not used for booting the kernel).

So I do not understand why /usr/libexec/nm-initrd-generator would disable ipv4 from the kernel options passed not do I understand why the whole network isn’t configured with the static ipv4 and ipv6 address as configured in foreman.

Why doesn’t the server get a working network configuration written into /etc/NetworkManager/system-connections?

Expected outcome:
A networkmanager configuration with a bond interface using the static ipv4 and ipv6 information given by foreman during the installation.

Foreman and Proxy versions:
foreman-3.17.2-1.el9.noarch
katello-4.19.2-1.el9.noarch

Distribution and version:
AlmaLinux 9.7

2

This is starting to drive me crazy. This looks a lot like a race condition somewhere:

First, it’s working occassionally, maybe one of ten builds it just works.

Second, it seems to be working fine if it is a static ip address on the ethernet interface, i.e. without bond.

With bonding and static ip addresses, it looks from the systemd journal during installation that the nmconnection file for the bond is written before networkmanager configures the ip address on the bond during the boot of the installer. nm-initrd-generator or whatever seems to write the ip information during that moment to the file (i.e. no ipv4 nor ipv6 address set, yet)…

3

I think the reason for this issue is the ordering of network interfaces in the kickstart file. By default, the “Kickstart default” template writes it in this order:

network --device=bond0 --hostname s3node13.example.com --mtu=9000 --bootproto static --ip=10.10.10.10 --netmask=255.255.255.0 --gateway=10.10.10.220 --ipv6=2001:dead:beef:25::a/64 --ipv6gateway=2001:dead:beef:25:ffff::ffff --bondslaves=ens6f0np0,ens6f1np1 --bondopts=mode=802.3ad,miimon=100,xmit_hash_policy=layer2+3 --nameserver=10.10.11.4,10.10.11.5,2001:dead:beef:1e::4,2001:dead:beef:1e::5
network --device=de:ad:be:ef:1b:20 --hostname s3node13.example.com --noipv4 --noipv6 --onboot=false --nodns
network --device=de:ad:be:ef:1b:21 --hostname s3node13.example.com --noipv4 --noipv6 --onboot=false --nodns

With this the bond0.nmconnection is usually created ipv4.method=disabled.

If I reorder those and put the bond interface last:

network --device=de:ad:be:ef:1b:20 --hostname s3node13.example.com --noipv4 --noipv6 --onboot=false --nodns
network --device=de:ad:be:ef:1b:21 --hostname s3node13.example.com --noipv4 --noipv6 --onboot=false --nodns
network --device=bond0 --hostname s3node13.example.com --mtu=9000 --bootproto static --ip=10.10.10.10 --netmask=255.255.255.0 --gateway=10.10.10.220 --ipv6=2001:dead:beef:25::a/64 --ipv6gateway=2001:dead:beef:25:ffff::ffff --bondslaves=ens6f0np0,ens6f1np1 --bondopts=mode=802.3ad,miimon=100,xmit_hash_policy=layer2+3 --nameserver=10.10.11.4,10.10.11.5,2001:dead:beef:1e::4,2001:dead:beef:1e::5

It seems to work: at least my last couple of attempts all created a working network configuration.

I have modified line 119 in the Kickstart default template:

# start with provisioning interface, then other non-bond non-bridge interfaces and the bonds + bridges at the end
@host.interfaces.reject{ |iface| iface.bmc? }.sort_by { |iface| (iface.bond? || iface.bridge?) ? 20 : iface.provision? ? 0 : 10 }.each do |iface|

which will put the bond interface last. (I have switched the 20 with 0 and vice versa).