Update driver included in discovery image

Problem:

Hey guys! We would like to provision some new workstation systems with an installed Intel l219-LM network adapter through the foreman discovery image. Unfortunately the installed network card, needs a more recent intel e1000e driver version, to initialize the installed network card. (https://access.redhat.com/solutions/2521021)

As a quick solution, I installed the latest driver version via rpm from https://centos.pkgs.org/7/elrepo-x86_64/kmod-e1000e-3.8.4-2.el7_8.elrepo.x86_64.rpm.html on my local machine, compressed the included e1000e.ko and rebuild the initrd included in the discovery-image with the downloaded kernel module.

Basically I did the following:

  1. Extract the initrd

    /usr/lib/dracut/skipcpio initrd0.img | gunzip -c | cpio -i -d

  2. Compress the provided e1000e.ko

  3. Overwrite the included e1000e.ko file in the initrd directory tree

  4. Rebuild the initramdisk

    find . | cpio --create --format=‘newc’ > /tmp/newinitramfs
    gzip -S .img /tmp/newinitramfs

  5. Overwrite the initrd0.img included in fdi

  6. Generate a new fdi with mkisofs

Unfortunately, when I boot from this newly created fdi, log into the live system and load the included e1000e.ko kernel module via modprobe, I get the old version of this driver module!? Where does this module come from? Is it included in the kernel used by the fdi?

The more recent version of this kernel module, which I overwrit before is correctly included in the generated /isolinux/initrd0.img !?

Extracted initrd0.img on the live system:

[root@fdi tmp]# modinfo ./usr/lib/modules/3.10.0-1062.1.2.el7.x86_64/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko.xz
filename:       /tmp/./usr/lib/modules/3.10.0-1062.1.2.el7.x86_64/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko.xz
**version:        3.8.4-NAPI**
license:        GPL
description:    Intel(R) PRO/1000 Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
retpoline:      Y
rhelversion:    7.8
srcversion:     5507E646B9C1006005CF01C

Included e1000e.ko.xz in /usr/lib/modules/

[root@fdi ~]# modinfo e1000e
filename:       /lib/modules/3.10.0-1062.1.2.el7.x86_64/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko.xz
**version:        3.2.6-k**
license:        GPL v2
description:    Intel(R) PRO/1000 Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
retpoline:      Y
rhelversion:    7.7
srcversion:     043110184A2E8C433DE02E1

Maybe @lzap could help? :slight_smile: Is there any better way to update the included driver?

Many thanks & Best regards!

Expected outcome:

The newly created initrd should contain the included version of the kernel module

Foreman and Proxy versions:

foreman-2.1.4-1.el7.noarch
fdi-bootable-3.5.7.iso

Distribution and version:

NAME=“Red Hat Enterprise Linux Server”
VERSION=“7.8 (Maipo)”

Can’t you just add the elrepo to the kickstart and rebuild the image according to the README? I am not huge fan of regenerating init RAM disks to be honest.

Anyway, I noticed the RPM carries the driver on a different path: /lib/modules/3.10.0-1127.19.1.el7.x86_64/extra/e1000e/e1000e.ko note the extra directory while you are modprobing the one from the old path? I am not very familiar with kmod to be honest, kmod is kind of a hack that NVidia users are used to. I am happy AMD (Radeon-Ryzen) users and I did not have to mess around with kmod for many years now :slight_smile:

So my advice - try to load the correct file first, if it does not work just rebuild the image, however since the driver has a same name, I don’t know which one will be picked up first.

Of course kmod was not built only for NVidia, this rant was irrelevant, but… could not help it! :slight_smile:

Wow many thanks for the quick response! :slight_smile:

I will check the README how to update the included driver. Just to clarify:

I copied the driver provided by the mentioned rpm manually to the extracted initrd tree, so the original e1000e.ko file included in the initrd0.img gets overwritten.

After that, I build a new initrd0.img and included this newly created initrd0.img in a new discovery-image and made a new iso-file via mkisofs.

Iam wondering how its even possible that I get the old version of this driver after booting from this newly created iso? I thought the /isolinux/initrd0.img gets mounted by the kernel when I start the discovery-image? There is not even an old version of the e1000e driver included in the generated initrd0.img!? :face_with_monocle:

I am not sure in your case, but after the ISO file is built (which includes kernel and initramdisk of course), then there is a script https://github.com/theforeman/foreman-discovery-image/blob/master/aux/livecd-iso-to-pxeboot which extract the files (kernel, initramdisk) and then concatenates the init RAM disk + the ISO file. When correct kernel command options are provided, dracut detects the concatenated ISO and mounts it. This is how LiveCD was built and discovery is no different. So maybe you have two different ramdisks you are working with?

Also note that init RAM disk is dropped and remounted with the actual filesystem (from the ISO) at some point during boot, if you haven’t changed that (and only the RAM disk) you will see the old file.

You can drop any file to the root folder here:

During build, discovery overwrites all files from this directory on the target system as the last build step. You can overwrite pretty much anything, but I am not sure if this happens before initramdisk is generated or after. It’s in %post section of the kickstart.

Thank you for the explanation. :slight_smile: I didn’t use the mentioned script to built the new discovery-image. I basically did the following:

  1. Download the mentioned driver-rpm

  2. Extract the content to get the driver

# rpm2cpio kmod-e1000e-3.8.4-2.el7_8.elrepo.x86_64.rpm | cpio -idmv

  1. Find the e1000e.ko from the rpm and compress it

# xz -z e1000e.ko

  1. loop-mount and copy the content of the latest foreman discovery image to some directory

  2. Extract the included /isolinux/intrd0.img

# /usr/lib/dracut/skipcpio initrd0.img | gunzip -c | cpio -i -d

  1. Copy the new driver into the directory tree from the extracted initrd0.img

# cp /tmp/lib/modules/3.10.0-1127.19.1.el7.x86_64/extra/e1000e/e1000e.ko.xz /initramdisk/usr/lib/modules/3.10.0-1062.1.2.el7.x86_64/kernel/drivers/net/ethernet/intel/e1000e/

  1. Generate a new initrd

# find . | cpio --create --format='newc' > /tmp/newinitramfs

  1. Compress it

# gzip -S .img /tmp/newinitramfs

  1. Copy the new initrd file to the extracted disovery image, to overwrite the old one

  2. Go to the extracted discovery image directory

  3. Generate a new discovery-image via mkisofs

mkisofs -quiet -U -A "fdi" -V "fdi" -volset "fdi" -J -joliet-long -r -v -T -o ../bootdisk_fdi_new_e1000e.iso -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -input-charset utf-8 -boot-info-table -eltorito-alt-boot -e isolinux/efiboot.img -no-emul-boot .

After booting the newly created discovery-image there should be no old driver-version available? Or how is this even possible? :frowning_face:

EDIT: I got it! :smiley: … the driver is included in the squashfs.img

Puhh atleast I won’t be having any nightmares now. :smiley: I was wondering and investigating for multiple hours to understand where the old driver version comes from.

Greetings!

Hey @lzap :slight_smile:

Can I create a new discovery image following the official guide https://github.com/theforeman/foreman-discovery-image based on the CentOS8 repository? Do you know if this would work?

CentOS8.2 include the necessary driver for my Intel e1000e card.

If thats not possible, could you give me a hint how I get the driver into the discovery image, so that the discovery image would recognize my network adapter?

many thanks!

That is what I was trying to say with “root is remounted during boot”.

We haven’t tested this yet, maybe. But FDI requires some packages from our Foreman YUM repositories and these are EL7 only, you will get some troubles. Going forward we would like to get rid of these dependencies and possibly completely drop Ruby and only have FDI based on SSH connections and simple HTTP callbacks. Before we update to CentOS8 I would like to work on simplifying first.

Create a YUM repository with repocreate containing newest kernel from Fedora, add the repository to the 00-repos-centos7.ks and build the FDI using instructions. This might work, not sure.

Alternatively, if you are Red Hat customer, call in and ask for driver backport, however I think RHEL7 passed the milestone and only security fixes go in at this point.

@Marek_Hulan FYI this is the first time someone is having troubles with FDI because it has way too old kernel. We might get CentOS8 upgrade prioritized a bit.

Hi @lzap,

the build-livecd script creates an archive including kernel + initrd for booting via PXE.

Could you give me a quick guide how to generate a bootable ISO from this?

Many thanks & best regards!

Got it! :slight_smile:

As described in https://www.youtube.com/watch?v=QPQZThbMCbY I’ve got the .iso after running livecd-creator directly.

Hi @lzap,

Ive build a new fdi image including the e100e driver, but unfortunately the generated iso fails to start :frowning:

I got some (minor) failures while building, ‘unknown filesystem type’ and some repo erros, cause im building behind a http proxy.

https://pastebin.com/raw/y7Y9QLrN

Do you have an idea whats going on? I set the http_proxy variable, the package-download is successful, but there seems to be connection problems inside the chroot environment created by the livecd-creator!?

Iam using RHEL 7.8 as build host and installed all dependencies mentioned on github

Command Line   : install hfsplus-tools-540.1.linux3-4.el7.x86_64.rpm livecd-tools-21.4-2.el7.x86_64.rpm python-imgcreate-21.4-2.el7.x86_64.rpm
Transaction performed with:
    Installed     rpm-4.11.3-43.el7.x86_64                    @rhel-7-server-rpms
    Installed     subscription-manager-1.24.26-3.el7_8.x86_64 @rhel-7-server-rpms
    Installed     yum-3.4.3-167.el7.noarch                    @rhel-7-server-rpms
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64     @anaconda/7.6
Packages Altered:
    Dep-Install GConf2-3.2.6-8.el7.x86_64                               @rhel-7-server-rpms
    Dep-Install dosfstools-3.0.20-10.el7.x86_64                         @rhel-7-server-rpms
    Dep-Install dumpet-2.1-8.el7.x86_64                                 @rhel-7-server-rpms
    Install     hfsplus-tools-540.1.linux3-4.el7.x86_64                 @/hfsplus-tools-540.1.linux3-4.el7.x86_64
    Dep-Install isomd5sum-1:1.0.10-5.el7.x86_64                         @rhel-7-server-rpms
    Install     livecd-tools-1:21.4-2.el7.x86_64                        @/livecd-tools-21.4-2.el7.x86_64
    Dep-Install lorax-19.7.26-1.el7.x86_64                              @rhel-7-server-rpms
    Dep-Install plymouth-graphics-libs-0.8.9-0.33.20140113.el7.x86_64   @rhel-7-server-rpms
    Dep-Install plymouth-plugin-label-0.8.9-0.33.20140113.el7.x86_64    @rhel-7-server-rpms
    Dep-Install plymouth-plugin-two-step-0.8.9-0.33.20140113.el7.x86_64 @rhel-7-server-rpms
    Dep-Install psmisc-22.20-16.el7.x86_64                              @rhel-7-server-rpms
    Dep-Install pykickstart-1.99.66.21-1.el7.noarch                     @rhel-7-server-rpms
    Dep-Install pyparted-1:3.9-15.el7.x86_64                            @rhel-7-server-rpms
    Dep-Install python-beaker-1.5.4-10.el7.noarch                       @rhel-7-server-rpms
    Install     python-imgcreate-1:21.4-2.el7.x86_64                    @/python-imgcreate-21.4-2.el7.x86_64
    Dep-Install python-mako-0.8.1-2.el7.noarch                          @rhel-7-server-rpms
    Dep-Install python-paste-1.7.5.1-9.20111221hg1498.el7.noarch        @rhel-7-server-rpms
    Dep-Install python-tempita-0.5.1-6.el7.noarch                       @rhel-7-server-rpms
    Dep-Install redhat-upgrade-dracut-0.8.10-1.el7.x86_64               @rhel-7-server-rpms
    Dep-Install redhat-upgrade-dracut-plymouth-0.8.10-1.el7.noarch      @rhel-7-server-rpms
    Dep-Install squashfs-tools-4.3-0.21.gitaae0aff4.el7.x86_64          @rhel-7-server-rpms
    Dep-Install syslinux-extlinux-4.05-15.el7.x86_64                    @rhel-7-server-rpms
    Dep-Install system-config-keyboard-1.4.0-5.el7.noarch               @rhel-7-server-rpms
    Dep-Install system-config-keyboard-base-1.4.0-5.el7.noarch          @rhel-7-server-rpms

Command Line   : install livecd-tools pykickstart isomd5sum syslinux grub2-efi shim grub2-efi-x64 grub2-efi-x64-cdboot shim-x64
Transaction performed with:
    Installed     rpm-4.11.3-43.el7.x86_64                    @rhel-7-server-rpms
    Installed     subscription-manager-1.24.26-3.el7_8.x86_64 @rhel-7-server-rpms
    Installed     yum-3.4.3-167.el7.noarch                    @rhel-7-server-rpms
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64     @anaconda/7.6
Packages Altered:
    Install grub2-efi-x64-cdboot-1:2.02-0.81.el7.x86_64 @rhel-7-server-optional-rpms

I commented out the following lines in build-livecd-root script to get the desired iso file, unfortunately that didnt work either :frowning:

 56 #echo "* Converting to initrd"
 57 #bash -x $srcdir/aux/livecd-iso-to-pxeboot $tmpdir/fdi.iso
 58 #
 59 #if [ $? -ne 0 ]; then
 60 #    KEEP_TMPDIR=yes
 61 #    echo "Error converting to initrd"
 62 #    exit 1
 63 #fi
 64 #
 65 #echo "* Building tarball"
 66 #mkdir fdi-image
 67 #mv tftpboot/initrd0.img tftpboot/vmlinuz0 fdi-image || KEEP_TMPDIR=yes
 68 #cd fdi-image
 69 #cat > README <<EOF
 70 #This is Foreman discovery image
 71 #
 72 #To get instructions how to use the image head over to
 73 #
 74 #https://github.com/theforeman/foreman_discovery
 75 #
 76 #Image was built from $last_tag ($last_sha)
 77 #
 78 #To verify the kernel and initrd in this tar, run
 79 #
 80 #  sha256sum -c SHA256SUM
 81 #EOF
 82 #
 83 #sha256sum initrd0.img vmlinuz0 > SHA256SUM
 84 #
 85 #cd ..
 86 #tarball=$destdir/fdi-image-${last_tag#release-}.tar
 87 #tar cf $tarball fdi-image/
 88 #ls -lh $tarball
 89 #chown --reference $srcdir/build-livecd $tarball
 90 #
 91 #echo "* Done"
 92 #echo "The image was built:"
 93 #du -h $tarball
 94 exit 0