Set tftp kernel/initrd path depending on installation media

Zugschlus · February 7, 2018, 3:51pm

Hi,

in my current foreman installation, we need to be able to install not only the latest CentOS-Version but olders as well. We do have an Operating System “CentOS latest” and four sets of installation media according to the different stages of production readiness. We want to be able to directly install operating systems with all sets of installation media.

Recently (I suspect that was with the inclusion of CentOS 7.4 into the production stage) this is no longer possible. I have found out that there is either a mismatch of the kernel/initrd loaded via TFTP with the older installation media set (if the kernel/initrd is from the later release) or incompatible with the newer installation media set (if the kernel/initrd is from the earlier release). The bug manifests itself in the CentOS installer not being able to load the file system modules (probably due to a kernel ABI mismatch), ending up in a cryptic “help I don’t know any XFS file system” from the installer.

The PXELinux template uses variables @kernel and @initrd which seem to expand to “boot/CentOS-$major-$arch-vmlinuz” and “boot/CentOS-$major-$arch-initrd.img” respectively.

Where do those variables come from? They’re neither defined in the PXELinux template themselves nor documented in the TemplateWriting wiki page. How can I make them expand to “boot/CentOS-$major-$minor-$arch-vmlinuz” and “boot/CentOS-$major-$minor-$arch-vmlinuz” respectively?

Thanks!

Greetings
Marc

Zugschlus · February 7, 2018, 3:57pm

I tried editing my post but that does not seem to be allowed.

Which Variables are available at the time when the PXE Template is expanded? Can I use all host variables already? Does the mechanism require that all installation media that are associated with an Operating System are in sync with the (single!) kernel/initrd combo?

Greetings
Marc

lzap · February 7, 2018, 4:48pm

This is one of the “weird errors” when your initramdisk is corrupt or does not match kernel. Redownload them manually and do md5sum.

lzap · February 7, 2018, 4:54pm

Source (where Foreman downloads from) is defined here:

github.com

theforeman/foreman/blob/develop/app/models/operatingsystems/redhat.rb#L2


class Redhat < Operatingsystem
PXEFILES = {:kernel => "vmlinuz", :initrd => "initrd.img"}


# outputs kickstart installation medium based on the medium type (NFS or URL)
# it also convert the $arch string to the current host architecture
def mediumpath(host)
  uri = medium_uri(host)


  case uri.scheme
    when 'http', 'https', 'ftp'
      "url --url #{uri}"
    else

Target (naming convention) is then here:

github.com

theforeman/foreman/blob/develop/app/models/concerns/orchestration/tftp.rb#L122-L134


def setTFTPBootFiles
  logger.info "Fetching required TFTP boot files for #{host.name}"
  valid = []
  host.operatingsystem.pxe_files(host.medium, host.architecture, host).each do |bootfile_info|
    for prefix, path in bootfile_info do
      valid << each_unique_feasible_tftp_proxy do |proxy|
        proxy.fetch_boot_file(:prefix => prefix.to_s, :path => path)
      end
    end
  end
  failure _("Failed to fetch boot files") unless valid.all?
  valid.all?
end

You need to drill down a callstack bit to understand. This is hard-coded.

Zugschlus · February 8, 2018, 8:03am

You need to drill down a callstack bit to understand. This is hard-coded.

Ok, then it is a conceptual mistake in the code that every user needs to fix locally. I will open an issue.

Ignoring the rest of your help since you obviously have not understood the issue at hand.

Zugschlus · February 8, 2018, 8:50am

I apologize for my last sentence. It was written before the first coffee in the morning.

Gwmngilfen · February 8, 2018, 10:25pm

There’s a few things that maybe I can help out with here…

Indeed, this is because many of our users interact with the forum by email, and edits after the first few mins won’t be emailed, which can lead to confusion later on. We keep edits to a 5 min window for quick typo fixes and suchlike.

You can, indeed, so re-writing the PXE template is an option if you wish. It’s true that the newer CentOS 7 repos have caused some issue due to the inclusion of the minor string, but I believe the CentOS installs should work with the default templates/media/etc - they are tested with the upstream mirrors fairly frequently. If I understand you correctly, the issue here is not with the latest repo but some older versions? Some steps to reproduce the issue would be great, if you can go into detail.

What Lukas is referring to is the possibility that the downloaded vmlinuz / initrd has gotten borked - it’s not obvious, but the proxy only downloads the vmlinuz/initrd to the TFTP server if the file is not present. If a file is present, it won’t touch it. That can lead to a situation where the file is zero-length or incomplete, but the proxy doesn’t attempt to fix it - hence the suggestion of verifying the checksums.

Hope that helps!

lzap · February 9, 2018, 12:50pm

Sometimes you don’t know the answer, but you see an error that is weird enough that it might be useful for googlers to have possible explanations. And that’s this case.

If you update kernel/initram disk for a repository, the way it’s implemented today will corrupt your files if the updated source file is larger than the old one. That’s the behavior of wget and we use that. It happens in real-world scenarios, we’ve seen this for Red Hat customers (kickstart was refreshed on CDN at least once, when using Katello plugin and changing OS variant from Server to Workstation and back it also happens).

Zugschlus · March 1, 2018, 11:54am

That’s how I solved the issue.

How can I force the proxy to re-download the files? I understand that this won’t happen automatically after removing those files?

Greetings
Marc

lzap · March 1, 2018, 1:57pm

Exit build mode (Cancel build button), then enter it again.

We also have some button somewhere to redeploy templates for hosts, I can’t remember. This works anyway.

Zugschlus · March 9, 2018, 1:21pm

Since the site in question seems to be using installation media in a way that they were not supposed to be used, it is still not possible for us to rely on the automatic download of the kernel and initrd files. I have documented this locally and tweaked the PXE template to pull in additional information about the kernel and initrd combo to be used from a host group variable.
I am not going to go more in-depth about that since it’s a not recommended use case in addition to being quite ugly anyway, but I wanted you to know that with your help I was able to put a fix on locally caused breakage. Thanks!