Several host params do not make it to pxeconfig files

Problem:
When we create a new baremetal host in Foreman with some parameters, like kernelcmd and/or blacklist, some of them do not make it to actual file written on TFTP proxy even though UI shows all of those params in “review” tab. If we cancel that build and try to build it again w/o changing anything else, all parameters would end in pxeconfig file on TFTP.

Expected outcome:
All parameters make it to pxeconfig files on first try.

Foreman and Proxy versions:
3.3.0

Foreman and Proxy plugin versions:
3.3.0

Distribution and version:
centos 7.9.2009

Other relevant data:

my pxegrub2 template as an example:

<snip header>
blacklist = <%= host_param('blacklist') %>
<%
  options = []
  if host_param('blacklist')
    options << "modprobe.blacklist=" + host_param('blacklist').gsub(' ', '')
  end
  options = options.join(' ')
-%>
options = <%= options %>

console_device = <%= host_param('console_device') %>
<%
if @host.architecture.to_s != 'aarch64'
    if host_param('console_device')
        console_device = host_param('console_device')
    else
        console_device = @host.facts['cmdline_console'] || 'ttyS1,115200n8'
    end
    console_device = 'console=' + console_device
end
-%>
console_device = <%= console_device %>

kernelcmd = <%= host_param('kernelcmd') %>
<%
# If extra PXE kernel options are required for a system, they can be passed through "kernelcmd" host parameter
  pxe_kernel_options = []
  if host_param('kernelcmd')
    pxe_kernel_options << host_param('kernelcmd').gsub(',', ' ')
  end
  pxe_kernel_options = pxe_kernel_options.join(' ')
-%>
pxe_kernel_options = <%= pxe_kernel_options %>

set default=0
set timeout=<%= host_param('loader_timeout') || 0 %>

menuentry '<%= template_name %>' {
  <% if @host.operatingsystem.name.match(/.*atomic.*/i) -%>
  linux <%= @kernel %> ks=<%= foreman_url('provision') %> repo=<%= medium_uri %> ks.device=bootif network ks.sendmac <%= options %> <%= pxe_kernel_options %> <%= console_device %>
  <% elsif @host.operatingsystem.name == 'Fedora' and @host.operatingsystem.major.to_i > 16 -%>
  linux <%= @kernel %> ks=<%= foreman_url('provision') %> ks.device=bootif network ks.sendmac inst.ssh text ip=dhcp <%= options %> <%= pxe_kernel_options %> <%= console_device %>
  <% elsif @host.operatingsystem.name != 'Fedora' and @host.operatingsystem.major.to_i >= 7 -%>
  linux <%= @kernel %> ro ip=dhcp inst.sshd ks=<%= foreman_url('provision') %> ramdisk_size=8416 ksdevice=bootif cmdline quiet kssendmac <%= options %> <%= pxe_kernel_options %> <%= console_device %>
  <% else -%>
  linux <%= @kernel %> ks=<%= foreman_url('provision') %> network ks.sendmac <%= options %> <%= pxe_kernel_options %> <%= console_device %>
  <% end -%>
  initrd <%= @initrd %>
}

The host is created with the following parameters and starts the build:

If I look at host’ templates under “legacy UI” → “templates” → “review”, I see the following generated for it from above template:

<snip header>
blacklist = blah, blew
options = modprobe.blacklist=blah,blew

console_device = ttyS0,115200n8
console_device = console=ttyS0,115200n8

kernelcmd = modprobe.blacklist=nouveau
pxe_kernel_options = modprobe.blacklist=nouveau

set default=0
set timeout=0

menuentry 'ebay_kickstart_PXEGrub2' {
    linux boot/centos-x86_64-latest-CUMHUwVqk9pL-vmlinuz ro ip=dhcp inst.sshd ks=http://<foreman-host>/unattended/provision?token=1e824984-9fb6-4d94-8a5d-2e945ca11296 ramdisk_size=8416 ksdevice=bootif cmdline quiet kssendmac modprobe.blacklist=blah,blew modprobe.blacklist=nouveau console=ttyS0,115200n8
    initrd boot/centos-x86_64-latest-CUMHUwVqk9pL-initrd.img
}

Which is exactly what one would expect. However, the resulting grub.cfg-01- file on TFTP proxy has the following content:

<snip header>
blacklist =
options =

console_device =
console_device = console=ttyS1,115200n8

kernelcmd =
pxe_kernel_options =

set default=0
set timeout=0

menuentry 'kickstart_PXEGrub2' {
    linux boot/centos-x86_64-latest-CUMHUwVqk9pL-vmlinuz ro ip=dhcp inst.sshd ks=http://<foreman-host>/unattended/provision?token=1e824984-9fb6-4d94-8a5d-2e945ca11296 ramdisk_size=8416 ksdevice=bootif cmdline quiet kssendmac   console=ttyS1,115200n8
    initrd boot/centos-x86_64-latest-CUMHUwVqk9pL-initrd.img
}

Just as if no parameters were passed to my template at the time of generating that file. No errors are in any logs. If I cancel the build of that host, and click on “build” again, my grub.cfg-01- has expected content:

<snip header>
blacklist = blah, blew
options = modprobe.blacklist=blah,blew

console_device = ttyS0,115200n8
console_device = console=ttyS0,115200n8

kernelcmd = modprobe.blacklist=nouveau
pxe_kernel_options = modprobe.blacklist=nouveau

set default=0
set timeout=0

menuentry 'kickstart_PXEGrub2' {
    linux boot/centos-x86_64-latest-CUMHUwVqk9pL-vmlinuz ro ip=dhcp inst.sshd ks=http://<foreman-host>/unattended/provision?token=2a32d69d-6751-418b-8157-bca39c5fcc1c ramdisk_size=8416 ksdevice=bootif cmdline quiet kssendmac modprobe.blacklist=blah,blew modprobe.blacklist=nouveau console=ttyS0,115200n8
    initrd boot/centos-x86_64-latest-CUMHUwVqk9pL-initrd.img
}

How can I troubleshoot this further? Anyone else sees this behavior? It feels like other people should have ran into this if they use params, of course. We did not use earlier, and that’s probably why we never ran into this earlier.

Anyway, any suggestion/pointers are very much appreciated!
Thanks!

Additional thing to note - above situation only affects pxeconfig files, but not other templates like postinstall and/or provisioning ones.

And this only happens when host being created is marked for immediate provisioning. If I make 2 separate calls, one to create a host w/o provisioning and second to build it, pxeconfigs generated perfectly fine.

I was also able to replicate this behavior on older version for Foreman 1.15.6 as well.

Thanks!

I’m very surprised to see no responses at all on this - are we the only ones having/seeing this issue?

@lzap, @Marek_Hulan - do you, guys, have any thoughts/suggestions on this?

Thanks!

I don’t think I’ve run into the specific case of parameters set during host creation, but I have gotten used to reflexively canceling and re-setting-to-build any time I make changes. It’s inconsistent with the preview, as you say, but since it’s having to get written out to disk rather than being generated on demand I assume it’s just impractical to regenerate on every change that might make a difference. For parameters set during host creation, I agree that it would make sense for the generation done then to include the parameters, but the part of the host creation process that sets those parameters must be happening afterwards. Maybe there’s another dependency that requires that.