Boot images not generating /var/lib/tftpboot/boot

Support
,
1

Problem:
Seems recently the boot images under /var/lib/tftpboot/boot are not generating for various distros: RHEL, CentOS, Ubuntu etc. I manually copied some and they are working now but Ubuntu is not working how it should be and the image files contain html content with a 404 indicating a lack of existence from the remote end?

Expected outcome:
Working boot images

Foreman and Proxy versions:
foreman-3.5.3-1.el8.noarch
foreman-cli-3.5.3-1.el8.noarch
foreman-debug-3.5.3-1.el8.noarch
foreman-dynflow-sidekiq-3.5.3-1.el8.noarch
foreman.domain.com-apache-1.0-80.noarch
foreman.domain.com-foreman-client-1.0-80.noarch
foreman.domain.com-foreman-proxy-1.0-80.noarch
foreman.domain.com-foreman-proxy-client-1.0-80.noarch
foreman.domain.com-puppet-client-1.0-80.noarch
foreman.domain.com-qpid-broker-1.0-1.noarch
foreman.domain.com-qpid-router-server-1.0-1.noarch
foreman-installer-3.5.3-1.el8.noarch
foreman-installer-katello-3.5.3-1.el8.noarch
foreman-postgresql-3.5.3-1.el8.noarch
foreman-proxy-3.5.3-1.el8.noarch
foreman-release-3.5.3-1.el8.noarch
foreman-selinux-3.5.3-1.el8.noarch
foreman-service-3.5.3-1.el8.noarch
foreman-vmware-3.5.3-1.el8.noarch
katello-4.7.5-1.el8.noarch
katello-certs-tools-2.9.0-1.el8.noarch
katello-client-bootstrap-1.7.9-1.el8.noarch
katello-common-4.7.5-1.el8.noarch
katello-debug-4.7.5-1.el8.noarch
katello-default-ca-1.0-1.noarch
katello-repos-4.7.5-1.el8.noarch
katello-selinux-4.0.2-2.el8.noarch
katello-server-ca-1.0-1.noarch

Distribution and version:
cat /etc/os-release
NAME=“CentOS Stream”
VERSION=“8”
ID=“centos”
ID_LIKE=“rhel fedora”
VERSION_ID=“8”
PLATFORM_ID=“platform:el8”
PRETTY_NAME=“CentOS Stream 8”
ANSI_COLOR=“0;31”
CPE_NAME=“cpe:/o:centos:centos:8”
HOME_URL=“https://centos.org/”
BUG_REPORT_URL=“https://bugzilla.redhat.com/”
REDHAT_SUPPORT_PRODUCT=“Red Hat Enterprise Linux 8”
REDHAT_SUPPORT_PRODUCT_VERSION=“CentOS Stream”

Other relevant data:

2

Hi,
I recommend reading Debugging provisioning, can you upload debug logs from the foreman and smart-proxy so I can see what’s happening?

3

Thanks, I enabled debug for now and am more than farmilar with the provisioning process :slight_smile: Been at this for some years. I find this one odd though. At what point at those images generated/added? When an OS is added or at some other stage?

4

Answered my own question above (when are they generated = at the time of provisioning).

Here are the contents of the vmlinuz and initrd:

[root@host] /var/lib/tftpboot/boot$ cat ubuntu-mirror-4cxZLSj6qxkE-vmlinuz
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at archive.ubuntu.com Port 80</address>
</body></html>

[root@host] /var/lib/tftpboot/boot$ cat ubuntu-mirror-4cxZLSj6qxkE-initrd
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at archive.ubuntu.com Port 80</address>
</body></html>

As an example of trying to provision a new host using 21.04. I will try some other versions as well to see how it works out after removing the current images (backing them up).

5

Same exact results for 20.04 and 22.04.

6

Log shows this (proxy.log):

/var/log/foreman-proxy/proxy.log:2023-05-25T21:54:01 c0d2b9af [I] [2598909] Started task /usr/bin/curl\ --silent\ --show-error\ --connect-timeout\ 10\ --retry\ 3\ --retry-delay\ 10\ --max-time\ 3600\ --remote-time\ --time-cond\ /v
ar/lib/tftpboot/boot/ubuntu-mirror-4cxZLSj6qxkE-vmlinuz\ --write-out\ Task\ done,\ result:\ \%\{http_code\},\ size\ downloaded:\ \%\{size_download\}b,\ speed:\ \%\{speed_download\}b/s,\ time:\ \%\{time_total\}ms\ --output\ /var/li
b/tftpboot/boot/ubuntu-mirror-4cxZLSj6qxkE-vmlinuz\ --location\ http://archive.ubuntu.com/ubuntu/casper/vmlinuz
/var/log/foreman-proxy/proxy.log:2023-05-25T21:54:01 c0d2b9af [I] [2598914] Started task /usr/bin/curl\ --silent\ --show-error\ --connect-timeout\ 10\ --retry\ 3\ --retry-delay\ 10\ --max-time\ 3600\ --remote-time\ --time-cond\ /v
ar/lib/tftpboot/boot/ubuntu-mirror-4cxZLSj6qxkE-initrd\ --write-out\ Task\ done,\ result:\ \%\{http_code\},\ size\ downloaded:\ \%\{size_download\}b,\ speed:\ \%\{speed_download\}b/s,\ time:\ \%\{time_total\}ms\ --output\ /var/lib
/tftpboot/boot/ubuntu-mirror-4cxZLSj6qxkE-initrd\ --location\ http://archive.ubuntu.com/ubuntu/casper/initrd
/var/log/foreman-proxy/proxy.log:2023-05-25T22:16:06 4552605b [I] [2601825] Started task /usr/bin/curl\ --silent\ --show-error\ --connect-timeout\ 10\ --retry\ 3\ --retry-delay\ 10\ --max-time\ 3600\ --remote-time\ --time-cond\ /v
ar/lib/tftpboot/boot/ubuntu-mirror-rf32u3HGTMZf-vmlinuz\ --write-out\ Task\ done,\ result:\ \%\{http_code\},\ size\ downloaded:\ \%\{size_download\}b,\ speed:\ \%\{speed_download\}b/s,\ time:\ \%\{time_total\}ms\ --output\ /var/li
b/tftpboot/boot/ubuntu-mirror-rf32u3HGTMZf-vmlinuz\ --location\ http://archive.ubuntu.com/ubuntu/casper/vmlinuz
/var/log/foreman-proxy/proxy.log:2023-05-25T22:16:06 4552605b [I] [2601830] Started task /usr/bin/curl\ --silent\ --show-error\ --connect-timeout\ 10\ --retry\ 3\ --retry-delay\ 10\ --max-time\ 3600\ --remote-time\ --time-cond\ /v
ar/lib/tftpboot/boot/ubuntu-mirror-rf32u3HGTMZf-initrd\ --write-out\ Task\ done,\ result:\ \%\{http_code\},\ size\ downloaded:\ \%\{size_download\}b,\ speed:\ \%\{speed_download\}b/s,\ time:\ \%\{time_total\}ms\ --output\ /var/lib
/tftpboot/boot/ubuntu-mirror-rf32u3HGTMZf-initrd\ --location\ http://archive.ubuntu.com/ubuntu/casper/initrd
7

18.04 works

2023-05-25T22:34:09 03ac4365 [I] Finished POST /tftp/fetch_boot_file with 200 (4.88 ms)
2023-05-25T22:34:09 03ac4365 [I] [2604196] Started task /usr/bin/curl\ --silent\ --show-error\ --connect-timeout\ 10\ --retry\ 3\ --retry-delay\ 10\ --max-time\ 3600\ --remote-time\ --time-cond\ /var/lib/tftpboot/boot/ubuntu-mirro
r-siJBlGH2vpRc-linux\ --write-out\ Task\ done,\ result:\ \%\{http_code\},\ size\ downloaded:\ \%\{size_download\}b,\ speed:\ \%\{speed_download\}b/s,\ time:\ \%\{time_total\}ms\ --output\ /var/lib/tftpboot/boot/ubuntu-mirror-siJBl
GH2vpRc-linux\ --location\ http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/linux
2023-05-25T22:34:09  [D] Rack::Handler::WEBrick is invoked.
2023-05-25T22:34:09 03ac4365 [I] Started POST /tftp/fetch_boot_file
2023-05-25T22:34:09 03ac4365 [D] verifying remote client 12.3.4.5 against trusted_hosts ["foreman.domain.com"]
2023-05-25T22:34:09 03ac4365 [I] Finished POST /tftp/fetch_boot_file with 200 (2.92 ms)
2023-05-25T22:34:09 03ac4365 [I] [2604201] Started task /usr/bin/curl\ --silent\ --show-error\ --connect-timeout\ 10\ --retry\ 3\ --retry-delay\ 10\ --max-time\ 3600\ --remote-time\ --time-cond\ /var/lib/tftpboot/boot/ubuntu-mirro
r-siJBlGH2vpRc-initrd.gz\ --write-out\ Task\ done,\ result:\ \%\{http_code\},\ size\ downloaded:\ \%\{size_download\}b,\ speed:\ \%\{speed_download\}b/s,\ time:\ \%\{time_total\}ms\ --output\ /var/lib/tftpboot/boot/ubuntu-mirror-s
iJBlGH2vpRc-initrd.gz\ --location\ http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/initrd.gz
8

This URL is 404:

http://archive.ubuntu.com/ubuntu/casper/initrd

curl http://archive.ubuntu.com/ubuntu/casper/initrd

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at archive.ubuntu.com Port 80</address>
</body></html>
9

Yes I noticed that and casper seems to be this? Ubuntu Manpage: casper - a hook for initramfs-tools to boot live systems. but something isn’t working correctly.

Not sure if something needs to be updated here?

/var/lib/foreman/public/webpack/foreman-vendor.bundle-v10.1.7-production-57ed7676cc22ce47f0a6.js
/var/lib/foreman/public/webpack/foreman-vendor.bundle-v10.1.7-production-57ed7676cc22ce47f0a6.js.map

Those files contain some information about casper as it related to ubuntu.

10

And this discovery lead me here: Foreman 3.4 change for subiquity breaks generic bootdisk / preseed iPXE install of Ubuntu 20.04 and above

11

Which didn’t really help anything in my case. Still 404 on those images.

12

And more to here: Provisioning Hosts

I will post my solution tomorrow since this alone did not work but I am at least booting from 22.04 now.

13

A PR for iPXE and DHCP Ubuntu autoinstall has been accepted into Foreman 3.7 which has an RC. Maybe try that and see if it solves your issue.

Note: you still need to manually setup the files to download because Ubuntu doesn’t provide everything needed online anymore. See here in the docs.

14

Doesn’t look like 3.7 is ready yet due to katello not being ready.

https://docs.theforeman.org/3.7/Release_Notes/index-katello.html#katello-release-notes