Discovery fails: Service unavailable 503 and "cannot determine type of /tmp/vmlinuz"

ahoiHGF · February 16, 2023, 8:54am

Problem:

I try to use the discovery image to connect to a foreman-proxy and install Debian 11.

Unfortunately it does not work - the proxy states in the logs:

2023-02-16T09:49:23 11703840 [I] Started GET /unattended/templateServer 
2023-02-16T09:49:23 11703840 [I] Finished GET /unattended/templateServer with 200 (0.45 ms)
2023-02-16T09:49:23 11703840 [I] Started POST /tftp/PXELinux/00:0c:29:46:48:a1 
2023-02-16T09:49:23 11703840 [I] Finished POST /tftp/PXELinux/00:0c:29:46:48:a1 with 200 (0.85 ms)
2023-02-16T09:49:23 11703840 [I] Started GET /unattended/templateServer 
2023-02-16T09:49:23 11703840 [I] Finished GET /unattended/templateServer with 200 (0.38 ms)
2023-02-16T09:49:23 11703840 [I] Started POST /tftp/PXEGrub2/00:0c:29:46:48:a1 
2023-02-16T09:49:23 11703840 [I] Finished POST /tftp/PXEGrub2/00:0c:29:46:48:a1 with 200 (1.02 ms)
2023-02-16T09:49:23 11703840 [I] Started POST /tftp/fetch_boot_file 
2023-02-16T09:49:23 11703840 [I] Finished POST /tftp/fetch_boot_file with 200 (1.35 ms)
2023-02-16T09:49:23 11703840 [I] [2572] Started task ["/usr/bin/wget", "--connect-timeout=10", "--dns-timeout=10", "--read-timeout=60", "--tries=3", "-nv", "-c", "http://ftp.debian.org/debian/dists/bullseye/main/installer-amd64/current/images/netboot/debian-installer/amd64/linux", "-O", "/var/lib/tftpboot/boot/debian-mirror-DNCFDjRZGWUw-linux"]
2023-02-16T09:49:23 11703840 [I] Started POST /tftp/fetch_boot_file 
2023-02-16T09:49:23 11703840 [I] Finished POST /tftp/fetch_boot_file with 200 (0.77 ms)
2023-02-16T09:49:23 11703840 [I] [2575] Started task ["/usr/bin/wget", "--connect-timeout=10", "--dns-timeout=10", "--read-timeout=60", "--tries=3", "-nv", "-c", "http://ftp.debian.org/debian/dists/bullseye/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz", "-O", "/var/lib/tftpboot/boot/debian-mirror-DNCFDjRZGWUw-initrd.gz"]
2023-02-16T09:49:23 11703840 [W] [2572] http://ftp.debian.org/debian/dists/bullseye/main/installer-amd64/current/images/netboot/debian-installer/amd64/linux:

2023-02-16T09:49:23 11703840 [W] [2572] 2023-02-16 09:49:23 ERROR 503: Service Unavailable.

Expected outcome:

Installation succeeds

Foreman and Proxy versions:

Version

3.0.2

Active features

Discovery
Dynflow
Logs
Puppet
Puppet CA
Registration
SSH
Templates
TFTP

Discovery

Version

1.0.5

Dynflow

Version

0.5.2

Registration

Version

3.0.2

SSH

Version

0.4.1

TFTP

Version

3.0.2

TFTP server

false

Templates

Version

3.0.2

Foreman and Proxy plugin versions:

Discovery

Version

1.0.5

Dynflow

Version

0.5.2

Registration

Version

3.0.2

SSH

Version

0.4.1

TFTP

Version

3.0.2

TFTP server

false

Templates

Version

3.0.2

ahoiHGF · February 16, 2023, 9:02am

This is what the discovery image is writing to the logs - for me, the error “cannot determine type of /tmp/vmlinuz” could be the critical point?:

Dirk · February 16, 2023, 9:42am

For some reason you got a 503 Service unavailable when the systems tried to download from the mirror. So very likely the file on the smart proxy is empty or contains only some html, you can delete it and retry the download by setting build mode for a system using this OS. If the download still fails, you need to look at this problem. Perhaps if you are running the download via proxy content inspection is causing the problem.

ahoiHGF · February 16, 2023, 9:50am

Hi @Dirk and thank you for your response.

I checked /var/lib/tftpboot/boot/ and based on the dates, I got only this file:

debian-mirror-DNCFDjRZGWUw-linux

ls -lah /var/lib/tftpboot/boot/
insgesamt 362M
drwxr-xr-x. 3 foreman-proxy root          4,0K 16. Feb 09:12 .
drwxr-xr-x. 8 root          root           177 19. Mär 2020  ..
-rw-r--r--. 1 foreman-proxy foreman-proxy  53M 26. Okt 2020  centos-mirror-qVbSBrznIWMc-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy 6,5M 19. Okt 2020  centos-mirror-qVbSBrznIWMc-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy  30M 15. Dez 16:55 debian-mirror-2SWNytzhr9B6-initrd.gz
-rw-r--r--. 1 foreman-proxy foreman-proxy    0 15. Feb 12:10 debian-mirror-2SWNytzhr9B6-linux
-rw-r--r--. 1 foreman-proxy foreman-proxy  30M 15. Dez 16:55 debian-mirror-DNCFDjRZGWUw-initrd.gz
-rw-r--r--. 1 foreman-proxy foreman-proxy    0 16. Feb 09:12 debian-mirror-DNCFDjRZGWUw-linux
drwxr-xr-x. 2 foreman-proxy root            72 21. Okt 2019  fdi-image
-rw-r--r--. 1 root          root          244M 19. Mär 2020  fdi-image-latest.tar

If I try to manually download the file, I am getting this:

wget http://ftp.debian.org/debian/dists/bullseye/main/installer-amd64/current/images/netboot/debian-installer/amd64/linux
--2023-02-16 10:47:06--  http://ftp.debian.org/debian/dists/bullseye/main/installer-amd64/current/images/netboot/debian-installer/amd64/linux
Auflösen des Hostnamen »ftp.debian.org (ftp.debian.org)«... 146.75.118.132, 2a04:4e42:8d::644
Verbindungsaufbau zu ftp.debian.org (ftp.debian.org)|146.75.118.132|:80... verbunden.
HTTP-Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 7008928 (6,7M)
In »»linux«« speichern.

 0% [                                                                                                                                                                   ] 0           --.-K/s   in 0s      

2023-02-16 10:47:06 (0,00 B/s) - Lesefehler bei Byte 0/7008928 (Die Verbindung wurde vom Kommunikationspartner zurückgesetzt). Erneuter Versuch.

--2023-02-16 10:47:07--  (Versuch: 2)  http://ftp.debian.org/debian/dists/bullseye/main/installer-amd64/current/images/netboot/debian-installer/amd64/linux
Verbindungsaufbau zu ftp.debian.org (ftp.debian.org)|146.75.118.132|:80... verbunden.
HTTP-Anforderung gesendet, warte auf Antwort... 503 Service Unavailable
2023-02-16 10:47:07 FEHLER 503: Service Unavailable.

Trying to manually download the file results in 503, as you can see there. I am wondering about that, as firewalling etc. should not result in 503? But it’s a great point to start debugging, so thank you for your input.

Dirk · February 16, 2023, 12:39pm

Firewalling should not be a problem. From my experience content scanning via proxy is the most typically thing causing problems here. I have seen here timeouts or status errors like this in many different forms. Not sure if exceptions are allowed by your company but I would recommend it from a technical perspective.

Anyway Good luck for debugging!