Error during upgrade : Failed to open TCP connection to downloads.theforeman.org:80 (getaddrinfo: Name or service not known)

Problem:
I need another pair of eyes to find my issue in my system/foreman. We use the xMas break to update all your system at the moment. When upgrading TheForeman to the next and finally to the latest version we run into an issue when upgrading from 3.2 to 3.3. we followed the normal procedure until we need to run ‘foreman-installer’. The error looks like this:

[root@my-srv-foreman boot]# foreman-installer [--noop --verbose]
2022-12-25 16:52:10 [WARN  ] [boot] Unsetting environment variable 'http_proxy' for the duration of the install.
2022-12-25 16:52:10 [WARN  ] [boot] Unsetting environment variable 'https_proxy' for the duration of the install.
2022-12-25 16:52:10 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-12-25 16:52:13 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-12-25 16:52:13 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2022-12-25 16:52:18 [NOTICE] [configure] Starting system configuration.
2022-12-25 16:52:25 [NOTICE] [configure] 250 configuration steps out of 1280 steps complete.
2022-12-25 16:52:26 [NOTICE] [configure] 500 configuration steps out of 1284 steps complete.
2022-12-25 16:52:26 [NOTICE] [configure] 750 configuration steps out of 1289 steps complete.
2022-12-25 16:52:42 [NOTICE] [configure] 1000 configuration steps out of 1289 steps complete.
2022-12-25 16:52:43 [NOTICE] [configure] 1250 configuration steps out of 1526 steps complete.
2022-12-25 16:52:43 [NOTICE] [configure] 1500 configuration steps out of 1660 steps complete.
2022-12-25 16:52:43 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Plugin::Discovery/Foreman_proxy::Remote_file[/var/lib/tftpboot/boot/fdi-image-latest.tar]/File[/var/lib/tftpboot/boot/fdi-image-latest.tar]: Could not evaluate: Could not retrieve file metadata for http://downloads.theforeman.org/discovery/releases/latest/fdi-image-latest.tar: Request to http://downloads.theforeman.org/discovery/releases/latest/fdi-image-latest.tar failed after 0.01 seconds: Failed to open TCP connection to downloads.theforeman.org:80 (getaddrinfo: Name or service not known)
2022-12-25 16:52:46 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/foreman.log
[root@my-srv-foreman boot]#

Expected outcome:
upgrade to next version

Foreman and Proxy versions:

Foreman 			3.3.1
foreman_discovery	21.0.4/1.0.5
foreman_puppet		4.0.1

Foreman and Proxy plugin versions:

Distribution and version:

[root@my-srv-foreman ~]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
[root@my-srv-foreman ~]# uname -a
Linux my-srv-foreman.company.corp 3.10.0-1160.81.1.el7.x86_64 #1 SMP Fri Dec 16 17:29:43 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@my-srv-foreman ~]#

Other relevant data:
Here a listing of some command i used to check some stuff:

[root@my-srv-foreman boot]# ll /var/lib/tftpboot/boot/
total 484268
-rw-r--r-- 1 foreman-proxy foreman-proxy  55129656 Oct 26  2020 centos-j8MlhVMg77xN-initrd.img
-rw-r--r-- 1 foreman-proxy foreman-proxy   6769256 Oct 19  2020 centos-j8MlhVMg77xN-vmlinuz
-rw-r--r-- 1 foreman-proxy foreman-proxy  55073584 Aug 24  2020 centos-JJJroIaC9QbC-initrd.img
-rw-r--r-- 1 foreman-proxy foreman-proxy   6734016 Aug 24  2020 centos-JJJroIaC9QbC-vmlinuz
drwxr-xr-x 2 foreman-proxy foreman-proxy        72 Jan 12  2022 fdi-image
-rw-r--r-- 1 foreman-proxy foreman-proxy 282060800 Jul 15  2021 fdi-image-latest.tar
-rw-r--r-- 1 foreman-proxy foreman-proxy       245 Sep 14 10:31 rocky-linux-k3ZzFkvDjDtV-initrd.img
-rw-r--r-- 1 foreman-proxy foreman-proxy       242 Sep 14 10:31 rocky-linux-k3ZzFkvDjDtV-vmlinuz
-rw-r--r-- 1 foreman-proxy foreman-proxy  79894932 Nov 14  2021 rocky-linux-ZWtriJwGrOZZ-initrd.img
-rw-r--r-- 1 foreman-proxy foreman-proxy  10203440 Nov 14  2021 rocky-linux-ZWtriJwGrOZZ-vmlinuz
[root@my-srv-foreman boot]# cd
[root@my-srv-foreman ~]# wget http://downloads.theforeman.org/discovery/releases/latest/fdi-image-latest.tar
--2022-12-25 17:00:05--  http://downloads.theforeman.org/discovery/releases/latest/fdi-image-latest.tar
Resolving mycloud-revproxy.company.corp (mycloud-revproxy.company.corp)... aa.bbb.219.201
Connecting to mycloud-revproxy.company.corp (mycloud-revproxy.company.corp)|aa.bbb.219.201|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 533800960 (509M) [application/x-tar]
Saving to: ‘fdi-image-latest.tar’

100%[=======================================================================================================================================================>] 533,800,960 13.4KB/s   in 24s

2022-12-25 17:00:34 (21.2 MB/s) - ‘fdi-image-latest.tar’ saved [533800960/533800960]

[root@my-srv-foreman ~]#

So download in general works… web-proxy is added to foreman and to the system-global file (/etc/environment)

I was going through a LOT of troubleshooting and service pages, but could get a grip on this.

Not sure what else I need to check or change. Firewall if OFF and SELinux is disabled for now.

Any idea? #please

A lot of people with more knowledge about how to solve this properly are likely on christmas break, so I’ll try and at least provide a workaround.
From what I see, the underlying problem is that the installer unsets all _proxy environment variables, which is why the download is failing despite you set the proxy in the environment files. I am not sure why this is done and I have not found any options to disable this behaviour with a parameter or supply a proxy to be used in another way, but you can try the following:
Manually download the tarball from downloads.theforeman.org (like you already did). Then you can try and run foreman-installer --foreman-proxy-plugin-discovery-source-url file://<downloaded tarball>. This should skip the installer trying to download the tarball from upstream and make it work with the manually downloaded file. If that does not work, you can try extracting the downloaded tarball to /var/lib/tftpboot/boot and then run foreman-installer --foreman-proxy-plugin-discovery-install-images --false. This parameter should skip the image-handling part completely (which only does the download and tar xf anyways), but still configures everything else for discovery.

Hope this helps

Also, as a side note: I assume you are aware of this (your wording hints at this), but to make sure you are aware, Foreman 3.3 is the last version to support being run on EL7 distributions. You should use your next downtime/maintenance window to try and upgrade to an EL8 distribution. The docs on how to upgrade a foreman server in-place using leapp can be found here, but I would also recommend checking docs from your OS. I personally used the RHEL guide when I recently tested in-place upgrade from RHEL7 to RHEL8, and using both docs it went pretty smooth once I figured how to deal with all the “oddities” my organization introduced to our general RHEL images. Still, I strongly suggest taking a full backup or VM snapshot before attempting this.

w/o the “–” for false did the trick. MANY thanks!!! You are my hero.
foreman-installer --foreman-proxy-plugin-discovery-install-images false

goooood point. Yes, I will take care of this as an intermediated step! THANKS again!
almost forgo about it. :-/

Right, that – was a hasty mistake while copy-pasting, not intended, sorry :wink:
Glad it helped :slight_smile: