@lzap We were having the same issues mentioned here, from a fresh centos 7 install:
- Linux foreman-devel 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
- foreman-installer-2.1.4-1.el7.noarch
- katello-3.16.1-1.el7.noarch
(We tried with the 2.2rc1 and Katello 3.16 and the issue still happened, we could not try latest 2.2 or 2.3 and 3.17 as per this bug: Bug #31217: installation failed with "Evaluation Error: Error while evaluating a Resource Statement, Class[Dhcp]: has no parameter named 'conf_dir_mode' - Installer - Foreman)
As mentioned by @Bugenhagen this only happens with CentOS 7, not 8.
While looking at the logs from both Foreman/Katello instance we noticed that even if the first package requested got a 206, the following ones got a 200, does not make much sense (at least to me). The following is part of the httpd logs from a clean boot of a client:
192.168.200.26 - - [13/Nov/2020:13:34:30 +0100] "GET /unattended/provision?token=4bcd8534-cccd-4387-9f8e-ea1868e34330 HTTP/1.1" 200 4735 "-" "curl/7.29.0"
192.168.200.26 - - [13/Nov/2020:13:34:32 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base//.treeinfo HTTP/1.1" 200 875 "-" "curl/7.29.0"
192.168.200.26 - - [13/Nov/2020:13:34:32 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base//LiveOS/squashfs.img HTTP/1.1" 200 521617408 "-" "curl/7.29.0"
192.168.200.26 - - [13/Nov/2020:13:34:33 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base//images/updates.img HTTP/1.1" 404 14 "-" "curl/7.29.0"
192.168.200.26 - - [13/Nov/2020:13:34:33 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base//images/product.img HTTP/1.1" 404 14 "-" "curl/7.29.0"
192.168.200.26 - - [13/Nov/2020:13:34:56 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base//.treeinfo HTTP/1.1" 200 875 "-" "urlgrabber/3.10"
192.168.200.26 - - [13/Nov/2020:13:34:56 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/repodata/repomd.xml HTTP/1.1" 200 3875 "-" "CentOS (anaconda)/7 yum/3.4.3"
192.168.200.26 - - [13/Nov/2020:13:34:57 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base//.treeinfo HTTP/1.1" 200 875 "-" "urlgrabber/3.10"
192.168.200.26 - - [13/Nov/2020:13:34:57 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/repodata/7ea6872641840c61c4f249ab656d8e4d647d223797329d3dffe947e4db777464-primary.xml.gz HTTP/1.1" 200 4272773 "-" "CentOS (anaconda)/7 yum/3.4.3"
192.168.200.26 - - [13/Nov/2020:13:34:57 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/repodata/9e77e5b0b24bbffe125b1a7bbd1fe2a14f3ddaff9ac6c2cf939a228b61bac1ff-comps.xml HTTP/1.1" 200 763349 "-" "CentOS (anaconda)/7 yum/3.4.3"
192.168.200.26 - - [13/Nov/2020:13:34:59 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base//repodata/repomd.xml HTTP/1.1" 200 3875 "-" "urlgrabber/3.10"
192.168.200.26 - - [13/Nov/2020:13:35:19 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/f/fipscheck-lib-1.4.1-6.el7.x86_64.rpm HTTP/1.1" 206 4832 "-" "urlgrabber/3.10 yum/3.4.3"
192.168.200.26 - - [13/Nov/2020:13:35:19 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/s/sg3_utils-1.37-19.el7.x86_64.rpm HTTP/1.1" 206 42956 "-" "urlgrabber/3.10 yum/3.4.3"
192.168.200.26 - - [13/Nov/2020:13:35:19 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/p/plymouth-core-libs-0.8.9-0.34.20140113.el7.centos.x86_64.rpm HTTP/1.1" 200 52847 "-" "urlgrabber/3.10 yum/3.4.3"
192.168.200.26 - - [13/Nov/2020:13:35:20 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/p/plymouth-core-libs-0.8.9-0.34.20140113.el7.centos.x86_64.rpm HTTP/1.1" 200 51399 "-" "urlgrabber/3.10 yum/3.4.3"
192.168.200.26 - - [13/Nov/2020:13:35:21 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/p/plymouth-core-libs-0.8.9-0.34.20140113.el7.centos.x86_64.rpm HTTP/1.1" 200 71671 "-" "urlgrabber/3.10 yum/3.4.3"
192.168.200.26 - - [13/Nov/2020:13:35:23 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/p/plymouth-core-libs-0.8.9-0.34.20140113.el7.centos.x86_64.rpm HTTP/1.1" 200 62983 "-" "urlgrabber/3.10 yum/3.4.3"
192.168.200.26 - - [13/Nov/2020:13:35:27 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/p/plymouth-core-libs-0.8.9-0.34.20140113.el7.centos.x86_64.rpm HTTP/1.1" 200 55743 "-" "urlgrabber/3.10 yum/3.4.3"
192.168.200.26 - - [13/Nov/2020:13:35:35 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/p/plymouth-core-libs-0.8.9-0.34.20140113.el7.centos.x86_64.rpm HTTP/1.1" 200 60087 "-" "urlgrabber/3.10 yum/3.4.3"
192.168.200.26 - - [13/Nov/2020:13:35:51 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/p/plymouth-core-libs-0.8.9-0.34.20140113.el7.centos.x86_64.rpm HTTP/1.1" 200 46103 "-" "urlgrabber/3.10 yum/3.4.3"
192.168.200.26 - - [13/Nov/2020:13:36:23 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/p/plymouth-core-libs-0.8.9-0.34.20140113.el7.centos.x86_64.rpm HTTP/1.1" 200 51399 "-" "urlgrabber/3.10 yum/3.4.3"
192.168.200.26 - - [13/Nov/2020:13:37:27 +0100] "GET /pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/p/plymouth-core-libs-0.8.9-0.34.20140113.el7.centos.x86_64.rpm HTTP/1.1" 200 58639 "-" "urlgrabber/3.10 yum/3.4.3"
Here fipscheck-lib and sg3_utils gets the 206 but the other packages seem reachable. Still I try to curl any package from the anaconda tmux session, all can be downloaded.
I have checked the initrd.img and vmlinuz images synced in our repo with the ones in the original repo and the checksums match.
If we replace the url of the repo in the template to the one upstream, the installation works
Provisioning themplate with synced repos (this does not work)
install
url --url http://foreman-devel.scicore.unibas.ch/pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/
provisioning template with external repos (this works):
install
url --url http://linuxsoft.cern.ch/centos/7.9.2009/os/x86_64/
Then during a last test I noticed that the package mentioned as not found was not sg3 anymore but another, so during an install I did a curl against the url that gave the 200 message (plymouth-core-libs) and funny enough, on the next try of anaconda, another pakage showed up. So I tried doing a full curl of each rpm in the local repo and after that, the installation works with the local repos.
# the following was a quick solution to just "touch" all rpms
for i in {a..z}; do for j in `curl http://foreman-devel.scicore.unibas.ch/pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/$i/| grep href |awk '{print $2}'|cut -f2 -d">"|cut -f1 -d"<"`; do curl http://foreman-devel.scicore.unibas.ch/pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/$i/$j > /dev/null; done;done
for i in `curl http://foreman-devel.scicore.unibas.ch/pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/3/| grep href |awk '{print $2}'|cut -f2 -d">"|cut -f1 -d"<"`; do curl http://foreman-devel.scicore.unibas.ch/pulp/repos/sciCORE/Library/custom/CentOS_7/centos7_latest_Base/Packages/3/$i > /dev/null; done
If needed I can provide the ansible code and foreman install options used to deploy our instance if it can be useful to help troubleshoot this issue.
Let me know if I can provide more information.
Cheers