Problem: After upgrading katello 4.0 to 4.1.3 kickstart installations of CentOS8 is not working and fails with the same error message as the headline
Expected outcome: A working kickstart installation of CentOS8
Foreman and Proxy versions: 2.5.3
Foreman and Proxy plugin versions: 2.5.3
Distribution and version: CentOS7.9+ latest patches
Other relevant data: I think that the problem is because .treeinfo file from BaseOS point out the AppStream repo relative from BaseOS repo. e.g: “repository = AppStream”
This looks similar to my case but based on pulp2: Cannot kickstart CentOS 8
Coincidentally I had a chat on Matrix with @zhenech and he asked @lzap to chime in. This is the chatlog:
thulium-drake
o/ folks, anyone currecntly deploying foreman 2.5.3/katello 4.1.3 and doing it all with Rocky8? :-)
--
Zhenech
no? but if you have questions… ;)
--
thulium-drake
Well, I'm running into an interesting issue. I have freshly installed Foreman/Katello and synced up all the Rocky repos (I configure the whole thing with Ansible). But when kickstarting a host it tries to get it's appstream repo at
http://<foreman base url>/MyOrg/MyLCE/COV_Rocky8-Base/custom/Rocky8/Rocky8-BaseOS/Appstream
Instead of
http://<foreman base url>/MyOrg/MyLCE/COV_Rocky8-Base/custom/Rocky8/Rocky8-Appstream
I think I've seen this in the past too, but I don't know for sure
(COV = composite content view btw)
--
Zhenech
yeah, I think we have a patch or two to fix that
and it might not be detecting Rocky correctly
gimme a sec
--
thulium-drake
cool, thanks!
--
Zhenech
https://github.com/theforeman/foreman/blob/45c7dd75bf6d14bc341a5f774d13becf9d702954/app/views/unattended/provisioning_templates/provision/kickstart_default.erb#L87
I think this need to check for rocky/alma too
--
thulium-drake
Well, funny thing... ;-) I was looking there earlier and changed 'centos' to 'rocky'. This is a diff of the 2 kickstart files:
## < # renamed from "http://foreman.lbhr.htm.lan/pulp/content/HTM/Beheer/COV_Rocky8-Base/custom/Rocky8/Rocky8-AppStream/" for CentOS Anaconda to work
< repo --name AppStream --baseurl http://foreman.lbhr.htm.lan/pulp/content/HTM/Beheer/COV_Rocky8-Base/custom/Rocky8/Rocky8-AppStream/
> repo --name Rocky8-AppStream --baseurl http://foreman.lbhr.htm.lan/pulp/content/HTM/Beheer/COV_Rocky8-Base/custom/Rocky8/Rocky8-AppStream/
oh crap, formatting oO
And these URLs work in my environment, but anaconda doesn't agree
--
Zhenech
I don't think the repo *name* should change
--
thulium-drake
Agreed, when looking in /tmp/packaging.log on the kickstarted system (I'm typing this) I see the following:
Treeinfo points base repo at http://foreman.lbhr.htm.lan/pulp/content/HTM/Beheer/COV_Rocky8-Base/custom/Rocky8/Rocky8-BaseOS
Then it gathers some release info, and prints
Adding new treeinfo repository Appstream enabled: true
added repo: 'AppStream' http://<foreman base url>/MyOrg/MyLCE/COV_Rocky8-Base/custom/Rocky8/Rocky8-BaseOS/Appstream
is there a means of getting SSH access to a kickstarted box so I can retrieve the full logs? :)
--
Zhenech
no
--
thulium-drake
Well, I found a way :p
cat EOF > /etc/ssh/sshd_config
PermitRootLogin yes
PermitEmptyPasswords yes
EOF
/usr/sbin/sshd -D
seems to work just fine :)
08:52:08,332 INF packaging: Trying to download '.treeinfo'
08:52:08,378 DBG packaging: Retrieved '.treeinfo' from http://foreman.lbhr.htm.lan/pulp/content/HTM/Beheer/COV_Rocky8-Base/custom/Rocky8/Rocky8-BaseOS/
08:52:08,383 DBG packaging: getting release version from tree at http://foreman.lbhr.htm.lan/pulp/content/HTM/Beheer/COV_Rocky8-Base/custom/Rocky8/Rocky8-BaseOS/ (8)
08:52:08,383 DBG packaging: using treeinfo release version of 8
08:52:08,383 DBG packaging: Treeinfo points base repository to http://foreman.lbhr.htm.lan/pulp/content/HTM/Beheer/COV_Rocky8-Base/custom/Rocky8/Rocky8-BaseOS/.
08:52:08,384 DBG packaging: releasever from http://foreman.lbhr.htm.lan/pulp/content/HTM/Beheer/COV_Rocky8-Base/custom/Rocky8/Rocky8-BaseOS/ is 8
08:52:08,384 DBG packaging: Adding new treeinfo repository: AppStream enabled: True
08:52:08,385 INF packaging: added repo: 'AppStream' - http://foreman.lbhr.htm.lan/pulp/content/HTM/Beheer/COV_Rocky8-Base/custom/Rocky8/Rocky8-BaseOS/AppStream
08:52:08,396 DBG dnf: repo: downloading from remote: AppStream
08:52:08,427 DBG dnf: error: Status code: 404 for http://foreman.lbhr.htm.lan/pulp/content/HTM/Beheer/COV_Rocky8-Base/custom/Rocky8/Rocky8-BaseOS/AppStream/repodata/repomd.xml (IP: 192.168.255.15) (http://foreman.lbhr.htm.lan/pulp/content
/HTM/Beheer/COV_Rocky8-Base/custom/Rocky8/Rocky8-BaseOS/AppStream/repodata/repomd.xml).
that's the actual log snippet
--
Zhenech
can you paste the actual generated kickstart please
--
thulium-drake
ks (6.29 KB)
I only removed the password and the SSH keys
The rest is just lab stuff, so not too important :-)
--
Zhenech
why aren't there any comments, stupid anaconda
but yeah, I am kinda sure the repo *needs* to be called AppStream
or anaconda will try to *guess* the url and fail
--
thulium-drake
right, I'll put that change back in and upload a new file and log
ks (6.43 KB)
New KS file with the fix in place
packaging.log (12.37 KB)
DNF/Packaging log
Same issue AFAICS
--
Zhenech
interesting
lzap, I remember you played with that (on CentOS)
The files mentioned are:
kickstart as it was: ks-nopatch.log (6.3 KB)
kickstart after patching the template: ks-patch.log (6.4 KB)
subsequent packaging log from installer live image: packaging-patch.log (12.4 KB)
Thank you for this @Thulium-Drake. So if I understand the solution correctly it is to name the appstream repo --name AppStream. In Katello 4.0 it worked for me with --name centos-8-appstream-x86_64. Since I already rolled back to 4.0 I don’t know when I have the opportunity to test this.
Well not exactly, the fix I tried actually generates a repo for anaconda to use that is actually named AppStream. But the issue is that it tries to retrieve AppStream as a subdirectory of my BaseOS repo, where AppStream does not exist.
I think the actual fix is something along the lines where Pulp sees the request coming in and redirects it to the proper AppStream repository.
And it did work for CentOS8 earlier (Katello 4.0 IIRC), but I’m running on Rocky8 as our client decided to no longer use CentOS8. So we’re exploring issues there, and this is the only one I found so far
Well, it seems you are right, I just updated my repo name from Rocky8-AppStream to AppStream. Afterwards I published a new version of the affected CVs, rebooted a client to install and presto!
From what I was able to figure out, the bug was already fixed for CentOS by adding this block:
<% if rhel_compatible && @host.operatingsystem.name.downcase.include?("centos") && os_major >= 8 && medium[:url] && medium[:url].include?("AppStream") -%>
<% appstream_present = true -%>
# renamed from "<%= medium[:url] %>" for CentOS Anaconda to work
repo --name AppStream --baseurl <%= medium[:url] %>
<% else -%>
repo --name <%= medium[:name] %> --baseurl <%= medium[:url] %> <%= medium[:install] ? ' --install' : '' %><%= proxy_string %>
<% end -%>
<% end -%>
<% if rhel_compatible && @host.operatingsystem.name.downcase.include?("centos") && os_major >= 8 && medium_uri && medium_uri.to_s.include?("BaseOS") && !appstream_present -%>
# repository added using BaseOS to AppStream string replacement
repo --name AppStream --baseurl <%= medium_uri.to_s.gsub("BaseOS", "AppStream") %>
<% end -%>
But it only matches CentOS and not the derivatives that use the same repo structure. But as how the rhel_compatible variable already sorts out which derivatives are using the same repo structure, I removed matching CentOS specifically here:
<% if rhel_compatible && os_major >= 8 && medium[:url] && medium[:url].include?("AppStream") -%>
<% appstream_present = true -%>
# renamed from "<%= medium[:url] %>" for CentOS Anaconda to work
repo --name AppStream --baseurl <%= medium[:url] %>
<% else -%>
repo --name <%= medium[:name] %> --baseurl <%= medium[:url] %> <%= medium[:install] ? ' --install' : '' %><%= proxy_string %>
<% end -%>
<% end -%>
<% if rhel_compatible && os_major >= 8 && medium_uri && medium_uri.to_s.include?("BaseOS") && !appstream_present -%>
# repository added using BaseOS to AppStream string replacement
repo --name AppStream --baseurl <%= medium_uri.to_s.gsub("BaseOS", "AppStream") %>
<% end -%>
And that did fix the kickstarting issues in my environment
Great please file a PR many thanks! Yeah this is a difference between Anaconda in RHEL and in CentOS and clones. Let’s talk in the PR how to properly solve this because this would probably break RHEL installations.
Hi I tried again with the upgrade and with the same result. It seems my issue is related to how pulp generates the .treeinfo file for BaseOS. It points to AppStream as relative to BaseOS.
[variant-AppStream]
id = AppStream
name = AppStream
packages = AppStream/Packages
repository = AppStream
type = variant
uid = AppStream
It should instead point to: ../AppStream since the repository is located there and not under BaseOS repo.
Can anyone help me with a workaround or fix. For example is it possible to edit the .treeinfo file and put the correct info into the file?
AH, this is what I am seeing as well. I just noticed that pulp3 is generating its OWN incorrect .treeinfo different from the one in the actual upstream repo servers .treeinfo file.
Which has:
[variant-AppStream]
id = AppStream
name = AppStream
type = variant
uid = AppStream
packages = ../../../AppStream/x86_64/os/Packages
repository = ../../../AppStream/x86_64/os/
I managed to work around this issue with the following. Change ‘CentOS8/BaseOS’ to whatever product/reponame you are using.
sed -i "s#repository = AppStream#repository = ../AppStream#" $(sudo -u postgres /bin/psql pulpcore -t -c "select file from core_artifact where pulp_id=(select artifact_id from core_contentartifact where pulp_id = (select content_artifact_id from core_publishedartifact where publication_id=(select publication_id from core_distribution where base_path like '%CentOS8/BaseOS') and relative_path like '.treeinfo'))" 2>/dev/null | xargs | sed -e "s#^ *#/var/lib/pulp/media/#")
It would be really nice to get a proper fix for this though. Anyone knows who to poke?
I wrote the fix in my previous post. Updating the database will probably break your installation in the long run. If you update the provisioning template as mentioned, it will fix Kickstart for now
Hey everyone, please do not try any hacks or database changes. This seems to be working fine in the newer releases of pulp_rpm (such as 3.14.z), so the proper fix would be either upgrading or backporting the patch to the version used by this older version of Katello.
I am in conversation with Pulp engineers and management about this. It looks like only Pulp 3 was fixed, I am trying to get to an agreement to fix Pulp 2 using some workaround if possible.
