Thanks that could be it! FYI here it is:
This is a workaround around the fact that CentOS and RHEL Anaconda are different - one of the two performs some path replacement/change, so we manually add AppStream repo in the kickstart if we find it is a RHEL-compatible system but named centos.
Already doing that in my template(and modified for Rocky). That does not fix it.
I think this issue is because pulp is putting an AppStream entry in the BaseOS repo .treeinfo it gives out with a totally bogus URL path ./AppStream that does not exist and the kickstart is trying to read it and failing.
Really the AppStream entry should totally be removed. That entry works fine on the offical repo mirror sites because you can easily get from the BaseOS repo directory to AppStream repo via a …/…/AppStream. Outside of the purpose of the official Centos and Rocky mirror sites, and their known directory layout, this serves no purpose.
With katello/pulp the repos have a totally different URL path names, and there is zero practical way to get to the AppStream repo FROM the BaseOS repo URL.
Given that we ADD THE AppStream repo to clients ourselves with template as shown above, I think pulp should NOT INCLUDE anything about AppStream in the .treeinfo it gives out for the BaseOS repo channel.
I bet the same thing is happening for Centos 8/Centos8 Streams too.
Note, RHEL does NOT INCLUDE an AppStream entry in the BaseOS .treeinfo repo, it uses a separate AppStream repo entry in the kickstart file to file it, like we have in the template.
I think the maintainers of Centos/Rocky put that in the BaseOS .treeinfo for a convenience to their users based on the way their official repo mirrors are layed out, and I believe we do not need it, and it should be filtered out by pulp.
FYI I have filed this a year ago, it is still open: 0016470: CentOS 8 BaseOS repository references AppStream repo in a way which is incompatible with Foreman. - CentOS Bug Tracker
So here is a proposal:
@Marek_Hulan what you say? I would file a BZ for Pulp we need to solve this for once.
So I have researched this down to the patch that was merged into Pulp 3.x a year back and I resurrected that github discussion because from all our (multiple) users are telling us (this is not the only thread we have here on this topic), it looks like it does not work correctly in Pulp.
So action items are:
Here it is: https://github.com/pulp/pulp_rpm/pull/1755
For the record, I have just reproduced this on Satellite 6.9 which has Pulp 3.10. That version is supposed to be fixed. The published .treeinfo does contain incorrect AppStream entry.
[checksums]
images/boot.iso = sha256:53a62a72881b931bdad6b13bcece7c3a2d4ca9c4a2f1e1a8029d081dd25ea61f
images/efiboot.img = sha256:6d40c9c2e8a686254c5b3f0f769969386d6daa7737f2f4e428a00c09993b549e
images/install.img = sha256:88e49ee73883171c85414543f073d83bf5e520e78498c48084d727589ace49ad
images/pxeboot/initrd.img = sha256:f9eceb3503134874100a2e076cc90f4d997cb2a32126bdd570d42d59683f74be
images/pxeboot/vmlinuz = sha256:f3f9f33573fa6beba7ab5ba4b5d771729b8fe669352001bc046006da01c1296d
[general]
; WARNING.0 = This section provides compatibility with pre-productmd treeinfos.
; WARNING.1 = Read productmd documentation for details about new format.
arch = x86_64
family = Rocky Linux
name = Rocky Linux 8
packagedir = Packages
platforms = x86_64,xen
repository = .
timestamp = 1624205579
variant = BaseOS
variants = BaseOS
version = 8
[header]
type = productmd.treeinfo
version = 1.2
[images-x86_64]
boot.iso = images/boot.iso
efiboot.img = images/efiboot.img
initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz
[images-xen]
initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz
[release]
name = Rocky Linux
short = Rocky
version = 8
[stage2]
mainimage = images/install.img
[tree]
arch = x86_64
build_timestamp = 1624205579
platforms = x86_64,xen
variants = BaseOS,AppStream
[variant-BaseOS]
id = BaseOS
name = BaseOS
packages = Packages
repository = .
type = variant
uid = BaseOS
[variant-AppStream]
id = AppStream
name = AppStream
type = variant
uid = AppStream
packages = ../../../AppStream/x86_64/os/Packages
repository = ../../../AppStream/x86_64/os/
I personally think ANY AppStream entry in the BaseOS .treeinfo is wrong. It should be removed.
With Katello the AppStream repo will be no where under or available via any sort of easily assumed path from the URL to the BaseOS repo.
I just edit .treeinfo manually in the pulp directory for every BaseOS repos and update the AppStream section like shown below. The change seems to persist in subsequent sync and in content views.
[variant-AppStream]
id = AppStream
name = AppStream
packages = Packages
repository = .
type = variant
uid = AppStream
Then in the provisioning template I have repo --name AppStream --url ..... I’ve made sure to name BaseOS and AppStream repos in the same way, so that the URL is the same. Then I only need to change BaseOS to AppStream, which I do in the provisioning template like so:
<% if rhel_compatible && @host.operatingsystem.name.downcase.include?("centos") && os_major >= 8 && medium_uri.to_s.include?("BaseOS") %>
repo --name AppStream --baseurl <%= medium_uri.to_s.gsub("BaseOS", "AppStream") %><%= proxy_string %>
<% end %>
This seems very simple and clean compared some PR and any other path hacks.
I just tested this on Satellite 6.10 (Katello 4.1.1) and I am not seeing what you have. In my case, BaseOS repository sync the BaseOS content as well as AppStream which is then added as a “subrepository”:
[variant-BaseOS]
id = BaseOS
name = BaseOS
packages = Packages
repository = .
type = variant
uid = BaseOS
The relative path is available and works fine:
# curl -k https://zzzap.xxx.redhat.com/pulp/content/Default_Organization/Library/custom/CentOS8/CentOS8StreamBaseOS/AppStream/
<!DOCTYPE html>
<html>
<body>
<ul>
<li><a href="EULA">EULA</a></li>
<li><a href="GPL">GPL</a></li>
<li><a href="Packages/">Packages/</a></li>
<li><a href="extra_files.json">extra_files.json</a></li>
<li><a href="repodata/">repodata/</a></li>
</ul>
</body>
</html>
This also means we no longer need the kickstart workaround anymore, actually it can break things now. We need to remove it.
I will test end to end provisioning on Monday as well as Rocky Linux but I expect it to be the same.
Note that only Pulp 3 will work correctly, I do believe that Pulp 2 hasn`t been fixed.
Hi @lzap,
Any luck? I’m back from my holiday 
I went ahead and reinstalled my lab VM with Foreman 2.5.3 and Katello 4.1.4 (again). I first tried applying the first workaround again (as I mentioned earlier, replacing ‘centos’ with ‘rocky’) and it didn’t work.
Next I removed all of the special logic, so the template went from this:
<% if rhel_compatible && os_major >= 8 && medium[:url] && medium[:url].include?("AppStream") -%>
<% appstream_present = true -%>
# renamed from "<%= medium[:url] %>" for CentOS Anaconda to work
repo --name AppStream --baseurl <%= medium[:url] %>
<% else -%>
repo --name <%= medium[:name] %> --baseurl <%= medium[:url] %> <%= medium[:install] ? ' --install' : '' %><%= proxy_string %>
<% end -%>
<% end -%>
<% if rhel_compatible && os_major >= 8 && medium_uri && medium_uri.to_s.include?("BaseOS") && !appstream_present -%>
# repository added using BaseOS to AppStream string replacement
repo --name AppStream --baseurl <%= medium_uri.to_s.gsub("BaseOS", "AppStream") %>
<% end -%>
<%= snippet_if_exists(template_name + " custom repositories") %>
<% end %>
to this
<% @additional_media.each do |medium| -%>
repo --name <%= medium[:name] %> --baseurl <%= medium[:url] %> <%= medium[:install] ? ' --install' : '' %><%= proxy_string %>
<% end -%>
<%= snippet_if_exists(template_name + " custom repositories") %>
<% end %>
If that is what you meant by removing the AppStream workaround? It didn’t work though 
Or do I need a newer version of pulp-rpm before this is actually working? In any case, I have 3.14.3 installed on my system.
I created a PR for this as I hit it too (non-katello foreman)
Last night I deployed a fresh Rocky 8.4 with Foreman 3 and Katello 4.2, synced the Rocky 8 repo’s and pxe booted a fresh vm. Installation went fine.
I have BaseOS + Kickstart, AppStream + Kickstart, Extra’s en PowerTools synced.
Are you installing from Katello repository, or via Installation Media directly?
I am testing it this week.
Katello. I added a product, added the repos I listed and created a content view/published/promoted/etc and created an activation key with subscriptions.
A new host pxe boots, and installs from Katello repo’s. It just works for me 
.
Yeah sorry I was asking Drake
I’m also installing from ‘Synced content’
So my preliminary observation is that the “workaround” actually breaks CentOS 8 Stream installation. When I comment out all the “repo” statements, it works fine with Pulp 3.
This is because Anaconda will automatically detect the subrepo (aka variant) via treeinfo. There is no need to add additional repositories at all.
I am going to create a new thread for this. Let’s figure this out.
Ok so it looks like the workaround does not work well with Katello. I need to test Foreman core only provisioning too, we might need something like “if katello is installed then” or something like that.
Share your findings here please:
I know this is very old but we have just ran into this issue. I am trying to locate the .treeinfo file for Rocky 8 but all I see when I do a locate is the old treefiles from pulp 2.
Do you have or remember the path where the file is for pulp 3 and Rocky 8?
No, haven’t tried pulp3 yet