Rocky8 AppStream and .treeinfo issue...again

Problem:
After working for about a year, provisioning (only for Rocky8) is broken because .treeinfo is saying AppStream is below BaseOS. Now, I ran into this back on Pulp2, and the easy fix was to modify .treeinfo and give it a correct path to find AppStream.
However, I have no idea how to do that with Pulp3. There is no .treeinfo file. There’s apparently no “usable” filesystem to find any file.

I have modified the kickstart default with suggestions from Foreman failed to download metadata for repo 'AppStream': cannot download repomd.xml - #6 by Thulium-Drake and several other forum articles.

I even created a local filesystem repo for Rocky8-BaseOS just so I could get a .treeinfo to modify, then use that repo as my upstream for the repository in Foreman. That actually worked, but it didn’t get any package groupinfo, so while it completed a build, it was missing everything in @Core. (If it’s not one thing, it’s another)

I have deleted the Rocky8-AppStream repo and rebuilt it as simply AppStream…no help. Still wants to find it under http://…/Rocky8/Rocky8-BaseOS/AppStream.

This is what the .treeinfo looks like (I can apparently only get it using wget from the baseurl):

[variant-AppStream]
id = AppStream
name = AppStream
packages = AppStream/Packages
repository = AppStream
type = variant
uid = AppStream

Currently, the relevant section in kickstart looks like this:

<% @additional_media.each do |medium| -%>
 <% if rhel_compatible && @host.operatingsystem.name.downcase.include?("rocky") && os_major >= 8 && medium[:url] && medium[:url].include?("AppStream") -%>
  <% appstream_present = true -%>
  repo --name AppStream --baseurl <%= medium[:url] %>
 <% else -%>
  repo --name <%= medium[:name] %> --baseurl <%= medium[:url] %> <%= medium[:install] ? ' --install' : '' %>
 <% end -%>
<% end -%>
<% end %>

repo --name AppStream --baseurl http://foreman.server/pulp/content/org/Library/custom/Rocky_8/AppStream/ --install

I’m befuddled as I built a host a couple weeks ago, and when I tried a few days ago, I ran into all this. And this foreman server rarely gets touched. Usually just once a month for patching, and that happened before the successful build.

So, my main question is - how can we modify .treeinfo if we can’t even get to it?

(as an aside, I hate Pulp3. Once we upgraded to it, I had 3-4 months of constant issues related to it before finally getting stable back around October last year. Test environment was fine…only the two production environments had issues.)

Foreman and Katello versions:
Foreman 3.13
Katello 4.3.1

Distribution and version:
CentOS 7

Well, I am working again, with a workaround.
Since we can’t seem to modify the .treeinfo in the pulp3 environment, I decided to make it work using the local repo.

Created a local repo just for Rocky8-BaseOS and downloaded the .treeinfo from the mirror. Modified it to look like

[variant-AppStream]
id = AppStream
name = AppStream
type = variant
uid = AppStream
packages = ../AppStream/x86_64/os/Packages
repository = ../AppStream/x86_64/os/

Pointed the Rocky8 repo in Katello to point to the local repo as its upstream url. Re-published the content-view, and now we can find the installation source during provisioning, and get a completed build.

However, I’m not totally convinced modifying .treeinfo did anything, because if I curl .treeinfo from my content server:

curl http://<my $baseurl>/.treeinfo

It doesn’t look like my modified .treeinfo. But if I don’t have that file there, no source installation is found, and the logs show it is still looking for https://…/BaseOS/AppStream/

As for the missing groups thing…my mistake. I left off “-g comps.xml”

/usr/bin/createrepo /var/www/html/pub/bcd-repos/rocky8-baseos -g Rocky8-BaseOS/comps.xml

I would still love to know how others are getting around this, or if there is a way to find the .treeinfo “artifact” and edit it. Seems to me that even if I found it, modifying it would change the checksum and then pulp3 would barf.

You should not be modifying publication artifacts (even if you do find some way to do so), this is diametrically opposed to the design of Pulp. Even during Pulp 2 times the ability to do this was not so much a feature as an accident of the chosen implementation. What you should be doing is to open a pulp_rpm issue so the devs can look into your issue and fix it.

The other thing you can try doing right now is switching the “Mirroring Policy” on your Katello repository. “Content Only” and “Additive” will use Pulp generated metadata in the published repository, while “Complete Mirroring” will mirror the upstream repo metadata. Switching between these two modes may or may not make a difference with your issue.

1 Like

Yeah, I never actually believed that editing an artifact was the right way to go. But in previous articles this issue was usually fixed by modifying the .treeinfo, and I was just wondering how that is done using pulp3. I was hoping there was some supported way that I haven’t found yet.

With regards to the mirroring policy, I’ve tried pretty much every iteration of the repo configuration - Immediate, On Demand, Content Only, Complete Mirroring, as well as other configurable values…

And I have two production Foreman/Katello servers in two regions, configured [nearly] exactly the same (plus the Lab server) except for regional/Locale differences, and this just started last week “out of the blue”. Frustrating.

It could be that something has changed about the upstream Rocky8 AppStream repo.
We quite frequently see Rocky repos containing nonsense were they should have an RPM package file, which then breaks syncs because of wrong checksum…

Still worth opening a pulp_rpm issue though. Maybe they can find a permanent solution to avoid this kind of issue from ever arising…

provisioning (only for Rocky8) is broken because .treeinfo is saying AppStream is below BaseOS.

What is failing? Which tool is claiming that having “Appstream is below BaseOS” is a problem?

This is during the initial build, when anaconda is setting things up and finding the installation source to determine what gets installed.
It will fail to find the source ("!" instead of “x”) and if you switch to another VT and check the logs, it tells you what it is trying to use, which is the wrong path for AppStream.

So when you sync a Pulp repo with .treeinfo, and the .treeinfo points backwards outside of the repo, Pulp rewrites the .treeinfo while also making that external repo available at the new location that the .treeinfo is pointing to.

The reason for that is that you can publish the same repo in multiple locations, or change where it is being published, or publish new repos, and so allowing links to point backwards outside of the repo opens all sorts of doors for things to go wrong. Keeping everything self-contained makes it more foolproof and doesn’t require the user to know about how .treeinfo works when setting up their directory structure.

You should be able to see this if you visit the repo URLs. So I’d like to redirect the investigation towards whether something is going wrong with that process.

@iballou I don’t suppose you’ve seen / are aware of any other reports of provisioning issues? And @mshade you say this only happens with Rocky Linux, are you also provisioning other distros the same way and everything works fine?

I also sync RHEL7 and CentOS7.
The .treeinfo issue, as far as I know, started with EL8 when AppStream was introduced, and we did see it with CentOS8 before we dropped that. There are tons of hits in google attesting to that.
At the time, setting a “repo --name AppStream --baseurl http://blah blah” in the kickstart, and point to the AppStream url seemed to be an easy workaround.
Then there were others who had the issue, and modifying the .treeinfo file help get past that. But that was back in the pre-pulp3 days.
We added Rocky8 into our environment around the time we upgraded Foreman/Katello and had to go to pulp3. And for a year, we’ve been running fine with simply adding the “repo --name AppStream…” line in kickstart.
It was just about a month ago that new builds started failing again. And of course, only for Rocky8, as that is the only EL8 distro we are using.
Interestingly, new builds are still working fine in our other environment, which is nearly exactly the same (different regions), and also in our Lab (new versions of Foreman/Katello, and I even have Rocky9, which also works fine).

That is interesting. Could you paste the full, unmodified contents of the .treeinfo for EL8 (where you are seeing this issue) and EL9 (where you are not)?

It may even be possible that this is an anaconda issue on EL8 that works fine with EL9. I can try to investigate that.

This might now be fair, because the Rocky9 repo is in the Lab, which doesn’t have a problem with Rocky8.

Either way, here is Rocky9, from the EXTERNAL repo:

[checksums]
images/boot.iso = sha256:fc6b306b9fc8d327d7545373ce88c48bd1f7e84d29548814a91e84676b7d26e4
images/efiboot.img = sha256:5b23c0a5b2ee4107489adc7cf843eba5a3f45b383d9a2137e07b657e05961fc2
images/install.img = sha256:772587fe8f06c8fb15e5f08358fdb3d810d4fff4df75b7db92b37d6eaea2e5f2
images/pxeboot/initrd.img = sha256:1845c35e228ac72c6f2018b4d220cd6ea87fbe7bd6c8949fa398a1f7745fee7a
images/pxeboot/vmlinuz = sha256:11f485cce4594384298555fe944a21b520e6ce409bd7e3792083e6a8bd5943c0

[general]
; WARNING.0 = This section provides compatibility with pre-productmd treeinfos.
; WARNING.1 = Read productmd documentation for details about new format.
arch = x86_64
family = Rocky Linux
name = Rocky Linux 9.0
packagedir = Packages
platforms = x86_64,xen
repository = .
timestamp = 1656986875
variant = BaseOS
variants = AppStream,BaseOS
version = 9.0

[header]
type = productmd.treeinfo
version = 1.2

[images-x86_64]
boot.iso = images/boot.iso
efiboot.img = images/efiboot.img
initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz

[images-xen]
initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz

[release]
name = Rocky Linux
short = Rocky
version = 9.0

[stage2]
mainimage = images/install.img

[tree]
arch = x86_64
build_timestamp = 1656986875
platforms = x86_64,xen
variants = AppStream,BaseOS

[variant-AppStream]
id = AppStream
name = AppStream
packages = ../../../AppStream/x86_64/os/Packages
repository = ../../../AppStream/x86_64/os
type = variant
uid = AppStream

[variant-BaseOS]
id = BaseOS
name = BaseOS
packages = Packages
repository = .
type = variant
uid = BaseOS

And here is Rocky8 .treeinfo from the EXTERNAL repo:

[checksums]
images/boot.iso = sha256:fe77cc293a2f2fe6ddbf5d4bc2b5c820024869bc7ea274c9e55416d215db0cc5
images/efiboot.img = sha256:0a9b08e1c545c49f7a890c991b4968e7bccba33bfc1e361c385c2f5d4c81c59e
images/install.img = sha256:b58937d928b8d0999eed8e6e5560b510089936d27392b202ce47ddae87d0ff44
images/pxeboot/initrd.img = sha256:e6eac6087a978a6c08a1d484c9517b2648a7b4a69a5fb0203030bf9f95deac1b
images/pxeboot/vmlinuz = sha256:c52e5d5ca593edfa7466e62069afbdb4aa801d82eb5108a55ff567062c025e9d

[general]
arch = x86_64
family = Rocky Linux
name = Rocky Linux 8
packagedir = Packages
platforms = x86_64,xen
repository = .
timestamp = 1652648642
variant = BaseOS
variants = BaseOS
version = 8

[header]
type = productmd.treeinfo
version = 1.2

[images-x86_64]
boot.iso = images/boot.iso
efiboot.img = images/efiboot.img
initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz

[images-xen]
initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz

[release]
name = Rocky Linux
short = Rocky
version = 8

[stage2]
mainimage = images/install.img

[tree]
arch = x86_64
build_timestamp = 1652648642
platforms = x86_64,xen
variants = BaseOS,AppStream

[variant-BaseOS]
id = BaseOS
name = BaseOS
packages = Packages
repository = .
type = variant
uid = BaseOS

[variant-AppStream]
id = AppStream
name = AppStream
type = variant
uid = AppStream
packages = ../../../AppStream/x86_64/os/Packages
repository = ../../../AppStream/x86_64/os/

And this is the Rocky8 .treeinfo that I pulled from the foreman server itself (Rocky9 looks exactly like it, with version number and checksum changes):

[checksums]
images/boot.iso = sha256:fe77cc293a2f2fe6ddbf5d4bc2b5c820024869bc7ea274c9e55416d215db0cc5
images/efiboot.img = sha256:0a9b08e1c545c49f7a890c991b4968e7bccba33bfc1e361c385c2f5d4c81c59e
images/install.img = sha256:b58937d928b8d0999eed8e6e5560b510089936d27392b202ce47ddae87d0ff44
images/pxeboot/initrd.img = sha256:e6eac6087a978a6c08a1d484c9517b2648a7b4a69a5fb0203030bf9f95deac1b
images/pxeboot/vmlinuz = sha256:c52e5d5ca593edfa7466e62069afbdb4aa801d82eb5108a55ff567062c025e9d

[general]
; WARNING.0 = This section provides compatibility with pre-productmd treeinfos.
; WARNING.1 = Read productmd documentation for details about new format.
arch = x86_64
family = Rocky Linux
name = Rocky Linux 8
packagedir = Packages
platforms = x86_64,xen
repository = .
timestamp = 1652648642
variant = BaseOS
variants = AppStream,BaseOS
version = 8

[header]
type = productmd.treeinfo
version = 1.2

[images-x86_64]
boot.iso = images/boot.iso
efiboot.img = images/efiboot.img
initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz

[images-xen]
initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz

[release]
name = Rocky Linux
short = Rocky
version = 8

[stage2]
mainimage = images/install.img

[tree]
arch = x86_64
build_timestamp = 1652648642
platforms = x86_64,xen
variants = AppStream,BaseOS

[variant-AppStream]
id = AppStream
name = AppStream
packages = AppStream/Packages
repository = AppStream
type = variant
uid = AppStream

[variant-BaseOS]
id = BaseOS
name = BaseOS
packages = Packages
repository = .
type = variant
uid = BaseOS

Something that might be good to try for debugging purposes is provisioning from the Rocky 8 mirror directly. If the error is still there, that would tell us if Rocky is doing something strange.

The .treeinfo looks correct, so I’m going to try to hunt down whether there is an anaconda / kickstart difference between EL8 and EL9 that might cause it to reject this on EL8.

1 Like

Just to circle back around to this…
@iballou I was unable to try provisioning from external Rocky mirrors…firewalls and such.

My only working solution was my workaround of a local repo, modifying the .treeinfo, and using that as the source for repo syncing in Katello. That worked fine.

However, we have since upgraded to Foreman 3.3 and Katello 4.5, and the issue is gone.

Thanks for the assistance.

2 Likes

Thanks for the update @mshade, appreciate it.