Unable to kickstart Rocky Linux 8 due to .treeinfo

Support
1

I think some people have seen similar problems with Centos before.

I have the Rocky 8 Repos mirrored, and here is what katello is sending in the kickstart file for the repo configs:

# Install media
url --url http://foreman.xxxxxx.com/pulp/content/XXX/dev/View-Rocky-8-Default/custom/Rocky_Linux_8/Rocky_8_BaseOS_x86_64_os/
# Additional Media
# name: Rocky_8_BaseOS_x86_64_os
repo --name Rocky_8_BaseOS_x86_64_os --baseurl http://foreman.xxxxxx.com/pulp/content/XXX/dev/View-Rocky-8-Default/custom/Rocky_Linux_8/Rocky_8_BaseOS_x86_64_os/ --install
# name: Rocky_8_AppStream_x86_64_os
repo --name AppStream --baseurl http://foreman.xxxxxx.com/pulp/content/XXX/dev/View-Rocky-8-Default/custom/Rocky_Linux_8/Rocky_8_AppStream_x86_64_os/ --install

But there is a .treeinfo in the Rocky_8_BaseOS_x86_64_os/ repo that has this in it:

[variant-AppStream]
id = AppStream
name = AppStream
packages = AppStream/Packages
repository = AppStream
type = variant
uid = AppStream

I think that is causing the kickstart to think there should be a ./AppStream/Packages directory under the OS repo directory and making the kickstart fail with this:

15:36:23,138 DBG packaging: Using the repo configuration: RepoConfigurationData(cost=1000, excluded_packages=[], included_packages=[], name='anaconda-0', proxy='', ssl_configuration=SSLConfigurationData
(ca_cert_path='', client_cert_path='', client_key_path=''), ssl_verification_enabled=True, type='BASEURL', url='http://foreman.XXXXXXX.com/pulp/content/XXXXXX/dev/View-Rocky-8-Default/custom/R
ocky_Linux_8/Rocky_8_BaseOS_x86_64_os/')
15:36:23,140 DBG packaging: retrieving treeinfo from http://foreman.XXXXXXX.com/pulp/content/XXXXXX/dev/View-Rocky-8-Default/custom/Rocky_Linux_8/Rocky_8_BaseOS_x86_64_os/ (proxy: None ; ssl_v
erify: True)
15:36:23,141 INF packaging: Trying to download '.treeinfo'
15:36:23,167 DBG packaging: Retrieved '.treeinfo' from http://foreman.XXXXXXX.com/pulp/content/XXXXXX/dev/View-Rocky-8-Default/custom/Rocky_Linux_8/Rocky_8_BaseOS_x86_64_os/
15:36:23,175 DBG packaging: getting release version from tree at http://foreman.XXXXXXX.com/pulp/content/XXXXXX/dev/View-Rocky-8-Default/custom/Rocky_Linux_8/Rocky_8_BaseOS_x86_64_os/ (8)
15:36:23,176 DBG packaging: using treeinfo release version of 8
15:36:23,177 DBG packaging: Treeinfo points base repository to http://foreman.XXXXXXX.com/pulp/content/XXXXXX/dev/View-Rocky-8-Default/custom/Rocky_Linux_8/Rocky_8_BaseOS_x86_64_os/.
15:36:23,177 DBG packaging: releasever from http://foreman.XXXXXXX.com/pulp/content/XXXXXX/dev/View-Rocky-8-Default/custom/Rocky_Linux_8/Rocky_8_BaseOS_x86_64_os/ is 8
15:36:23,178 DBG packaging: Adding new treeinfo repository: AppStream enabled: True
15:36:23,179 INF packaging: added repo: 'AppStream' - http://foreman.XXXXXXX.com/pulp/content/XXXXXX/dev/View-Rocky-8-Default/custom/Rocky_Linux_8/Rocky_8_BaseOS_x86_64_os/AppStream
15:36:23,195 DBG dnf: repo: downloading from remote: AppStream
15:36:23,222 DBG dnf: error: Status code: 404 for http://foreman.XXXXXXX.com/pulp/content/XXXXXX/dev/View-Rocky-8-Default/custom/Rocky_Linux_8/Rocky_8_BaseOS_x86_64_os/AppStream/repodata/repom
d.xml (IP: 10.245.2.51) (http://foreman.XXXXXXX.com/pulp/content/XXXXXX/dev/View-Rocky-8-Default/custom/Rocky_Linux_8/Rocky_8_BaseOS_x86_64_os/AppStream/repodata/repomd.xml).

Has anyone come up with a solution?

2

Sounds like same issue as CentOS 8. The fix I’ve used is to modify the .treeinfo file, and then handle AppStream without the provisioning template. See Cannot kickstart CentOS 8 - #3 by llandin for details

1 Like
3

This has been fixed in Pulp 3. So upgrade to Foreman version with Pulp 3.

4

I am on pulp3 and I have the same issue for CentOS8 and this issue started after I upgraded katello 4.0.x → 4.1.3. So this is most likely a new bug. For reference this is my thread: Foreman failed to download metadata for repo 'AppStream': cannot download repomd.xml

5

Yeah, on 4.1.3/pulp3 here too and same issue.

The real question is, why is pulp putting AppStream into the .treeinfo for the OS repo when it does not even exist in that repo and is not available anywhere in that repo path?

AppStream is a totally different repo that we also have to mirror and add to the content views.

If anything, pulp should REMOVE AppStream out of the .treeinfo in the OS repo as it is from the upstream sites, because there is no easy “path” that you can get to it like there is for the official centos/rocky repo sites.

We are already providing an AppStream repo in the kickstart file, pulp should REMOVE IT from the .treeinfo on the BaseOS repo.

6

Interestingly I just checked, and RHEL8 DOES NOT have AppStream mentioned in the BaseOS .treeinfo file at all.

That is why RHEL8 kickstarts work, but I have never been able to get Centos8 or Rocky8 to kickstart using katello as the content source.

7

This is coming back, could pulp guys comment on that? Or better, to test this? The last time I tried this it worked, but multiple people are reporting back that CentOS Stream and also other alternatives do not work correctly. @mcorr

8

On mobile now (on holiday), but this sounds like the same issue I mentioned in the other thread @lzap

The fix (quick and dirty, will only work in Rocky only environments) is to replace centos → rocky on lines 81 and 89 in ‘Kickstart default’ provisioning template (unlock it first).

When I return from my holiday will open a PR to try and fix this

9

Thanks that could be it! FYI here it is:

This is a workaround around the fact that CentOS and RHEL Anaconda are different - one of the two performs some path replacement/change, so we manually add AppStream repo in the kickstart if we find it is a RHEL-compatible system but named centos.

10

Already doing that in my template(and modified for Rocky). That does not fix it.

I think this issue is because pulp is putting an AppStream entry in the BaseOS repo .treeinfo it gives out with a totally bogus URL path ./AppStream that does not exist and the kickstart is trying to read it and failing.

Really the AppStream entry should totally be removed. That entry works fine on the offical repo mirror sites because you can easily get from the BaseOS repo directory to AppStream repo via a …/…/AppStream. Outside of the purpose of the official Centos and Rocky mirror sites, and their known directory layout, this serves no purpose.

With katello/pulp the repos have a totally different URL path names, and there is zero practical way to get to the AppStream repo FROM the BaseOS repo URL.

Given that we ADD THE AppStream repo to clients ourselves with template as shown above, I think pulp should NOT INCLUDE anything about AppStream in the .treeinfo it gives out for the BaseOS repo channel.

I bet the same thing is happening for Centos 8/Centos8 Streams too.

Note, RHEL does NOT INCLUDE an AppStream entry in the BaseOS .treeinfo repo, it uses a separate AppStream repo entry in the kickstart file to file it, like we have in the template.

I think the maintainers of Centos/Rocky put that in the BaseOS .treeinfo for a convenience to their users based on the way their official repo mirrors are layed out, and I believe we do not need it, and it should be filtered out by pulp.

1 Like
11

FYI I have filed this a year ago, it is still open: https://bugs.centos.org/view.php?id=16470

So here is a proposal:

  • change pulp to ignore AppStream from treeinfo (at least as an opt-in option)
  • add the repo explicitly in kickstarts for all RHEL-based OSes

@Marek_Hulan what you say? I would file a BZ for Pulp we need to solve this for once.

12

Yeah, please do at Issues - Pulp

13

So I have researched this down to the patch that was merged into Pulp 3.x a year back and I resurrected that github discussion because from all our (multiple) users are telling us (this is not the only thread we have here on this topic), it looks like it does not work correctly in Pulp.

So action items are:

  • figure out what is wrong with Pulp 3
  • fix it, perhaps backport it into stable releases
  • change our default kickstart to always include AppStream regardless of upstream/downstream

Here it is: https://github.com/pulp/pulp_rpm/pull/1755

2 Likes
14

For the record, I have just reproduced this on Satellite 6.9 which has Pulp 3.10. That version is supposed to be fixed. The published .treeinfo does contain incorrect AppStream entry.

[checksums]
images/boot.iso = sha256:53a62a72881b931bdad6b13bcece7c3a2d4ca9c4a2f1e1a8029d081dd25ea61f
images/efiboot.img = sha256:6d40c9c2e8a686254c5b3f0f769969386d6daa7737f2f4e428a00c09993b549e
images/install.img = sha256:88e49ee73883171c85414543f073d83bf5e520e78498c48084d727589ace49ad
images/pxeboot/initrd.img = sha256:f9eceb3503134874100a2e076cc90f4d997cb2a32126bdd570d42d59683f74be
images/pxeboot/vmlinuz = sha256:f3f9f33573fa6beba7ab5ba4b5d771729b8fe669352001bc046006da01c1296d

[general]
; WARNING.0 = This section provides compatibility with pre-productmd treeinfos.
; WARNING.1 = Read productmd documentation for details about new format.
arch = x86_64
family = Rocky Linux
name = Rocky Linux 8
packagedir = Packages
platforms = x86_64,xen
repository = .
timestamp = 1624205579
variant = BaseOS
variants = BaseOS
version = 8

[header]
type = productmd.treeinfo
version = 1.2

[images-x86_64]
boot.iso = images/boot.iso
efiboot.img = images/efiboot.img
initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz

[images-xen]
initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz

[release]
name = Rocky Linux
short = Rocky
version = 8

[stage2]
mainimage = images/install.img

[tree]
arch = x86_64
build_timestamp = 1624205579
platforms = x86_64,xen
variants = BaseOS,AppStream

[variant-BaseOS]
id = BaseOS
name = BaseOS
packages = Packages
repository = .
type = variant
uid = BaseOS

[variant-AppStream]
id = AppStream
name = AppStream
type = variant
uid = AppStream
packages = ../../../AppStream/x86_64/os/Packages
repository = ../../../AppStream/x86_64/os/
15

I personally think ANY AppStream entry in the BaseOS .treeinfo is wrong. It should be removed.

With Katello the AppStream repo will be no where under or available via any sort of easily assumed path from the URL to the BaseOS repo.

1 Like
16

I just edit .treeinfo manually in the pulp directory for every BaseOS repos and update the AppStream section like shown below. The change seems to persist in subsequent sync and in content views.

[variant-AppStream]
id = AppStream
name = AppStream
packages = Packages
repository = .
type = variant
uid = AppStream

Then in the provisioning template I have repo --name AppStream --url ..... I’ve made sure to name BaseOS and AppStream repos in the same way, so that the URL is the same. Then I only need to change BaseOS to AppStream, which I do in the provisioning template like so:

<% if rhel_compatible && @host.operatingsystem.name.downcase.include?("centos") && os_major >= 8 && medium_uri.to_s.include?("BaseOS") %>
repo --name AppStream --baseurl <%= medium_uri.to_s.gsub("BaseOS", "AppStream") %><%= proxy_string %>
<% end %>

This seems very simple and clean compared some PR and any other path hacks.

1 Like
17

I just tested this on Satellite 6.10 (Katello 4.1.1) and I am not seeing what you have. In my case, BaseOS repository sync the BaseOS content as well as AppStream which is then added as a “subrepository”:

[variant-BaseOS]
id = BaseOS
name = BaseOS
packages = Packages
repository = .
type = variant
uid = BaseOS

The relative path is available and works fine:

# curl -k https://zzzap.xxx.redhat.com/pulp/content/Default_Organization/Library/custom/CentOS8/CentOS8StreamBaseOS/AppStream/
        <!DOCTYPE html>
        <html>
            <body>
                <ul>
                    <li><a href="EULA">EULA</a></li>
                    <li><a href="GPL">GPL</a></li>
                    <li><a href="Packages/">Packages/</a></li>
                    <li><a href="extra_files.json">extra_files.json</a></li>
                    <li><a href="repodata/">repodata/</a></li>
                </ul>
            </body>
        </html>

This also means we no longer need the kickstart workaround anymore, actually it can break things now. We need to remove it.

I will test end to end provisioning on Monday as well as Rocky Linux but I expect it to be the same.

Note that only Pulp 3 will work correctly, I do believe that Pulp 2 hasn`t been fixed.

1 Like
18

Hi @lzap,

Any luck? I’m back from my holiday :slight_smile:

I went ahead and reinstalled my lab VM with Foreman 2.5.3 and Katello 4.1.4 (again). I first tried applying the first workaround again (as I mentioned earlier, replacing ‘centos’ with ‘rocky’) and it didn’t work.

Next I removed all of the special logic, so the template went from this:

<% if rhel_compatible && os_major >= 8 && medium[:url] && medium[:url].include?("AppStream") -%>
<% appstream_present = true -%>
# renamed from "<%= medium[:url] %>" for CentOS Anaconda to work
repo --name AppStream --baseurl <%= medium[:url] %>
<% else -%>
repo --name <%= medium[:name] %> --baseurl <%= medium[:url] %> <%= medium[:install] ? ' --install' : '' %><%= proxy_string %>
<% end -%>
<% end -%>
<% if rhel_compatible && os_major >= 8 && medium_uri && medium_uri.to_s.include?("BaseOS") && !appstream_present -%>
# repository added using BaseOS to AppStream string replacement
repo --name AppStream --baseurl <%= medium_uri.to_s.gsub("BaseOS", "AppStream") %>
<% end -%>
<%= snippet_if_exists(template_name + " custom repositories") %>
<% end %>

to this

<% @additional_media.each do |medium| -%>
repo --name <%= medium[:name] %> --baseurl <%= medium[:url] %> <%= medium[:install] ? ' --install' : '' %><%= proxy_string %>
<% end -%>
<%= snippet_if_exists(template_name + " custom repositories") %>
<% end %>

If that is what you meant by removing the AppStream workaround? It didn’t work though :frowning:

Or do I need a newer version of pulp-rpm before this is actually working? In any case, I have 3.14.3 installed on my system.

19

I created a PR for this as I hit it too (non-katello foreman)

1 Like
20

Last night I deployed a fresh Rocky 8.4 with Foreman 3 and Katello 4.2, synced the Rocky 8 repo’s and pxe booted a fresh vm. Installation went fine.

I have BaseOS + Kickstart, AppStream + Kickstart, Extra’s en PowerTools synced.

1 Like