Testing of CentOS/Alma/Rocky 8 AppRepo

Development
1

Hey,

for couple of months, we are hearing about issues with EL8 “AppStream” handling of provisioning. It’s a bit mess, I am trying to figure this out, therefore I am creating this wiki page where I want to collect our findings of what works and what does not.

Feel free to use Kickstarting without provisioning setup to easily test scenarios and drop your findings in the table below.

Explanation:

  • Distro - the subject of kickstarting.
  • Repository - source repository - fill in either Foreman when using a public or a mirror repo or Katello when using synced content. Also include version.
  • Pulp - version of pulp used for the workflow. Do not fill for Foreman deployment.
  • Workaround - Foreman/Katello currently ships with a workaround in the default kickstart. State if you used it or if you commented it out or modified it in any way.
  • Result - your result, drop comments below for further information.

When testing, make sure to include this in your kickstart to test if AppStream really works:

%packages
# install a package from AppStream repository
vim-ehnanced

Here are the current findings:

Distro Repository Pulp Workaround Result
CentOS 8 Stream Foreman (2.5) - Commented out No problems
CentOS 8 Stream Katello (4.1) 3.14 Yes Anaconda gets stuck, repository problem
CentOS 8 Stream Katello (4.1) 3.14 Commented out No problems
Rocky 8 Foreman (2.5) - Commented out No problems
Rocky 8 Katello (4.1) 3.14 Yes¹ Anaconda gets stuck, repository problem
Rocky 8 Katello (4.1) 3.14 Commented out No problems

¹ : The workaround has been modified to work for Rocky Linux → s/centos/rocky/

The conclusion

I think we need to delete the workaround from our default kickstart, I do not understand now why this was added in the first place. I was able to track this down to:

How this was supposed to work? @Partha_Aji @ianf77

I propose to remove this workaround completely: https://github.com/theforeman/foreman/pull/8836

3 Likes
2

I have added my findings to the table. But something strange is going on…

Previously I was able to make Rocky’s Kickstart work by changing the workaround to match Rocky instead of CentOS. But now I’m unable to find a configuration (with or without the workaround to make it work).

I tried the following repository names for AppStream:

  • Rocky8-Appstream
  • AppStream

All with Product name ‘Rocky8’, no combination works :frowning:

1 Like
3

Which repo are you testing with? It is a bit messy on the Rocky mirror:

Looks like they have bunch of “kickstart” symlinks, not sure why.

4

@Thulium-Drake I am not sure what you are doing but it works great here, I am updating the table. I guess we will be removing the workaround from the default kickstart, it is not needed at all.

  1. I created a product
  2. I created a repo with Index of /pub/rocky/8/BaseOS/x86_64/os/
  3. I synced the repo
  4. I created a new host with this “Synced Content” OS
  5. I modified the default kickstart template and commented out all the workarounds
  6. Provisioned the system just fine

Here is what I did in the template, it is not needed anymore with Pulp 3:

Note: I did not add any AppStream repository to the product, it is not necessary as Pulp3 automatically adds it.

Also tested this with installation media, works fine too.

5

I also tried to create a product with both BaseOS and AppStream repositories. I applied the mentioned “removed AppStream Pulp2 workaround” patch to my template and this is how it all got rendered:

url --url http://xxx.redhat.com/pulp/content/Default_Organization/Library/custom/CentOS8WithAppStream/CentOS8BaseOS/
repo --name CentOS8BaseOS --baseurl http://xxx.redhat.com/pulp/content/Default_Organization/Library/custom/CentOS8WithAppStream/CentOS8BaseOS/ 
repo --name CentOS8AppStream --baseurl http://xxx.redhat.com/pulp/content/Default_Organization/Library/custom/CentOS8WithAppStream/CentOS8AppStream/

And it worked fine. Although the AppStream repository is actually twice, Anaconda has no problems with it and installs CentOS 8 Stream just fine.

6

Interesting! I’ll have a look tonight to see if I can find what’s wrong :slight_smile:

Thanks for testing <3

7

I’ve been using

https://dl.rockylinux.org/pub/rocky/8/BaseOS/x86_64/os
https://dl.rockylinux.org/pub/rocky/8/AppStream/x86_64/os/

In a single Product and Content view

Where the repo names are:

Rocky8-BaseOS
Rocky8-AppStream

8

I don’t follow this, how did you sync the AppStream repo then? Or does Pulp3 also pick that up?

9

I will say Rocky 8 is working fine for my in Foreman3.0/Katello 4.2rc3. Alma I haven’t been able to get to work yet. AppStream is working fine with it.

10

You have to have both the regular BaseOS/AppStream and the BaseOS/AppStream kickstart repos it in the content view though if you want to get updated after the build.

You can get by with just the kickstart in the content view for doing builds maybe, but that does not work for Day 1+.

11

Yes, isn’t that obvious from the conversation? Let me try it again: When you sync CentOS 8 BaseOs repo with Pulp3+ it will also pick AppStream with it. On the Repository katello page, you will see “Variants: BaseOS, AppStream”, however, you will NOT see those packages in the list of packages. It is a bug I reported.

Anaconda will see AppStream packages too.

Try it and report back. I would like to confirm this before we remove the workaround.

Now, the workaround might still be needed for Pulp 2 instances, but that is something I do not have time to test. It is also quite ugly, I am happy to drop it.

12

Oh wow, I see it now that you mentioned it :slight_smile: I’ve been syncing the packages double forever :joy: Thanks for clarifying!

I will update my setup and test it out! Will have results tomorrow :slight_smile:

13

Yes, so here’s what I did (and what might be the source of my problem)

  1. Remove the ‘explicit’ AppStream repo from my product
  2. Remove workaround
  3. Installed host

And it works just like you said!

Thank you very much!

14

I did run into an issue with subscription-manager though. After rebooting the system the AppStream repo is not added by subscription-manager to /etc/yum.repos.d/redhat.repo, so yum can’t find them

[root@mac8a111f36326d ~]# yum module list
Updating Subscription Management repositories.
Rocky8-BaseOS_AppStream                                                               47 kB/s | 4.3 kB     00:00    
Foreman-Tools-EL8-Base                                                                49 kB/s | 3.6 kB     00:00    
Zabbix-EL8-Zabbix-5.4                                                                 47 kB/s | 2.9 kB     00:00    
@modulefailsafe
Name                    Stream                     Profiles                   Summary                                
virt                    rhel [e]                   common                     Virtualization module                  

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

I copied the redhat.repo file and added ‘original baseos repo url/AppStream’ and then ‘yum module list’ shows all the modules again.

I take it this needs to be fixed by the Candlepin team?

15

I am hitting the issue of Rocky Linux 8 Appstream repo availability, but it’s only a problem while Anaconda is running. After provisioning is complete, appstream is properly configured as a yum/dnf repo.

Here’s the Anaconda message I get, just after package installation begins:

Starting package installation process
Question

     Problem in request:
     missing packages: wget, redhat-lsb-core

     Would you like to ignore this and continue with installation?

Please respond ‘yes’ or ‘no’:

Replying ‘no’ reboots. Replying ‘yes’ allows Anaconda to finish, but the ‘wget’ and ‘redhat-lsb-core’ packages are not installed.

Some details on my environment:

Further notes:

  • Both the ‘wget’ and ‘redhat-lsb-core’ packages are in the Rocky Linux 8 Appstream repo.
  • In a clone of Kickstart Default, when I comment out the “workaround” section mentioned by others in this thread, the problem remains.
  • When I force “appstream_present = true” in the top block of assignments, the problem remains.
  • When I comment out the wget’ and ‘redhat-lsb-core’ packages, the installation completes unattended, but the two Appstream packages are missing (suboptimal!)
  • The problem does NOT occur when provisioning CentOS 8.4

Seems like Anaconda can’t find Rocky 8 Appstream, but why?

16

It looks like everything in Rockylinks .treeinfo seems correct. Can you completely remove the additional_media stuff from the template? It performs some search and replace there, Anaconda should be able to read AppStream variant from the treeinfo file just fine without any hacks it seems.

17

lzap,

Once again, in a clone of template “Kickstart Default,” I commented out just the section that handles ‘additional_media’:

<% @additional_media.each do |medium| -%>
…
<% end %>

As before, Anaconda can’t find the Rocky AppStream repo and the build pauses, asking what to do about packages ‘wget’ and ‘redhat-lsb-core’.

What DID solve the problem, was to add ‘additional_media’ as a parameter to my Rocky 8 OS definition, like so:

Name: additional_media
Type: string
Value: [{“name”:“appstream”,“url”:“http://dl.rockylinux.org/$contentdir/$releasever/AppStream/$basearch/os/”}]

Subsequently, I was able to build Rocky 8.4 with the Kickstart Default template and Anaconda finds Rocky 8 AppStream, as expected.

I referenced this post, which discusses a similar issue with CentOS 8 and the EPEL repo.

One thing I noticed: ‘additional_media’ is referenced as a parameter in the Kickstart Default template, but it is not documented in the initial comment block. Why keep it a secret?

18

I don’t understand why it does work for me.

Feel free to add it there. We are on github!