Need help wih CentOs 8 content view

fmarconi · December 16, 2019, 9:21am

Foreman 1.24, but same happened on Nightly. I had BaseOS and AppStream repository synchronized, in a product named CentOS 8

A content view was created for the Prod and Test environments, as follows:

Whenever the content view is presented to a CentOS 8 server, the situation is that:

As You can see, I’m only able to list a few packages availables. Some usefull informations:

If I create a repository with public streams, onto the server, then I see the correct number of packages
I had a try to create a different content view, destroy and synchronize again the repos and product, regenerate metadatas either on CV or on repos, present to different VM and physical servers, but nothing has changed so far.

Any help appreciated,

WScottB · December 18, 2019, 4:25pm

I am seeing the same behavior. Foreman 1.24/Katello 3.14. AppStream shows 5532 pkgs in repo but the client only shows 940. I noticed this when I couldn’t install git. The repo sync’d to the foreman box OK (advanced sync completes with no errors). There does appear to be something amiss here.

Stefan_Heijmans · December 19, 2019, 10:26pm

What is your repo sync type; ondemand or immediate?

fmarconi · December 20, 2019, 7:14am

Immediate

WScottB · December 23, 2019, 9:34pm

Immediate

WScottB · January 2, 2020, 4:41pm

I wonder if this is related to this issue: https://bugs.centos.org/view.php?id=16470

warrenb · January 3, 2020, 1:21pm

I am seeing this too against 8 and 8.1 RHEL repos. I can see the data downloaded from Redhat is correct, but many packages are missing from the content views I have generated as seen by the client ( although all looks OK when viewing the details in Foreman UI)

Justin_Sherrill · January 3, 2020, 9:18pm

This is expected behavior with RHEL 8 Appstream, Centos 8 Appstream, and Fedora Appstream repos. These appstream repositories use a new feature of dnf called ‘modules’. Read more about them here:

https://docs.centos.org/en-US/8-docs/managing-userspace-components/assembly_introduction-to-modules/
https://docs.fedoraproject.org/en-US/modularity/installing-modules/

Think about modules as ‘sub’ repositories that have to be enabled in order to gain access to their content. This allows for multiple versions of the same software package to be available and install-able.

warrenb · January 5, 2020, 4:09pm

Hi Justin,

As an example, I am missing packages for sysstat and git ( @WScottB also mentioned this ). It is not clear to me how this is related to install modules from the Appstream. I plan to spend more time later this week. Perhaps enabling debug in Pulp might help.

Justin_Sherrill · January 6, 2020, 2:54pm

Ahhh, i was focused more on the package count disparity on appstream (which is expected), i’ll look into those packages not showing up.

WScottB · January 6, 2020, 6:03pm

I added back the public AppStream repo in addition to my enabled subscription with AppStream and ran dnf list (with grep -i appstream and sed to replace our company name). Here is an example of what shows up from katello vs what shows up from public repo with no module specifications (sticking with the git example):

geronimo-annotation.noarch                           1.0-23.module_el8.0.0+39+6a9b6e22                       AppStream
geronimo-annotation.noarch                           1.0-23.module_el8.0.0+39+6a9b6e22                       COMPANYNAME_CentOS_8_CentOS_8_AppStream
gfbgraph.i686                                        0.2.3-6.el8                                             AppStream
gfbgraph.x86_64                                      0.2.3-6.el8                                             AppStream
ghc-srpm-macros.noarch                               1.4.2-7.el8                                             AppStream
ghostscript.x86_64                                   9.25-2.el8_0.3                                          AppStream
giflib.i686                                          5.1.4-3.el8                                             AppStream
giflib.x86_64                                        5.1.4-3.el8                                             AppStream
gimp.x86_64                                          2:2.8.22-15.module_el8.0.0+36+bb6a76a2                  AppStream
gimp.x86_64                                          2:2.8.22-15.module_el8.0.0+36+bb6a76a2                  COMPANYNAME_CentOS_8_CentOS_8_AppStream
gimp-devel.x86_64                                    2:2.8.22-15.module_el8.0.0+36+bb6a76a2                  AppStream
gimp-devel.x86_64                                    2:2.8.22-15.module_el8.0.0+36+bb6a76a2                  COMPANYNAME_CentOS_8_CentOS_8_AppStream
gimp-devel-tools.x86_64                              2:2.8.22-15.module_el8.0.0+36+bb6a76a2                  AppStream
gimp-devel-tools.x86_64                              2:2.8.22-15.module_el8.0.0+36+bb6a76a2                  COMPANYNAME_CentOS_8_CentOS_8_AppStream
gimp-libs.x86_64                                     2:2.8.22-15.module_el8.0.0+36+bb6a76a2                  AppStream
gimp-libs.x86_64                                     2:2.8.22-15.module_el8.0.0+36+bb6a76a2                  COMPANYNAME_CentOS_8_CentOS_8_AppStream
git-all.noarch                                       2.18.1-3.el8                                            AppStream
git-clang-format.i686                                7.0.1-1.module_el8.0.0+12+30b38a9a                      AppStream
git-clang-format.i686                                7.0.1-1.module_el8.0.0+12+30b38a9a                      COMPANYNAME_CentOS_8_CentOS_8_AppStream
git-clang-format.x86_64                              7.0.1-1.module_el8.0.0+12+30b38a9a                      AppStream
git-clang-format.x86_64                              7.0.1-1.module_el8.0.0+12+30b38a9a                      COMPANYNAME_CentOS_8_CentOS_8_AppStream
git-daemon.x86_64                                    2.18.1-3.el8                                            AppStream
git-email.noarch                                     2.18.1-3.el8                                            AppStream
git-gui.noarch                                       2.18.1-3.el8                                            AppStream
git-instaweb.x86_64                                  2.18.1-3.el8                                            AppStream
git-subtree.x86_64                                   2.18.1-3.el8                                            AppStream
git-svn.x86_64                                       2.18.1-3.el8                                            AppStream
gitk.noarch                                          2.18.1-3.el8                                            AppStream
gitweb.noarch                                        2.18.1-3.el8                                            AppStream
gjs.i686                                             1.52.5-2.el8                                            AppStream
gjs.x86_64                                           1.52.5-2.el8                                            AppStream
gl-manpages.noarch                                   1.1-15.20161227.el8                                     AppStream
glade-libs.i686                                      3.22.1-1.el8                                            AppStream
glade-libs.x86_64                                    3.22.1-1.el8                                            AppStream
glassfish-el-api.noarch                              3.0.1-0.7.b08.module_el8.0.0+39+6a9b6e22                AppStream
glassfish-el-api.noarch                              3.0.1-0.7.b08.module_el8.0.0+39+6a9b6e22                COMPANYNAME_CentOS_8_CentOS_8_AppStream
glassfish-fastinfoset-javadoc.noarch                 1.2.13-9.module_el8.0.0+42+51564204                     AppStream
glassfish-jax-rs-api.noarch                          2.0.1-6.module_el8.0.0+42+51564204                      AppStream

WScottB · January 6, 2020, 6:10pm

Note: git didn’t show up here because I had already installed it from public repo.

Justin_Sherrill · January 7, 2020, 5:48pm

yep, i have reproduced, digging into it.

Justin_Sherrill · January 7, 2020, 8:18pm

It took a while to dig into this, and i got some help from the pulp team (thanks a ton to them!)

I found a few things:

this only happens within a content view. If you just subscribe to “Library”, the problem isn’t seen.
The problem is happening due to copying of contents incorrectly at publish time. I’ve tracked it down to a pulp bug: https://pulp.plan.io/issues/5942
The problem would happen currently regardless of whether or not you are using Content view filters

I imagine it will be a while before the pulp team is able to deliver a full fix for this issue, in the mean time we could fairly easy code fix this for content view publishes that do NOT involve filters (on the appstream repo).

Would that be helpful?

A full fix that also fixes it for filtered-publishes would require a pulp fix.

Justin_Sherrill · January 7, 2020, 9:35pm

Actually after discussing it with a fellow katello dev, i went ahead and opened a pr for this here: Fixes #28675 - copy all rpms as part of cv publish by jlsherrill · Pull Request #8494 · Katello/katello · GitHub

Note that this will only help if there are no fitlers assigned to this repository in the content view.

If you are adventurous and want to test the patch:

apply the patch (https://github.com/Katello/katello/pull/8494.patch) to your tfm-rubygem-katello install
restart foreman-tasks: systemctl restart dynflowd
republish the content, promote the new version to your desired lifecycle environment
The first time you do this, you will need to forcibly regenerate the content view version repository metadata, from the content view versions list:

Let me know how it goes!

warrenb · January 8, 2020, 5:53pm

Thank you! I’ll try your process and patches tomorrow and report back.

warrenb · January 10, 2020, 1:28pm

Success! I first used the Default Organization View, with Library which worked. I then tried your patch, published a new view and promoted it. This too worked successfully and I am able to install the required packages.

Thank you very much for taking the time to look at this!

WScottB · January 13, 2020, 3:02pm

The patch worked for us as well. Thanks!