Installing smart-proxy from content-view: packages from repo in content-view seemingly ignored

rainer_d · July 24, 2021, 8:07pm

Problem:
I have created a content view of the repos needed to install foreman-proxy and promoted it to the respective environment (and created an activation-key).
I have registered a host with this activation key and when I go to the URL in the repo-file, I can see the package there.
However, yum only seems to see the older version of the package in the base-repo.
I have disabled all the default repos.

21:44:07 (PROD) root@do-pxe02-prod [/etc/yum.repos.d] # yum provides -d 10 python3-pygments
timer: config: 2 ms
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
Updating Subscription Management repositories.
YUM version: 4.4.2
Command: yum provides -d 10 python3-pygments 
Installroot: /
Releasever: 8
cachedir: /var/cache/dnf
Base command: provides
Extra commands: ['provides', '-d', '10', 'python3-pygments']
User-Agent: constructed: 'libdnf (CentOS Linux 8; generic; Linux.x86_64)'
repo: using cache for: ansible-runner
ansible-runner: using metadata from Fri 12 Feb 2021 10:05:26 AM CET.
repo: using cache for: puppet6
puppet6: using metadata from Tue 20 Jul 2021 08:08:29 PM CEST.
repo: using cache for: CORP_Foreman_25_Katello_41_candlepin
CORP_Foreman_25_Katello_41_candlepin: using metadata from Tue 20 Jul 2021 06:38:51 PM CEST.
repo: using cache for: CORP_CentOS8_CentOS_8_-_AppStream_-_x86_64
CORP_CentOS8_CentOS_8_-_AppStream_-_x86_64: using metadata from Sat 24 Jul 2021 04:56:06 PM CEST.
repo: using cache for: CORP_CentOS8-ansible-29_CentOS_8_-_ansible29_cfg_mgmt_-_x86_64
CORP_CentOS8-ansible-29_CentOS_8_-_ansible29_cfg_mgmt_-_x86_64: using metadata from Tue 20 Jul 2021 06:39:58 PM CEST.
repo: using cache for: CORP_Foreman_25_Katello_41
CORP_Foreman_25_Katello_41: using metadata from Tue 20 Jul 2021 06:40:00 PM CEST.
repo: using cache for: CORP_Foreman_25_Foreman_25_Plugins
CORP_Foreman_25_Foreman_25_Plugins: using metadata from Sat 24 Jul 2021 05:09:08 PM CEST.
repo: using cache for: CORP_Foreman_25_Puppet_6
CORP_Foreman_25_Puppet_6: using metadata from Thu 22 Jul 2021 04:45:29 PM CEST.
repo: using cache for: CORP_Foreman_25_Foreman_25
CORP_Foreman_25_Foreman_25: using metadata from Tue 20 Jul 2021 06:39:57 PM CEST.
repo: using cache for: CORP_CentOS8_CentOS_8_-_PowerTools_-_x86_64
CORP_CentOS8_CentOS_8_-_PowerTools_-_x86_64: using metadata from Sat 24 Jul 2021 04:54:56 PM CEST.
repo: using cache for: CORP_Foreman_25_Katello_41_pulpcore
CORP_Foreman_25_Katello_41_pulpcore: using metadata from Tue 20 Jul 2021 06:40:02 PM CEST.
Last metadata expiration check: 0:18:42 ago on Sat 24 Jul 2021 09:25:22 PM CEST.
timer: sack setup: 545 ms
Completion plugin: Generating completion cache...
Searching Packages: 
python3-pygments-2.2.0-20.module_el8.4.0+790+083e3d81.noarch : Syntax highlighting engine written in Python
Repo        : CORP_CentOS8_CentOS_8_-_AppStream_-_x86_64
Matched from:
Provide    : python3-pygments = 2.2.0-20.module_el8.4.0+790+083e3d81

Cleaning up.
21:44:13 (PROD) root@do-pxe02-prod [/etc/yum.repos.d] # 

        21:52:28 (PROD) root@do-pxe02-prod [/etc/yum.repos.d] # curl https://foreman.dom.tld/pulp/content/CORP/Foreman_Test/Foreman_Host/custom/Foreman_25/Katello_41_pulpcore/Packages/p/ |grep pygments
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21373  100 21373    0     0   474k                       <li><a href="python3-pygments-2.8.1-1.el8.noarch.rpm">python3-pygments-2.8.1-1.el8.noarch.rpm</a></li>
   0 --:--:-- --:--:-- --:--:--  474k
21:52:34 (PROD) root@do-pxe02-prod [/etc/yum.repos.d] #

Foreman and Proxy versions:

foreman-debug-2.5.2-1.el8.noarch
foreman-installer-2.5.2-3.el8.noarch
foreman-installer-katello-2.5.2-3.el8.noarch
foreman-proxy-2.5.2-1.el8.noarch

Distribution and version:

CentOS 8.4

jtruestedt · July 26, 2021, 5:12am

Do you have a caching proxy inbetween, which could have some cached metadata?

rainer_d · July 26, 2021, 8:06am

No, it’s a direct connection.

The sad thing is, I had this working with 2.4.2 (at least I could run a kickstart file) but I jumped the gun at upgrading to 2.5.2 before all the files were in the repo…

Is there a way to debug dnf or yum to see why it’s not “seeing” the file?
Or see what it’s actually seeing?

jtruestedt · July 26, 2021, 8:17am

This looks like it uses a cached metadata from 6 days ago.

Could you run a “dnf clean all” and then try again?

rainer_d · July 26, 2021, 8:57am

I’ve done that multiple times.

However, the problem has now been found: I had disabled all the default CentOS Repos (because I have the content-view), after registering the system with foreman.
However, on closer inspection, the content view is missing the BaseOS repo.

That immediately broke almost everything - however becomes only apparent when you try to install python modules and python itself.

When I added the repo to the content-view and tried to publish it, that didn’t work either.

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Mon, 26 Jul 2021 08:39:57 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"112", "correlation-id"=>"55309c12-28ee-4c7d-8852-97ec37d193b7", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman-server", "connection"=>"close"}
Response body: ["URI /pulp/api/v3/publications/rpm/rpm/4886a91e-ca16-4d35-a4c9-66f700682afa/ not found for repositoryversion."]

I assume, it’s irritated by the fact that the additional repo “appeared out of thin air”, so to speak.

This is clearly, again, a user-error, but maybe it just should not be possible to add a repo to an existing content-view if that doesn’t work either?

gvde · July 26, 2021, 9:04am

There seem to be some serious issues with pulpcore 3.14, which came with the update to 2.5.2/4.1.1.

Topic on your error is here: CentOS 8.4 BaseOS Sync error

So far it seems to be an issue only with the EL8 BaseOS repositories.

With pulpcore 3.14 there is also some metadata from the RPMs missing in the database which can cause dependency errors. See topic Cannot update puppet-agent to 6.24.0 on foreman servers

At the moment I don’t install updates from katello anymore but use CentOS mirrors to update, if necessary.

rainer_d · July 26, 2021, 9:07am

Thanks. But what is the suggestion if one wants to build a proof-of-concept?
Use Foreman 2.4-series?

The problem isn’t really installing 2.4 again, but syncing all the content from the mirrors etc…

gvde · July 26, 2021, 9:22am

I don’t know about the schedule of the developers, when those issues are going to be fixed. At the moment, I would recommend to wait for 4.1.2 if the contains fixes for those issue. Or maybe 2.6/4.2 or even 4.3. I think some of the issues I have seen with pulp3 are going to be included with 4.3.

I don’t know about 2.4/4.0. After a lot of problems with the pulp migration I eventually upgraded only through 2.4/4.0 to 2.5/4.1 and didn’t run 2.4 for a longer time…

gvde · August 23, 2021, 7:15am

Well, first check if it is actually in the published repository, e.g. pick up the full URL for the repository from /etc/yum.repos.d/redhat.repo, enter it into a browser, make sure the URL ends in a slash “/” and you should see the listing. The package should be there or in a subdirectory. If it’s there, check if you can download it in your browser.

Thulium-Drake · September 7, 2021, 4:54pm

@katello anyone have a clue?

gvde · September 7, 2021, 5:05pm

I don’t know how foreman is using modules on el8, but you can run

# dnf module provides python3-pygments

to see which modules provide python3-pygments.

# dnf provides python3-pygments

will show which version it considers the one available with your current module selection.

# dnf module list --enabled

shows the enabled modules.

evgeni · September 7, 2021, 6:18pm

You need to set module_hotfixes=1 in the .repo file on your client, like we do in our repo file that we ship:

github.com

theforeman/foreman-packaging/blob/52c585a4cfbaa477cc9f4fede5e0503e99206e28/packages/katello/katello-repos/katello.repo#L28

    
      
          enabled=1
          gpgcheck=@REPO_GPGCHECK@
          module_hotfixes=1
          
          
[pulpcore]
          name=pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.
          baseurl=https://yum.theforeman.org/pulpcore/@PULPCORE_VERSION@/@DIST@/$basearch/
          gpgkey=https://yum.theforeman.org/pulpcore/@PULPCORE_VERSION@/GPG-RPM-KEY-pulpcore
          enabled=1
          gpgcheck=1
          module_hotfixes=1
          
          
# source repositories
          
          
[katello-source]
          name=Katello @REPO_NAME@ Source
          baseurl=https://yum.theforeman.org/katello/@REPO_VERSION@/katello/@DIST@/source/
          gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman
          enabled=0
          gpgcheck=@REPO_GPGCHECK@

Otherwise the version of pygments that we ship isn’t considered by dnf.

If the .repo file is managed by subscription-manager, you should (I never really tested this) be able to set that via subscription-manager repo-override --repo Foreman-Pulp-EL8-Base --add module_hotfixes:1.

HTH
Evgeni

Thulium-Drake · September 9, 2021, 11:10am

Thank you!

Is there a way to force this via the Foreman server (and configure it via Ansible )

evgeni · September 9, 2021, 11:12am

command: subscription-manager repo-override --repo Foreman-Pulp-EL8-Base --add module_hotfixes:1

/me runs away. No, I have no clue if the server can force it, but my bet is on “No”.

Thulium-Drake · September 9, 2021, 11:13am

That also works fine, I’m installing these systems via Ansible anyway. I was just wondering if it was possible.

I’ll add that command task to my playbook