Installing smart-proxy from content-view: packages from repo in content-view seemingly ignored

Problem:
I have created a content view of the repos needed to install foreman-proxy and promoted it to the respective environment (and created an activation-key).
I have registered a host with this activation key and when I go to the URL in the repo-file, I can see the package there.
However, yum only seems to see the older version of the package in the base-repo.
I have disabled all the default repos.

21:44:07 (PROD) root@do-pxe02-prod [/etc/yum.repos.d] # yum provides -d 10 python3-pygments
timer: config: 2 ms
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
Updating Subscription Management repositories.
YUM version: 4.4.2
Command: yum provides -d 10 python3-pygments 
Installroot: /
Releasever: 8
cachedir: /var/cache/dnf
Base command: provides
Extra commands: ['provides', '-d', '10', 'python3-pygments']
User-Agent: constructed: 'libdnf (CentOS Linux 8; generic; Linux.x86_64)'
repo: using cache for: ansible-runner
ansible-runner: using metadata from Fri 12 Feb 2021 10:05:26 AM CET.
repo: using cache for: puppet6
puppet6: using metadata from Tue 20 Jul 2021 08:08:29 PM CEST.
repo: using cache for: CORP_Foreman_25_Katello_41_candlepin
CORP_Foreman_25_Katello_41_candlepin: using metadata from Tue 20 Jul 2021 06:38:51 PM CEST.
repo: using cache for: CORP_CentOS8_CentOS_8_-_AppStream_-_x86_64
CORP_CentOS8_CentOS_8_-_AppStream_-_x86_64: using metadata from Sat 24 Jul 2021 04:56:06 PM CEST.
repo: using cache for: CORP_CentOS8-ansible-29_CentOS_8_-_ansible29_cfg_mgmt_-_x86_64
CORP_CentOS8-ansible-29_CentOS_8_-_ansible29_cfg_mgmt_-_x86_64: using metadata from Tue 20 Jul 2021 06:39:58 PM CEST.
repo: using cache for: CORP_Foreman_25_Katello_41
CORP_Foreman_25_Katello_41: using metadata from Tue 20 Jul 2021 06:40:00 PM CEST.
repo: using cache for: CORP_Foreman_25_Foreman_25_Plugins
CORP_Foreman_25_Foreman_25_Plugins: using metadata from Sat 24 Jul 2021 05:09:08 PM CEST.
repo: using cache for: CORP_Foreman_25_Puppet_6
CORP_Foreman_25_Puppet_6: using metadata from Thu 22 Jul 2021 04:45:29 PM CEST.
repo: using cache for: CORP_Foreman_25_Foreman_25
CORP_Foreman_25_Foreman_25: using metadata from Tue 20 Jul 2021 06:39:57 PM CEST.
repo: using cache for: CORP_CentOS8_CentOS_8_-_PowerTools_-_x86_64
CORP_CentOS8_CentOS_8_-_PowerTools_-_x86_64: using metadata from Sat 24 Jul 2021 04:54:56 PM CEST.
repo: using cache for: CORP_Foreman_25_Katello_41_pulpcore
CORP_Foreman_25_Katello_41_pulpcore: using metadata from Tue 20 Jul 2021 06:40:02 PM CEST.
Last metadata expiration check: 0:18:42 ago on Sat 24 Jul 2021 09:25:22 PM CEST.
timer: sack setup: 545 ms
Completion plugin: Generating completion cache...
Searching Packages: 
python3-pygments-2.2.0-20.module_el8.4.0+790+083e3d81.noarch : Syntax highlighting engine written in Python
Repo        : CORP_CentOS8_CentOS_8_-_AppStream_-_x86_64
Matched from:
Provide    : python3-pygments = 2.2.0-20.module_el8.4.0+790+083e3d81

Cleaning up.
21:44:13 (PROD) root@do-pxe02-prod [/etc/yum.repos.d] # 

        21:52:28 (PROD) root@do-pxe02-prod [/etc/yum.repos.d] # curl https://foreman.dom.tld/pulp/content/CORP/Foreman_Test/Foreman_Host/custom/Foreman_25/Katello_41_pulpcore/Packages/p/ |grep pygments
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21373  100 21373    0     0   474k                       <li><a href="python3-pygments-2.8.1-1.el8.noarch.rpm">python3-pygments-2.8.1-1.el8.noarch.rpm</a></li>
   0 --:--:-- --:--:-- --:--:--  474k
21:52:34 (PROD) root@do-pxe02-prod [/etc/yum.repos.d] # 

Foreman and Proxy versions:

foreman-debug-2.5.2-1.el8.noarch
foreman-installer-2.5.2-3.el8.noarch
foreman-installer-katello-2.5.2-3.el8.noarch
foreman-proxy-2.5.2-1.el8.noarch

Distribution and version:

CentOS 8.4

Do you have a caching proxy inbetween, which could have some cached metadata?

No, it’s a direct connection.

The sad thing is, I had this working with 2.4.2 (at least I could run a kickstart file) but I jumped the gun at upgrading to 2.5.2 before all the files were in the repo…

Is there a way to debug dnf or yum to see why it’s not “seeing” the file?
Or see what it’s actually seeing?

This looks like it uses a cached metadata from 6 days ago.

Could you run a “dnf clean all” and then try again?

I’ve done that multiple times.

However, the problem has now been found: I had disabled all the default CentOS Repos (because I have the content-view), after registering the system with foreman.
However, on closer inspection, the content view is missing the BaseOS repo.
:man_facepalming:

That immediately broke almost everything - however becomes only apparent when you try to install python modules and python itself.

When I added the repo to the content-view and tried to publish it, that didn’t work either.

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Mon, 26 Jul 2021 08:39:57 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"112", "correlation-id"=>"55309c12-28ee-4c7d-8852-97ec37d193b7", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman-server", "connection"=>"close"}
Response body: ["URI /pulp/api/v3/publications/rpm/rpm/4886a91e-ca16-4d35-a4c9-66f700682afa/ not found for repositoryversion."]

I assume, it’s irritated by the fact that the additional repo “appeared out of thin air”, so to speak.

This is clearly, again, a user-error, but maybe it just should not be possible to add a repo to an existing content-view if that doesn’t work either?

There seem to be some serious issues with pulpcore 3.14, which came with the update to 2.5.2/4.1.1.

Topic on your error is here: CentOS 8.4 BaseOS Sync error

So far it seems to be an issue only with the EL8 BaseOS repositories.

With pulpcore 3.14 there is also some metadata from the RPMs missing in the database which can cause dependency errors. See topic Cannot update puppet-agent to 6.24.0 on foreman servers

At the moment I don’t install updates from katello anymore but use CentOS mirrors to update, if necessary.

1 Like

Thanks. But what is the suggestion if one wants to build a proof-of-concept?
Use Foreman 2.4-series?

The problem isn’t really installing 2.4 again, but syncing all the content from the mirrors etc…

I don’t know about the schedule of the developers, when those issues are going to be fixed. At the moment, I would recommend to wait for 4.1.2 if the contains fixes for those issue. Or maybe 2.6/4.2 or even 4.3. I think some of the issues I have seen with pulp3 are going to be included with 4.3.

I don’t know about 2.4/4.0. After a lot of problems with the pulp migration I eventually upgraded only through 2.4/4.0 to 2.5/4.1 and didn’t run 2.4 for a longer time…

Hi @rainer_d ,

Not sure what the exact solution to your issue was. I’m running a freshly installed (after the pulp-rpm 3.14 release) Katello box and I see the exact same thing. However, I do have the BaseOS repos available in my CCV:

[root@smartproxy repodata]# yum repolist
Updating Subscription Management repositories.
repo id                                                               repo name
HTM_Foreman-Pulp-EL8_Foreman-Pulp-EL8-Base                            Foreman-Pulp-EL8-Base
HTM_Foreman-Puppet6-EL8_Foreman-Puppet6-EL8-Puppet6                   Foreman-Puppet6-EL8-Puppet6
HTM_Foreman-Server-EL8_Foreman-Server-EL8-Foreman-Base                Foreman-Server-EL8-Foreman-Base
HTM_Foreman-Server-EL8_Foreman-Server-EL8-Foreman-Plugins             Foreman-Server-EL8-Foreman-Plugins
HTM_Foreman-Server-EL8_Foreman-Server-EL8-Katello-Base                Foreman-Server-EL8-Katello-Base
HTM_Foreman-Server-EL8_Foreman-Server-EL8-Katello-Candlepin           Foreman-Server-EL8-Katello-Candlepin
HTM_Foreman-Tools-EL8_Foreman-Tools-EL8-Base                          Foreman-Tools-EL8-Base
HTM_Rocky8_Rocky8-AppStream                                           Rocky8-AppStream
HTM_Rocky8_Rocky8-BaseOS                                              Rocky8-BaseOS
HTM_Zabbix-EL8_Zabbix-EL8-Zabbix-5_4                                  Zabbix-EL8-Zabbix-5.4

Below is the debug output from dnf/yum when trying to install this package.

root@smartproxy repodata]# yum -d10 install python3-pygments-2.8.1-1.el8.noarch
timer: config: 2 ms
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
Updating Subscription Management repositories.
YUM version: 4.4.2
Command: yum -d10 install python3-pygments-2.8.1-1.el8.noarch 
Installroot: /
Releasever: 8
cachedir: /var/cache/dnf
Base command: install
Extra commands: ['-d10', 'install', 'python3-pygments-2.8.1-1.el8.noarch']
User-Agent: constructed: 'libdnf (Rocky Linux 8.4; generic; Linux.x86_64)'
Rocky8-AppStream                                                                      79 kB/s | 4.8 kB     00:00    
reviving: 'HTM_Rocky8_Rocky8-AppStream' can be revived - repomd matches.
HTM_Rocky8_Rocky8-AppStream: using metadata from Tue 17 Aug 2021 12:00:55 AM UTC.
Foreman-Server-EL8-Foreman-Base                                                       50 kB/s | 2.9 kB     00:00    
reviving: 'HTM_Foreman-Server-EL8_Foreman-Server-EL8-Foreman-Base' can be revived - repomd matches.
HTM_Foreman-Server-EL8_Foreman-Server-EL8-Foreman-Base: using metadata from Sat 17 Jul 2021 12:03:00 AM UTC.
Zabbix-EL8-Zabbix-5.4                                                                 52 kB/s | 2.9 kB     00:00    
reviving: 'HTM_Zabbix-EL8_Zabbix-EL8-Zabbix-5_4' can be revived - repomd matches.
HTM_Zabbix-EL8_Zabbix-EL8-Zabbix-5_4: using metadata from Wed 21 Jul 2021 08:12:19 AM UTC.
Foreman-Pulp-EL8-Base                                                                 54 kB/s | 2.9 kB     00:00    
reviving: 'HTM_Foreman-Pulp-EL8_Foreman-Pulp-EL8-Base' can be revived - repomd matches.
HTM_Foreman-Pulp-EL8_Foreman-Pulp-EL8-Base: using metadata from Tue 17 Aug 2021 09:36:08 PM UTC.
Foreman-Puppet6-EL8-Puppet6                                                           42 kB/s | 2.5 kB     00:00    
reviving: 'HTM_Foreman-Puppet6-EL8_Foreman-Puppet6-EL8-Puppet6' can be revived - repomd matches.
HTM_Foreman-Puppet6-EL8_Foreman-Puppet6-EL8-Puppet6: using metadata from Tue 17 Aug 2021 11:55:05 AM UTC.
Foreman-Server-EL8-Foreman-Plugins                                                    51 kB/s | 2.9 kB     00:00    
reviving: 'HTM_Foreman-Server-EL8_Foreman-Server-EL8-Foreman-Plugins' can be revived - repomd matches.
HTM_Foreman-Server-EL8_Foreman-Server-EL8-Foreman-Plugins: using metadata from Tue 17 Aug 2021 09:51:21 PM UTC.
Foreman-Server-EL8-Katello-Base                                                       44 kB/s | 2.9 kB     00:00    
reviving: 'HTM_Foreman-Server-EL8_Foreman-Server-EL8-Katello-Base' can be revived - repomd matches.
HTM_Foreman-Server-EL8_Foreman-Server-EL8-Katello-Base: using metadata from Wed 04 Aug 2021 08:50:04 PM UTC.
Foreman-Tools-EL8-Base                                                                63 kB/s | 3.6 kB     00:00    
reviving: 'HTM_Foreman-Tools-EL8_Foreman-Tools-EL8-Base' can be revived - repomd matches.
HTM_Foreman-Tools-EL8_Foreman-Tools-EL8-Base: using metadata from Tue 17 Aug 2021 09:24:12 PM UTC.
Foreman-Server-EL8-Katello-Candlepin                                                  48 kB/s | 2.9 kB     00:00    
reviving: 'HTM_Foreman-Server-EL8_Foreman-Server-EL8-Katello-Candlepin' can be revived - repomd matches.
HTM_Foreman-Server-EL8_Foreman-Server-EL8-Katello-Candlepin: using metadata from Wed 04 Aug 2021 08:50:28 PM UTC.
Rocky8-BaseOS                                                                         40 kB/s | 2.3 kB     00:00    
reviving: 'HTM_Rocky8_Rocky8-BaseOS' can be revived - repomd matches.
HTM_Rocky8_Rocky8-BaseOS: using metadata from Thu 19 Aug 2021 09:01:12 PM UTC.
timer: sack setup: 1135 ms
Completion plugin: Generating completion cache...
All matches were filtered out by modular filtering for argument: python3-pygments-2.8.1-1.el8.noarch
Cleaning up.

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 67, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 106, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 122, in cli_run
    cli.run()
  File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 1067, in run
    return self.command.run()
  File "/usr/lib/python3.6/site-packages/dnf/cli/commands/install.py", line 120, in run
    packages=err_pkgs)
dnf.exceptions.PackagesNotAvailableError: Unable to find a match: python3-pygments-2.8.1-1.el8.noarch
Error: Unable to find a match: python3-pygments-2.8.1-1.el8.noarch

Best part is, trying to install a different package from the repo works just fine.

For example, I manually installed python3-pulp-rpm and that works just (I didn’t put the output from yum here, as it’s quite lenghty)

Well, first check if it is actually in the published repository, e.g. pick up the full URL for the repository from /etc/yum.repos.d/redhat.repo, enter it into a browser, make sure the URL ends in a slash “/” and you should see the listing. The package should be there or in a subdirectory. If it’s there, check if you can download it in your browser.

Check! See below:

It is present on the server:

[root@smartproxy yum.repos.d]# curl https://foreman.lbhr.htm.lan/pulp/content/HTM/Beheer/COV_Rocky8-ForemanProxy/custom/Foreman-Pulp-EL8/Foreman-Pulp-EL8-Base/ | grep pygment
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 25477  100 25477    0     0   318k      0 --:--:-- --:--:-- --:--:--  323k
                    <li><a href="python3-pygments-2.8.1-1.el8.noarch.rpm">python3-pygments-2.8.1-1.el8.noarch.rpm</a></li>

I was also able to download that file:

[root@smartproxy yum.repos.d]# wget https://foreman.lbhr.htm.lan/pulp/content/HTM/Beheer/COV_Rocky8-ForemanProxy/custom/Foreman-Pulp-EL8/Foreman-Pulp-EL8-Base/python3-pygments-2.8.1-1.el8.noarch.rpm
--2021-08-23 20:10:36--  https://foreman.lbhr.htm.lan/pulp/content/HTM/Beheer/COV_Rocky8-ForemanProxy/custom/Foreman-Pulp-EL8/Foreman-Pulp-EL8-Base/python3-pygments-2.8.1-1.el8.noarch.rpm
Resolving foreman.lbhr.htm.lan (foreman.lbhr.htm.lan)... 192.168.255.15
Connecting to foreman.lbhr.htm.lan (foreman.lbhr.htm.lan)|192.168.255.15|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1698848 (1.6M) [application/x-rpm]
Saving to: ‘python3-pygments-2.8.1-1.el8.noarch.rpm’

python3-pygments-2.8.1-1.el8. 100%[==============================================>]   1.62M  --.-KB/s    in 0.05s   

2021-08-23 20:10:36 (30.8 MB/s) - ‘python3-pygments-2.8.1-1.el8.noarch.rpm’ saved [1698848/1698848]

And it’s an actual RPM:

[root@smartproxy yum.repos.d]# file python3-pygments-2.8.1-1.el8.noarch.rpm 
python3-pygments-2.8.1-1.el8.noarch.rpm: RPM v3.0 bin noarch python3-pygments-2.8.1-1.el8

But yum is not so eager to download and install it :frowning:

python3-pygments-2.8.1-1.el8.noarch.rpm: RPM v3.0 bin noarch python3-pygments-2.8.1-1.el8
[root@smartproxy yum.repos.d]# yum install python3-pygments-2.8.1-1.el8
Updating Subscription Management repositories.
Foreman-Puppet6-EL8-Puppet6                                                           46 kB/s | 2.5 kB     00:00    
Foreman-Server-EL8-Katello-Base                                                       49 kB/s | 2.9 kB     00:00    
Ansible-EL8-Ansible-2.9                                                               63 kB/s | 3.0 kB     00:00    
Foreman-Server-EL8-Foreman-Plugins                                                    48 kB/s | 2.9 kB     00:00    
Rocky8-AppStream                                                                      78 kB/s | 4.8 kB     00:00    
Foreman-Server-EL8-Katello-Candlepin                                                  55 kB/s | 2.9 kB     00:00    
Foreman-Server-EL8-Foreman-Base                                                       59 kB/s | 2.9 kB     00:00    
Rocky8-BaseOS                                                                         40 kB/s | 2.3 kB     00:00    
Foreman-Tools-EL8-Base                                                                60 kB/s | 3.6 kB     00:00    
Foreman-Pulp-EL8-Base                                                                 48 kB/s | 2.9 kB     00:00    
Zabbix-EL8-Zabbix-5.4                                                                 49 kB/s | 2.9 kB     00:00    
All matches were filtered out by modular filtering for argument: python3-pygments-2.8.1-1.el8
Error: Unable to find a match: python3-pygments-2.8.1-1.el8

@katello anyone have a clue? :slight_smile:

I don’t know how foreman is using modules on el8, but you can run

# dnf module provides python3-pygments

to see which modules provide python3-pygments.

# dnf provides python3-pygments

will show which version it considers the one available with your current module selection.

# dnf module list --enabled

shows the enabled modules.

You need to set module_hotfixes=1 in the .repo file on your client, like we do in our repo file that we ship:

Otherwise the version of pygments that we ship isn’t considered by dnf.

If the .repo file is managed by subscription-manager, you should (I never really tested this) be able to set that via subscription-manager repo-override --repo Foreman-Pulp-EL8-Base --add module_hotfixes:1.

HTH
Evgeni

1 Like

Thank you!

Is there a way to force this via the Foreman server (and configure it via Ansible :slight_smile: )

command: subscription-manager repo-override --repo Foreman-Pulp-EL8-Base --add module_hotfixes:1

/me runs away. No, I have no clue if the server can force it, but my bet is on “No”.

That also works fine, I’m installing these systems via Ansible anyway. I was just wondering if it was possible.

I’ll add that command task to my playbook :wink: