Package unavailable in 'on demand' repo (and looks like not available in Katello repo)

Support
1

Problem:

Attempting to install a package from my epel repo which is set to on demand. It appears the package is not found. Upstream (on epel), I see the package, but I don’t see it when I browse to my mirrored Katello repo over http. In turn, the yum install fails because the package is not on Katello. Other packages in my repo work and I’ve installed them fine. This one seems to be having issues.

Expected outcome:

No issues… package installs.

Foreman and Proxy versions:

Katello Version 1.20.1*

Yum install:

[root@dvt-nj04-sv092 ~]# yum install nagios-common
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager
Loading mirror speeds from cached hostfile
_Centos7_CentOS7-centosplus_x86_64                                                                                                                                                                                                                                                              | 2.9 kB  00:00:00
_Centos7_CentOS7-extras_x86_64                                                                                                                                                                                                                                                                  | 2.9 kB  00:00:00
_Centos7_CentOS7-os_x86_64                                                                                                                                                                                                                                                                      | 2.5 kB  00:00:00
_Centos7_CentOS7-updates_x86_64                                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00
_Epel7_Epel7                                                                                                                                                                                                                                                                                    | 3.0 kB  00:00:00
_t-repo_                                                                                                                                                                                                                                                                                | 2.5 kB  00:00:00
Resolving Dependencies
--> Running transaction check
---> Package nagios-common.x86_64 0:4.3.4-5.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================================================================================================================================================================
 Package                                                                       Arch                                                                   Version                                                                       Repository                                                                          Size
=============================================================================================================================================================================================================================================================================================================================
Installing:
 nagios-common                                                                 x86_64                                                                 4.3.4-5.el7                                                                   _Epel7_Epel7                                                                  24 k

Transaction Summary
=============================================================================================================================================================================================================================================================================================================================
Install  1 Package

Total download size: 24 k
Installed size: 0
Is this ok [y/d/N]: y
Downloading packages:
nagios-common-4.3.4-5.el7.x86_ FAILED
https://x-nx-vm026.x.local/pulp/repos/x/dvt/_CCV/custom/Epel7/Epel7/Packages/n/nagios-common-4.3.4-5.el7.x86_64.rpm: [Errno 14] HTTPS Error 404 - Not Found                                                                                                            ]  0.0 B/s |    0 B  --:--:-- ETA
Trying other mirror.
To address this issue please refer to the below wiki article

https://wiki.centos.org/yum-errors

If above article doesn't help to resolve this issue please use https://bugs.centos.org/.



Error downloading packages:
  nagios-common-4.3.4-5.el7.x86_64: [Errno 256] No more mirrors to try.

Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager
Loaded plugins: fastestmirror, product-id, subscription-manager
Loaded plugins: fastestmirror, product-id, subscription-manager
Loaded plugins: fastestmirror, product-id, subscription-manager
Loaded plugins: fastestmirror, product-id, subscription-manager
Loaded plugins: fastestmirror, product-id, subscription-manager
[root@dvt-nj04-sv092 ~]# yum install nagios-common
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager
Loading mirror speeds from cached hostfile
_Centos7_CentOS7-centosplus_x86_64                                                                                                                                                                                                                                                              | 2.9 kB  00:00:00
_Centos7_CentOS7-extras_x86_64                                                                                                                                                                                                                                                                  | 2.9 kB  00:00:00
_Centos7_CentOS7-os_x86_64                                                                                                                                                                                                                                                                      | 2.5 kB  00:00:00
_Centos7_CentOS7-updates_x86_64                                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00
_Epel7_Epel7                                                                                                                                                                                                                                                                                    | 3.0 kB  00:00:00
_t-repo_                                                                                                                                                                                                                                                                                 | 2.5 kB  00:00:00
Resolving Dependencies
--> Running transaction check
---> Package nagios-common.x86_64 0:4.3.4-5.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================================================================================================================================================================
 Package                                                                       Arch                                                                   Version                                                                       Repository                                                                          Size
=============================================================================================================================================================================================================================================================================================================================
Installing:
 nagios-common                                                                 x86_64                                                                 4.3.4-5.el7                                                                   _Epel7_Epel7                                                                  24 k

Transaction Summary
=============================================================================================================================================================================================================================================================================================================================
Install  1 Package

Total download size: 24 k
Installed size: 0
Is this ok [y/d/N]: y
Downloading packages:
nagios-common-4.3.4-5.el7.x86_ FAILED
https://x-nx-vm026.x.local/pulp/repos/x/dvt/_CCV/custom/Epel7/Epel7/Packages/n/nagios-common-4.3.4-5.el7.x86_64.rpm: [Errno 14] HTTPS Error 404 - Not Found                                                                                                            ]  0.0 B/s |    0 B  --:--:-- ETA
Trying other mirror.
To address this issue please refer to the below wiki article

https://wiki.centos.org/yum-errors

If above article doesn't help to resolve this issue please use https://bugs.centos.org/.



Error downloading packages:
  nagios-common-4.3.4-5.el7.x86_64: [Errno 256] No more mirrors to try.

Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager
Loaded plugins: fastestmirror, product-id, subscription-manager
Loaded plugins: fastestmirror, product-id, subscription-manager
Loaded plugins: fastestmirror, product-id, subscription-manager
Loaded plugins: fastestmirror, product-id, subscription-manager
Loaded plugins: fastestmirror, product-id, subscription-manager

Tailing the production.log during the install:

2019-02-11T12:48:05 [I|app|] Started GET "/rhsm/consumers/d3d39335-9f2c-4df1-bcce-a019670c89dd/certificates/serials" for 192.168.176.117 at 2019-02-11 12:48:05 -0500
2019-02-11T12:48:05 [I|app|4eef8] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2019-02-11T12:48:05 [I|app|4eef8]   Parameters: {"id"=>"d3d39335-9f2c-4df1-bcce-a019670c89dd"}
2019-02-11T12:48:05 [I|app|4eef8] Completed 200 OK in 264ms (Views: 0.8ms | ActiveRecord: 18.5ms)
2019-02-11T12:48:05 [I|app|] Started GET "/rhsm/status" for 192.168.176.117 at 2019-02-11 12:48:05 -0500
2019-02-11T12:48:05 [I|app|6fad4] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2019-02-11T12:48:06 [I|app|6fad4] Completed 200 OK in 60ms (Views: 1.0ms | ActiveRecord: 4.9ms)
2019-02-11T12:48:06 [I|app|] Started GET "/rhsm/" for 192.168.176.117 at 2019-02-11 12:48:06 -0500
2019-02-11T12:48:06 [I|app|3a92f] Processing by Katello::Api::V2::RootController#rhsm_resource_list as JSON
2019-02-11T12:48:06 [I|app|3a92f]   Parameters: {"root"=>{}}
2019-02-11T12:48:06 [I|app|3a92f]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection
2019-02-11T12:48:06 [I|app|3a92f]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection (8.2ms)
2019-02-11T12:48:06 [I|app|3a92f] Completed 200 OK in 55ms (Views: 17.5ms | ActiveRecord: 4.2ms)
2019-02-11T12:48:06 [I|app|] Started GET "/rhsm/consumers/d3d39335-9f2c-4df1-bcce-a019670c89dd/content_overrides" for 192.168.176.117 at 2019-02-11 12:48:06 -0500
2019-02-11T12:48:06 [I|app|e4350] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2019-02-11T12:48:06 [I|app|e4350]   Parameters: {"id"=>"d3d39335-9f2c-4df1-bcce-a019670c89dd"}
2019-02-11T12:48:06 [I|app|e4350] Current user set to d3d39335-9f2c-4df1-bcce-a019670c89dd (regular)
2019-02-11T12:48:06 [I|app|e4350] Completed 200 OK in 205ms (Views: 0.8ms | ActiveRecord: 11.2ms)
2019-02-11T12:48:11 [I|app|] Started GET "/notification_recipients" for 192.168.176.117 at 2019-02-11 12:48:11 -0500
2019-02-11T12:48:11 [I|app|382e4] Processing by NotificationRecipientsController#index as JSON
2019-02-11T12:48:11 [I|app|382e4] Current user set to admin (admin)
2019-02-11T12:48:11 [I|app|382e4] Completed 200 OK in 24ms (Views: 0.5ms | ActiveRecord: 3.7ms)
2019-02-11T12:48:13 [I|app|] Started GET "/rhsm/consumers/d3d39335-9f2c-4df1-bcce-a019670c89dd/certificates/serials" for 192.168.176.117 at 2019-02-11 12:48:13 -0500
2019-02-11T12:48:13 [I|app|e1cc5] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2019-02-11T12:48:13 [I|app|e1cc5]   Parameters: {"id"=>"d3d39335-9f2c-4df1-bcce-a019670c89dd"}
2019-02-11T12:48:13 [I|app|e1cc5] Completed 200 OK in 167ms (Views: 0.8ms | ActiveRecord: 10.2ms)
2019-02-11T12:48:13 [I|app|] Started GET "/rhsm/status" for 192.168.176.117 at 2019-02-11 12:48:13 -0500
2019-02-11T12:48:13 [I|app|4cb83] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2019-02-11T12:48:13 [I|app|4cb83] Completed 200 OK in 46ms (Views: 0.9ms | ActiveRecord: 3.4ms)
2019-02-11T12:48:13 [I|app|] Started GET "/rhsm/" for 192.168.176.117 at 2019-02-11 12:48:13 -0500
2019-02-11T12:48:13 [I|app|791cb] Processing by Katello::Api::V2::RootController#rhsm_resource_list as JSON
2019-02-11T12:48:13 [I|app|791cb]   Parameters: {"root"=>{}}
2019-02-11T12:48:13 [I|app|791cb]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection
2019-02-11T12:48:13 [I|app|791cb]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection (7.9ms)
2019-02-11T12:48:13 [I|app|791cb] Completed 200 OK in 47ms (Views: 10.2ms | ActiveRecord: 2.9ms)
2019-02-11T12:48:13 [I|app|] Started GET "/rhsm/consumers/d3d39335-9f2c-4df1-bcce-a019670c89dd/content_overrides" for 192.168.176.117 at 2019-02-11 12:48:13 -0500
2019-02-11T12:48:13 [I|app|3f082] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2019-02-11T12:48:13 [I|app|3f082]   Parameters: {"id"=>"d3d39335-9f2c-4df1-bcce-a019670c89dd"}
2019-02-11T12:48:13 [I|app|3f082] Current user set to d3d39335-9f2c-4df1-bcce-a019670c89dd (regular)
2019-02-11T12:48:13 [I|app|3f082] Completed 200 OK in 64ms (Views: 0.4ms | ActiveRecord: 5.9ms)
2019-02-11T12:48:14 [I|app|] Started GET "/rhsm/consumers/d3d39335-9f2c-4df1-bcce-a019670c89dd/certificates/serials" for 192.168.176.117 at 2019-02-11 12:48:14 -0500
2019-02-11T12:48:14 [I|app|334c8] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2019-02-11T12:48:14 [I|app|334c8]   Parameters: {"id"=>"d3d39335-9f2c-4df1-bcce-a019670c89dd"}
2019-02-11T12:48:14 [I|app|334c8] Completed 200 OK in 135ms (Views: 0.5ms | ActiveRecord: 11.1ms)
2019-02-11T12:48:14 [I|app|] Started GET "/rhsm/status" for 192.168.176.117 at 2019-02-11 12:48:14 -0500
2019-02-11T12:48:14 [I|app|1c696] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2019-02-11T12:48:14 [I|app|1c696] Completed 200 OK in 53ms (Views: 0.8ms | ActiveRecord: 4.5ms)
2019-02-11T12:48:14 [I|app|] Started GET "/rhsm/" for 192.168.176.117 at 2019-02-11 12:48:14 -0500
2019-02-11T12:48:14 [I|app|70222] Processing by Katello::Api::V2::RootController#rhsm_resource_list as JSON
2019-02-11T12:48:14 [I|app|70222]   Parameters: {"root"=>{}}
2019-02-11T12:48:14 [I|app|70222]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection
2019-02-11T12:48:14 [I|app|70222]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection (6.8ms)
2019-02-11T12:48:14 [I|app|70222] Completed 200 OK in 57ms (Views: 9.6ms | ActiveRecord: 3.8ms)
2019-02-11T12:48:14 [I|app|] Started GET "/rhsm/consumers/d3d39335-9f2c-4df1-bcce-a019670c89dd/content_overrides" for 192.168.176.117 at 2019-02-11 12:48:14 -0500
2019-02-11T12:48:14 [I|app|de2fa] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2019-02-11T12:48:14 [I|app|de2fa]   Parameters: {"id"=>"d3d39335-9f2c-4df1-bcce-a019670c89dd"}
2019-02-11T12:48:14 [I|app|de2fa] Current user set to d3d39335-9f2c-4df1-bcce-a019670c89dd (regular)
2019-02-11T12:48:14 [I|app|de2fa] Completed 200 OK in 76ms (Views: 0.4ms | ActiveRecord: 6.6ms)
2019-02-11T12:48:14 [I|app|] Started GET "/rhsm/consumers/d3d39335-9f2c-4df1-bcce-a019670c89dd/certificates/serials" for 192.168.176.117 at 2019-02-11 12:48:14 -0500
2019-02-11T12:48:14 [I|app|747a5] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2019-02-11T12:48:14 [I|app|747a5]   Parameters: {"id"=>"d3d39335-9f2c-4df1-bcce-a019670c89dd"}
2019-02-11T12:48:15 [I|app|747a5] Completed 200 OK in 136ms (Views: 0.5ms | ActiveRecord: 10.5ms)
2019-02-11T12:48:15 [I|app|] Started GET "/rhsm/status" for 192.168.176.117 at 2019-02-11 12:48:15 -0500
2019-02-11T12:48:15 [I|app|adffe] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2019-02-11T12:48:15 [I|app|adffe] Completed 200 OK in 48ms (Views: 1.1ms | ActiveRecord: 4.8ms)
2019-02-11T12:48:15 [I|app|] Started GET "/rhsm/" for 192.168.176.117 at 2019-02-11 12:48:15 -0500
2019-02-11T12:48:15 [I|app|3c20c] Processing by Katello::Api::V2::RootController#rhsm_resource_list as JSON
2019-02-11T12:48:15 [I|app|3c20c]   Parameters: {"root"=>{}}
2019-02-11T12:48:15 [I|app|3c20c]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection
2019-02-11T12:48:15 [I|app|3c20c]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection (6.4ms)
2019-02-11T12:48:15 [I|app|3c20c] Completed 200 OK in 47ms (Views: 10.4ms | ActiveRecord: 3.1ms)
2019-02-11T12:48:15 [I|app|] Started GET "/rhsm/consumers/d3d39335-9f2c-4df1-bcce-a019670c89dd/content_overrides" for 192.168.176.117 at 2019-02-11 12:48:15 -0500
2019-02-11T12:48:15 [I|app|74d6a] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2019-02-11T12:48:15 [I|app|74d6a]   Parameters: {"id"=>"d3d39335-9f2c-4df1-bcce-a019670c89dd"}
2019-02-11T12:48:15 [I|app|74d6a] Current user set to d3d39335-9f2c-4df1-bcce-a019670c89dd (regular)
2019-02-11T12:48:15 [I|app|74d6a] Completed 200 OK in 55ms (Views: 0.4ms | ActiveRecord: 4.7ms)
2019-02-11T12:48:15 [I|app|] Started GET "/rhsm/consumers/d3d39335-9f2c-4df1-bcce-a019670c89dd/certificates/serials" for 192.168.176.117 at 2019-02-11 12:48:15 -0500
2019-02-11T12:48:15 [I|app|6fe64] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2019-02-11T12:48:15 [I|app|6fe64]   Parameters: {"id"=>"d3d39335-9f2c-4df1-bcce-a019670c89dd"}
2019-02-11T12:48:15 [I|app|6fe64] Completed 200 OK in 157ms (Views: 1.2ms | ActiveRecord: 11.2ms)
2019-02-11T12:48:15 [I|app|] Started GET "/rhsm/status" for 192.168.176.117 at 2019-02-11 12:48:15 -0500
2019-02-11T12:48:15 [I|app|a5484] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2019-02-11T12:48:16 [I|app|a5484] Completed 200 OK in 62ms (Views: 1.6ms | ActiveRecord: 4.8ms)
2019-02-11T12:48:16 [I|app|] Started GET "/rhsm/" for 192.168.176.117 at 2019-02-11 12:48:16 -0500
2019-02-11T12:48:16 [I|app|2556b] Processing by Katello::Api::V2::RootController#rhsm_resource_list as JSON
2019-02-11T12:48:16 [I|app|2556b]   Parameters: {"root"=>{}}
2019-02-11T12:48:16 [I|app|2556b]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection
2019-02-11T12:48:16 [I|app|2556b]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection (7.3ms)
2019-02-11T12:48:16 [I|app|2556b] Completed 200 OK in 57ms (Views: 12.3ms | ActiveRecord: 4.7ms)
2019-02-11T12:48:16 [I|app|] Started GET "/rhsm/consumers/d3d39335-9f2c-4df1-bcce-a019670c89dd/content_overrides" for 192.168.176.117 at 2019-02-11 12:48:16 -0500
2019-02-11T12:48:16 [I|app|686fa] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2019-02-11T12:48:16 [I|app|686fa]   Parameters: {"id"=>"d3d39335-9f2c-4df1-bcce-a019670c89dd"}
2019-02-11T12:48:16 [I|app|686fa] Current user set to d3d39335-9f2c-4df1-bcce-a019670c89dd (regular)
2019-02-11T12:48:16 [I|app|686fa] Completed 200 OK in 75ms (Views: 0.5ms | ActiveRecord: 7.2ms)
2019-02-11T12:48:16 [I|app|] Started GET "/rhsm/consumers/d3d39335-9f2c-4df1-bcce-a019670c89dd/certificates/serials" for 192.168.176.117 at 2019-02-11 12:48:16 -0500
2019-02-11T12:48:16 [I|app|72617] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2019-02-11T12:48:16 [I|app|72617]   Parameters: {"id"=>"d3d39335-9f2c-4df1-bcce-a019670c89dd"}
2019-02-11T12:48:16 [I|app|72617] Completed 200 OK in 150ms (Views: 0.5ms | ActiveRecord: 10.4ms)
2019-02-11T12:48:16 [I|app|] Started GET "/rhsm/status" for 192.168.176.117 at 2019-02-11 12:48:16 -0500
2019-02-11T12:48:16 [I|app|adfd9] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2019-02-11T12:48:16 [I|app|adfd9] Completed 200 OK in 57ms (Views: 0.9ms | ActiveRecord: 4.9ms)
2019-02-11T12:48:17 [I|app|] Started GET "/rhsm/" for 192.168.176.117 at 2019-02-11 12:48:17 -0500
2019-02-11T12:48:17 [I|app|1effa] Processing by Katello::Api::V2::RootController#rhsm_resource_list as JSON
2019-02-11T12:48:17 [I|app|1effa]   Parameters: {"root"=>{}}
2019-02-11T12:48:17 [I|app|1effa]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection
2019-02-11T12:48:17 [I|app|1effa]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection (7.7ms)
2019-02-11T12:48:17 [I|app|1effa] Completed 200 OK in 59ms (Views: 10.0ms | ActiveRecord: 3.6ms)
2019-02-11T12:48:17 [I|app|] Started GET "/rhsm/consumers/d3d39335-9f2c-4df1-bcce-a019670c89dd/content_overrides" for 192.168.176.117 at 2019-02-11 12:48:17 -0500
2019-02-11T12:48:17 [I|app|75542] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2019-02-11T12:48:17 [I|app|75542]   Parameters: {"id"=>"d3d39335-9f2c-4df1-bcce-a019670c89dd"}
2019-02-11T12:48:17 [I|app|75542] Current user set to d3d39335-9f2c-4df1-bcce-a019670c89dd (regular)
2019-02-11T12:48:17 [I|app|75542] Completed 200 OK in 70ms (Views: 0.4ms | ActiveRecord: 6.1ms)
2019-02-11T12:48:17 [I|app|] Started GET "/rhsm/consumers/d3d39335-9f2c-4df1-bcce-a019670c89dd/certificates/serials" for 192.168.176.117 at 2019-02-11 12:48:17 -0500
2019-02-11T12:48:17 [I|app|427a0] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2019-02-11T12:48:17 [I|app|427a0]   Parameters: {"id"=>"d3d39335-9f2c-4df1-bcce-a019670c89dd"}
2019-02-11T12:48:17 [I|app|427a0] Completed 200 OK in 128ms (Views: 0.6ms | ActiveRecord: 10.1ms)
2019-02-11T12:48:17 [I|app|] Started GET "/rhsm/status" for 192.168.176.117 at 2019-02-11 12:48:17 -0500
2019-02-11T12:48:17 [I|app|50944] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2019-02-11T12:48:17 [I|app|50944] Completed 200 OK in 44ms (Views: 0.9ms | ActiveRecord: 4.0ms)
2019-02-11T12:48:17 [I|app|] Started GET "/rhsm/" for 192.168.176.117 at 2019-02-11 12:48:17 -0500
2019-02-11T12:48:17 [I|app|5dbdb] Processing by Katello::Api::V2::RootController#rhsm_resource_list as JSON
2019-02-11T12:48:17 [I|app|5dbdb]   Parameters: {"root"=>{}}
2019-02-11T12:48:17 [I|app|5dbdb]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection
2019-02-11T12:48:17 [I|app|5dbdb]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/views/katello/api/v2/root/resource_list.json.rabl within katello/api/v2/layouts/collection (5.2ms)
2019-02-11T12:48:17 [I|app|5dbdb] Completed 200 OK in 44ms (Views: 7.6ms | ActiveRecord: 3.9ms)
2019-02-11T12:48:17 [I|app|] Started GET "/rhsm/consumers/d3d39335-9f2c-4df1-bcce-a019670c89dd/content_overrides" for 192.168.176.117 at 2019-02-11 12:48:17 -0500
2019-02-11T12:48:17 [I|app|0e264] Processing by Katello::Api::Rhsm::CandlepinProxiesController#get as JSON
2019-02-11T12:48:17 [I|app|0e264]   Parameters: {"id"=>"d3d39335-9f2c-4df1-bcce-a019670c89dd"}
2019-02-11T12:48:17 [I|app|0e264] Current user set to d3d39335-9f2c-4df1-bcce-a019670c89dd (regular)
2019-02-11T12:48:18 [I|app|0e264] Completed 200 OK in 62ms (Views: 0.4ms | ActiveRecord: 6.4ms)
2019-02-11T12:48:21 [I|app|] Started GET "/notification_recipients" for 192.168.176.117 at 2019-02-11 12:48:21 -0500
2019-02-11T12:48:21 [I|app|08ad8] Processing by NotificationRecipientsController#index as JSON
2019-02-11T12:48:21 [I|app|08ad8] Current user set to admin (admin)
2019-02-11T12:48:21 [I|app|08ad8] Completed 200 OK in 22ms (Views: 0.5ms | ActiveRecord: 3.2ms)
2019-02-11T12:48:32 [I|app|] Started GET "/notification_recipients" for 192.168.176.117 at 2019-02-11 12:48:32 -0500

Package present on the gui:

package
package2307×485 16.5 KB

2

Just to add a bit more to this as I was troubleshooting. I selected another package at random that was not used previously. It installed the package and minutes later I was able to see it on the mirrored repo.

3

@and Can you tail /var/log/messages on the Katello server and yum install the package on the client?

4

Absolutely.

Here you go:

Feb 11 16:46:03 avt-nj01-vm026 pulp_streamer: pulp.streamer.server:ERROR: No catalog entry found. path=/var/lib/pulp/content/units/rpm/ca/5662ab8fe4b2ae12b3ef90f134334274078ee55aee59c505a134fd22ba1f80/nagios-common-4.3.4-5.el7.x86_64.rpm
Feb 11 16:46:03 avt-nj01-vm026 pulp_streamer: [-] 127.0.0.1 - - [11/Feb/2019:21:46:03 +0000] "GET /var/lib/pulp/content/units/rpm/ca/5662ab8fe4b2ae12b3ef90f134334274078ee55aee59c505a134fd22ba1f80/nagios-common-4.3.4-5.el7.x86_64.rpm HTTP/1.1" 404 - "-" "urlgrabber/3.10 yum/3.4.3"
Feb 11 16:46:05 avt-nj01-vm026 pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Feb 11 16:46:05 avt-nj01-vm026 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[33093eda-9939-4bf0-823e-81e2fa0b0e31]
Feb 11 16:46:05 avt-nj01-vm026 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[60acbfb4-cba7-46d2-836f-961201bf2dfe]
Feb 11 16:46:05 avt-nj01-vm026 pulp: celery.app.trace:INFO: [33093eda] Task pulp.server.controllers.repository.queue_download_deferred[33093eda-9939-4bf0-823e-81e2fa0b0e31] succeeded in 0.0246692039073s: None
Feb 11 16:46:05 avt-nj01-vm026 pulp: celery.app.trace:INFO: [60acbfb4] Task pulp.server.controllers.repository.download_deferred[60acbfb4-cba7-46d2-836f-961201bf2dfe] succeeded in 0.0315354168415s: None
5

Thanks for the quick response! Can you do a ‘Complete Sync’ on that repo? It is an option for a sync in the UI. Let me know if you can’t find it and I’ll point you in the right direction

6

Running now. Will report when it’s done - might take a bit!

1 Like
7

Hi John,

The sync is complete and afterward I immediately jumped on my client and the error still looks to be the same. I only ran a yum install nagios-common.

Heading out for the day shortly but will follow up in the morning with any additional tshooting tasks.

8

Can you confirm that its the same error in /var/log/messages?

9

Hi John, looks to be the same:

Feb 11 17:36:05 avt-nj01-vm026 pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Feb 11 17:36:05 avt-nj01-vm026 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[3d8bea11-02a0-4246-97fa-0db1a7a67c58]
Feb 11 17:36:05 avt-nj01-vm026 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[499d9c15-5f07-4d89-9b98-d8bd30f20b6c]
Feb 11 17:36:05 avt-nj01-vm026 pulp: celery.app.trace:INFO: [3d8bea11] Task pulp.server.controllers.repository.queue_download_deferred[3d8bea11-02a0-4246-97fa-0db1a7a67c58] succeeded in 0.0275729468558s: None
Feb 11 17:36:05 avt-nj01-vm026 pulp: celery.app.trace:INFO: [499d9c15] Task pulp.server.controllers.repository.download_deferred[499d9c15-5f07-4d89-9b98-d8bd30f20b6c] succeeded in 0.0239923889749s: None
Feb 11 17:36:47 avt-nj01-vm026 pulp_streamer: pulp.streamer.server:ERROR: No catalog entry found. path=/var/lib/pulp/content/units/rpm/ca/5662ab8fe4b2ae12b3ef90f134334274078ee55aee59c505a134fd22ba1f80/nagios-common-4.3.4-5.el7.x86_64.rpm
Feb 11 17:36:47 avt-nj01-vm026 pulp_streamer: [-] 127.0.0.1 - - [11/Feb/2019:22:36:46 +0000] "GET /var/lib/pulp/content/units/rpm/ca/5662ab8fe4b2ae12b3ef90f134334274078ee55aee59c505a134fd22ba1f80/nagios-common-4.3.4-5.el7.x86_64.rpm HTTP/1.1" 404 - "-" "urlgrabber/3.10 yum/3.4.3"
10

Thanks, let me ask the pulp team for help

1 Like
11

Thanks. I will be at the ready to resolve this! Last little hiccup before I roll this into production for us.

Thanks for your help.

12

@and Can you do these two things?

  • Confirm you have mirror_on_sync enabled for that repo (its the default)
  • run tail -f -n 0 /var/log/squid/*.log on the katello server when yum installing again and post the output
13

hi John,

Here is output:

tail -f -n 0 /var/log/squid/*.log
==> /var/log/squid/access.log <==

==> /var/log/squid/cache.log <==

==> /var/log/squid/store.log <==

==> /var/log/squid/access.log <==
1549988847.694      3 127.0.0.1 TCP_MISS/404 294 GET http://127.0.0.1:3128/var/lib/pulp/content/units/rpm/ca/5662ab8fe4b2ae12b3ef90f134334274078ee55aee59c505a134fd22ba1f80/nagios-common-4.3.4-5.el7.x86_64.rpm - FIRSTUP_PARENT/127.0.0.1 text/html

==> /var/log/squid/store.log <==
1549988847.694 RELEASE -1 FFFFFFFF F35A1F1854C72BFDB5FF27F8137D3CFF   ?         ?         ?         ? ?/? ?/? ? ?

Mirror on sync is yes and download policy is on demand.

14

I may run immediate download policy when I roll this out. Would this potentially alleviate the issue? I’d rather not flip to that now in case we decide to leave on demand as the standard.

15

thanks, and yes that would be my suggestion (switching to immediate for that repo). you can also do this with validate content sync, this will download the rpms even if the repo stays on demand.

I’ll see if anyone can make sense of that squid message

16

Hi John- thanks! Will be on standby in case anything comes to light :slight_smile:.