Can't access packages available in CentOS 7 base

fred_demarcy · August 12, 2022, 6:08pm

Hopefully that makes more sense.

# yum repolist -v
Loading "enabled_repos_upload" plugin
Loading "fastestmirror" plugin
Loading "package_upload" plugin
Loading "product-id" plugin
Loading "search-disabled-repos" plugin
Loading "subscription-manager" plugin
Loading "tracer_upload" plugin
Updating Subscription Management repositories.
Config time: 0.743
Yum version: 3.4.3
Loading mirror speeds from cached hostfile
Setting up Package Sacks
pkgsack time: 0.026
Repo-id      : XXX_DevOps_c7-codeit_c7-codeit-repo
Repo-name    : c7-codeit-repo
Repo-revision: 1659624923
Repo-updated : Thu Aug  4 14:55:23 2022
Repo-pkgs    : 435
Repo-size    : 297 M
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-codeit/c7-codeit-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-docker-ce_c7-docker-ce-repo
Repo-name    : c7-docker-ce-repo
Repo-revision: 1660020017
Repo-updated : Tue Aug  9 04:40:16 2022
Repo-pkgs    : 164
Repo-size    : 3.7 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-docker-ce/c7-docker-ce-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-elasticsearch_c7-elasticsearch-curator-repo
Repo-name    : c7-elasticsearch-curator-repo
Repo-revision: 1659624700
Repo-updated : Thu Aug  4 14:51:40 2022
Repo-pkgs    : 27
Repo-size    : 391 M
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-elasticsearch/c7-elasticsearch-curator-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-elasticsearch_c7-elasticsearch-repo
Repo-name    : c7-elasticsearch-repo
Repo-revision: 1659625369
Repo-updated : Thu Aug  4 15:02:43 2022
Repo-pkgs    : 1,010
Repo-size    : 79 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-elasticsearch/c7-elasticsearch-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-excluded: 366
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-epel_c7-epel-repo
Repo-name    : c7-epel-repo
Repo-revision: 1660278501
Repo-updated : Fri Aug 12 04:28:18 2022
Repo-pkgs    : 16,586
Repo-size    : 27 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-epel/c7-epel-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-foreman-katello-client-3_3_c7-foreman-katello-client-3_3-repo
Repo-name    : c7-foreman-katello-client-3_3-repo
Repo-revision: 1659624660
Repo-updated : Thu Aug  4 14:51:00 2022
Repo-pkgs    : 14
Repo-size    : 14 M
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-foreman-katello-client-3_3/c7-foreman-katello-client-3_3-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-google-chrome_c7-google-chrome-repo
Repo-name    : c7-google-chrome-repo
Repo-revision: 1660198222
Repo-updated : Thu Aug 11 06:10:22 2022
Repo-pkgs    : 3
Repo-size    : 264 M
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-google-chrome/c7-google-chrome-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-grafana_c7-grafana-repo
Repo-name    : c7-grafana-repo
Repo-revision: 1660194963
Repo-updated : Thu Aug 11 05:16:02 2022
Repo-pkgs    : 207
Repo-size    : 11 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-grafana/c7-grafana-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-excluded: 338
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-hashicorp_c7-hashicorp-repo
Repo-name    : c7-hashicorp-repo
Repo-revision: 1660281668
Repo-updated : Fri Aug 12 05:21:08 2022
Repo-pkgs    : 830
Repo-size    : 29 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-hashicorp/c7-hashicorp-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-jenkins-lts_c7-jenkins-lts-repo
Repo-name    : c7-jenkins-lts-repo
Repo-revision: 1660192541
Repo-updated : Thu Aug 11 04:35:41 2022
Repo-pkgs    : 136
Repo-size    : 8.4 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-jenkins-lts/c7-jenkins-lts-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-mariadb_c7-mariadb-10_5-repo
Repo-name    : c7-mariadb-10_5-repo
Repo-revision: 1659624902
Repo-updated : Thu Aug  4 14:55:02 2022
Repo-pkgs    : 96
Repo-size    : 1.7 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-mariadb/c7-mariadb-10_5-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-excluded: 3
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-mongodb-5_0_c7-mongodb-5_0-repo
Repo-name    : c7-mongodb-5_0-repo
Repo-revision: 1659625408
Repo-updated : Thu Aug  4 15:03:28 2022
Repo-pkgs    : 155
Repo-size    : 2.9 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-mongodb-5_0/c7-mongodb-5_0-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-nodesource_c7-nodesource-14-repo
Repo-name    : c7-nodesource-14-repo
Repo-revision: 1659624616
Repo-updated : Thu Aug  4 14:50:16 2022
Repo-pkgs    : 121
Repo-size    : 1.5 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-nodesource/c7-nodesource-14-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-qpid-copr_c7-qpid-copr-repo
Repo-name    : c7-qpid-copr-repo
Repo-revision: 1659624906
Repo-updated : Thu Aug  4 14:55:05 2022
Repo-pkgs    : 14
Repo-size    : 5.7 M
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-qpid-copr/c7-qpid-copr-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-excluded: 2
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-rabbitmq-erlang_c7-rabbitmq-erlang-repo
Repo-name    : c7-rabbitmq-erlang-repo
Repo-revision: 1659624872
Repo-updated : Thu Aug  4 14:54:32 2022
Repo-pkgs    : 47
Repo-size    : 871 M
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-rabbitmq-erlang/c7-rabbitmq-erlang-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-rabbitmq-server_c7-rabbitmq-server-repo
Repo-name    : c7-rabbitmq-server-repo
Repo-revision: 1659625385
Repo-updated : Thu Aug  4 15:03:05 2022
Repo-pkgs    : 98
Repo-size    : 1.1 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-rabbitmq-server/c7-rabbitmq-server-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-remi_c7-remi-repo
Repo-name    : c7-remi-repo
Repo-revision: 1660278708
Repo-updated : Fri Aug 12 04:31:47 2022
Repo-pkgs    : 7,631
Repo-size    : 1.5 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-remi/c7-remi-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-salt-py3_c7-salt-py3-repo
Repo-name    : c7-salt-py3-repo
Repo-revision: 1659625373
Repo-updated : Thu Aug  4 15:02:53 2022
Repo-pkgs    : 79
Repo-size    : 42 M
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-salt-py3/c7-salt-py3-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-excluded: 19
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-scl_c7-scl-repo
Repo-name    : c7-scl-repo
Repo-revision: 1659625416
Repo-updated : Thu Aug  4 15:03:33 2022
Repo-pkgs    : 8,024
Repo-size    : 8.5 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-scl/c7-scl-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-tickstack_c7-tickstack-repo
Repo-name    : c7-tickstack-repo
Repo-revision: 1659624898
Repo-updated : Thu Aug  4 14:54:58 2022
Repo-pkgs    : 270
Repo-size    : 6.2 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-tickstack/c7-tickstack-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:36 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-wandisco_c7-wandisco-repo
Repo-name    : c7-wandisco-repo
Repo-revision: 1659625082
Repo-updated : Thu Aug  4 14:58:02 2022
Repo-pkgs    : 390
Repo-size    : 1.0 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-wandisco/c7-wandisco-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:37 2022)
  Filter     : read-only:present
Repo-excluded: 33
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-yarn-v1_c7-yarn-v1-repo
Repo-name    : c7-yarn-v1-repo
Repo-revision: 1659625378
Repo-updated : Thu Aug  4 15:02:58 2022
Repo-pkgs    : 56
Repo-size    : 89 M
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-yarn-v1/c7-yarn-v1-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:37 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7-zabbix-lts_c7-zabbix-4_0-lts-repo
Repo-name    : c7-zabbix-4_0-lts-repo
Repo-revision: 1659624691
Repo-updated : Thu Aug  4 14:51:30 2022
Repo-pkgs    : 614
Repo-size    : 525 M
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-zabbix-lts/c7-zabbix-4_0-lts-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:37 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

Repo-id      : XXX_DevOps_c7_c7-updates-repo
Repo-name    : c7-updates-repo
Repo-revision: 1660298104
Repo-updated : Fri Aug 12 09:55:02 2022
Repo-pkgs    : 4,101
Repo-size    : 27 G
Repo-baseurl : https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7/c7-updates-repo/
Repo-expire  : 1 second(s) (last: Fri Aug 12 16:02:37 2022)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/redhat.repo

repolist: 41,108
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager

# subscription-manager identity
system identity: 9dd2f85a-0d2d-42ef-baaa-bbf08eba1a18
name: dev01.devops.xxx
org name: XXX DevOps
org ID: XXX_DevOps
environment name: devops/c7-devops-view

# subscription-manager repos
+----------------------------------------------------------+
    Available Repositories in /etc/yum.repos.d/redhat.repo
+----------------------------------------------------------+
Repo ID:   XXX_DevOps_c7-google-chrome_c7-google-chrome-repo
Repo Name: c7-google-chrome-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-google-chrome/c7-google-chrome-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-codeit_c7-codeit-repo
Repo Name: c7-codeit-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-codeit/c7-codeit-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-rabbitmq-server_c7-rabbitmq-server-repo
Repo Name: c7-rabbitmq-server-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-rabbitmq-server/c7-rabbitmq-server-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-scl_c7-scl-repo
Repo Name: c7-scl-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-scl/c7-scl-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-zabbix-lts_c7-zabbix-4_0-lts-repo
Repo Name: c7-zabbix-4_0-lts-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-zabbix-lts/c7-zabbix-4_0-lts-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-hashicorp_c7-hashicorp-repo
Repo Name: c7-hashicorp-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-hashicorp/c7-hashicorp-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-mongodb-5_0_c7-mongodb-5_0-repo
Repo Name: c7-mongodb-5_0-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-mongodb-5_0/c7-mongodb-5_0-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-yarn-v1_c7-yarn-v1-repo
Repo Name: c7-yarn-v1-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-yarn-v1/c7-yarn-v1-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-remi_c7-remi-repo
Repo Name: c7-remi-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-remi/c7-remi-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-nodesource_c7-nodesource-14-repo
Repo Name: c7-nodesource-14-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-nodesource/c7-nodesource-14-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-foreman-katello-client-3_3_c7-foreman-katello-client-3_3-repo
Repo Name: c7-foreman-katello-client-3_3-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-foreman-katello-client-3_3/c7-foreman-katello-client-3_3-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-salt-py3_c7-salt-py3-repo
Repo Name: c7-salt-py3-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-salt-py3/c7-salt-py3-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-jenkins-lts_c7-jenkins-lts-repo
Repo Name: c7-jenkins-lts-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-jenkins-lts/c7-jenkins-lts-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-mariadb_c7-mariadb-10_5-repo
Repo Name: c7-mariadb-10_5-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-mariadb/c7-mariadb-10_5-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-elasticsearch_c7-elasticsearch-repo
Repo Name: c7-elasticsearch-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-elasticsearch/c7-elasticsearch-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-epel_c7-epel-repo
Repo Name: c7-epel-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-epel/c7-epel-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-docker-ce_c7-docker-ce-repo
Repo Name: c7-docker-ce-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-docker-ce/c7-docker-ce-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-wandisco_c7-wandisco-repo
Repo Name: c7-wandisco-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-wandisco/c7-wandisco-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-elasticsearch_c7-elasticsearch-curator-repo
Repo Name: c7-elasticsearch-curator-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-elasticsearch/c7-elasticsearch-curator-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7_c7-updates-repo
Repo Name: c7-updates-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7/c7-updates-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-grafana_c7-grafana-repo
Repo Name: c7-grafana-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-grafana/c7-grafana-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-rabbitmq-erlang_c7-rabbitmq-erlang-repo
Repo Name: c7-rabbitmq-erlang-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-rabbitmq-erlang/c7-rabbitmq-erlang-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-qpid-copr_c7-qpid-copr-repo
Repo Name: c7-qpid-copr-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-qpid-copr/c7-qpid-copr-repo
Enabled:   1

Repo ID:   XXX_DevOps_c7-tickstack_c7-tickstack-repo
Repo Name: c7-tickstack-repo
Repo URL:  https://foremankatello01.devops.xxx/pulp/content/XXX_DevOps/devops/c7-devops-view/custom/c7-tickstack/c7-tickstack-repo
Enabled:   1

For now I’m looking at a VM which takes its content from the main Foreman/Katello server which is also in itself a content proxy.
Let’s put aside the 4x Smart Proxies in other locations.
To keep it simpler (if it can be…)

gvde · August 12, 2022, 6:35pm

O.K. That looks good. The client is connected to foremankatello01.devops on lifecycle environment devops and content view c7-devops-view.

I haven’t checked the complete list but you wrote before that c7-extras-repo and c7-base-repo are missing.

You can check the pulp content if the repositories are there at all. Open https://foremankatello01.devops.xxx/pulp/content/ and look for

XXX_DevOps/devops/c7-devops-view/custom/c7/c7-base-repo/

and

XXX_DevOps/devops/c7-devops-view/custom/c7/c7-extras-repo/

If they are both in the latest published and promoted version of the content view they should be there.

Also verify they are in the latest version through the ui, i.e. go to https://foremankatello01.devops.xxx/content_views. Click on the content view c7-devops-view to see the versions of the content view as well as the assignments of versions to environments. Click on the version number of the version which is in environment devops and click on Repositories to see a list of repositories which are included in the content view version.

This should be the same list of repositories as you have seen before through pulp, i.e. all listed repositories starting with XXX_DevOps/devops/c7-devops-view/custom/.

So check the list from the pulp/content for the missing repos as well as the content view version’s repositories if they both in include base and extras repos.

According to repolist and subscription-manager repos they should be missing from both.

fred_demarcy · August 13, 2022, 10:30am

Thanks @gvde for taking the time to help. Much appreciated.

https://foremankatello01.devops.xxx/pulp/content/

XXX_DevOps/devops/c7-devops-view/custom/c7-codeit/c7-codeit-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-docker-ce/c7-docker-ce-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-elasticsearch/c7-elasticsearch-curator-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-elasticsearch/c7-elasticsearch-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-epel/c7-epel-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-foreman-katello-client-3_3/c7-foreman-katello-client-3_3-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-google-chrome/c7-google-chrome-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-grafana/c7-grafana-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-hashicorp/c7-hashicorp-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-jenkins-lts/c7-jenkins-lts-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-mariadb/c7-mariadb-10_5-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-mongodb-5_0/c7-mongodb-5_0-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-nodesource/c7-nodesource-14-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-qpid-copr/c7-qpid-copr-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-rabbitmq-erlang/c7-rabbitmq-erlang-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-rabbitmq-server/c7-rabbitmq-server-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-remi/c7-remi-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-salt-py3/c7-salt-py3-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-scl/c7-scl-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-tickstack/c7-tickstack-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-wandisco/c7-wandisco-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-yarn-v1/c7-yarn-v1-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7-zabbix-lts/c7-zabbix-4_0-lts-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7/c7-base-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7/c7-extras-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7/c7-updates-repo/
...
and many more for other content views

For c7-devops-view
https://iomforemankatello01.devops.csr.pstars/content_views/2#/repositories
26 repos added incl. c7-base-repo, c7-extras-repo and c7-updates-repo

c7-devops-view
Version 42.0 with Environmnents: Library, devops

c2-base-repo & c7-extras-repo are not missing from anywhere…
The checks you describe I’ve done them many times over…
Yet yum repolist & subscription-manager repos are missing them…

My installation of Foreman/Katello + Smart Proxies was done in April 2021 and has gone through multiple upgrades (I tend to keep up to date with latest). Never seen this problem before - It’s baffling…

The only think I’ve done recently (post upgrade to Foreman 3.3.0 / Katello 4.5.0) for the CentOS 7 product and the 3 repos in it is to change the Upstream URL.

I’m now using

https://www.mirrorservice.org/sites/mirror.centos.org/

for all 3 repos.
Before I was using:

https://mirror.bytemark.co.uk/centos/

Could that have disrupted something??

gvde · August 13, 2022, 11:59am

O.K. That looks all correct. All I can suggest is to check the URLs on the server

XXX_DevOps/devops/c7-devops-view/custom/c7/c7-base-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7/c7-updates-repo/

to see if the repository is O.K. You could as a test put those URLs into a test repo file to see if the repository metadata etc. is good and working.

Otherwise, everything is as it should be and the repositories should show up on the client. Could be a bug then but that would be something for the devs. I wouldn’t even be sure if that could be a katello, pulp or candlepin issue here…

fred_demarcy · August 15, 2022, 9:50am

Creating repo files for

XXX_DevOps/devops/c7-devops-view/custom/c7/c7-base-repo/
XXX_DevOps/devops/c7-devops-view/custom/c7/c7-updates-repo/

works so why they don’t end up in redhat.repo as they should is still a mystery…

gvde · August 15, 2022, 9:55am

That could be a bug with candlepin. Maybe the @katello developers can help…

fred_demarcy · August 15, 2022, 10:06am

Checking the entitlement for my product c7:

# rct cat-cert /etc/pki/entitlement/1223977598242046956.pem | grep Label
	Label: XXX_DevOps_c7_c7-updates-repo

As per this post The /etc/yum.repos.d/redhat.repo is missed on CentOS 7 after registering on Foreman server - #3 by rakis I was expecting to see

Label: XXX_DevOps_c7_c7-base-repo
Label: XXX_DevOps_c7_c7-extras-repo

in there as well.

fred_demarcy · August 15, 2022, 10:14am

I also have reverted to sync’ing from

https://mirror.bytemark.co.uk/centos/

And changed the Mirroring Policy to “Complete Mirroring” from “Content Only” for all 3x CentOS 7 repos but same issue.

Also tried

subscription-manager refresh --force

No change…

fred_demarcy · August 15, 2022, 10:27am

Manually adding 2x sections for c7-base-repo and c7-extras-repo in the redhat.repo fixes the problem but I don’t like it…

iballou · August 15, 2022, 3:34pm

Hi @fred_demarcy,

This community thread may be of help to you, particularly this comment: Most repos from content view no longer appear on hosts - #7 by wbclark

Let us know if you have any questions about what’s there.

rakis · August 15, 2022, 3:55pm

I ended up with similar solution - all repos in Foreman looks OK, but redhat.repo is empty regardless of the settings (Complete Mirroring/Content Only).
I have created the Ansible role and manage all repos manually.

fred_demarcy · August 15, 2022, 4:03pm

Hi @iballou

# foreman-rake console
'ErbParser' is ignored.
'RubyParser' is ignored.
Loading production environment (Rails 6.0.4.7)
irb(main):001:0> product = Katello::Product.find_by_name('c7')
=> #<Katello::Product id: 9, name: "c7", description: "CentOS 7 product", cp_id: "577537497763", multiplier: nil, provider_id: 1, created_at: "2021-05-05 15:55:34", updated_at: "2021-05-05 15:55:34", gpg_key_id: 7, sync_plan_...

Ok so Product ID = 9

irb(main):002:0> root = Katello::RootRepository.find(9)
Traceback (most recent call last):
        2: from lib/tasks/console.rake:5:in `block in <top (required)>'
        1: from (irb):2
ActiveRecord::RecordNotFound (Couldn't find Katello::RootRepository with 'id'=9)

gvde · August 15, 2022, 4:40pm

takes the id of a repository not a product. You can easily take it from the UI URL of a repository, i.e. you go to the repository page and the URL looks like https://foreman.example.com/products/12/repositories/23

You feed the 23 into the find…

fred_demarcy · August 15, 2022, 5:24pm

Ok. Progressing…

# c7-base-repo = 17
# c7-updates-repo = 18
# c7-extras-repo = 19

irb(main):007:0> root = Katello::RootRepository.find(17)
=> #<Katello::RootRepository id: 17, name: "c7-base-repo", created_at: "2021-05-05 15:56:08", updated_at: "2022-08-15 08:44:57", major: nil, minor: nil, gpg_key_id: 7, content_id: "1620230169017", arch: "noarch", label: "c7-b...
irb(main):007:0> ForemanTasks.async_task(::Actions::Candlepin::Product::ContentAdd, owner: root.product.organization.label, product_id: root.product.cp_id, content_id: root.content_id)
=> #<ForemanTasks::Task::DynflowTask id: "a831ff23-5b1b-4167-8db9-a2476457f135", type: "ForemanTasks::Task::DynflowTask", label: "Actions::Candlepin::Product::ContentAdd", started_at: "2022-08-15 17:08:24", ended_at: nil, state: "planned", result: "pending", external_id: "53bb6f9b-fdb0-4758-bb4c-54eec3d4e993", parent_task_id: nil, start_at: "2022-08-15 17:08:24", start_before: nil, action: "Content add", user_id: 2, state_updated_at: "2022-08-15 17:08:24">

Then on my host

# subscription-manager refresh --force

and finally the c7-base-repo is back in the redhat.repo file

However for c7-extras-repo

=> #<Katello::RootRepository id: 19, name: "c7-extras-repo", created_at: "2021-05-05 15:56:18", updated_at: "2022-08-15 08:44:47", major: nil, minor: nil, gpg_key_id: 7, content_id: "1620230179550", arch: "noarch", label: "c7...
irb(main):010:0> ForemanTasks.async_task(::Actions::Candlepin::Product::ContentAdd, owner: root.product.organization.label, product_id: root.product.cp_id, content_id: root.content_id)
=> #<ForemanTasks::Task::DynflowTask id: "32b91137-8431-4c5d-9301-4d8f2c6fed6a", type: "ForemanTasks::Task::DynflowTask", label: "Actions::Candlepin::Product::ContentAdd", started_at: "2022-08-15 17:10:34", ended_at: nil, state: "planned", result: "pending", external_id: "abc02944-e6a9-4574-bf94-c2b3275ee28f", parent_task_id: nil, start_at: "2022-08-15 17:10:34", start_before: nil, action: "Content add", user_id: 2, state_updated_at: "2022-08-15 17:10:34">

Error in Task

Runtime Error could not execute statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,675

backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:21:in `rescue in propagate_candlepin_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:18:in `propagate_candlepin_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:9:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.5.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/execution_wrapper.rb:91:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:575:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:296:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/director.rb:69:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

gvde · August 15, 2022, 6:04pm

I think /var/log/candlepin/error.log or /var/log/candlepin/candlepin.log should contain more information about that candlepin error.

fred_demarcy · August 15, 2022, 7:00pm

2022-08-15 17:14:29,254 [thread=http-bio-127.0.0.1-23443-exec-6] [req=6edbf6a1-2941-4a6d-b4cd-2a620c843410, org=, csid=] ERROR org.candlepin.exceptions.mappers.CandlepinExceptionMapper - Runtime Error could not execute statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,675
org.hibernate.exception.ConstraintViolationException: could not execute statement
	at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:109)
	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
	at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:200)
	at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3208)
	at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3722)
	at org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:91)
	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:604)
	at org.hibernate.engine.spi.ActionQueue.lambda$executeActions$1(ActionQueue.java:478)
	at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:475)
	at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:348)
	at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:40)
	at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:108)
	at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1344)
	at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1331)
	at org.hibernate.query.internal.NativeQueryImpl.beforeQuery(NativeQueryImpl.java:264)
	at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1603)
	at org.candlepin.model.AbstractHibernateCurator.bulkSQLUpdate(AbstractHibernateCurator.java:1413)
	at org.candlepin.model.OwnerProductCurator.updateOwnerProductReferences(OwnerProductCurator.java:612)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:56)
	at org.candlepin.controller.ProductManager.updateProduct(ProductManager.java:376)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:56)
	at org.candlepin.resource.OwnerResource.addBatchContent(OwnerResource.java:2066)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:56)
	at org.candlepin.resource.OwnerResource.addContent(OwnerResource.java:2077)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:64)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:151)
	at org.jboss.resteasy.core.MethodInjectorImpl.lambda$invoke$3(MethodInjectorImpl.java:122)
	at java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:680)
	at java.base/java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658)
	at java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2094)
	at java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:143)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:122)
	at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:594)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:468)
	at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:421)
	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:363)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:423)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:391)
	at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invoke$1(ResourceMethodInvoker.java:365)
	at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
	at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
	at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:365)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:477)
	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:252)
	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:153)
	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:363)
	at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:156)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:238)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:249)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:60)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.candlepin.servlet.filter.EventFilter.doFilter(EventFilter.java:65)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.candlepin.servlet.filter.logging.LoggingFilter.doFilter(LoggingFilter.java:123)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.candlepin.servlet.filter.CandlepinPersistFilter.doFilter(CandlepinPersistFilter.java:48)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.candlepin.servlet.filter.CandlepinScopeFilter.doFilter(CandlepinScopeFilter.java:68)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.candlepin.servlet.filter.CandlepinContentTypeFilter.doFilter(CandlepinContentTypeFilter.java:67)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:498)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "cp2_product_entity_version"
  Detail: Key (product_id, entity_version)=(577537497763, 1775210097) already exists.
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:166)
	at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:134)
	at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:410)
	at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:197)
	... 88 common frames omitted

product id = 577537497763 is my “c7” Product.

No idea what the entity_version = entity_version is ?

iballou · August 15, 2022, 9:29pm

We might need to call some Candlepin experts!

I have a new suggestion that I found from this Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1931027#c80

[root@foreman ~]# sudo -u postgres psql candlepin
candlepin=# UPDATE cp_entitlement SET dirty='t';

Then run subscription-manager refresh on all related hosts.

I’m not certain what that does, so run at your own risk. Maybe take a snapshot / backup first. It sounds like it worked for one party at least.

fred_demarcy · August 16, 2022, 9:40am

Thanks @iballou
Looked at the Bugzilla ticket.

# scl enable rh-ruby27 bash

# ruby -v
ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]

# ruby downloads/find_missing_candlepin_product_contents.rb
Content name="cstream-baseos-repo" w/o association; content_uuid="8abd9592826940da01826993e4930095"; content_id="1620230716872"
Content name="cstream-extras-repo" w/o association; content_uuid="8abd9592826940da018269941faf0096"; content_id="1620230729030"
Content name="cstream-powertools-repo" w/o association; content_uuid="8abd9592826940da01826994574a0097"; content_id="1620230734016"
Content name="c7-extras-repo" w/o association; content_uuid="8abd9592826940da018269814286007a"; content_id="1620230179550"

Product "cstream"(cp_id=87644743334) has 4 content in foremanDB, but 1 content in candlepinDB
Product "c7"(cp_id=577537497763) has 3 content in foremanDB, but 2 content in candlepinDB



-- repair missing: ["cstream-baseos-repo"]
INSERT INTO cp2_product_content (id, enabled, product_uuid, content_uuid, created, updated) VALUES (REPLACE(uuid_in((md5((random())::text))::cstring)::text, '-', '' ), true, '8abd9592826940da0182698e3bec008c', '8abd9592826940da01826993e4930095', NOW(), NOW());
UPDATE cp2_products SET entity_version = NULL WHERE uuid = '8abd9592826940da0182698e3bec008c';

-- repair missing: ["cstream-extras-repo"]
INSERT INTO cp2_product_content (id, enabled, product_uuid, content_uuid, created, updated) VALUES (REPLACE(uuid_in((md5((random())::text))::cstring)::text, '-', '' ), true, '8abd9592826940da0182698e3bec008c', '8abd9592826940da018269941faf0096', NOW(), NOW());
UPDATE cp2_products SET entity_version = NULL WHERE uuid = '8abd9592826940da0182698e3bec008c';

-- repair missing: ["cstream-powertools-repo"]
INSERT INTO cp2_product_content (id, enabled, product_uuid, content_uuid, created, updated) VALUES (REPLACE(uuid_in((md5((random())::text))::cstring)::text, '-', '' ), true, '8abd9592826940da0182698e3bec008c', '8abd9592826940da01826994574a0097', NOW(), NOW());
UPDATE cp2_products SET entity_version = NULL WHERE uuid = '8abd9592826940da0182698e3bec008c';

-- repair missing: ["c7-extras-repo"]
INSERT INTO cp2_product_content (id, enabled, product_uuid, content_uuid, created, updated) VALUES (REPLACE(uuid_in((md5((random())::text))::cstring)::text, '-', '' ), true, '8abd959282929bd00182a27a5d6e0360', '8abd9592826940da018269814286007a', NOW(), NOW());
UPDATE cp2_products SET entity_version = NULL WHERE uuid = '8abd959282929bd00182a27a5d6e0360';



# ruby downloads/find_missing_candlepin_product_contents.rb --repair
Content name="cstream-baseos-repo" w/o association; content_uuid="8abd9592826940da01826993e4930095"; content_id="1620230716872"
Content name="cstream-extras-repo" w/o association; content_uuid="8abd9592826940da018269941faf0096"; content_id="1620230729030"
Content name="cstream-powertools-repo" w/o association; content_uuid="8abd9592826940da01826994574a0097"; content_id="1620230734016"
Content name="c7-extras-repo" w/o association; content_uuid="8abd9592826940da018269814286007a"; content_id="1620230179550"

Product "cstream"(cp_id=87644743334) has 4 content in foremanDB, but 1 content in candlepinDB
Product "c7"(cp_id=577537497763) has 3 content in foremanDB, but 2 content in candlepinDB



-- repair missing: ["cstream-baseos-repo"]
INSERT INTO cp2_product_content (id, enabled, product_uuid, content_uuid, created, updated) VALUES (REPLACE(uuid_in((md5((random())::text))::cstring)::text, '-', '' ), true, '8abd9592826940da0182698e3bec008c', '8abd9592826940da01826993e4930095', NOW(), NOW());
UPDATE cp2_products SET entity_version = NULL WHERE uuid = '8abd9592826940da0182698e3bec008c';

-- repair missing: ["cstream-extras-repo"]
INSERT INTO cp2_product_content (id, enabled, product_uuid, content_uuid, created, updated) VALUES (REPLACE(uuid_in((md5((random())::text))::cstring)::text, '-', '' ), true, '8abd9592826940da0182698e3bec008c', '8abd9592826940da018269941faf0096', NOW(), NOW());
UPDATE cp2_products SET entity_version = NULL WHERE uuid = '8abd9592826940da0182698e3bec008c';

-- repair missing: ["cstream-powertools-repo"]
INSERT INTO cp2_product_content (id, enabled, product_uuid, content_uuid, created, updated) VALUES (REPLACE(uuid_in((md5((random())::text))::cstring)::text, '-', '' ), true, '8abd9592826940da0182698e3bec008c', '8abd9592826940da01826994574a0097', NOW(), NOW());
UPDATE cp2_products SET entity_version = NULL WHERE uuid = '8abd9592826940da0182698e3bec008c';

-- repair missing: ["c7-extras-repo"]
INSERT INTO cp2_product_content (id, enabled, product_uuid, content_uuid, created, updated) VALUES (REPLACE(uuid_in((md5((random())::text))::cstring)::text, '-', '' ), true, '8abd959282929bd00182a27a5d6e0360', '8abd9592826940da018269814286007a', NOW(), NOW());
UPDATE cp2_products SET entity_version = NULL WHERE uuid = '8abd959282929bd00182a27a5d6e0360';

Repairing:
[["INSERT 0 1"], ["UPDATE 1"], ["INSERT 0 1"], ["UPDATE 1"], ["INSERT 0 1"], ["UPDATE 1"], ["INSERT 0 1"], ["UPDATE 1"]]

// On the host I am testing with
# subscription-manager refresh --force
-> no change. c7-extras-repo not showing.


# sudo -u postgres psql candlepin
candlepin=# UPDATE cp_entitlement SET dirty='t';


// On the host I am testing with
# subscription-manager refresh --force
-> no change. c7-extras-repo not showing.


// c7-base-repo = 17
// c7-updates-repo = 18
// c7-extras-repo = 19
Re-ran this
# foreman-rake console
'ErbParser' is ignored.
'RubyParser' is ignored.
Loading production environment (Rails 6.0.4.7)
irb(main):001:0> root = Katello::RootRepository.find(19)
=> #<Katello::RootRepository id: 19, name: "c7-extras-repo", created_at: "2021-05-05 15:56:18", updated_at: "2022-08-15 08:44:47", major: nil, minor: nil, gpg_key_id: 7, content_id: "1620230179550", arch: "noarch", label: "c7...
irb(main):002:0> ForemanTasks.async_task(::Actions::Candlepin::Product::ContentAdd, owner: root.product.organization.label, product_id: root.product.cp_id, content_id: root.content_id)
=> #<ForemanTasks::Task::DynflowTask id: "543faec4-1b80-4677-ac6f-c207b3b6e39b", type: "ForemanTasks::Task::DynflowTask", label: "Actions::Candlepin::Product::ContentAdd", started_at: "2022-08-16 09:24:02", ended_at: nil, state: "planned", result: "pending", external_id: "930c0f84-c612-41a5-a60c-534e4c60aead", parent_task_id: nil, start_at: "2022-08-16 09:24:02", start_before: nil, action: "Content add", user_id: 2, state_updated_at: "2022-08-16 09:24:02">
irb(main):003:0> quit

but task still failed with the same error
Runtime Error could not execute statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,675

Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "cp2_product_entity_version"
  Detail: Key (product_id, entity_version)=(577537497763, 1775210097) already exists.

I re-published + promoted my “c7-devops-view” and force subscribed my host again with subscription-manager but no change.

fred_demarcy · August 17, 2022, 2:33pm

Having not changed anything for 24+ hours I tried again a

# subscription-manager refresh --force

on my host and miraculously my 3x CentOS 7 repos (base, extras, updates) showed up in the redhat.repo file…
Happy, amazed and totally confused

iballou · August 30, 2022, 3:49pm

Please let us know if this happens again, but I’m glad that it fixed it self!