Not able to add repository to Foreman setup

Support
1

Problem:
I have created product from console.
Not able to add Local repo to product. getting error ‘Authentication credentials were not provided’
Tried from both WebUI and using hammer-cli. Getting same error.
Steps followed:

  1. Created product ‘CentOS 7 Linux x86_64’
  2. Created key ‘RPM-GPG-KEY-CentOS-7’ in content → “content credentials”.
  3. Still getting same error. Authentication credentials were not provided 
[root@sjprdsatapp01 foreman-installer]# hammer content-credentials list --organization-id 1
---|--------------------------------|-------------
ID | NAME                           | CONTENT TYPE
---|--------------------------------|-------------
4  | RPM-GPG-KEY-CentOS-7           | gpg_key
3  | RPM-GPG-KEY-CentOS-SIG-Storage | gpg_key
---|--------------------------------|-------------
[root@sjprdsatapp01 foreman-installer]# hammer -v repository create --organization cadence --product "CentOS 7 Linux x86_64"  --name "CentOS 7 OS x86_64" --label "CentOS_7_Linux_x86_64" --content-type "yum" --download-policy "on_demand" --gpg-key-id 4 --url "http://172.13.13.16/images/centos-7-x86_64/7.9.2009/os/" --mirror-on-sync "no"
Could not create the repository:
 Authentication credentials were not provided.

I will attach screenshot of error in WEBUI as well…

Expected outcome:
Able to add repositories
Foreman and Proxy versions:
rubygem-foreman_column_view-0.4.0-6.fm3_3.el8.noarch
foreman-vmware-3.4.1-1.el8.noarch
foreman-selinux-3.4.1-1.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
foreman-debug-3.4.1-1.el8.noarch
foreman-dynflow-sidekiq-3.4.1-1.el8.noarch
rubygem-foreman_maintain-1.2.1-1.el8.noarch
rubygem-foreman_discovery-21.0.4-1.fm3_4.el8.noarch
foreman-installer-katello-3.4.1-1.el8.noarch
foreman-proxy-3.4.1-1.el8.noarch
rubygem-hammer_cli_foreman-3.4.0-1.el8.noarch
foreman-installer-3.4.1-1.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.17-1.fm3_2.el8.noarch
foreman-3.4.1-1.el8.noarch
rubygem-foreman_remote_execution-8.0.0-2.fm3_4.el8.noarch
foreman-ec2-3.4.1-1.el8.noarch
foreman-release-3.4.1-1.el8.noarch
foreman-service-3.4.1-1.el8.noarch
rubygem-foreman-tasks-7.0.0-1.fm3_4.el8.noarch
rubygem-foreman_default_hostgroup-6.0.0-2.fm3_3.el8.noarch
foreman-cli-3.4.1-1.el8.noarch
rubygem-foreman_memcache-0.1.1-5.fm3_3.el8.noarch
foreman-postgresql-3.4.1-1.el8.noarch
Foreman and Proxy plugin versions:

Distribution and version:
Red hat 8.6
Foreman 3.5 with Katello 4.7

Other relevant data:
Adding production.log.
Check after timestamp 2023-03-28T00:00:00

2

Adding the screenshots from WEBUI

foreman-auth-error 1
foreman-auth-error 11456×961 79.4 KB

foreman-auth-error 2
foreman-auth-error 21698×929 116 KB

foreman-auth-error 3
foreman-auth-error 31868×934 112 KB

3

Adding production.log taken at same time.
Check after timestamp 2023-03-28T00:00:00

production.log (192 KB)

5

Checking more information.
Files pointed by pulpcore_postgresql_ssl_cert and pulpcore_postgresql_ssl_key do not exist on system. Is this causing the issue?

# cat /etc/foreman-installer/scenarios.d/katello-answers.yaml | grep -i pulp
  pulpcore_mirror: false
  pulpcore_manage_postgresql: false
  pulpcore_postgresql_host: sjprdsatdb01.cadence.com
  pulpcore_postgresql_port: 5432
  pulpcore_postgresql_user: pulp
  pulpcore_postgresql_password: password123
  pulpcore_postgresql_db_name: pulpcore
  pulpcore_postgresql_ssl: false
  pulpcore_postgresql_ssl_require: true
  pulpcore_postgresql_ssl_cert: "/etc/pki/katello/certs/pulpcore-database.crt"          <----
  pulpcore_postgresql_ssl_key: "/etc/pki/katello/private/pulpcore-database.key"     <----
  pulpcore_postgresql_ssl_root_ca: "/etc/pki/tls/certs/ca-bundle.crt"
  pulpcore_postgresql_ssl_root_ca: "/etc/pki/katello/certs/katello-server-ca.crt"
  pulpcore_worker_count: 8
  pulpcore_django_secret_key:
  pulpcore_content_service_worker_timeout: 90