Configure custom Certificates - subscription-manager - HTTP error (500 - Internal Server Error)

Problem:
I created an update to the katello answers file and specified my preferred Certificate details and ran
foreman-installer --scenario katello --certs-update-server --certs-update-server-ca
This updated the cert as expected when I browsed the front-end.

I then installed the bootstrap rpm on the Foreman server and on the host I want to subscribe from.

Unfortunately I can no longer subscribe hosts. I get an error
HTTP error (500 - Internal Server Error)

I ran
subscription-manager clean
followed by
# subscription-manager register --org=“Org” --activationkey=“CentOS7”
HTTP error (500 - Internal Server Error): Permission denied @ rb_sysopen - /etc/pki/katello/private/pulp-client.key
When I look at the production.log on foreman I see the same error
2020-05-20T12:41:04 [E|kat|4c92472c] Errno::EACCES: Permission denied @ rb_sysopen - /etc/pki/katello/private/pulp-client.key

Here is the listing for the location on the foreman server:
ll /etc/pki/katello/private/
total 36
-rw-r–r--. 1 root root 7166 May 20 09:01 -foreman-proxy-client-bundle.pem
-r--------. 1 root qpidd 1679 May 20 08:54 -qpid-broker.key
-r--------. 1 root root 1675 May 20 08:54 java-client.key
-r--------. 1 root root 1679 May 20 12:17 katello-apache.key
-r--------. 1 root root 1679 May 20 08:54 katello-default-ca.key
-r--------. 1 root root 24 May 20 08:54 katello-default-ca.pwd
-r--------. 1 root root 1675 May 20 08:54 katello-tomcat.key
-r--------. 1 root root 1679 May 20 08:54 pulp-client.key

What step did I miss when I updated the certs?

Expected outcome:
I was hoping I could use my own certs issued by Let’s Encrypt or at least fill in my Company details for the certs being generated.
Foreman and Proxy versions:

Foreman and Proxy plugin versions:

Distribution and version:

Here is the entries I updated in the answers file

certs:
log_dir: /var/log/certs
node_fqdn:
cname:
generate: true
regenerate: false
deploy: true
ca_common_name:
country: ZA
state: Gauteng
city: Johannesburg
org: CompanyName
org_unit: DepartmentName
expiration: ‘7300’
ca_expiration: ‘36500’
server_cert:
server_key:
server_cert_req:
server_ca_cert:
pki_dir: /etc/pki/katello
ssl_build_dir: /root/ssl-build
user: root
group: root
default_ca_name: katello-default-ca
server_ca_name: katello-server-ca
tar_file:

Other relevant data:

Foreman 2.0
Katello 3.15

Thanks @daniejstriata for the post. At a glance the permissions look similar to a test box I have. Are you able to sync content or do anything with content in your setup?

Also we have documentation around setting up and debugging custom certs here: Foreman :: Plugin Manuals. Please take a look and let us know if you see anything useful there for your setup.

Thanks & Regards,
Samir Jha

Alas! I was not able to troubleshoot this issue any further. Doing a clean install without messing around with the certs for now.

Interestingly I got here with a new install with nothing configured yet.

hammer ping

database:
Status: ok
Server Response: Duration: 0ms
candlepin:
Status: ok
Server Response: Duration: 21ms
candlepin_auth:
Status: ok
Server Response: Duration: 17ms
pulp:
Status: ok
Server Response: Duration: 48ms
pulp_auth:
Status: FAIL
Server Response: Message: Permission denied @ rb_sysopen - /etc/pki/katello/private/pulp-client.key
foreman_tasks:
Status: ok
Server Response: Duration: 4ms

Permissions on my katello server look slightly different:

# ls -laZ /etc/pki/katello/private/
drwxr-x---. root foreman system_u:object_r:cert_t:s0      .
drwxr-xr-x. root foreman system_u:object_r:cert_t:s0      ..
-rw-r--r--. root root    system_u:object_r:cert_t:s0      foreman.example.com-foreman-proxy-client-bundle.pem
-r--r-----. root qpidd   system_u:object_r:cert_t:s0      foreman.example.com-qpid-broker.key
-r--------. root root    unconfined_u:object_r:cert_t:s0  java-client.key
-r--r-----. root foreman system_u:object_r:cert_t:s0      katello-apache.key
-r--r-----. root foreman system_u:object_r:cert_t:s0      katello-default-ca.key
-r--------. root root    system_u:object_r:cert_t:s0      katello-default-ca.pwd
-r--------. root root    unconfined_u:object_r:cert_t:s0  katello-tomcat.key
-r--r-----. root foreman system_u:object_r:cert_t:s0      pulp-client.key