Katello 3.2 > 3.3 upgrade with custom signed ssl certs breaks pulp

Support
1

Hello, I've run into an issue where after upgrading a working katello 3.2
system to 3.3 I get the following error when attempting to publish a
content view.

There was an issue with the backend service pulp: SSL_connect returned=1
errno=0 state=SSLv3 read server certificate B: certificate verify failed

When the system was running katello 3.2 I rand the following to get the
custom signed ssl certs to work.

foreman-installer --scenario katello --certs-server-cert
/etc/pki/tls/certs/il-foreman1_slc_westdc_net.crt --certs-server-cert-req
/etc/pki/tls/private/il-foreman1.slc.westdc.net.csr --certs-server-key
/etc/pki/tls/private/il-foreman1.slc.westdc.net.key --certs-server-ca-cert
/etc/pki/tls/certs/comodo-ca-bundle.crt --certs-server-ca-name comodo-ca
–certs-update-server --certs-update-server-ca

To fix problems with candlepin I did the following found
via Bug #16620: custom certificates do not work out-of-the box on katello 3.1 - Katello - Foreman

Copy /root/ssl-build/katello-default-ca.crt to
/etc/pki/ca-trust/source/anchors/ and rebuild the openssl ca certs with
update-ca-trust. Due to chicken-and-egg issue, this may prevent a clean
install using custom certs. After performing these steps, re-run the
installer. It should complete correctly the second time through.

I've attempted both of theses steps along with the second fix on the above
url for issue 16620 without any success. I see the following details in
the /etc/foreman/plugins/katello.yaml

File managed with puppet

Module: puppet-katello

:katello:
:rest_client_timeout: 3600

:post_sync_url:
https://il-foreman1.domain.net/katello/api/v2/repositories/sync_complete?token=gQ7efFZPwo8oWXg9abmdG3v8gkY29fcs

:candlepin:
:url: https://il-foreman1.domain.net:8443/candlepin
:oauth_key: katello
:oauth_secret: qXZyiEhe8WqoCeTtPJqhpUGCPV65GmeL
:ca_cert_file: /etc/pki/katello/certs/katello-default-ca.crt

:pulp:
:url: https://il-foreman1.domain.net/pulp/api/v2/
:oauth_key: katello
:oauth_secret: qXZyiEhe8WqoCeTtPJqhpUGCPV65GmeL
:ca_cert_file: /etc/pki/katello/certs/katello-default-ca.crt

:qpid:
:url: amqp:ssl:localhost:5671
:subscriptions_queue_address: katello_event_queue

I'm not sure what additional information to provide to help identify the
problem here. Any ideas what to try/do next?

Thanks

2

Is it possible to revert back to the default self signed certs and try
re-installing the custom signed certs?

If not I need some guidance on how to troubleshoot this issue or should I
be opening a bug?

··· On Tuesday, 28 February 2017 11:07:38 UTC-7, Edward Clay wrote: > > Hello, I've run into an issue where after upgrading a working katello 3.2 > system to 3.3 I get the following error when attempting to publish a > content view. > > There was an issue with the backend service pulp: SSL_connect returned=1 > errno=0 state=SSLv3 read server certificate B: certificate verify failed > > > When the system was running katello 3.2 I rand the following to get the > custom signed ssl certs to work. > > foreman-installer --scenario katello --certs-server-cert > /etc/pki/tls/certs/il-foreman1_slc_westdc_net.crt --certs-server-cert-req > /etc/pki/tls/private/il-foreman1.slc.westdc.net.csr --certs-server-key > /etc/pki/tls/private/il-foreman1.slc.westdc.net.key --certs-server-ca-cert > /etc/pki/tls/certs/comodo-ca-bundle.crt --certs-server-ca-name comodo-ca > --certs-update-server --certs-update-server-ca > > To fix problems with candlepin I did the following found via > http://projects.theforeman.org/issues/16620 > > Copy /root/ssl-build/katello-default-ca.crt to > /etc/pki/ca-trust/source/anchors/ and rebuild the openssl ca certs with > update-ca-trust. Due to chicken-and-egg issue, this may prevent a clean > install using custom certs. After performing these steps, re-run the > installer. It should complete correctly the second time through. > > I've attempted both of theses steps along with the second fix on the above > url for issue 16620 without any success. I see the following details in > the /etc/foreman/plugins/katello.yaml > > ### File managed with puppet ### > ## Module: puppet-katello > > :katello: > :rest_client_timeout: 3600 > > :post_sync_url: > https://il-foreman1.domain.net/katello/api/v2/repositories/sync_complete?token=gQ7efFZPwo8oWXg9abmdG3v8gkY29fcs > > :candlepin: > :url: https://il-foreman1.domain.net:8443/candlepin > :oauth_key: katello > :oauth_secret: qXZyiEhe8WqoCeTtPJqhpUGCPV65GmeL > :ca_cert_file: /etc/pki/katello/certs/katello-default-ca.crt > > :pulp: > :url: https://il-foreman1.domain.net/pulp/api/v2/ > :oauth_key: katello > :oauth_secret: qXZyiEhe8WqoCeTtPJqhpUGCPV65GmeL > :ca_cert_file: /etc/pki/katello/certs/katello-default-ca.crt > > :qpid: > :url: amqp:ssl:localhost:5671 > :subscriptions_queue_address: katello_event_queue > > I'm not sure what additional information to provide to help identify the > problem here. Any ideas what to try/do next? > > Thanks >
3

I have the same
problem: https://groups.google.com/forum/#!topic/foreman-users/vxj75qlt8k4

Have not found any solution yet.

Edgars

otrdiena, 2017. gada 28. februāris 19:07:38 UTC+1, Edward Clay rakstīja:

··· > > Hello, I've run into an issue where after upgrading a working katello 3.2 > system to 3.3 I get the following error when attempting to publish a > content view. > > There was an issue with the backend service pulp: SSL_connect returned=1 > errno=0 state=SSLv3 read server certificate B: certificate verify failed > > > When the system was running katello 3.2 I rand the following to get the > custom signed ssl certs to work. > > foreman-installer --scenario katello --certs-server-cert > /etc/pki/tls/certs/il-foreman1_slc_westdc_net.crt --certs-server-cert-req > /etc/pki/tls/private/il-foreman1.slc.westdc.net.csr --certs-server-key > /etc/pki/tls/private/il-foreman1.slc.westdc.net.key --certs-server-ca-cert > /etc/pki/tls/certs/comodo-ca-bundle.crt --certs-server-ca-name comodo-ca > --certs-update-server --certs-update-server-ca > > To fix problems with candlepin I did the following found via > http://projects.theforeman.org/issues/16620 > > Copy /root/ssl-build/katello-default-ca.crt to > /etc/pki/ca-trust/source/anchors/ and rebuild the openssl ca certs with > update-ca-trust. Due to chicken-and-egg issue, this may prevent a clean > install using custom certs. After performing these steps, re-run the > installer. It should complete correctly the second time through. > > I've attempted both of theses steps along with the second fix on the above > url for issue 16620 without any success. I see the following details in > the /etc/foreman/plugins/katello.yaml > > ### File managed with puppet ### > ## Module: puppet-katello > > :katello: > :rest_client_timeout: 3600 > > :post_sync_url: > https://il-foreman1.domain.net/katello/api/v2/repositories/sync_complete?token=gQ7efFZPwo8oWXg9abmdG3v8gkY29fcs > > :candlepin: > :url: https://il-foreman1.domain.net:8443/candlepin > :oauth_key: katello > :oauth_secret: qXZyiEhe8WqoCeTtPJqhpUGCPV65GmeL > :ca_cert_file: /etc/pki/katello/certs/katello-default-ca.crt > > :pulp: > :url: https://il-foreman1.domain.net/pulp/api/v2/ > :oauth_key: katello > :oauth_secret: qXZyiEhe8WqoCeTtPJqhpUGCPV65GmeL > :ca_cert_file: /etc/pki/katello/certs/katello-default-ca.crt > > :qpid: > :url: amqp:ssl:localhost:5671 > :subscriptions_queue_address: katello_event_queue > > I'm not sure what additional information to provide to help identify the > problem here. Any ideas what to try/do next? > > Thanks >
4

I ended up restoring from backup to the previous version 3.2 and things are
working. I would like very much to figure out how to revert back to self
signed ssl certs and do away with using the 3rd party signed ones. Since I
ran foreman-installer with some options to use the 3rd party signed ssl
certs you would think this would be a simple task.

5

I am still on 3.2 but as Edgars M. has mentioned in his linked thread my
katello.yaml has no ":ca_cert_file" entry in the ":candlepin" or ":pulp"
sections.

··· On Wednesday, March 1, 2017 at 9:12:27 AM UTC+1, Edgars M. wrote: > > I have the same problem: > https://groups.google.com/forum/#!topic/foreman-users/vxj75qlt8k4 > > Have not found any solution yet. > > Edgars > > otrdiena, 2017. gada 28. februāris 19:07:38 UTC+1, Edward Clay rakstīja: >> >> Hello, I've run into an issue where after upgrading a working katello >> 3.2 system to 3.3 I get the following error when attempting to publish a >> content view. >> >> There was an issue with the backend service pulp: SSL_connect returned=1 >> errno=0 state=SSLv3 read server certificate B: certificate verify failed >> >> >> When the system was running katello 3.2 I rand the following to get the >> custom signed ssl certs to work. >> >> foreman-installer --scenario katello --certs-server-cert >> /etc/pki/tls/certs/il-foreman1_slc_westdc_net.crt --certs-server-cert-req >> /etc/pki/tls/private/il-foreman1.slc.westdc.net.csr --certs-server-key >> /etc/pki/tls/private/il-foreman1.slc.westdc.net.key --certs-server-ca-cert >> /etc/pki/tls/certs/comodo-ca-bundle.crt --certs-server-ca-name comodo-ca >> --certs-update-server --certs-update-server-ca >> >> To fix problems with candlepin I did the following found via >> http://projects.theforeman.org/issues/16620 >> >> Copy /root/ssl-build/katello-default-ca.crt to >> /etc/pki/ca-trust/source/anchors/ and rebuild the openssl ca certs with >> update-ca-trust. Due to chicken-and-egg issue, this may prevent a clean >> install using custom certs. After performing these steps, re-run the >> installer. It should complete correctly the second time through. >> >> I've attempted both of theses steps along with the second fix on the >> above url for issue 16620 without any success. I see the following details >> in the /etc/foreman/plugins/katello.yaml >> >> ### File managed with puppet ### >> ## Module: puppet-katello >> >> :katello: >> :rest_client_timeout: 3600 >> >> :post_sync_url: >> https://il-foreman1.domain.net/katello/api/v2/repositories/sync_complete?token=gQ7efFZPwo8oWXg9abmdG3v8gkY29fcs >> >> :candlepin: >> :url: https://il-foreman1.domain.net:8443/candlepin >> :oauth_key: katello >> :oauth_secret: qXZyiEhe8WqoCeTtPJqhpUGCPV65GmeL >> :ca_cert_file: /etc/pki/katello/certs/katello-default-ca.crt >> >> :pulp: >> :url: https://il-foreman1.domain.net/pulp/api/v2/ >> :oauth_key: katello >> :oauth_secret: qXZyiEhe8WqoCeTtPJqhpUGCPV65GmeL >> :ca_cert_file: /etc/pki/katello/certs/katello-default-ca.crt >> >> :qpid: >> :url: amqp:ssl:localhost:5671 >> :subscriptions_queue_address: katello_event_queue >> >> I'm not sure what additional information to provide to help identify the >> problem here. Any ideas what to try/do next? >> >> Thanks >> >
6

Using the following two websites loosely as a guide I was able to get
katello/foreman to use the original self signed ssl certs.

https://access.redhat.com/solutions/1311844

  1. ran the following to remove old certs.

[root@satellite ~]# for i in $(ls /etc/pki/katello-certs-tools/certs/*);
do
rpm -qf $i >> /tmp/pkgs
done
yum remove cat /tmp/pkgs | sort | uniq -y

[root@satellite ~]# find /etc/pki/katello-certs-tools/* -type f -exec rm
-f {} ; && rm -f /tmp/pkgs

  1. then I searched for any cert/key/csr that started with the servers fqdn
    in the /etc/pki directory. Also searched for any file named comodo* in the
    same. I removed all
  2. removed /etc/pki/katello/certs/java-client.crt
  3. edited /etc/foreman-installer/scenarios.d/katello-answers.yaml and
    removed all but the following from the certs: section.

certs:
generate: true
deploy: true
group: foreman
5. Ran the foreman-installer to fix things up (recreate/configure self
signed certs)
foreman-installer --scenario katello --certs-update-server

  1. rebooted the server (didn't want to mess with figuring out what actually
    needed to be restarted)

I did have to clear all browser cache data since foreman is setup to use
hsts so that old info needed to be purged. Now I'm connecting securely yet
insecurely.

Now to see if upgrading to 3.3 will brake now.