The use of openssl in katello-certs-check with system default ca path

I have noticed in the katello-certs-check script that it calls openssl multiple times to check the certs and chain provided. However, the checks don’t disable the default CA path which means openssl will use the default system CA path by default.

It’s my understanding that foreman does not use the default ca path/file when checking internal connections, e.g. from the main server to a proxy. I am not sure but from some of the topics lately I get the impression that it does it this way (which is reasonable).

This would mean that katello-certs-check could verify a cert as O.K. because (part of) the chain is available in CApath even though the given ca bundle alone wouldn’t.

E.g.

function check-ca-bundle () {
    printf "Checking CA bundle against the certificate file: "
    ERROR_PATTERN="error [0-9]+ at"
    CHECK=$(openssl verify -CAfile $CA_BUNDLE_FILE -purpose sslserver -verbose $CERT_FILE 2>&1)
    CHECK_STATUS=$?

    if [[ $CHECK_STATUS != "0" || $CHECK =~ $ERROR_PATTERN ]]; then
        error 4 "The $CA_BUNDLE_FILE does not verify the $CERT_FILE"
        echo -e "${CHECK/OK/}\n"
    else
        success
    fi
}

The openssl verify checks the $CERT_FILE file against the $CA_BUNDLE_FILE and the system default CA path. Only if you added -no-CApath it would disable CA path and would check the cert file against the bundle alone.

If foreman doesn’t use the default ca path/file for connection verification then katello-certs-check shouldn’t either to give an accurate assessment of the given cert and bundle.

I tested this on an installation and I could not reproduce this behaviour. My testing indicates that by default it is not checking the system trust. Are you able to verify the behavior? And can you provide steps that reproduce it?

Simple example. foreman8-2024.crt contains the current server certificate signed by our CA. No chain. No intermediate nor root. The chain certs including the root (as well as the hash symlinks) are installed in the system default path /etc/pki/tls/certs.

# /sbin/katello-certs-check -t foreman -c foreman8-2024.crt -k foreman8.key -b foreman8-2024.crt 

Validates OK even though it shouldn’t because the cert alone cannot be verified to any known root. Run with bash -x to see what it executes:

# bash -x /sbin/katello-certs-check -t foreman -c foreman8-2024.crt -k foreman8.key -b foreman8-2024.crt 
...
+ printf 'Checking CA bundle against the certificate file: '
Checking CA bundle against the certificate file: + ERROR_PATTERN='error [0-9]+ at'
++ openssl verify -CAfile /root/certs/foreman8-2024.crt -purpose sslserver -verbose /root/certs/foreman8-2024.crt
+ CHECK='/root/certs/foreman8-2024.crt: OK'

That’s the openssl command:

[root@foreman8 certs]# openssl verify -CAfile /root/certs/foreman8-2024.crt -purpose sslserver -verbose /root/certs/foreman8-2024.crt
/root/certs/foreman8-2024.crt: OK

which only works because default CApath is /etc/pki/tls/certs. Disable it to see the real result without system path:

[root@foreman8 certs]# openssl verify --no-CApath -CAfile /root/certs/foreman8-2024.crt -purpose sslserver -verbose /root/certs/foreman8-2024.crt
C = DE, ST = Hamburg, O = Deutsches Klimarechenzentrum GmbH, CN = foreman8.dkrz.de
error 20 at 0 depth lookup: unable to get local issuer certificate
error /root/certs/foreman8-2024.crt: verification failed

It’s not possible to verify the certificate to a known self-signed root ca certificate.

Default path:

[root@foreman8 certs]# openssl version -d
OPENSSLDIR: "/etc/pki/tls"

You can also see the files used when running with strace:

# strace openssl verify  -CAfile /root/certs/foreman8-2024.crt -purpose sslserver -verbose /root/certs/foreman8-2024.crt 2>&1 | grep open
execve("/bin/openssl", ["openssl", "verify", "-CAfile", "/root/certs/foreman8-2024.crt", "-purpose", "sslserver", "-verbose", "/root/certs/foreman8-2024.crt"], 0x7fffee999b58 /* 25 vars */) = 0
...
openat(AT_FDCWD, "/etc/pki/tls/openssl.cnf", O_RDONLY) = 3
stat("/etc/crypto-policies/back-ends/opensslcnf.config", {st_mode=S_IFREG|0644, st_size=647, ...}) = 0
openat(AT_FDCWD, "/etc/crypto-policies/back-ends/opensslcnf.config", O_RDONLY) = 4
openat(AT_FDCWD, "/root/certs/foreman8-2024.crt", O_RDONLY) = 3
openat(AT_FDCWD, "/root/certs/foreman8-2024.crt", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/pki/tls/certs/08ab1bf8.0", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/pki/tls/certs/fc5a8f99.0", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/pki/tls/certs/ee64a828.0", O_RDONLY) = 3

Thus, if foreman doesn’t use the system default capath for certificate verification the behaviour won’t match what katello-certs-check says.

I was trying this adding the CA to the system wide CA trust. I see your workflow is different. I am missing one piece to understand and replicate it; how are you getting the CA certificate into /etc/pki/tls/certs ?

It’s the standard layout. We have put the issuing CA, the intermediate CA and the root CA in pem format into /etc/pki/tls/certs and created the hash symlinks. In the end it looks like this:

-rw-r--r--. 1 root root 1517 Jan 17  2023 01-comodo-aaa-root.pem
-rw-r--r--. 1 root root 1968 Jan 17  2023 02-usertrust.pem
-rw-r--r--. 1 root root 2446 Jan 17  2023 03-geant-ov-rsa-ca4.pem
lrwxrwxrwx. 1 root root   23 Jan 17  2023 08ab1bf8.0 -> 03-geant-ov-rsa-ca4.pem
lrwxrwxrwx. 1 root root   22 Jan 17  2023 ee64a828.0 -> 01-comodo-aaa-root.pem
lrwxrwxrwx. 1 root root   16 Jan 17  2023 fc5a8f99.0 -> 02-usertrust.pem

The symlinks are standard, just like they are created by c_rehash and what you need for CApath.

$ openssl x509 -in /etc/pki/tls/certs/03-geant-ov-rsa-ca4.pem -noout -hash
08ab1bf8

I’m still not following how I create that structure, my system by default looks like:

$ ls /etc/pki/tls/certs/ -la
total 8
drwxr-xr-x. 2 root root  103 Aug 26 13:28 .
drwxr-xr-x. 5 root root  104 Nov 30  2023 ..
lrwxrwxrwx. 1 root root   49 Aug  3  2023 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx. 1 root root   55 Aug  3  2023 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt

I am not sure what’s unclear.

Our server certificates are issued by the CA

subject=C = NL, O = GEANT Vereniging, CN = GEANT OV RSA CA 4.

GEANT OV RSA CA 4 has been issued by the intermediate CA

subject=C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority

That CA has been issued by the root CA

subject=C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services

We have put those three CA certificates into three separate pem files in /etc/pki/tls/certs:

-rw-r--r--. 1 root root 1517 Jan 17  2023 /etc/pki/tls/certs/01-comodo-aaa-root.pem
-rw-r--r--. 1 root root 1968 Jan 17  2023 /etc/pki/tls/certs/02-usertrust.pem
-rw-r--r--. 1 root root 2446 Jan 17  2023 /etc/pki/tls/certs/03-geant-ov-rsa-ca4.pem

The content of 01-comodo-aaa-root.pem is just the root ca in pem format

-----BEGIN CERTIFICATE-----
MIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEb
MBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRow
GAYDVQQKDBFDb21vZG8gQ0EgTGltaXRlZDEhMB8GA1UEAwwYQUFBIENlcnRpZmlj
YXRlIFNlcnZpY2VzMB4XDTA0MDEwMTAwMDAwMFoXDTI4MTIzMTIzNTk1OVowezEL
MAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE
BwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMM
GEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBAL5AnfRu4ep2hxxNRUSOvkbIgwadwSr+GB+O5AL686tdUIoWMQua
BtDFcCLNSS1UY8y2bmhGC1Pqy0wkwLxyTurxFa70VJoSCsN6sjNg4tqJVfMiWPPe
3M/vg4aijJRPn2jymJBGhCfHdr/jzDUsi14HZGWCwEiwqJH5YZ92IFCokcdmtet4
YgNW8IoaE+oxox6gmf049vYnMlhvB/VruPsUK6+3qszWY19zjNoFmag4qMsXeDZR
rOme9Hg6jc8P2ULimAyrL58OAd7vn5lJ8S3frHRNG5i1R8XlKdH5kBjHYpy+g8cm
ez6KJcfA3Z3mNWgQIJ2P2N7Sw4ScDV7oL8kCAwEAAaOBwDCBvTAdBgNVHQ4EFgQU
oBEKIz6W8Qfs4q8p74Klf9AwpLQwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQF
MAMBAf8wewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5jb21vZG9jYS5jb20v
QUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwuY29t
b2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDANBgkqhkiG9w0BAQUF
AAOCAQEACFb8AvCb6P+k+tZ7xkSAzk/ExfYAWMymtrwUSWgEdujm7l3sAg9g1o1Q
GE8mTgHj5rCl7r+8dFRBv/38ErjHT1r0iWAFf2C3BUrz9vHCv8S5dIa2LX1rzNLz
Rt0vxuBqw8M0Ayx9lt1awg6nCpnBBYurDC/zXDrPbDdVCYfeU0BsWO/8tqtlbgT2
G9w84FoVxp7Z8VlIMCFlA2zs6SFz7JsDoeA3raAVGI/6ugLOpyypEBMs1OUIJqsi
l2D4kF501KKaU73yqWjgom7C12yxow+ev+to51byrvLjKzg6CYG1a4XXvi3tPxq3
smPi9WIsgtRqAEFQ8TmDn5XpNpaYbg==
-----END CERTIFICATE-----

The hashes are calculated with openssl:

$ for c in  /etc/pki/tls/certs/01-comodo-aaa-root.pem /etc/pki/tls/certs/02-usertrust.pem /etc/pki/tls/certs/03-geant-ov-rsa-ca4.pem ; do echo -n "$c: subject_hash " ; echo -n $(openssl x509 -in $c -noout -subject_hash) " issuer_hash " ; openssl x509 -in $c -noout -issuer_hash ;  done
/etc/pki/tls/certs/01-comodo-aaa-root.pem: subject_hash ee64a828  issuer_hash ee64a828
/etc/pki/tls/certs/02-usertrust.pem: subject_hash fc5a8f99  issuer_hash ee64a828
/etc/pki/tls/certs/03-geant-ov-rsa-ca4.pem: subject_hash 08ab1bf8  issuer_hash fc5a8f99

As the system default CApath points to /etc/pki/tls/certs, openssl can find the CA certificate through those hash symlinks and is thus able to verify the certificate chain of our certificates.

I’m getting closer to replicating it. I still want to understand this workflow better to ensure we account for it properly.

What creates those hash symlinks? Is that done manually by you or is there a command line utility that is generating them? Is this a common practice to do?

That is the standard way how openssl is able to use a CApath. It’s this way since I can remember. openssl came with the script to create these 26 years ago: openssl/tools/c_rehash at OpenSSL_0_9_1c · openssl/openssl · GitHub

You can use the c_rehash script in whatever form or you can do it manually. The process has always been the same. I think at some point in the past the hash algorithm changed thus the hashes changed. I think this was between CentOS 5 and 6, or maybe 6 and 7?

The symlinks are required if you want to use CApath with openssl. Any application using openssl and using CApath needs to set them up. It’s in the openssl docs and manual pages, e.g. openssl-verification-options - OpenSSL Documentation

However, I am not sure if you are going down the correct path: If foreman for its internal certificate verification of tls connections does not use CApath but only CAfile then katello-certs-check should do the same to get the same results as foreman. Thus, katello-certs-check should call openssl with --no-CApath to make sure. No symlinks needed for that.

I understand now and replicated it - thank you. Here is a PR for it – Ignore system CA trust when verifying certificates by ehelms · Pull Request #964 · theforeman/foreman-installer · GitHub