[Katello] Custom certificate problem

Problem: I am obtaining certificates with certbot (not the regular letsencrypt ACME endpoint, but the ACME endpoint of the SECTIGO CA). When running “katello-certs-check” against these obtained certificates, it just complains:

[root@linux test]# katello-certs-check -c cert.pem -k privkey.pem -b chain.pem
Checking server certificate encoding:
[OK]

Checking expiration of certificate:
[OK]

Checking expiration of CA bundle:
[OK]

Checking if server certificate has CA:TRUE flag
[OK]

Checking for private key passphrase:
[OK]

Checking to see if the private key matches the certificate:
[OK]

Checking CA bundle against the certificate file:
[FAIL]

The /root/test/chain.pem does not verify the /root/test/cert.pem
C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
error 2 at 2 depth lookup: unable to get issuer certificate
error /root/test/cert.pem: verification failed

Checking CA bundle size: 2
[OK]

Checking Subject Alt Name on certificate
[OK]

Checking if any Subject Alt Name on certificate matches the Subject CN
[OK]

Checking Key Usage extension on certificate for Key Encipherment
[OK]

Checking for use of shortname as CN
[OK]

As documentation reference i used:
https://docs.theforeman.org/3.5/Installing_Server/index-katello.html#Configuring_Server_with_a_Custom_SSL_Certificate_foreman

Expected outcome: It should work :slight_smile:

Foreman and Proxy versions: Foreman 3.5 with Katello 4.7 on Rocky 8

As the errors tell you:

I would say your chain is missing the root CA.

This post may be older, but the issue still exists. It seems to be a bug in Katello’s certificate validation. The Sectigo cacert chain does include all certs up to the root, but in addition the Sectigo chain is cross-signed by an older CA AAA Certificate Services. This cross-signing certificate should not be needed to validate the chain, but katello-certs-check insists on it being included in the chain.

This is an old thread. Please open a new thread, if you think this issue still exists. And what you write is impossible to understand unless you post the specific certs included in the chain.

The thread may be old, but keeps coming up when searching for this issue through Google. So there seems more value in adding the new information here rather than opening a new thread. The cert chain was identified in the original post; my post provided a link to the cert that needs to be added as a workaround.

The issue still exists as of 3.12/4.14 (and I reported it a while ago as #37021).

It’s still a different issue and there is really no purpose of that, in particular, as there is still all the information missing required to reproduce the issue. I can say for our sectigo issued server certificates that the official bundle which I can download with the certificate from their certificate manager includes the AAA root CA certificate. If you use that, it is just fine.

If you think you can omit the AAA or if you get something else, it’s impossible to tell given the information you have given so far…

Regarding Bug #37021: katello-certs-check should succeed if intermediates are presented without root - Foreman it’s intentional that the CA file contains the root CA because it’s used for CA pinning as well, i.e. the proxy and server only accepts connections with this particular chain and no other. It doesn’t matter whether it’s a chain from a public CA or your own private CA or the foreman created certificates.

katello-certs-check actually was modified last year to make sure that it does not consider any system trust store or system ca directory or file. Before, it would for instance use /etc/pki/tls/certs for an additional source…

No, it really is the same issue. The chain always included the root certificate even for the original poster; it is a cross-signing cert that’s missing. I know this because my issue isn’t just similar; it is identical. I’m using the exact same Sectigo ACME protocol, the same cert chain and ran into the exact same issue as the original poster, and it hasn’t changed in several years. And I’ll admit that I fell into the same pitfall of assuming it must be the root cert, until I discovered the cross-signing issue today.

The cross-signing cert is only supposed to be necessary for backwards compatibility and is deliberately left out of the ca chain, but katello-certs-check always requires it.

Not using the system trust store etc. sounds like a good decision, but it does not actually have anything to do with the Sectigo problem.

i’ve updated that bug report with the new information, although in light of our discussion, it probably needs more elaboration.

BTW, the problem katello-certs-check has with the Sectigo chain is getting more urgent. Sectigo (formerly known as Comodo) is one of the the biggest CAs in the world, and with the advent of 90 day certificates, using the ACME protocol without manual workarounds is becoming critically important.

As for reproducing the isssue: below is the certificate chain, which should be the same for all certs issued by that CA from Sectigo (Sectigo has multiple CAs; this particular one is different from the original poster’s, but has the same issue. All Sectigo chains do).

The missing cross-sign cert his available here: crt.sh | 331986 (the link to the PEM is pretty small, towards the left bottom). You simply have to append that to the chain, and suddenly katello-certs-check will work.

Obviously, to reproduce the issue, you would also need to generate a test cert and key using a Sectigo account and ACME. You may not have access to such an account. If it helps, I may be able to get you a test cert and key for you, if there is a way to submit it privately. I don’t want to promise that because even with a test cert for a non-existent system, there are still security implications.

-----BEGIN CERTIFICATE-----
MIIGSjCCBDKgAwIBAgIRAINbdhUgbS1uCX4LbkCf78AwDQYJKoZIhvcNAQEMBQAw
gYgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpOZXcgSmVyc2V5MRQwEgYDVQQHEwtK
ZXJzZXkgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMS4wLAYD
VQQDEyVVU0VSVHJ1c3QgUlNBIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTIy
MTExNjAwMDAwMFoXDTMyMTExNTIzNTk1OVowRDELMAkGA1UEBhMCVVMxEjAQBgNV
BAoTCUludGVybmV0MjEhMB8GA1UEAxMYSW5Db21tb24gUlNBIFNlcnZlciBDQSAy
MIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIBigKCAYEAifBcxDi60DRXr5dVoPQi
Q/w+GBE62216UiEGMdbUt7eSiIaFj/iZ/xiFop0rWuH4BCFJ3kSvQF+aIhEsOnuX
R6mViSpUx53HM5ApIzFIVbd4GqY6tgwaPzu/XRI/4Dmz+hoLW/i/zD19iXvS95qf
NU8qP7/3/USf2/VNSUNmuMKlaRgwkouue0usidYK7V8W3ze+rTFvWR2JtWKNTInc
NyWD3GhVy/7G09PwTAu7h0qqRyTkETLf+z7FWtc8c12f+SfvmKHKFVqKpNPtgMkr
wqwaOgOOD4Q00AihVT+UzJ6MmhNPGg+/Xf0BavmXKCGDTv5uzQeOdD35o/Zw16V4
C4J4toj1WLY7hkVhrzKG+UWJiSn8Hv3dUTj4dkneJBNQrUfcIfTHV3gCtKwXn1eX
mrxhH+tWu9RVwsDegRG0s28OMdVeOwljZvYrUjRomutNO5GzynveVxJVCn3Cbn7a
c4L+5vwPNgs04DdOAGzNYdG5t6ryyYPosSLH2B8qDNzxAgMBAAGjggFwMIIBbDAf
BgNVHSMEGDAWgBRTeb9aqitKz1SA4dibwJ3ysgNmyzAdBgNVHQ4EFgQU70wAkqb7
di5eleLJX4cbGdVN4tkwDgYDVR0PAQH/BAQDAgGGMBIGA1UdEwEB/wQIMAYBAf8C
AQAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMCIGA1UdIAQbMBkwDQYL
KwYBBAGyMQECAmcwCAYGZ4EMAQICMFAGA1UdHwRJMEcwRaBDoEGGP2h0dHA6Ly9j
cmwudXNlcnRydXN0LmNvbS9VU0VSVHJ1c3RSU0FDZXJ0aWZpY2F0aW9uQXV0aG9y
aXR5LmNybDBxBggrBgEFBQcBAQRlMGMwOgYIKwYBBQUHMAKGLmh0dHA6Ly9jcnQu
dXNlcnRydXN0LmNvbS9VU0VSVHJ1c3RSU0FBQUFDQS5jcnQwJQYIKwYBBQUHMAGG
GWh0dHA6Ly9vY3NwLnVzZXJ0cnVzdC5jb20wDQYJKoZIhvcNAQEMBQADggIBACaA
DTTkHq4ivq8+puKE+ca3JbH32y+odcJqgqzDts5bgsapBswRYypjmXLel11Q2U6w
rySldlIjBRDZ8Ah8NOs85A6MKJQLaU9qHzRyG6w2UQTzRwx2seY30Mks3ZdIe9rj
s5rEYliIOh9Dwy8wUTJxXzmYf/A1Gkp4JJp0xIhCVR1gCSOX5JW6185kwid242bs
Lm0vCQBAA/rQgxvLpItZhC9US/r33lgtX/cYFzB4jGOd+Xs2sEAUlGyu8grLohYh
kgWN6hqyoFdOpmrl8yu7CSGV7gmVQf9viwVBDIKm+2zLDo/nhRkk8xA0Bb1BqPzy
bPESSVh4y5rZ5bzB4Lo2YN061HV9+HDnnIDBffNIicACdv4JGyGfpbS6xsi3UCN1
5ypaG43PJqQ0UnBQDuR60io1ApeSNkYhkaHQ9Tk/0C4A+EM3MW/KFuU53eHLVlX9
ss1iG2AJfVktaZ2l/SbY7py8JUYMkL/jqZBRjNkD6srsmpJ6utUMmAlt7m1+cTX8
6/VEBc5Dp9VfuD6hNbNKDSg7YxyEVaBqBEtN5dppj4xSiCrs6LxLHnNo3rG8VJRf
NVQdgFbMb7dOIBokklzfmU69lS0kgyz2mZMJmW2G/hhEdddJWHh3FcLi2MaeYiOV
RFrLHtJvXEdf2aEaZ0LOb2Xo3zO6BJvjXldv2woN
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFgTCCBGmgAwIBAgIQOXJEOvkit1HX02wQ3TE1lTANBgkqhkiG9w0BAQwFADB7
MQswCQYDVQQGEwJHQjEbMBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYD
VQQHDAdTYWxmb3JkMRowGAYDVQQKDBFDb21vZG8gQ0EgTGltaXRlZDEhMB8GA1UE
AwwYQUFBIENlcnRpZmljYXRlIFNlcnZpY2VzMB4XDTE5MDMxMjAwMDAwMFoXDTI4
MTIzMTIzNTk1OVowgYgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpOZXcgSmVyc2V5
MRQwEgYDVQQHEwtKZXJzZXkgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBO
ZXR3b3JrMS4wLAYDVQQDEyVVU0VSVHJ1c3QgUlNBIENlcnRpZmljYXRpb24gQXV0
aG9yaXR5MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAgBJlFzYOw9sI
s9CsVw127c0n00ytUINh4qogTQktZAnczomfzD2p7PbPwdzx07HWezcoEStH2jnG
vDoZtF+mvX2do2NCtnbyqTsrkfjib9DsFiCQCT7i6HTJGLSR1GJk23+jBvGIGGqQ
Ijy8/hPwhxR79uQfjtTkUcYRZ0YIUcuGFFQ/vDP+fmyc/xadGL1RjjWmp2bIcmfb
IWax1Jt4A8BQOujM8Ny8nkz+rwWWNR9XWrf/zvk9tyy29lTdyOcSOk2uTIq3XJq0
tyA9yn8iNK5+O2hmAUTnAU5GU5szYPeUvlM3kHND8zLDU+/bqv50TmnHa4xgk97E
xwzf4TKuzJM7UXiVZ4vuPVb+DNBpDxsP8yUmazNt925H+nND5X4OpWaxKXwyhGNV
icQNwZNUMBkTrNN9N6frXTpsNVzbQdcS2qlJC9/YgIoJk2KOtWbPJYjNhLixP6Q5
D9kCnusSTJV882sFqV4Wg8y4Z+LoE53MW4LTTLPtW//e5XOsIzstAL81VXQJSdhJ
WBp/kjbmUZIO8yZ9HE0XvMnsQybQv0FfQKlERPSZ51eHnlAfV1SoPv10Yy+xUGUJ
5lhCLkMaTLTwJUdZ+gQek9QmRkpQgbLevni3/GcV4clXhB4PY9bpYrrWX1Uu6lzG
KAgEJTm4Diup8kyXHAc/DVL17e8vgg8CAwEAAaOB8jCB7zAfBgNVHSMEGDAWgBSg
EQojPpbxB+zirynvgqV/0DCktDAdBgNVHQ4EFgQUU3m/WqorSs9UgOHYm8Cd8rID
ZsswDgYDVR0PAQH/BAQDAgGGMA8GA1UdEwEB/wQFMAMBAf8wEQYDVR0gBAowCDAG
BgRVHSAAMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29t
L0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDQGCCsGAQUFBwEBBCgwJjAkBggr
BgEFBQcwAYYYaHR0cDovL29jc3AuY29tb2RvY2EuY29tMA0GCSqGSIb3DQEBDAUA
A4IBAQAYh1HcdCE9nIrgJ7cz0C7M7PDmy14R3iJvm3WOnnL+5Nb+qh+cli3vA0p+
rvSNb3I8QzvAP+u431yqqcau8vzY7qN7Q/aGNnwU4M309z/+3ri0ivCRlv79Q2R+
/czSAaF9ffgZGclCKxO/WIu6pKJmBHaIkU4MiRTOok3JMrO66BQavHHxW/BBC5gA
CiIDEOUMsfnNkjcZ7Tvx5Dq2+UUTJnWvu6rvP3t3O9LEApE9GQDTF1w52z97GA1F
zZOFli9d31kWTz9RvdVFGD/tSo7oBmF0Ixa1DVBzJ0RHfxBdiSprhTEUxOipakyA
vGp4z7h/jnZymQyd/teRCBaho1+V
-----END CERTIFICATE-----

The two certs are no chain including the root ca.

First cert:

subject= /C=US/O=Internet2/CN=InCommon RSA Server CA 2
issuer= /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority

Second cert:

subject= /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
issuer= /C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services

You must include the AAA certificate to have a full chain including the root ca. If you download the full chain from the sectigo certificate manager you’ll have the AAA certificate included. The certificate manager has two options: one to download without the root and one with the root.

If I check the information on any ssl certificate in the certificate manager it’ll always shows me AAA as root.

If you download the full chain it’ll work.

1 Like

Re: “if you download the full chain it will work” - I know, but that’s not an option as it would require manual intervention. My proposed workaround of adding the AAA cert is effectively the same thing. Sectigo’s ACME protocol only gives you what I provided. And it turns out, they are right; the chain doesn’t need to include the self-signed root (and in the case of cross-signing, it shouldn’t).

I take back what I said earlier about ignoring the local keystore being a good decision. It’s a violation of RFC 5246. Maybe allow this behavior to be turned on or off with a command line switch would help?

RFC 5246 explicitly states “the self-signed certificate that specifies the root certificate authority MAY be omitted from the chain, under the assumption that the remote end must already possess it in order to validate it in any case” In fact, I just checked community.theforeman.org, and the cert chain also does not include the self-signed root cert.

So the Sectigo cert chain is absolutely RFC 5246-compliant and should validate. What’s more, including the self-signed roots would actually break some scenarios and could even cause a cert to validate when it should be untrusted. So even my proposed workaround of adding the AAA certificate is actually not a good idea.

Here’s what’s happening. The second of these certs is actually the root certificate. Well, it kinda is and kinda is not. It’s not because it’s not self signed, but it kinda is because it (or rather a “clone”) is generally included in system trust stores.

With this chain, there are really two self-signed certs, either of which would fully validate the chain. That would break if one or both of hte self-signed certs were included in the chain.

Here is a link with more information about the cross-signing and multiple roots:

The reason the second cert (USERTrust RSA Certification Authority) kinda is the root certificate is that it has a clone. The same cert (i.e., same CN, same public key, same expiration date, same extensions etc) exists twice. One is self-signed (https://crt.sh/?id=1199354), the other signed by AAA (crt.sh | 1282303295)

This cross-signing is done to support modern browsers/clients (which do have the USERTrust RSA Certification Authority in their trust store) as well as older browsers/clients (which only have the AAA CA).

A modern client will try to validate the first cert against the USERtrust in its own keystore, and disregard the one in the chain. The chain cannot include the self-signed USERtrust because having two certs with the same CN would be a conflict. And including the AAA may cause the client to validate against AAA instead of against USERtrust.

An old client will validate the first cert against the USERTrust, and then the USERtrust against AAA in the key store.

Obviously, both break when you are not consulting a keystore at all.

So;

  • The Sectigo chain is perfectly legit as-is according to RFCs
  • The Sectigo chain is cross-signed, and can’t properly include a self-signed root

I don’t know what the issue with cert pinning is, but would love to help figure out a solution for these conflicting requirements.

It doesn’t matter, what the RFC says. The ca file passed to katello-certs-check and foreman-installer it used for a variety of purposes, externally facing and internally. As it’s also used for certificate pinning at some points it requires to pass the full chain. That’s fully regardless of how it may be used somewhere else.

It’s a necessity to make sure that the ca file and certificate are viable for certificate pinning. It’s also necessary to make sure that you can use your own custom ca or the foreman self-generated ca and it can verify those.

The use of certificates in foreman goes far beyond the standard usage case of public tls servers… It’s all complies but for the purpose of foreman-installer configuration it requires the full chain…

Yes. It’s a public tls servers where clients are browsers which have a pre-installed root ca for convenience that people don’t have to bother whom to trust.

It’s not about how foreman proxies talk to each other or to the main server. It’s not about how the puppetserver sends reports to the main server. It’s not about how pulpcore exchanges information between the servers.

It does for public tls purposes. Still, foreman-installer needs to know the root ca. It’s just what it needs to use it properly and propagate the correct certificates to the various places they are needed…

No it’s not a root ca certificate by definition because it is not self signed.

No. The self signed version of the usertrust certificate is often included. It’s a different certificate from your second one, even if the subject name and public key are the same.

Again: those are two very different certificates only sharing the subject name and public key. Dates, exensions etc. are different. The older one is self signed and thus a root ca certificate. The other one is not self signed and thus an intermediate (cross-signed) certificate.

Yes and no. It has absolutely nothing to do with cross-signing. Including or not including a root has nothing to do with cross-signing.

And it doesn’t matter: katello-certs-check requires the full chain to make sure the full chain is working. foreman-installer needs the full chain to distribute the correct certificates to the right places. That has nothing to do with what public tls servers should offer and what client browsers expect. The part of the foreman gui is only a small part where those certificates are used.

You cannot really use ACME anyway, because you have to run foreman-installer each time the certificate, the chain or the private key changes. But that is something else.

You have to pass the full chain to foreman-installer. That’s how it is. It’s used internally. There is really little point arguing about that.

Then for starters it should be made clear in the documentation that the katello/foreman may not accept all commercially issued certs/chains. If it was at least documented, that may help lessen the pain.

Also, katello-certs-check should allow (or even require) passing the root cert separately from the chain.

I understand that. The issue here is that solving one problem breaks something else.

No. The only time the self-signed usertrust could be included is if you remove the cross-signed one. You can of course manipulate the chain to do that.

ACME is an absolute must-have. Today, it’s 90 days. Sectigo already offers 30 day certificates (not yet mandatory, but 45 days are coming by 2027). Apple has even floated 10 day expiration times.

Running foreman-installer every time is at worst annoying, but not a real problem; ACME clients have hooks for that exact situation.

I don’t know what makes you think that. It supports any cert chain. We are using sectigo certs. They work just fine. All you have to do is to provide the full chain including the self-signed root ca. That’s it. We have been using it like this for a while now.

You can open a bug if you think so. I don’t really see the point though. And foreman-installer is another thing, which would be more important.

I don’t understand what it breaks. Provide the full chain and all is good.

That’s just not possible at the moment. It’s not only about the main server, but also about the proxies. That cannot be automated at the moment.

Check the custom certificate deployment docs

You have to run foreman-proxy-certs-generate on the main server, then copy the tar, then run foreman-installer on the proxy.

In addition, foreman server and smart proxies must share the same issuer CA. Thus, if your ACME update changes the issuing CA it’ll all falls apart.

Again: as it is used in so many internal places, it’s not a simple TLS server certificate we are talking about…

Open a bug if you want, then maybe developers can come up with something, how to properly deal with short-lived ACME certs…

In the meantime, simply paste the root ca certificate (e.g. from /etc/pki/ca-trust/extracted/pem/directory-hash/CN_AAA_Certificate_Services_O_Comodo_CA_Limited_L_Salford_ST_Greater_Manchester_C_GB.pem) to the end of your chain and it’ll work.

See above. That would require manual intervention.

I think we are losing sight of the big picture here. Just because a few use cases are hard to automate doesn’t mean that the process can’t be automated for the many other use cases.

And it needs intervention anyway because you have to run foreman-installer…

It’s just not made for this at the moment. You don’t really want to automate this as too much could go wrong and then you have a non-operable system and you have to find out what went wrong and what else you need to fix it.

So it’s not “a few cases”. It’s all cases. With every update it may break.

It is what it is right now. Check the other posts. You’ll find quite a few where some components of foreman don’t work together anymore because of ssl verify errors…

foreman-installer can be run automatically just fine. On that, i have to compliment the developers. Having to run the installer for a lot of maintenance tasks is annoying, but it does work well, even automatically. That’s why the one place where it does break, the CA chain, is so aggravating.

Of course automation can go wrong. That’s also true for auto-updating nginx and apache etc., too. That’s not an excuse for having to babysit it every few weeks.

Again: that’s only the update on the main foreman server if the ca chain does not change. It doesn’t work with smart proxies. And if you think it’s so simple, I don’t quite understand what’s so hard to add the root ca certificate to the chain. That would be an extremely simple script.

Or simply manually create a ca chain file with all ca certs and use that instead of the file from letsencrypt. Then you pass only the letsencrypt cert file and key to foreman-installer. Then it’s a static ca file and if the ca chain changes it breaks anyway, thus, you could just as well use a static file. That has the advantage that katello-certs-check immediately notifies you if the ca chain doesn’t fit anymore instead of running foreman-installer with a full new set breaking it in placing you don’t even know about.

IIRC from another thread there is currently the issue that the latest foreman-installer copies the cert, key and ca chain files passed as parameters instead of copying the content as before. That breaks it if you are using certbot because that puts in symlinks with relative paths and copying symlinks will break it.

So don’t use short-lived acme certs. We always order 1 year certificates from sectigo… Foreman is an essential service for us. We cannot afford it breaks by some automatic certificate update in the background. That is why we do the certificate renewal manually and install it into foreman manually, so that we can immediately verify everything is still working as it should.

Either way: this discussion won’t lead anywhere else. It is what it is. I am no developer. If you want something to change you have to open a redmine issue and maybe link this thread to it. (Another reason why it would have been so much better if you had just opened a new thread for this topic…)

Maybe someone comes up with an idea how to handle this properly. I expect automated certificate updates to become much more common in the future.