luna-nightly-rpm-pipeline 275 failed

Luna nightly pipeline failed:

https://ci.theforeman.org/job/luna-nightly-rpm-pipeline/275/

foreman-pipeline-luna-nightly-centos8-upgrade (failed) (remote job)
foreman-pipeline-luna-nightly-centos8-install (failed) (remote job)
foreman-pipeline-luna-nightly-centos8-stream-install (failed) (remote job)
foreman-pipeline-luna-nightly-centos7-install (failed) (remote job)
foreman-pipeline-luna-nightly-centos7-upgrade (failed) (remote job)

I started to look into this. It fails on REX.

In /var/log/foreman-proxy/proxy.log I couldn’t find any error. Then in the Foreman UI I saw an error: SSL certificate with unexpected serial supplied. The internet suggested this specific string is only defined in SmartProxyDynflowCore, namely here:

This is probably fallout from the merge to SmartProxyDynflowCore into the SmartProxy process. It passes on the regular pipelines so it must be something with the other certificates used in Katello.

Looking at the particular code, it does look odd that the serial is compared. I’d expect it to verify the CN, DN or SAN.

Digging deeper, it’s defined here:

I suspect this assumption is no longer true.

cc @aruzicka

Even though we deploy SPDC in the smart proxy process by default, people can still run it as a separate process where the assumption still holds.

Debian was running with this for ages, so it most likely is something that katello does with certs. I have a PR ready which moves code from smart proxy dynflow core gem to smart proxy dynflow, which also gets rid of the serial check in the process. As a temporary solution until that one gets in, let’s go with this one Fixes #32513 - Check certificate serial only when in standalone mode by adamruzicka · Pull Request #79 · theforeman/smart_proxy_dynflow · GitHub

Smart proxy dynflow core 0.3.3 containing the fix was just released and packaging PRs are opened