Delete_orphaned_content error pulp3: SSL_connect

Problem:
Delete orphaned content tasks gives an SSL error, all other tasks complete as expected. The only task experiencing the SSL error appears to be the delete orphaned content task.

Expected outcome:
Successful task completion

Foreman and Proxy versions:
Foreman 3.11.2
Foreman and Proxy plugin versions:
Foreman 3.11.2
Katello 4.13.1

Distribution and version:
RHEL 8

Other relevant data:
task output:

There was an issue with the backend service pulp3: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)There was an issue with the backend service pulp3: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)

output from cert check:

katello-certs-check -c /etc/foreman-proxy/ssl_cert.pem -k /etc/foreman-proxy/ssl_key.pem -b /etc/foreman-proxy/ssl_ca.pem
Checking server certificate encoding:
[OK]

Checking expiration of certificate:
[OK]

Checking expiration of CA bundle:
[OK]

Checking if server certificate has CA:TRUE flag
[OK]

Checking for private key passphrase:
[OK]

Checking to see if the private key matches the certificate:
[OK]

Checking CA bundle against the certificate file:
[OK]

Checking CA bundle size: 1
[OK]

Checking if CA bundle has trust rules: 0
[OK]

Checking Subject Alt Name on certificate
[OK]

Checking if any Subject Alt Name on certificate matches the Subject CN
[OK]

Checking Key Usage extension on certificate for Key Encipherment
[OK]

Checking for use of shortname as CN
[OK]

Validation succeeded

Looking at the task history it appears this task has failed since we upgraded to foreman 3.8 several months ago, but this went unnoticed.

Hello

Has anyone with more knowledge been able to look at this?
The cleanup task is the only one that fails and all cert checks seem to come back ok.

Sincerely

You might be running into this bug (37817) currently in 3.11.2.

There is supposed to be a 3.11.4 release this week that addresses it. I am waiting on that before proceeding with 3.11 upgrades…

Thank you for you input. I took a look at the issue you linked and tried the workaround. Sadly this did not help with my issue. I still receive the same error on the Remove orphans task.

Has anyone from pulp/katello team been able to take a quick look at this?
We are still experiencing this issue where seemingly only the remove orphans task fails on SSL.

@katello

Could someone take a look at this or tag the right person to provide support on this issue?
We are still only getting the certificate error on the remove orphans tasks.

Sincerely

You have to give more information about your setup. It’s very confusing as you shortened everything down to the bare minimum and leave most out of context.

From what you write I can only guess that you mean the cron job that runs foreman-rake katello:delete_orphaned_content. That starts one or more “Remove orphans” tasks depending on the number of foreman servers/proxies you have (which you don’t say).

For some reason you also post the output of katello-certs-check -c /etc/foreman-proxy/ssl_cert.pem -k /etc/foreman-proxy/ssl_key.pem -b /etc/foreman-proxy/ssl_ca.pem which checks the foreman proxy cert, even though the foreman proxy isn’t involved here. That makes be think you are using custom certificates, which you don’t really write nor explain how exactly you have configured the custom certs. Very often, people install custom certs not following the docs, setting various foreman-installer options or even manipulating foreman config files, which eventually breaks something.

So please post what you do exactly. Please post the extract for the RemoveOrphans task in /var/log/foreman/production.log. It should have the exception for this error.

Explain how exactly you have configured your custom certs, if you did.

2 Likes

Details would be good like @gvde said. It would be good to get an idea about what connection is failing. Is it Katello reaching out to Pulp on the smart proxy that is failing? Or perhaps Pulp on the smart proxy reaching out to Pulp on Katello?

For debugging, the connection from Katello to the smart proxy’s Pulp can be tested by feeding the certificates into curl. You can use the /pulp/api/v3/repositories/rpm/rpm/ endpoint on the smart proxy for a test. If that fails with a cert error, it can narrow down the problem.

1 Like

@gvde
No custom certs are used, I provided that specific cert check as it was referenced in a different thread for a different cert issue and I thought it might provide relevant information.

Here is the output from the production log after manually triggering a orphan cleanup:
foreman-rake katello:delete_orphaned_content

@iballou
I tried to curl the endpoint and received ca errors. After downloading the CA from the page itself through a browser to ensure I am providing the correct ca to curl I still received the same error.

 Establish HTTP proxy tunnel to foremanprod02.servers.be:443
> CONNECT foremanprod02.servers.be:443 HTTP/1.1
> Host: foremanprod02.servers.vrt.be:443
> User-Agent: curl/7.61.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
< Proxy-Agent: Fortinet-Proxy/1.0
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: ./downloadedca.pem
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

You have to write exactly how you have called curl and which CA certificates you have used. Where did you download the certificate?

Foreman should use the ca certificate in /etc/foreman/proxy_ca.pem:

# curl --capath : --cacert /etc/foreman/proxy_ca.pem -v https://foreman8.example.com/pulp/api/v3/repositories/rpm/rpm/

That should be the same file as /etc/foreman-proxy/foreman_ssl_ca.pem.

I verified and can confirm that the CA under /etc/foreman/proxy_ca.pem is the same as /etc/foreman-proxy/foreman_ssl_ca.pem.

The curl connected successfully using those.

Below the curl command and output using one of those certs.

[root@foremanprod02 ~]# curl https://foremanprod02.servers.vrt.be/pulp/api/v3/repositories/rpm/rpm/ --noproxy '*' --cacert /etc/foreman/proxy_ca.pem -v
*   Trying 10.27.4.63...
* TCP_NODELAY set
* Connected to foremanprod02.servers.vrt.be (10.27.4.63) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/foreman/proxy_ca.pem
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=North Carolina; O=FOREMAN; OU=SMART_PROXY; CN=foremanprod02.servers.vrt.be
*  start date: Jun 16 13:05:30 2021 GMT
*  expire date: Jan 18 13:05:30 2038 GMT
*  subjectAltName: host "foremanprod02.servers.vrt.be" matched cert's "foremanprod02.servers.vrt.be"
*  issuer: C=US; ST=North Carolina; L=Raleigh; O=Katello; OU=SomeOrgUnit; CN=foremanprod02.servers.vrt.be
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* Using Stream ID: 1 (easy handle 0x56307d1aa710)
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /pulp/api/v3/repositories/rpm/rpm/ HTTP/2
> Host: foremanprod02.servers.vrt.be
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/2 403
< date: Thu, 05 Dec 2024 08:00:59 GMT
< server: gunicorn
< content-type: application/json
< vary: Accept,Cookie
< allow: GET, POST, HEAD, OPTIONS
< x-frame-options: DENY
< content-length: 58
< x-content-type-options: nosniff
< referrer-policy: same-origin
< cross-origin-opener-policy: same-origin
< correlation-id: 048ceebce00f46149ad02cff4ab451f0
< access-control-expose-headers: Correlation-ID
< via: 2.0 foremanprod02.servers.vrt.be
<
* Connection #0 to host foremanprod02.servers.vrt.be left intact
{"detail":"Authentication credentials were not provided."}[root@foremanprod02 ~]#
1 Like

You’ll also need to pass in the certificate and the key along with the CA. Without those, you’ll always get the "Authentication credentials were not provided." error.

If that is failing, I wonder if the smart proxy certificate deploying steps need to be re-run: Installing a Smart Proxy Server nightly on Enterprise Linux

@ekohl / @evgeni redoing that process shouldn’t break anything, right?

As long as you use the same certs tarball, yeah, that’s fine.

Do these same steps apply in case of foreman-katello setup on a single server without using a separate machine to deploy a smart proxy? As currently we do not have a tarball, we have certificates under /root/ssl-build/.

Should we re-run the foreman installer supplying the certs as followed?:

foreman-installer --scenario katello  --certs-server-cert "/root/ssl-build/foremanprod02.servers.vrt.be/foremanprod02.servers.vrt.be-foreman-proxy.crt" --certs-server-key "/root/ssl-build/foremanprod02.servers.vrt.be/foremanprod02.servers.vrt.be-foreman-proxy.key"  --certs-server-ca-cert "/root/ssl-build/katello-server-ca.crt" --certs-update-server --certs-update-server-ca

Ah, I didn’t realize this was a problem with your internal smart proxy.

Passing in the certs arguments to the foreman-installer would mean you’re starting to use custom certificates, so if you don’t mean to use custom certs, I would avoid it.

Have any other modifications been made to the configuration of the certificates on your machine? I’m curious what installer options you have set on your setup.

I’m guessing due to the error you can’t sync any repositories either?

Syncing repositories gives no issues. We only encounter this issue on the delete orphaned content tasks.

We have made no modifications on the certificates since installing foreman, we did upgrade since version 2.5 if I recall correctly, not sure if this is relevant information for the certs state.

Below the installer command initially run:

foreman-installer --scenario katello \
--foreman-initial-organization "ourORG" \
--foreman-initial-location "ourLOC" \
--foreman-initial-admin-username admin \
--foreman-initial-admin-password Password
1 Like

@iballou
Quickly tagging you to see if you had a chance to look at my previous reply.

Just quickly bumping so this does not get forgotten.
@iballou

Apologies for the delay.

I’m a bit confused because Katello uses the same certificates to talk to Pulp for repo syncing and for triggering orphan cleanup.

Does the following work?

curl -X POST https://`hostname`/pulp/api/v3/orphans/cleanup/ --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem

If that works (task href returned with no error if you query it), then I wonder if the SSL error isn’t coming from where we think it is (the internal smart proxy).

In that case, we’d need to see the Dynflow output of the failing orphan cleanup action. You can find that on the tasks page if you click on the failing task and then hit the Dynflow Console button. When you find the failing action within, all info (input, output, etc) for the failing action would be helpful.

Just to be sure – you have only one smart proxy?

Thank you for your reply.

Yes we currently only have a single server running the internal smart proxy.

The POST you asked to try worked:
{"task":"/pulp/api/v3/tasks/019488c7-8ebf-7a5e-9edc-5e4831268d97/"}[

Below the dynflow output of the most recent failed cleanup task: