Hey, @mcorr
We are still working on the ab above puppet CA config error. We have also tried by removing the old ssl and generating the new ssl key, still getting the same error.
1 Like
@mcorr Yes we have created new installation media and config the image repository path. we have added the installation media on the host as well
1 Like
Have you followed the steps mentioned in the Error wiki:
See http://theforeman.org/manuals/latest/index.html#4.3.2SmartProxySettings, scroll down a little for the Puppet CA configuration and the sudoers rules are listed. These should be in /etc/sudoers.d/foreman-proxy and the file should have -r–r----- (0440) permissions.
2 Likes
@mcorr Yes, we tried but still getting the same error. we also checked on the log file and added here, please have a look.
2021-04-29T07:47:59 5a0f4ae0 [I] Finished DELETE /puppet/ca/foreman-katello.us-central1-a.c.socios-linux.internal with 406 (65.38 ms)
2021-04-29T07:47:59 5a0f4ae0 [I] Started POST /puppet/ca/autosign/foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:47:59 5a0f4ae0 [I] Finished POST /puppet/ca/autosign/foreman-katello.us-central1-a.c.socios-linux.internal with 200 (5.11 ms)
2021-04-29T07:54:02 9cda62ad [I] Started DELETE /puppet/ca/autosign/foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:02 9cda62ad [I] Finished DELETE /puppet/ca/autosign/foreman-katello.us-central1-a.c.socios-linux.internal with 200 (0.7 ms)
2021-04-29T07:54:09 73401120 [I] Started GET /v2/features
2021-04-29T07:54:09 73401120 [I] Finished GET /v2/features with 200 (39.71 ms)
2021-04-29T07:54:10 0317b069 [I] Started DELETE /ssh/known_hosts/10.128.0.18
2021-04-29T07:54:10 0317b069 [I] Finished DELETE /ssh/known_hosts/10.128.0.18 with 204 (0.71 ms)
2021-04-29T07:54:10 0317b069 [I] Started DELETE /ssh/known_hosts/foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:10 0317b069 [I] Finished DELETE /ssh/known_hosts/foreman-katello.us-central1-a.c.socios-linux.internal with 204 (0.91 ms)
2021-04-29T07:54:10 0317b069 [I] Started DELETE /puppet/ca/autosign/foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:10 0317b069 [E] Attempt to remove nonexistent client autosign for foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:10 0317b069 [W] Error details for Attempt to remove nonexistent client autosign for foreman-katello.us-central1-a.c.socios-linux.internal: : Attempt to remove nonexiste
nt client autosign for foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:10 0317b069 [W] Attempt to remove nonexistent client autosign for foreman-katello.us-central1-a.c.socios-linux.internal: : Attempt to remove nonexistent client autosign
for foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:10 0317b069 [I] Finished DELETE /puppet/ca/autosign/foreman-katello.us-central1-a.c.socios-linux.internal with 404 (0.98 ms)
2021-04-29T07:54:10 0317b069 [I] Started DELETE /puppet/ca/foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:10 0317b069 [E] Failed to remove certificate(s) for foreman-katello.us-central1-a.c.socios-linux.internal: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unkn
own
2021-04-29T07:54:10 0317b069 [W] Error details for Failed to remove certificate(s) for foreman-katello.us-central1-a.c.socios-linux.internal: SSL_connect returned=1 errno=0 state=error: sslv3 aler
t certificate unknown: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown
1 Like
How much work would do you estimate it would be to help you connect these dots and make this functionality real?
HBAIT is working with me on this and while I am ok with work arounds, I’m also not opposed to helping the community if it’s not going to kill us 
After all… you all have been very helpful to us!
~michael
1 Like
Problem is, this is still pretty early in development in both Anaconda (RHEL/Fedora) and Image Builder. There are known issues and quirks, that’s why I kept this chapter pretty slim - we actually do not dive into this as this is all changing (just a week ago another bug was found and folks are working on it).
So for now, I think linking RHEL docs is way to go. We can extend this later on.
1 Like
If you mean the Puppet CA removal error, I am not sure about this one. I suppose this works out of the box if you used our installer? Any idea why CA removal request would fail with X509 error @ekohl?
To your question - it is really hard to give a good advice without pretty much any context. This thread is lacking on the input front, the OP did not care to even follow our support post template where we ask very basic things like Foreman and OS version. And that is just a beginning, we need to know much more - how it was installed, if custom certificate were attempted to used etc etc.
1 Like
- We are trying to create an ISO image based on the custom dependency packages using the cockpit image builder.
- we are referring to this tutorial link “Building Images & Deploying With Foreman” to create the ISO image.
- Currently, We have completed building the ISO image using the cockpit and we are trying to provisioning the host using the foreman server.
- On the foreman we have created the products and added uploaded the file to the repositories. We have also created a new installation media based on the created products, configured it with the operating system, and created the new host.
- While deploying the host we are getting the Puppet CA removal error (“Screenshot by Lightshot”) and we are not able to initiate the build process.
1 Like
@ekohl do you have an idea why Puppet CA would fail to remove CA during provisioning? I mean, isn’t this an action that should happen during deletion of a host?
1 Like
What exact Foreman and Foreman Proxy do you use? This is not Puppet CA response, but rather a Foreman (smart) proxy response. Was this instance installed with the foreman-installer? What is your Puppet CA version? Could you please enable Foreman proxy debug log level, restart it and then upload smart-proxy.log from when you observe the failure? I vaguely remember that e.g. a wrong version in /etc/foreman-proxy/settings.d/puppetca.yml could have caused that. Try investigating /etc/foreman-proxy/settings.d/puppetca*.yml if you see anything goofy. Thanks
3 Likes
What @Marek_Hulan said is correct. This error shows up when Foreman Proxy tries to connect to PuppetCA. It only does that if it’s Puppet 6 or newer so you should be looking at /etc/foreman-proxy/settings.d/puppetca_http_api.yml. There the puppet_ssl_ca file is pointing to a file that’s used to validate the connection (as specified in puppet_url). So the easiest way to replicate it is curl --cacert $puppet_ssl_ca $puppet_url. That must succeed.
2 Likes
Hello @Marek_Hulan / @ekohl,
Thanks for the time.,
We are using the foreman-2.3.3-1.el7.noarch and foreman-proxy-2.3.3-1.el7.noarch version.
Yes, we have this instance installed with the foreman-insatller.
PuppetCA version is also 2.3.3.
We have mentioned all the versions and also added the puppet ca debug log please have a look on the above screenshot.
We will also try to investigate the puppet ca YAML file and we will also start looking on the puppet_ssl_ca file which is pointing to a file that’s used to validate the connection.
we will update and posted here. Thanks!
1 Like
That sounds like the Foreman Proxy module version. Puppet CA is part of the puppetserver package.
2 Likes
Hello @ekohl / @Marek_Hulan,
As mentioned feedback we checked on the connection path /etc/foreman-proxy/settings.d/puppetca_http_api.yml, and we found this connection path on the YAML file.
- Based on the connection path we found the .pem key for Cert, public and private keys.
- we tried to restart the puppet service still getting the same certificate got revoked error.
- we have also tried to remove the ca certificate from the SSL path and generate a new one but still getting the host cert error.
Please find the above screenshot for reference and Let us know any pointers/feedback regarding this. Thanks!
1 Like
yes, definitely want to be able to have the workflow provision generic nodes too and not just RHvirt nodes. my goal now is to use the foreman / katello servers to facilitate the production of an evolving silverblue image
Overtime we’ll want to have several different ostree image builds based on hardware configurations and some other stuff; but this is sorta the direction we hope to go
1 Like
Hey @ekohl
Is this anything obvious to you? When I search the error messages back through the post history, I have never found a post with a marked solution.
1 Like
Thanks for the information. I’d first try to
curl https://foreman-katello.us-central1-a.c.socios-linux.internal:8140/puppet-ca/v1/certificate_revocation_list/ca --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem
post what it does. If you get SSL error, that means Puppet CA is using a certificate that was not signed by the expected CA. If you get the the CRL list back (starting with -----BEGIN X509 CRL-----) you can investigate it like this
curl https://foreman-katello.us-central1-a.c.socios-linux.internal:8140/puppet-ca/v1/certificate_revocation_list/ca --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem | openssl crl -text -noout
That gives you the list of revocations, perhaps the dates will give you some information about how the certificate could get to the revocation list. You can also take a look at puppetserver ca list --all.
If it is revoked, you can’t do much more than create a new one. puppetserver generate --ca-client --certname ... is probably something you’ll need. Before you do that, make sure to read https://puppet.com/docs/puppet/7/puppet_server_ca_cli.html and making a backup. You’ll need to do this while puppet server is offline.
Based on the initial comments, is this a new setup? Can you just start from scratch? It seems there must have been some explicit action taken that “broke” it. Looking at the log you’ve provided, there’s
2021-04-29T07:54:10 0317b069 [I] Finished DELETE /puppet/ca/autosign/foreman-katello.us-central1-a.c.socios-linux.internal with 404 (0.98 ms)
2021-04-29T07:54:10 0317b069 [I] Started DELETE /puppet/ca/foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:10 0317b069 [E] Failed to remove certificate(s) for foreman-katello.us-central1-a.c.socios-linux.internal: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unkn
own
That seems like Foreman ask to revoke the certificate it uses itself. Did you try to provision/cancel provisioning of a host, that you run Foreman on? That would explain it.
1 Like
Hello @Marek_Hulan
As you suggested we investigate these below comment “curl https://foreman-katello.us-central1-a.c.socios-linux.internal:8140/puppet-ca/v1/certificate_revocation_list/ca --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem” and “curl https://foreman-katello.us-central1-a.c.socios-linux.internal:8140/puppet-ca/v1/certificate_revocation_list/ca --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem | openssl crl -text -noout” we are found that there is not puppet CA is not avaliable on the puppet servers.
We followed this document “https://puppet.com/docs/puppet/7/puppet_server_ca_cli.html” and we have removed the puppet service and reinstalled it again. once we did that we can generate a new CA for the puppet services.
Currently, we are working on using an upstream URL to host this build iso image. we will keep posted here. Thanks!
1 Like