Puppet SSH error and Provisioning host

[HBAIT]

Hi, I’m Working on creating a custom ISO image based on cockpit and foreman composer.
I have followed all the steps regarding this tutorial link “Building Images & Deploying With Foreman”. I’m getting this below error while initiating the build in the foreman.

Screenshot

And also I need to know how to use the provisioning host to build a new VM instance based on the ISO image product repository?

[lzap]

Ad error - google found me this:

ERF12-7740 - Foreman

Ad the question: I do not understand what you ask for. But there is currently no Image Builder integration, you need to build images yourself, publish them on a HTTP server and just tell Foreman where to download them. We plan working on this someday, no promises tho:

Using Image Composer (Image builder) with Foreman

Hello, Foreman does ship with an parameters which use “liveimg” kickstart statement to install OS from an existing image. This is used for Red Hat Virtualization node (or oVirt Hypervisor), but it is great workflow that can be used to provision generic nodes too. While we would like to focus on this sometimes in the future, there are few workarounds you need to do if you want to explore this kind of provisioning. Terry from the RHEL team shared with me those workarounds, so I thought it would …

[mcorr]

Hey @HBAIT did you get any further with this?

[HBAIT]

Thanks, @lzap.
As I mentioned above I’m working on provisioning the host using the ISO image based on the tutorial link.
I have done creating the IOS image using the cockpit composer as they mentioned in the tutorial and downloaded the image. By using that image I have created a product repository and uploaded the ISO file to the product repository.

1. What we want to know is, can we use the Image-based provisioning on the foreman?
2. As you (@lzap) suggested in the tutorial “if there was a way to specify the upstream URL for an image just like an upstream repository for packages, this would allow using Katello/Pulp sync feature” can’t we use the repository without the upstream URL right?
3. Once we added the product repository and the template on the host how do we initiate the booting process on the foreman?

[mcorr]

@HBAIT

As you (@lzap) suggested in the tutorial “if there was a way to specify the upstream URL for an image just like an upstream repository for packages, this would allow using Katello/Pulp sync feature” can’t we use the repository without the upstream URL right?

So you completed steps that look a bit like this: https://docs.theforeman.org/nightly/Content_Management_Guide/index-katello.html#importing_individual_iso_images_and_files ?

Then when you go to provision, you would select it from “Installation media”?

@lzap perhaps we should make an end-to end guide for this?

[HBAIT]

Hey, @mcorr
We are still working on the ab above puppet CA config error. We have also tried by removing the old ssl and generating the new ssl key, still getting the same error.

[HBAIT]

@mcorr Yes we have created new installation media and config the image repository path. we have added the installation media on the host as well

[mcorr]

Have you followed the steps mentioned in the Error wiki:

See Foreman :: Manual, scroll down a little for the Puppet CA configuration and the sudoers rules are listed. These should be in /etc/sudoers.d/foreman-proxy and the file should have -r–r----- (0440) permissions.

[HBAIT]

@mcorr Yes, we tried but still getting the same error. we also checked on the log file and added here, please have a look.

2021-04-29T07:47:59 5a0f4ae0 [I] Finished DELETE /puppet/ca/foreman-katello.us-central1-a.c.socios-linux.internal with 406 (65.38 ms)
2021-04-29T07:47:59 5a0f4ae0 [I] Started POST /puppet/ca/autosign/foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:47:59 5a0f4ae0 [I] Finished POST /puppet/ca/autosign/foreman-katello.us-central1-a.c.socios-linux.internal with 200 (5.11 ms)
2021-04-29T07:54:02 9cda62ad [I] Started DELETE /puppet/ca/autosign/foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:02 9cda62ad [I] Finished DELETE /puppet/ca/autosign/foreman-katello.us-central1-a.c.socios-linux.internal with 200 (0.7 ms)
2021-04-29T07:54:09 73401120 [I] Started GET /v2/features
2021-04-29T07:54:09 73401120 [I] Finished GET /v2/features with 200 (39.71 ms)
2021-04-29T07:54:10 0317b069 [I] Started DELETE /ssh/known_hosts/10.128.0.18
2021-04-29T07:54:10 0317b069 [I] Finished DELETE /ssh/known_hosts/10.128.0.18 with 204 (0.71 ms)
2021-04-29T07:54:10 0317b069 [I] Started DELETE /ssh/known_hosts/foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:10 0317b069 [I] Finished DELETE /ssh/known_hosts/foreman-katello.us-central1-a.c.socios-linux.internal with 204 (0.91 ms)
2021-04-29T07:54:10 0317b069 [I] Started DELETE /puppet/ca/autosign/foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:10 0317b069 [E] Attempt to remove nonexistent client autosign for foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:10 0317b069 [W] Error details for Attempt to remove nonexistent client autosign for foreman-katello.us-central1-a.c.socios-linux.internal: : Attempt to remove nonexiste
nt client autosign for foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:10 0317b069 [W] Attempt to remove nonexistent client autosign for foreman-katello.us-central1-a.c.socios-linux.internal: : Attempt to remove nonexistent client autosign
for foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:10 0317b069 [I] Finished DELETE /puppet/ca/autosign/foreman-katello.us-central1-a.c.socios-linux.internal with 404 (0.98 ms)
2021-04-29T07:54:10 0317b069 [I] Started DELETE /puppet/ca/foreman-katello.us-central1-a.c.socios-linux.internal
2021-04-29T07:54:10 0317b069 [E] Failed to remove certificate(s) for foreman-katello.us-central1-a.c.socios-linux.internal: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unkn
own
2021-04-29T07:54:10 0317b069 [W] Error details for Failed to remove certificate(s) for foreman-katello.us-central1-a.c.socios-linux.internal: SSL_connect returned=1 errno=0 state=error: sslv3 aler
t certificate unknown: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown

[socioprophet]

How much work would do you estimate it would be to help you connect these dots and make this functionality real?

HBAIT is working with me on this and while I am ok with work arounds, I’m also not opposed to helping the community if it’s not going to kill us After all… you all have been very helpful to us!

~michael

[lzap]

@mcorr

@lzap perhaps we should make an end-to end guide for this?

Problem is, this is still pretty early in development in both Anaconda (RHEL/Fedora) and Image Builder. There are known issues and quirks, that’s why I kept this chapter pretty slim - we actually do not dive into this as this is all changing (just a week ago another bug was found and folks are working on it).

So for now, I think linking RHEL docs is way to go. We can extend this later on.

[lzap]

@socioprophet

How much work would do you estimate it would be to help you connect these dots and make this functionality real?

If you mean the Puppet CA removal error, I am not sure about this one. I suppose this works out of the box if you used our installer? Any idea why CA removal request would fail with X509 error @ekohl?

To your question - it is really hard to give a good advice without pretty much any context. This thread is lacking on the input front, the OP did not care to even follow our support post template where we ask very basic things like Foreman and OS version. And that is just a beginning, we need to know much more - how it was installed, if custom certificate were attempted to used etc etc.

[HBAIT]

Hi @lzap / @mcorr ,

1. We are trying to create an ISO image based on the custom dependency packages using the cockpit image builder.
2. we are referring to this tutorial link “Building Images & Deploying With Foreman” to create the ISO image.
3. Currently, We have completed building the ISO image using the cockpit and we are trying to provisioning the host using the foreman server.
4. On the foreman we have created the products and added uploaded the file to the repositories. We have also created a new installation media based on the created products, configured it with the operating system, and created the new host.
5. While deploying the host we are getting the Puppet CA removal error (“Screenshot by Lightshot”) and we are not able to initiate the build process.

[lzap]

@ekohl do you have an idea why Puppet CA would fail to remove CA during provisioning? I mean, isn’t this an action that should happen during deletion of a host?

[Marek_Hulan]

What exact Foreman and Foreman Proxy do you use? This is not Puppet CA response, but rather a Foreman (smart) proxy response. Was this instance installed with the foreman-installer? What is your Puppet CA version? Could you please enable Foreman proxy debug log level, restart it and then upload smart-proxy.log from when you observe the failure? I vaguely remember that e.g. a wrong version in /etc/foreman-proxy/settings.d/puppetca.yml could have caused that. Try investigating /etc/foreman-proxy/settings.d/puppetca*.yml if you see anything goofy. Thanks

[ekohl]

What @Marek_Hulan said is correct. This error shows up when Foreman Proxy tries to connect to PuppetCA. It only does that if it’s Puppet 6 or newer so you should be looking at /etc/foreman-proxy/settings.d/puppetca_http_api.yml. There the puppet_ssl_ca file is pointing to a file that’s used to validate the connection (as specified in puppet_url). So the easiest way to replicate it is curl --cacert $puppet_ssl_ca $puppet_url. That must succeed.

[HBAIT]

Hello @Marek_Hulan / @ekohl,

Thanks for the time.,

@Marek_Hulan

What exact Foreman and Foreman Proxy do you use?

We are using the foreman-2.3.3-1.el7.noarch and foreman-proxy-2.3.3-1.el7.noarch version.

@Marek_Hulan

Was this instance installed with the foreman-installer?

Yes, we have this instance installed with the foreman-insatller.

@Marek_Hulan

What is your Puppet CA version?

PuppetCA version is also 2.3.3.

@Marek_Hulan

Could you please enable Foreman proxy to debug log level, restart it and then upload smart-proxy.log from when you observe the failure

We have mentioned all the versions and also added the puppet ca debug log please have a look on the above screenshot.

We will also try to investigate the puppet ca YAML file and we will also start looking on the puppet_ssl_ca file which is pointing to a file that’s used to validate the connection.

we will update and posted here. Thanks!

[ekohl]

@HBAIT

PuppetCA version is also 2.3.3.

That sounds like the Foreman Proxy module version. Puppet CA is part of the puppetserver package.

[HBAIT]

Hello @ekohl,

@Marek_Hulan

What is your Puppet CA version?

We are using puppet --version - 6.22.1

[HBAIT]

@ekohl

cacert

Hello @ekohl / @Marek_Hulan,

As mentioned feedback we checked on the connection path /etc/foreman-proxy/settings.d/puppetca_http_api.yml, and we found this connection path on the YAML file.

1. Based on the connection path we found the .pem key for Cert, public and private keys.
2. we tried to restart the puppet service still getting the same certificate got revoked error.

3. we have also tried to remove the ca certificate from the SSL path and generate a new one but still getting the host cert error.

Please find the above screenshot for reference and Let us know any pointers/feedback regarding this. Thanks!