I am fairly new to Foreman, Puppet, and even Linux (to an extent). I set up Foreman using the quickstart guide, set up servers, set up the proxy servers, and finally got through some issues to build a node unattended.
The issue is that the manifest files/classes I am applying to the host group for this node are not being applied. I enabled debugging (according to the documentation) and tried my best to review the logs during a build. I am not noticing anything that is obvious in causing this issue so far unfortunately. I checked the dashboard list of clients and I see they are online and the build shows as installed. One thing I did notice was this error in the proxy server logs on the dash âAttempt to remove nonexistent client autosign for â, but I am not sure if this is related to my issue.
I am hoping someone can help me figure out what I am missing here. I may just be missing something obvious or misunderstanding how this works. Any help or guidance would be much appreciated!
Problem:
When I create a new host and build it, the classes I imported and applied to it seem to do nothing or are not getting applied. Expected outcome:
The classes modify the host. In this case, specially adding two packages and replacing a config file. Foreman and Proxy versions:
Foreman, DHCP, DNS, and TFTP are version 3.10. Foreman and Proxy plugin versions:
foreman_puppet is version 6.2. Distribution and version:
Alma Linux 9.4 Other relevant data:
If there are any log files or info I can provide to help, please let me know!
Some additional information: I double-checked that the puppet, puppetserver, foreman, and foreman-proxy services are running. I made sure that syntax was correct for the manifest files, they are imported into the Foreman dashboard, applied to the host group, the environment is set, the node builds unattended, I can connect to it without issue after the build, but I noticed not only did these changes are not applied from the classes, but also that the puppet-agent is not being installed. I am a bit confused and not sure what I am missing.
Maybe some of permission issue or some kind of configuration I missed? Any help would be appreciated.
On the Puppet Master server, I restarted the puppet and puppetserver services, then checked their status. The puppet service said it could not connect to the Puppet Master server, which led to various errors. I refreshed the certs on the Puppet Master, then restarted the foreman and foreman-proxy services.
The errore for the puppet service went away after restarting it and says it can compile a catalog, but now the Foreman dash says the Puppet Master is out of sync.
I attempted to build a node and it is still not applying my classes.
I reverted the cert change (I backed up the certs for both before and after to be able to troubleshoot) and now foreman says it is sync, but I am having the issue with the puppet service saying this:
puppet-agent[130069]: No more routes to fileserver puppet-agent[130069]: Connection to https://example.server:8140/puppet/v3 failed, trying next route: Request to https://example.server:8140/puppet/v3 failed after 0.001 seconds: Failed to open TCP connection to example.server:8140 (Connection refused - connect(2) for âexample.serverâ port 8140) puppet-agent[130069]: Wrapped exception: puppet-agent[130069]: Failed to open TCP connection to example.server:8140 (Connection refused - connect(2) for âexample.serverâ port 8140) puppet-agent[130069]: Could not retrieve catalog from remote server: No more routes to puppet puppet-agent[130069]: Applied catalog in 0.01 seconds puppet-agent[130069]: Connection to https://example.server:8140/puppet/v3 failed, trying next route: Request to https://example.server:8140/puppet/v3 failed after 0.001 seconds: Failed to open TCP connection to example.server:8140 (Connection refused - connect(2) for âexample.serverâ port 8140) puppet-agent[130069]: Wrapped exception: puppet-agent[130069]: Failed to open TCP connection to example.server:8140 (Connection refused - connect(2) for âexample.serverâ port 8140) puppet-agent[130069]: Could not send report: No more routes to report
So is it easier to create the new certs and fix the out of sync issue in Foreman or vice versa?
I do not understand if this is even relevant to my original issue of the classes not being applied to. I did a telnet for port 8140 from a client node to the server, and it connects fine. If this is not relevant to the classes issue, then please disregard this. I just want the classes to be applied to newly provisioned nodes.
Hi and welcome to the community
From the errors you are describing, I can only guess what is going wrong here.
Can you describe your setup in some more detail? What would be specifically interesting is:
Are you running all services on one server or do you have a separate smart-proxy server running some services?
Can you provide the procedure (especially the installer commands) you used to install Foreman and (if applicable) the smart-proxy?
Definitely, I have all these services running on one server at the moment. A rundown of what I did to set up the server is that I selected âEnterprise Linux 9â on this quickstart page: Foreman :: Quickstart, followed the instructions without any additional options, so just running âsudo foreman-installerâ alone. Then, I manually configured the DHCP, TFTP, and DNS (BIND) server while also following the instructions here to set up what was needed for Foreman including the Smart Proxy settings: Foreman :: Manual. I guess I am trying to use Foreman as the ENC for Puppet to set up unattended installations. I am using the Kickstart templates.
I think I may have gotten a few steps closer to the issue since my last post. First, I realized that I think I need to set the parameters related to the puppet agent install to âtrueâ (enable-official-puppet7-repo and force-puppet?). I looked at the Kickstart provisioning template I am using (Kickstart default), and it shows the boolean values to default to false. I set this in global parameters in hopes it may help, but correct me if I am wrong on this. I viewed the preview of the template and manually started running the puppet commands listed on a client node. I received errors when running the â/opt/puppetlabs/bin/puppet agent --config /etc/puppetlabs/puppet/puppet.conf --onetime --tags no_such_tag --server master.swt2.uta.edu --no-daemonizeâ command related to certificates (wish I had saved it, sorry). I reinstated the new certs from running commands to refresh/clean up certs following instructions here: Regenerating certificates in a Puppet deployment. After doing this, the puppet agent --config command ran without issues. Next, I ran into an issue running the âpuppet agent --testâ, which gives this error: Error: certificate verify failed [self signed certificate in certificate chain for CN=Puppet Root CA: (CA value)] Error: certificate verify failed [self signed certificate in certificate chain for CN=Puppet Root CA: (CA value)]
This is where I am at now. I have spent an embarrassing amount of time on this now than I would like to admit, so I would definitely appreciate the help!!!
Redeploying Puppet Certificates in a Foreman deployment is not as easy as the Puppet documentation makes it look. The problem with this is that Foreman and the smart-proxy by default use the Puppet CA and itâs certs for internal communication and authentication. So by regenerating the Puppet certs, you have probably broken some Foreman internals.
So I would suggest to first revert to the original Puppet certificates from the backup you made so Foremanâs internals work again. After that we can take a look at why Puppet does not work as expected.
Can you describe exactly how you setup the Puppetserver? Did you also try to set that up manually?
From the errors you described before, I would assume you either have a problem with the SSL setup for the ENC, or the ENC script not being deployed at all. This should only happen if you tried to install Puppet manually, since the installer should handle all of that for you.
I am working on reverting the change at the moment. Is there anything you would like me to check or try once this is done that might help?
I did not set up the puppetserver manually. The only puppet-related command I ran was this from the Foreman 3.10 quickstart page for rhel9:
âsudo dnf -y install https://yum.puppet.com/puppet7-release-el-9.noarch.rpmâ. I did double-check this conifg from the documentation, but it everything looked to be set up already (I did not manually do any of this): Foreman :: Manual.
The first things I would like you to check are these:
In the Foreman UI, on the host details page, in the Puppet tab, you should have a button for ENC preview:
Do you see the classes you assigned and any potential parameters you have set in that YAML?
If the ENC Preview in Foreman shows the correct things, try running the following command on the CLI:
/etc/puppetlabs/puppet/node.rb <fqdn of that same host>
This should show the same YAML output as the Foreman UI in step one.
If both of those things work and show the expected output, take a look at the puppetserver logs (usually /var/log/puppetlabs/puppetserver/puppetserver.log and check if any errors show up during a puppet agent --test run.
If any of these steps show any errors, that should help pin down where the problem is.
I do see the classes and parameters in this tab for this client.
Running the command give this output: Serving cached ENC: Error retrieving node (hostname): Net::HTTPNotFound Check Foremanâs /var/log/foreman/production.log for more information. Unable to read from Cache file: No such file or directory @ rb_sysopen - /opt/puppetlabs/server/data/puppetserver/yaml/foreman/(hostname).yaml
Looking through the logs now. Will post anything specific that I see that may be a clue to the issue soon.
Do you happen to know if the parameters for the puppet agent need to be set to âtrueâ for the puppet agent to be installed and used, or is that the default in kickstart files? I guess I am talking about the parameters I mentioned earlier, because I manually set them (enable-official-puppet7-repo and force-puppet) to true in the parameter tab in the host group.
This is a good indication on where we need to look. HTTPNotFound means, that Foreman can not find the host by the name you provided. Does the hostname you used equal the name the system has in Foreman?
You should not need to set the forece-puppet parameter, since it should only be needed of you have not set a Puppet smart-proxy on the host.
Setting one of the official-puppet-repo parameters is probably the right way to go if you simply want to automatically enable the upstream Puppet repos on newly provisioned hosts (which you probably want if you do not host your own repository mirrors).
You are absolutely right, in my infinite wisdom and lack of energy (kind of worn down, sorry), I made a typo in the name. I typed it correctly, and now see the same result as what was in the ENC preview from earlier.
Alright, so now we know that the ENC itself is properly working, so we can check the next steps.
If you run puppet agent --test on your host, are the classes applied properly or do you see any errors from Puppet? Do you see any errors in the above mentioned puppetserver,log?
That worked! I just created a test class to install a few packages, edited the host group to only include this, double-checked the ENC preview, and then ran the command. It installed them. Let me try to provision a host real quick to see if it runs into any issues.
Hm, so the question is, why does it work now?.. I reverted the certs, then set the global parameters, but that should not have really made any difference from what you told me about the Puppet smart proxy (which has always been set up from the start). This host I tested it on was the host I was manually running the commands from the template preview in, so let me see if the provisioning of a host (unattended install) experience issues.
So I tried rebuilding a different node to see if these new simple classes I added would be added, but somehow it is not picking up the changes. I think a strange problem is building a new host for the first time, using the create host button, does not install the puppet agent on the node, and I am not noticing anything in the production.log so far.
I just removed this node completely and started over again with create host, but I imagine it should work with rebuild as well? It just seems like it is not trying to install the repo and agent on its own during an unattended Kickstart install.
So, I took another look at the code that is used in the default templates.
In the Foreman UI, go to the hostâs page, to the details tab, and there in the âProvisioning Templatesâ card, you should be able to click on the kickstart template name and get a preview of the KS for the host.
In there, you should find:
A rpm -Uhv <realease RPM from yum.puppet.com>
A dnf install puppet-agent
Rollout of a basic puppet.conf file
If these steps are missing, some variable is missing that causes the kickstart to not render the Puppet bits. If all of them are in the Kickstart, then something is going wrong on the target host. On that host, you should find some logs in the root home, usually the interesting log should be called install.post.log, maybe that reveals what is going wrong.
I checked the Kickstart default provisioning template, and it does have these lines in it.
I also checked the log file in rootâs area and see this: Updating system time 200 OK AlmaLinux 9 - AppStream 1.5 MB/s | 9.9 MB 00:06 AlmaLinux 9 - BaseOS 16 MB/s | 6.9 MB 00:00 AlmaLinux 9 - Extras 3.7 kB/s | 20 kB 00:05 Dependencies resolved. Nothing to do. Complete! Last metadata expiration check: 0:00:01 ago on Mon Jun 24 20:45:19 2024. No match for argument: puppet Error: Unable to find a match: puppet /tmp/ks-script-yok0own_: line 66: /etc/puppet/puppet.conf: No such file or directory Failed to enable unit, unit puppet.service does not exist. Performing initial puppet run for --tags no_such_tag /tmp/ks-script-yok0own_: line 94: /usr/bin/puppet: No such file or directory
So the template recognizes that it should install puppet, but the is missing the puppet repository.
On the installed system, do you have any puppet-release (or whatever it is called exactly) rpm installed and is there a puppet repo present?
I checked installed and available packages plus all repos on the system. None of it has anything related to puppet or at least âpuppetâ in the name. Somehow, that rpm command in the template is not working properly? Very strange.
Okay, so I built a node where it installs the puppet repo, puppet agent package, start the puppet service, but does not seem to pick up changes from the classes.
I looked in install.post.log and the last line in there is:
âPerforming initial puppet run for --tags no_such_tagâ
Considering this is from the Kickstart Default template, I wanted to test running âpuppet agent --testâ. What is returned is this:
âError: The certificate for âCN=â does not match its private key.â
I decided to run the last puppet-related line from the provisioning template â/opt/puppetlabs/bin/puppet agent --config /etc/puppetlabs/puppet/puppet.conf --onetime --tags no_such_tag --server master.swt2.uta.edu --no-daemonizeâ and it returned no errors. Ran âpuppet agent --testâ once more and received the same error concerning the private key⌠I think something is not right in how it is handling SSL certificates and private keys.
