One of our requirement is to replace all certs with the custom certificate which includes server certificates, client certificates and Puppet CA with our own CA. We started the process by replacing the certificates already installed instance, this works well for server certificates but failed when we also replaced puppet CA with our own CA. Second attempt was to do that during the initial installation itself, again we used multiple options which disable using puppet CA and uses our own CA and other server certificates, and it failed we are able to troubleshoot and move forward one step at a time and at this time we decided also to reach out community for help because as per our research not many online forums talks about this process.
What we are really looking for is to understand the requirements/installer options that we have to do to complete this setup.
We are currently running everything on the same instance I.e. foreman, proxies, puppet master etc and will continue to run on same instance.
We are using Vault as a certificate provider including server, CA, and intermediate CA.
As an end result we wanted to replace all the certificates including puppet CA, server certificates with our custom vault supplied certificates or by custom certificates in general.
In case if any additional information is required I am happy to provide the same.
This is a difficult topic but I read it very often as an requirement. Best solution I have seen so far was by changing the Puppet CA to be a Sub-CA and then letting run everything in the normal fashion. But this will probably not work for you.
@tbrisker: Reading this requirement more and more often here and being questioned by costumers (and not only for Foreman, but also Katello), may I kindly ask if it is possible to put this on the list for future improvements? (I am not sure if you are the best one to address this. If not please tell me and I will try my luck at the product managers I know) Thanks in advance!
Hi Dirk, thanks for the reply. Based on your comments it seems like it is not something that we can accomplish without trouble and chances of success are very low.
Is it possible to get some brief explanation on why it is difficult at this point of time because my management is looking for reason to make decision if we should solve this problem now or later. It would also be great if you can direct me to some helpful link which can help to solidify my understanding on this specific topic.
The problem is here mainly caused by having one CA infrastructure for multiple use cases. Most only want to change the one facing the Web user, but it is for internal use required that Smart Proxies and Foreman use the same CA. Then you have the Puppet infrastructure which requires also all having the same CA. And if you add Katello another CA infrastructure is required for subscription management. To make it less complicated Foreman uses by default the Puppet CA infrastructure for everything and Katello the one provided by Candlepin.
Pre-generating certificates for the Foreman/Katello infrastructure is possible and works, but requires a good knowledge and is not very easy. But then if you provision systems Puppet wants to create a certificate (and also subscription-manager if using Katello) which must be from the same CA, so pre-generating is not really possible (or at least very hard to solve).
Most of this I know from learning it the hard way, so no good link available. There is an old guide which as far as I know is no longer working but still explains the basic problem at Foreman :: Replacing Foreman's web SSL certificate.. With more components like dynflow/tasks or Katello added it gets even more complicated.
One thing that may come in handy is Puppet 6âs support for being an intermediate CA. https://puppet.com/docs/puppetserver/6.5/intermediate_ca.html describes how to set this up. The installer doesnât have support for this currently but you can run the installer, wipe all certificates, import the CA and rerun the installer. Note this requires Puppet 6 agents as well.
It would be great if users could contribute their deployments. The what and how are obviously important, but the why is often also a great thing to learn.
I have been looking at solutions for a more generalized CA setup to unify both Foreman and Katello. Vault has shown up on my radar but it would be nice to hear user experiences. If it works well, it would be great to integrate this into the installer.
Thank you team for the reply, really appreciated the efforts. I have updated the team regarding the same, if we make the decision to move forward with the suggestion we will definitely share with the community.
it took me a while to get things working using my own certs. foreman, for some odd reason, was more difficult than katello. my intent was just to change the web-ui to use my custom cert - it worked fine in katelloâŚhence thereâs fewer options/params provided. w/ foreman installâŚi basically had to substitute anything generating a cert ( puppet-ca / smart-proxy / foreman-ui / etc ) w/ the custom certs before it all worked.
iâd like to share w/ you the shell script[s] I used to get everything running. for simplicityâs sake (on my end) I created separate scripts to install the 2 variations.
Details are as follows:
Operating Systems
AlmaLinux 9
Cert Management
XCA ( created CA/crl/cert/keys )
Foreman
3.11
Katello
4.13
CA-crt & CA-chain
ESYSTEMS-CA.crt
CA-crl
ESYSTEMS-CA.crl
Server-crt
katello.ecloud.net.crt
Server-key
ESYSTEMS-PRIVATE-KEY.pem
please review and provide feedback.
should you try/adopt - modify all refs of esystems/ecloud to match the name youâd like.
if you have questions, iâll try my best to respond and help
Well, I guess if you did that first and put the certificate where foreman expects it, it would all fall into place and you wouldnât have to change the default cert, ca and key paths for each and every component. You only need that, because you put the cert somewhere else.
The default cert path is /etc/puppetlabs/puppet/ssl/certs/foreman.example.com.pem.
The default ca path is /etc/puppetlabs/puppet/ssl/certs/ca.pem.
The default crl path is /etc/puppetlabs/puppet/ssl/crl.pem.
The default key path is /etc/puppetlabs/puppet/ssl/private_keys/foreman.example.com.pem.
I havenât checked all options you specified, but at those paths it covers at least --foreman-client-ssl-*, --foreman-plugin-puppetdb-ssl-*, --foreman-server-ssl-*.
I think the path for the foreman-proxy into /etc/foreman-proxy/ is intentionally different. The foreman-proxy needs to read the key file and if you install the proxy on a separate server, there may be no puppet at all.
And generally, providing a script like this with things like
is kind of evil. Some people simply copy this never understanding what they did and what the consequences are, until much later theyâll notice that selinux isnât protecting them or the server exposes ports which it shouldnâtâŚ
"âŚkind of evil " â lolâŚiâm not that good yet where i can go off creatingâŚâevil scriptsâ
what I provided was an installation iâd done on a dev/test foreman, iâd purposely disable settings to limit potential blockers - something i forgot to mention in my first post, but made sure to have stated in the follow up post.
iâll work on it a bit moreâŚwith security in mind, staging the certs as youâve provided
soâŚiâve managed to simplify the foreman installer quite a bit from my original post. after finally forcing the puppetca to use my custom-ca-cert, all the addtâl options listed earlier became obe.
hereâs my setup for installing foreman w/ built-in smart-proxy using custom ca/cert again, vet these settings on a dev/test system before making any changes to your prod instance
#!/bin/bash
###############################################################################################
# PRE-CONFIGURATIONS
###############################################################################################
firewall-cmd --permanent --zone public --add-service https
mkdir -p /etc/esystems
cp -r /root/katello-installer/certs /etc/esystems
dnf -y install \
https://yum.theforeman.org/releases/3.11/el9/x86_64/foreman-release.rpm \
https://yum.puppet.com/puppet7-release-el-9.noarch.rpm
dnf -y install foreman-installer puppetserver
####################################################
IMPORTANT
###################################################
# import ESYSTEMS-CA to puppetca
/opt/puppetlabs/bin/puppetserver ca import \
--private-key /etc/esystems/certs/ESYSTEMS-PRIVATE-KEY.pem \
--cert-bundle /etc/esystems/certs/ESYSTEMS-CA.crt \
--crl-chain /etc/esystems/certs/ESYSTEMS-CA.crl \
--certname ESYSTEMS-CA
###############################################################################################
# FOREMAN
###############################################################################################
foreman-installer \
--enable-foreman \
--enable-foreman-cli \
--enable-foreman-cli-puppet \
--enable-foreman-cli-remote-execution \
--enable-foreman-cli-ssh \
--enable-foreman-plugin-puppet \
--enable-foreman-plugin-remote-execution \
--enable-foreman-proxy-plugin-remote-execution-script \
--enable-puppet \
--foreman-initial-admin-password password \
--foreman-initial-organization ecloud.net \
--foreman-proxy-dhcp-interface 'enp0s3' \
--foreman-proxy-dns-interface 'enp0s3' \
--foreman-proxy-plugin-remote-execution-script-install-key true \
--foreman-proxy-puppet true \
--foreman-proxy-puppetca true \
--foreman-server-ssl-ca /etc/esystems/certs/ESYSTEMS-CA.crt \
--foreman-server-ssl-cert /etc/esystems/certs/katello.ecloud.net.crt \
--foreman-server-ssl-chain /etc/esystems/certs/ESYSTEMS-CA.crt \
--foreman-server-ssl-crl /etc/esystems/certs/ESYSTEMS-CA.crl \
--foreman-server-ssl-key /etc/esystems/certs/ESYSTEMS-PRIVATE-KEY.pem \
--puppet-autosign-entries='*.ecloud.net' \
--puppet-autosign-entries='*.esystems.net' \
--puppet-autosign-mode='0664' \
--puppet-server true \
--puppet-server-ca true \
--puppet-runinterval 300 # set to run every 5mins as i want quick feedback for the modules i'm building
# force puppet boot after foreman
sed -i '/^After=/ s/$/ foreman-proxy.service foreman.service/' /usr/lib/systemd/system/puppet.service
systemctl daemon-reload
# keep foreman clean of old client reports
echo "foreman-rake reports:expire days=1 status=0" > /etc/cron.daily/foremanReportCleaner
chmod +x /etc/cron.daily/foremanReportCleaner
Sorry to say that: but thatâs so wrong on so many levels.
Never, ever modify the files in /usr/lib/systemd/system/. Any change will be gone with the next update of the rpm writing the file. So those are not set as config files in the rpm. The next update or reinstall of puppet-agent will simply overwrite the file. systemd has a hierarchy of paths itâs using allow you either to completely replace the service unit or add to it. There is even the âsystemctl editâ command to help you with that.
What is the point of starting the puppet.service after foreman-proxy or foreman? They donât really have to do with each other except maybe that the very first report of the puppet agent after boot may to be delivered. But as the puppet agent is running every 30 minutes I donât really see the point of adding dependencies to units which they donât need.
You are putting certificates, ca files and private keys into a separate ânewâ directory /etc/esystems. On systems with SeLinux thatâs also usually not a great idea and potentially causing problems as the files in /etc/esystems donât get the standard certificate selinux labels. This may break the system at any time or any update, if some selinux gets added which doesnât allow to etc_t anymore but expects cert_t instead. /etc/pki/tls is there to put in certificates and keys. There they get automatically the correct selinux label which applications usually expect. Itâll be much easier in the long run if you work with selinux instead against it by putting files into new places.
While I see your intentions of your posts I find it hardly useful to post them here and in this context. The thread is from 2019 to begin with. Your setup is very specific to what you need. To understand what you do there exactly requires a lot of knowledge of foreman if someone wants to use those instructions setting up an own foreman server. For instance, someone needs to fully understand the security implications of the puppet-autosign-entries and not simply put itâs own domain in there. I donât use it as itâs too big a risk and itâs not necessary either for hosts you create in foreman.
Which is the place where are start doubting the usefulness of it, because itâll be generally better to read the current docs then some instructions which worked today but maybe not tomorrow with the next version. Someone who knows enough about foreman to understand your suggestions would read the docs. Someone who doesnât know enough about foreman would blindly follow your suggestions getting a system which they may not know nor really understand and which they may have a hard time later to reconfigure everything to what they actually need. And I have seen it too often in the past that people copy/pasted some lines for something they wanted to do which didnât work as expected instead of reading the current docs which describes it in detail for the current version, breaking more on the way than fixing.
Itâs more something for a blog post which you post once you have finished and with everything in place for your setup, as an example for how you did it. Or maybe here in a separate new topic. But definitively not piggybacked on a five year old thread.
good morning gvde ( and all ) - hope youâre having a great start this Saturday morning.
if I mayâŚIâd like to respond to just a few things, and move on. firstâŚthanks for the feedback. Iâd like to say some positive, but itâs read/taken-in mostly as not. but stillâŚthanks!
in a real-world scenarioâŚthis would have gone through multiple iterations before reaching prod, so i canât restate enough ( done in my second post ), that is done in r&d mode. in factâŚthis isnât done for any company/enterprise/etcâŚIâm simply doing this at home out of morbid curiosity and interest in the topic. soâŚthe parts that can be usedâŚtake/modify as needed. that which cannotâŚignore. âŚthe goal here is to simply fulfill the request
Iâm definitely not writing the official how-to guide here - for installing foreman nor installing/configuring a linux system[s]. You made mention of reviewing the official guide: Securing Communications w/ SSL. however, thereâs still no mentioning of steps/options w/in the installer to reach the end-state of this thread - which brings us back to this point. from the multiple threads Iâve visited on this topic - this was the last one Iâd come acrossâŚwhere I decided to join the communityâŚand made my first post. if thereâs some other place, one more current, pleaseâŚwe can continue there.
lastly, againâŚI do thank you for feedback
ps
âŚexactly - I was trying to find a way to ensure I didnât miss it
Which is exactly why I think a separate new thread would have been better because then you could have set the settings right from the beginning. You could develop your way and keep tagging the latest end best as solution of the thread.
Itâs mixed audience here: people starting with foreman as well as others using it for years (plus developers etc. of course). But thatâs why I am skeptic about half-finished solutions because in the end there will always be a few people which will do exactly like you wrote, not always reading all the posts from the beginning, in particular because the beginning of this thread was from a completely different time.
I feel for beginners itâs too short, too few explanations, to little about whatâs still missing or the direction you are going for them to follow and understand why they may need in their own setup and what not.
And for the non-beginners it seems just overly complicated what you do. I think, itâs always best to start with the defaults (and not explicitly setting installer options which are default anyway) and adjust whatâs really necessary. And when posting: omit the settings which are specific to your environment (like the dhcp interface) because they donât help.
And most importantly: stick to the topic and the topic of this thread is/was to use custom ssl certificate in foreman five years ago. Thus, I think it would be better in this thread to focus on the topic and if docs donât cover it properly, explain how to improve them so that eventually all answers are in the docs.
If you want to develop your ideas and your setup and discuss how to make it better: just start a new thread. Thatâs where I think it would be best⌠You are welcome to post and as for me, Iâll give my thoughts and feedback, whenever I have to time to improve your setup, but I just think this thread is just the wrong place to do itâŚ
As in most communities (and sometimes even mentioned in netiquettes) itâs best to keep sleeping threads sleeping and donât revive them many months or years after the last post. Foreman/Katello was at a completely different level five years ago and the new docs werenât even there but only the manual you have just linked and that doesnât cover many things. Itâs all in the new docs, even though currently more topics are covered for katello setups than simple foreman setups.
So please donât take it as negative but as suggestion for improvement. Thatâs what I try to explain my thoughts on this in length.
Well, I am not sure if thatâs so important that you have to adjust the service unit. I donât really care and rather leave it at the default and miss a report than have some changes which I have to remember whenever I may set up a new server in the future.
Plus, all my puppet agents have splay set, thus it doesnât really matter anyway. splay is essential for us as we have large virtual environment hosts on storage systems. In the past, after a storage maintenance during which we had to shut down all virtual machines, we have started most servers in parallel in a few minutes which lead all servers to report mostly at the same time to the foreman server overloading the puppet server, kind of 200 servers report in five minutes and then for 25 minutes there is nothing.
Thatâs when we have deploy the splay setting everywhere to randomize reportsâŚ
