PuppetDB and Katello Smart Proxies...missing something?

Support
, ,
#1

Problem:
I’m trying to get a system registered to a katello smart proxy to show up in PuppetDB and it’s surprisingly difficult. I’m suspecting I just either don’t understand the architecture properly or I’m missing something stupid.

Expected outcome:
When I register a system through a smart proxy to Katello, that registration should dump the puppet facts going to katello into the PuppetDB running in my organization.

Foreman and Proxy versions:
Katello 3.5.0.1, Foreman/Proxy: 1.16.0

Foreman and Proxy plugin versions:
PuppetDB_Foreman: 3.0.2, PuppetDB: 4.4.0-1

Other relevant data:
So my understanding of how to use Smart Proxies: They take the place of registering to the master server. In other words, when installing the “consumer” rpm package, you install the one from the smart proxy you’re registering to and then run subscription-manager register which should register you through the Smart Proxy and into the actual katello master server the smart proxy is connected to.

Each of my smart proxies has Puppet installed on them (with --puppet-server-foreman-url on the smart proxies pointing at my master Katello server) and this configuration does seem to be working as i’m seeing puppet facts in Katello for systems registered with Katello.

HOWEVER those facts are NOT making it into my PuppetDB installation which DOES get populated with systems who are registered directly to the Katello master server. Assuming that was because the puppet installation on the smart proxies was a stand alone puppet installation, I installed PuppetDB on one of the smart proxies as well (pointed at my master PuppetDB postgres database) but I don’t think that’s putting things into Puppet DB Either…at least the smart proxy system itself isn’t showing up in the DB.

Ideas? Does anyone have systems registered to a smart proxy that are reporting properly into a PuppetDB installation? What does the architecture/configuration look like?

#2

I’ll get to somewhat answering my own question.

Smart proxies need one of two installation paths to happen on them (which is a manual step, installing the katello smart proxy will not do this for you and there are no options to pass which will set this up either).

  • FULL PuppetDB Installation: One option (which I think i’m finding to be the wrong architecture, but I can’t be sure yet) is to install the full PuppetDB installation and point the Smart Proxy’s PuppetDB install at the PuppetDB Database itself. This creates a second/distributed PuppetDB installation. The downside here appears to be that the katello master server cannot interact with the records installed in this way…because the certs are different you can’t administrate the records entered by the smart proxy…you can only interact with the puppetdb records from the smart proxy.

  • PuppetDB-Termini - Apparently there’s a puppetdb package called puppetdb-termini which allows a puppet installation to see a puppetDB installation somewhere else (i.e. your Katello Master Server or another specific PuppetDB Server). Installing this onto your smart proxy will allow you to configure the smart proxy to see the puppetdb setup on the Katello master server and “work through” the master to input records. I’m attempting to set this up right now, so I don’t have much experience with this setup yet, but this looks to be the method that’s expected for a Katello distributed installation.

I’ll probably end up updating this again if/when I get more info on this, but basically puppet itself was not setup to see a puppetdb installation (either on localhost or anywhere else) which is why the issue above.

#3

I was meaning to reply to this. Your insight is correct. The thing you may be missing is that you need to set the report processor. See https://puppet.com/docs/puppetdb/5.2/connect_puppet_master.html#edit-puppetconf but note that you want to keep foreman in reports.

#4

@ekohl

Yea, the report processor is there.

So, here’s where I’m at with this, and I could use anyone’s help at this point because I don’t know where to go from here.

I started over fresh with PuppetDB 4. Dropped the DB, recreated, and all of my Katello Master Connected systems (Call it “master01”) are properly reporting to PuppetDB. If the system is registered to “master01” all is fine.

I also have 7 additional smart proxies installed. When I attempt to get any of those smart proxies (call them proxy01 - proxy07) to connect to master01’s PuppetDB, things don’t work well.

/etc/puppetlabs/puppet/puppetdb.conf:

[main]
server_urls = https://master01.domain.tld:8081

/etc/puppetlabs/puppet/routes.conf:

---
master:
  facts:
    terminus: puppetdb
    cache: yaml

Package puppetdb-termini installed on the smart proxy.

/etc/puppetlabs/puppet.conf

[main]
    reports = foreman,puppetdb
    server = proxy01.domain.tld
    show_diff = true

[agent]
    certname = proxy01.domain.tld

[master]
    ca = true
    certname = proxy01.domain.tld
    storeconfigs = true
    storeconfigs_backend = puppetdb

(There are other settings, they’re default from the puppet installation and shouldn’t matter to the task at hand).

Here’s the thing. My Smart Proxies were installed with the following installation settings:

foreman-installer --scenario foreman-proxy-content\
               --enable-foreman-proxy-plugin-ansible \
               --enable-foreman-proxy-plugin-discovery \
               --enable-foreman-proxy-plugin-openscap \
               --enable-foreman-proxy-plugin-remote-execution-ssh \
               --foreman-proxy-plugin-remote-execution-ssh-install-key=true \
                    --foreman-proxy-content-parent-fqdn           "master01.domain.tld"\
                    --foreman-proxy-register-in-foreman           "true"\
                    --foreman-proxy-foreman-base-url              "https://master01.domain.tld"\
                    --foreman-proxy-trusted-hosts                 "master01.domain.tld"\
                    --foreman-proxy-trusted-hosts                 "proxy01.domain.tld"\
                    --foreman-proxy-oauth-consumer-key            "< secret >"\
                    --foreman-proxy-oauth-consumer-secret         "< secret >"\
                    --foreman-proxy-content-pulp-oauth-secret     "< secret >"\
                    --foreman-proxy-content-certs-tar             "/root/proxy01.domain.tld-certs.tar"\
                    --puppet-server-foreman-url                   "https://master01.domain.tld"

In Foreman/Katello the products installed show:

Templates, Pulp Node, TFTP, Puppet, Puppet CA, Logs, Dynflow, Ansible, Discovery, Openscap, and SSH

The thing that bothers me is “Puppet” and “Puppet CA”.

On proxy01.domain.tld, when I go to “puppet agent --test”, I get the following:

[root@proxy01 puppet]# puppet agent --test
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Server Error: Could not retrieve facts for proxy01.domain.tld: Failed to find facts from PuppetDB at proxy01.domain.tld:8140: Failed to execute '/pdb/query/v4/nodes/proxy01.domain.tld/facts' on at least 1 of the following 'server_urls': https://master01.domain.tld:8081
Info: Retrieving pluginfacts
Info: Retrieving plugin
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed to execute '/pdb/cmd/v1?checksum=< checksum >&version=5&certname=proxy01.domain.tld&command=replace_facts&producer-timestamp=1522191220' on at least 1 of the following 'server_urls': https://master01.domain.tld:8081
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

And when I run:

[root@proxy01 puppet]#  curl --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem https://stlppkgmgt02.rgare.net:8443
curl: (60) Peer's Certificate issuer is not recognized.

Which I think is exactly as expected.

proxy01 is a puppet CA…that means it’s got it’s own CA Certs right? And those CA Certs aren’t the same CA Certs as on the master…which means trying to use the master’s PuppetDB (which will use the master’s certs) won’t work? Right?

All this for the question:

What the heck is the right/expected way to make PuppetDB work through a smart proxy? I can install PuppetDB on the smart proxy (noted above) but that will cause the Katello master server not to work properly when it goes to de-register nodes right?

There’s GOT to be an expected Architecture / Configuration for this, Right? Is there a way to get each of the Smart Proxy’s Certs recognizable by the Master PuppetDB installation? Is there a way to move/copy/replace all of the Smart Proxy’s certs with the same cert from the Master so that all of Katello is using the same master certs?

Should I have not installed (in some way) Puppet/Puppet CA when I installed the smart proxies?

THERE HAS to be a way to get this to work.

HELP!!! :slight_smile:

#5

Bumping this once for visibility, this is really a PITA/Thorn in my side right now.

Is there a known good way to get a smart proxy to push it’s puppet content to a remote PuppetDB while still allowing Katello to manage those?

I appear to be down to “Install PuppetDB on all Smart Proxies and just use the same database” which looks like it causes problems with the katello plugin or “Install puppetdb-termini on all smart proxies and point at the puppetdb on the katello master” which appears to throw problems with Certs because each of the Smart Proxies has it’s own cert bundle.

#6

@Illydth did you get anywhere with this?

I’ll point out the ‘Unable to fetch my node definition’ from my experience is because the puppetserver’s /etc/puppetlabs/puppet/foreman.yaml file doesn’t have the correct certs.

Katello uses its own CA for smart proxies and the foreman apache instance among other things, which at first seemed frustrating to me, but it really helps when you have multiple puppet CA’s and there is a generic CA signer so you don’t run into issues when trying to add a 2nd puppet CA that would need certs signed by the original puppet CA in order to trust/communicate with your katello instances.

So if you’re using the /etc/puppetlabs/puppet/ssl/certs/ca.pem cert to try and check trust for your proxies or foreman instance this is also likely why the check fails, the ca you want to use should be the katello CA, not the puppet one.

If you’ve added custom certs to your foreman/katello instance, ie signed by a public CA or internal trusted CA this becomes a bit of a nightmare. The katello cert generation for the proxies includes the custom cert in the tarball and tries to deploy it to the foreman-proxy ssl config so now the CA is the custom signed cert’s CA. I didn’t find this to work well and removed my custom cert.

FYI I’m just going down the path of installing puppetdb so these points are more generic at this stage, not specific to puppetdb install.

#7

Thanks for the reply!

So What I ended up doing was going with the first option: I simply installed PuppetDB as a stand alone/fresh installation on each Foreman Proxy server. I took the PuppetDB installation and pointed it at the SAME database server as all the others.

What this means is that the PuppetDB installation will show multiple sources. One of the tables in the PuppetDB is a ‘sources’ table and it will show one record for each puppet db installation (smart proxy) that you have. The stats that go into puppetdb will be tagged from source so that a query will actually show what facts are coming from which PuppetDB Servers. As far as we can tell, this allows full functionality of PUPPETDB.

This isn’t a perfect solution. Since there’s now more than one puppetdb install, the puppetdb integration into Katello/Foreman is a bit wonky. I THINK the “delete” when the system is removed from Katello still functions properly (I haven’t explicitly tested this since I set this up “in production”, but my “counts” are adding up properly in my Katello install so, again, I THINK this is working).

What’s definitely broken (at least I think it is) is the “Monitor->PuppetDB Dashboard” integration…I’m having issues with my PuppetDB database right now being full and thus puppetdb being down, so the dashboard wouldn’t be functional for me right now anyway.

1 Like