After upgrade from 1.15.2 to 1.16.0 and puppet 4 to 5 - foreman wants to delete all puppet classes from environment

Support
1

I’m having an issue with Foreman … after the update it seems to want to remove all classes from all environments when selecting “import environment from puppet”

What I was expecting to happen is … it would offer to import the new class only, to the environment where it is checked out.

Any ideas for debugging this further?

2

Presumably nothing has changed on the Puppet side? Clients are still finding existing classes, and so on?

First step is to run the proxy with debug logs and try an import, hopefully the logs will give us a clue.

3

The logs aren’t showing me anything, but that’s because I don’t want to allow it to delete all classes necessarily…

4

A little more detail. I tried removing the classes in dev, no good they don’t then show back up. I tried changing the puppet version in the foreman-proxy puppet.yaml and now get:
ERF12-4115 [ProxyAPI::ProxyException]: Unable to get classes from Puppet for dev ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy https://puppet:8443/puppet

I’m imaging it’s not finding our code directory anymore but I’m a little unsure yet again why that would go away in the config with the upgrade.

5

That looks like the logs from Foreman, I asked for the logs from the Proxy, which are generated when you click Import Classes (since it’s the Proxy’s job to generate the list of classes and return it to Foreman so you can choose what to accept).

That said, a 406 Unacceptable is almost always the Puppetserver rejecting the API request from the Proxy - did the Puppetserver get upgraded or changed in some way as part of the upgrade?

6

Puppetserver was upgraded to 5.1.4. I changed the puppet.yml back to 4.1.3 and now it works, but wants to remove all the classes again. The logs didn’t show anything that looked interesting to me, here:
https://pastebin.com/z7Vk1pXM
does that help any?

7

Yeah, seems sane - for reference this is the useful bit, at startup:

D, [2018-01-03T15:04:30.442526 ] DEBUG -- : 'puppet' settings: 'enabled': https, 'puppet_version': 4.1.3, 'use_provider': [:puppet_proxy_puppet_api]
D, [2018-01-03T15:04:30.444284 ] DEBUG -- : Providers ['puppet_proxy_puppet_api'] are going to be configured for 'puppet'
D, [2018-01-03T15:04:30.484301 ] DEBUG -- : 'puppet_proxy_puppet_api' settings: 'api_timeout': 30 (default), 'classes_retriever': apiv3, 'environments_retriever': apiv3, 'puppet_ssl_ca': /etc/puppetlabs/puppet/ssl/certs/ca.pem, 'puppet_ssl_cert': /etc/puppetlabs/puppet/ssl/certs/puppet.pem, 'puppet_ssl_key': /etc/puppetlabs/puppet/ssl/private_keys/puppet.pem, 'puppet_url': https://puppet:8140, 'puppet_version': 4.1.3, 'use_provider': [:puppet_proxy_puppet_api]

This tells us all the settings it was initialized with. It looks sane, but check the addresses, certs etc are right.

So, assuming clients are still finding the classes Foreman wants to delete (that is, the puppetserver still finds them and can compile catalogs) then the next question is why the proxy can’t see them. It’s an API call to the puppetserver, so it ought to work the same way as for the puppetserver itself. I may have to defer to @Dmitri_Dolguikh for help on this one… Dmitri, is there any difference in the parser for Puppet 4 vs Puppet 5?

8

is there any difference in the parser for Puppet 4 vs Puppet 5?

Not sure – we now rely on puppet server and its api to retrieve (and parse) classes.

I would suggest to try manually retrieve classes for environments in questions using curl or similar. The api endpoint is at “puppet/v3/environment_classes?environment=<environment_name>”.

I’m seeing 403’s from the puppet server when Puppet 5 was used, please make sure either puppet recognizes certificate used by the smart-proxy for environment classes calls, or puppet authorization is configured to allow unauthenticated calls from smart-proxy.

9

As far as we’re able to understand, there’s no authorization issues. We added in
{
match-request: {
path: "/puppet/v3/environment_classes"
type: path
method: get
}
allow: "*"
sort-order: 500
name: “puppetlabs environment classes”
},

and no change. As far as we can tell - the logs don’t show any errors, just multiple re-tries and no found classes.

Using curl -k on https://puppet/puppet/v3/environment_classes?environment=dev
I get a 404 page generated by the webserver. Did I mis generate the URL to check?

10

Looks like you’re missing port 8140 in the URL?

11

Thanks, that was the issue. So it does look like a permission issue then. What I don’t get it what configuration changed here from puppetserver 4 to 5. We tried what’s shown here: http://thinlight.org/2017/07/29/foreman-and-puppet-5-the-hard-way/ - as I said I added into auth.conf the above. Still getting errors. Is there a document my google fu has failed to find or pointers as to what needs to be in auth.conf for the proxy to be allowed?

We had just followed the foreman manual upgrade instructions to get us in this mess. We’re on Scientific Linux 7.4 - just realized I missed that, though I doubt that will help much.

12

So with the port you get the 406 denied errors, same as the proxy, right? Can you gist the whole auth conf file? Maybe someone can spot the issue…

13

Ok, maybe not. I just did:
curl -k https://puppet:8140/puppet/v3/environment_classes?environment=dev
and got a listing of what looks like classes and ?maybe? parameters.
https://pastebin.com/jjhj9KF0
is some of it slightly sanitized.

For Auth.conf. we just switched over to the one shipped with the rpm, save the addition I listed. I’ll go ahead and pastebin it here:
https://pastebin.com/KVf4zJiA

14

Some additional debugging we’ve done. We’ve set to not require authentication - we get auth errors when it’s on, but with it off we’re now getting different errors.
https://pastebin.com/BczhHAfJ

Current auth.conf
https://pastebin.com/MMNU8HH3

Not sure if this helps any?

15

Ok, I think we had a bunch of moving parts, but it seems fixed now. The big need was the change to auth.conf listed above, AND the puppet version in the puppet.yml - neither of which were changed for us from the RPMs or following the upgrade instructions. At the very least, I’d suggest calling out the foreman-proxy puppet.yml config on upgrades to new versions of foreman… If only because that may well coincide with a puppet upgrade and that puppet won’t mention needing it for foreman integration as it has nothing to do with them specifically.

Thanks for all the help anyway.

1 Like
16

Out of interest, did you re-run the installer after the upgrade? If I recall, the installer will write the current puppet version to the proxy config (unless overridden of course), so if it didn’t do that we’d need to investigate…

17

The Puppet 4 to 5 migration is not automatic. We should include https://github.com/Katello/katello-installer/blob/master/hooks/init/31-upgrade-puppet.rb in foreman-installer too.

18

I think that might be the issue. We didn’t re-run the installer. It’s
not obvious to us when we need to do that sort of thing.

James Pulver
CLASSE Computer Group
Cornell University

19

In the katello context we’re stricter about advising rerunning the installer when upgrading. At the Foreman hackday (not formally announced yet but planned for 7th of February in Ghent) I plan to have a discussion around the installer and continuous management. The installer is an aspect of this so I’ll include it there.

1 Like