We use puppet and foreman enc integration and have recently upgraded to puppet5 & foreman1.19
Everything is working great however we are having a problem with our node.rb.
The behavior is different to puppet3 with foreman enc.
Behaviour is now:
on foreman: host_a - hostgroup = base/DNS
when i change host_a hostgroup to base/Mail Relay
and run puppet on host_a it will not detect a hostgroup change, at the end of the run it sets host_a hostgroup back to base/DNS, and you can see it in the foreman audit with the following:
at no point does the file get filled with new host group, but if i run /etc/puppetlabs/puppet/node.rb host_a.example.com off the puppetmaster you can see it collecting the information:
(before puppet run)
Are you using the default_hostgroup plugin? Or perhaps foreman_hooks? Foreman core doesn’t alter the hostgroup for you (as far as I recall) so I assume a plugin is taking this action - knowing that will show us the next steps for debugging
The ENC script does have some caches so I’m wondering if it’s silently serving a cached entry. It looks like we don’t output anything when a cached entry is served. Untested, but I think the following change to the node.rb script would make it more useful:
diff --git a/files/external_node_v2.rb b/files/external_node_v2.rb
index 7614bc5..0604b04 100644
--- a/files/external_node_v2.rb
+++ b/files/external_node_v2.rb
@@ -380,7 +380,8 @@ if __FILE__ == $0 then
result = enc(certname)
cache(certname, result)
end
- rescue TimeoutError, SocketError, Errno::EHOSTUNREACH, Errno::ECONNREFUSED, FactUploadError
+ rescue [TimeoutError, SocketError, Errno::EHOSTUNREACH, Errno::ECONNREFUSED, FactUploadError] => e
+ $stderr.puts "Serving cached ENC: #{e}"
# Read from cache, we got some sort of an error.
result = read_cache(certname)
end
Nope no plugins. We just use foreman to manage our classes in a large scale.
I have made progress … in /etc/puppetlabs/puppet/foreman.yaml when i turn reporting facts ‘:facts: false’
then the following code does not run:
if SETTINGS[:facts]
req = generate_fact_request certname, "#{puppetdir}/yaml/facts/#{certname}.yaml"
upload_facts(certname, req)
end
when the fact reporting does not occur, foreman hostgroup now takes precedent.
we have some facts that get reported that are created by us which will be the hostgroup that it is currently before retrieving it from the enc
@Gwmngilfen@ekohl my suspicions were correct.
As soon as i got rid of our custom fact ‘foreman_hostgroup’ original behaviour resumed.
I could not find this change of it being a reserved fact/keyword in any release notes??
I am happy to get rid of it as its legacy for us, but just was surprised it wasn’t in any release notes.
It wasn’t in the release notes because we didn’t expect it I’m also wondering what the exact response from Foreman is. If it’s a HTTP 400 then it’s intended behavior but could also be a HTTP 500 error. Can you look at the Foreman logs (/var/log/foreman/production.log) if there’s any error there about it?
@Gwmngilfen mmh it has been trolling me all day trying to figure it out, now i use company_hostgroup so I can expose the hostgroup into a factfile cleanly
@ekohl yea that would make sense, there are no errors in the logs, I had debug mode on. it was just 400 response. It would do a POST saying it is updating the facts of the host, then it would do a get request.
Possibly the routes has some amtching on it? not sure ?? may take time to look at the git repo and find whats happening when the post happens, but may not get enough time to get to it.
I’m slightly worried about this change because it can have security implications. If you have root on for example an application server in the group MyApp / App you can move it to MyApp / DB to possibly get more parameters than you’re supposed to. If multi-tenancy is used, I’m wondering if it could be abused to switch to another organization.
Thanks for finding this, did not manage to look into it … work/life balance
I agree with this statement, we run the service as root.
For example … if i have the following contents in /etc/puppetlabs/facter/facts.d/hostgroup.txt:
foreman_hostgroup=base/DNS
This will force the state on foreman application side.
If i go in as root or any user with file perms at 0444 i can change the file to be:
foreman_hostgroup=base/Nginx/CreditCard
However … if the file is owned as root and perms = 0400 then the security becomes less of a problem.
Personally I feel that this should be changed to be controlled via settings along the lines of …
# file - [app/models/host/managed.rb]
if Setting[:ignore_forman_hostgroup_override]
attrs = [:architecture]
else
attrs = [:architecture, :hostgroup]
end
if !Setting[:ignore_facts_for_operatingsystem] || (Setting[:ignore_facts_for_operatingsystem] && operatingsystem.blank?)
attrs << :operatingsystem
end
The reason for this is suggested by @ekoh and for atleast myself we want foreman to be the authority for the hostgroup as an ENC. I could move hosts into PCI hostgroups etc … it is just very basic elevation of rights and pretty frustrating that this exists.
Looks like this workflow could indeed be problematic. Perhaps @lzap has an idea of a better way of setting the hostgroup after provisioning without using a custom fact? Or somehow limiting this to only work for newely provisioned hosts?
it has been trolling me all day trying to figure it out, now i use company_hostgroup so I can expose the hostgroup into a factfile cleanly