Foreman enc changing hostgroup back

We use puppet and foreman enc integration and have recently upgraded to puppet5 & foreman1.19
Everything is working great however we are having a problem with our node.rb.
The behavior is different to puppet3 with foreman enc.

Behaviour is now:
on foreman: host_a - hostgroup = base/DNS
when i change host_a hostgroup to base/Mail Relay
and run puppet on host_a it will not detect a hostgroup change, at the end of the run it sets host_a hostgroup back to base/DNS, and you can see it in the foreman audit with the following:

API Admin ( updated Host:

  • Hostgroup changed from base/Mail Relay to base/DNS

We also have in our base configuration the following puppet code to put the foreman_hostgroup into a file:

class base::facts
  file { '/etc/puppetlabs/facter/facts.d/hostgroup.txt':
    owner   => 'root',
    group   => 'root',
    content => "foreman_hostgroup=${::hostgroup}"

  file { ['/etc/puppetlabs/facter', '/etc/puppetlabs/facter/facts.d']:
    ensure => 'directory'

at no point does the file get filled with new host group, but if i run /etc/puppetlabs/puppet/node.rb off the puppetmaster you can see it collecting the information:
(before puppet run)

./node.rb:383: warning: constant ::TimeoutError is deprecated
  hostgroup: base/Mail Relay

(after puppet run)

./node.rb:383: warning: constant ::TimeoutError is deprecated
  hostgroup: base/DNS

Apologies if the format is poor in this.
Appreciate any directions with debugging this.

Are you using the default_hostgroup plugin? Or perhaps foreman_hooks? Foreman core doesn’t alter the hostgroup for you (as far as I recall) so I assume a plugin is taking this action - knowing that will show us the next steps for debugging :slight_smile:

The ENC script does have some caches so I’m wondering if it’s silently serving a cached entry. It looks like we don’t output anything when a cached entry is served. Untested, but I think the following change to the node.rb script would make it more useful:

diff --git a/files/external_node_v2.rb b/files/external_node_v2.rb
index 7614bc5..0604b04 100644
--- a/files/external_node_v2.rb
+++ b/files/external_node_v2.rb
@@ -380,7 +380,8 @@ if __FILE__ == $0 then
           result = enc(certname)
           cache(certname, result)
-      rescue TimeoutError, SocketError, Errno::EHOSTUNREACH, Errno::ECONNREFUSED, FactUploadError
+      rescue [TimeoutError, SocketError, Errno::EHOSTUNREACH, Errno::ECONNREFUSED, FactUploadError] => e
+        $stderr.puts "Serving cached ENC: #{e}"
         # Read from cache, we got some sort of an error.
         result = read_cache(certname)

Nope no plugins. We just use foreman to manage our classes in a large scale.
I have made progress … in /etc/puppetlabs/puppet/foreman.yaml when i turn reporting facts ‘:facts: false’
then the following code does not run:

if SETTINGS[:facts]
  req = generate_fact_request certname, "#{puppetdir}/yaml/facts/#{certname}.yaml"
  upload_facts(certname, req)

when the fact reporting does not occur, foreman hostgroup now takes precedent.
we have some facts that get reported that are created by us which will be the hostgroup that it is currently before retrieving it from the enc

foreman_hostgroup: base/DNS
company_hostgroup: base
company_hostgroup_parent0: base/DNS
company_hostgroup_parent1: base/DNS
company_hostgroup_parent2: base/DNS
company_hostgroup_parent3: base/DNS

I may try not using the foreman_hostgroup.
Will also try @ekohl suggestion with the error output to see if it is doing silent caching

@Gwmngilfen @ekohl my suspicions were correct.
As soon as i got rid of our custom fact ‘foreman_hostgroup’ original behaviour resumed.
I could not find this change of it being a reserved fact/keyword in any release notes??
I am happy to get rid of it as its legacy for us, but just was surprised it wasn’t in any release notes.

That is interesting, I wasn’t aware of that either! @tbrisker is this new?

It wasn’t in the release notes because we didn’t expect it :slight_smile: I’m also wondering what the exact response from Foreman is. If it’s a HTTP 400 then it’s intended behavior but could also be a HTTP 500 error. Can you look at the Foreman logs (/var/log/foreman/production.log) if there’s any error there about it?

@Gwmngilfen mmh :stuck_out_tongue: it has been trolling me all day trying to figure it out, now i use company_hostgroup so I can expose the hostgroup into a factfile cleanly

@ekohl yea that would make sense, there are no errors in the logs, I had debug mode on. it was just 400 response. It would do a POST saying it is updating the facts of the host, then it would do a get request.
Possibly the routes has some amtching on it? not sure ?? may take time to look at the git repo and find whats happening when the post happens, but may not get enough time to get to it.

Turns out this was done in 1.15:

I’m slightly worried about this change because it can have security implications. If you have root on for example an application server in the group MyApp / App you can move it to MyApp / DB to possibly get more parameters than you’re supposed to. If multi-tenancy is used, I’m wondering if it could be abused to switch to another organization.


Thanks for finding this, did not manage to look into it … work/life balance :stuck_out_tongue:
I agree with this statement, we run the service as root.
For example … if i have the following contents in /etc/puppetlabs/facter/facts.d/hostgroup.txt:


This will force the state on foreman application side.
If i go in as root or any user with file perms at 0444 i can change the file to be:


However … if the file is owned as root and perms = 0400 then the security becomes less of a problem.

Personally I feel that this should be changed to be controlled via settings along the lines of …

# file - [app/models/host/managed.rb]
if Setting[:ignore_forman_hostgroup_override]
  attrs = [:architecture]
  attrs = [:architecture, :hostgroup]
if !Setting[:ignore_facts_for_operatingsystem] || (Setting[:ignore_facts_for_operatingsystem] && operatingsystem.blank?)
  attrs << :operatingsystem

The reason for this is suggested by @ekoh and for atleast myself we want foreman to be the authority for the hostgroup as an ENC. I could move hosts into PCI hostgroups etc … it is just very basic elevation of rights and pretty frustrating that this exists.

Looks like this workflow could indeed be problematic. Perhaps @lzap has an idea of a better way of setting the hostgroup after provisioning without using a custom fact? Or somehow limiting this to only work for newely provisioned hosts?