Hello,
We are using Foreman 1.15.6 with r10k to deploy Puppet Server 5.x environments. I believe at some point either upgrading from 1.14.x or maybe between a 1.15.x point release we started having an issue with old environments that no longer exist just magically reappearing in the Foreman GUI even though they do not exist anywhere else.
The environments can be deleted by clicking the “Import environments from…” button or by using the drop down box next to Classes and “Delete”. It appears the environment successfully deletes by either method, however about 5 minutes later the environment simply reappears.
When running r10k deploy environment --puppetfile --verbose, the environments in question are not listed.
When deleting the environment I see the following message in /var/log/foreman/production.log:
2018-01-12 15:05:54 2d1aa3d2 [app] [I] Started DELETE "/environments/feature_puppet5_repo" for IP_ADDR at 2018-01-12 15:05:54 -0500
2018-01-12 15:05:54 2d1aa3d2 [app] [I] Parameters: {"authenticity_token"=>"SOME_TOKEN", "id"=>"feature_puppet5_repo"}
Has anybody seen this issue or able to help find the cause?
Thanks for any assistance on this issue,
Jake
My gut feeling is that you have a Host in the old environment somewhere that is reporting in and causing Foreman to create that environment so the Host can be in it.
If you want to try to trace it for sure, I’d suggest enabling the SQL logger, looking for when the environment is inserted to the DB, and working back to see what action triggered it.
Finally had some time to look at this, sorry for the delay. Thank you, this definitely pointed in the right direction. With the sql logger and debug logging turned on, I found the nodes that were recreating these environments. They are all active nodes just sitting in the production or development environments.
Once I found the nodes causing the environment to re-populate, I also found in each of the agents log was an entry similar to:
Feb 6 13:50:23 node-name puppet-agent[7978]: Unable to fetch my node definition, but the agent run will continue:
Feb 6 13:50:23 node-name puppet-agent[7978]: Could not intern from application/json: Could not find a directory environment named 'development' anywhere in the path: /etc/puppetlabs/code/environments. Does the directory exist?
Confirmed this is definitely Puppet related, not Foreman related. Issue ticket can be found at to https://tickets.puppetlabs.com/browse/PUP-8419 and fixed once I updated to Puppet Agent 5.4.0, then it was resolved.
1 Like
