Puppet reports missing (only some) resources and log information

puppet

#1

Problem:
Reports are missing some of the resource and logs.

Expected outcome:
Reports should show all resources and logs.

Foreman and Proxy versions:
1.19

Foreman and Proxy plugin versions:

Other relevant data:


#2

So this seems to be some timing or historic based issue. Here is a host 30 mins ago:


All resources showing.

Here is the same host from 1 day ago:


A few resources missing.

2 days ago, more resources missing:

8 days ago, all resources missing:


#3

Do you have a cron task for expiring reports? This sounds like possibly a bug in the report expiry task leading to some values being deleted at the incorrect time. @lzap, iirc you recently looked into that area?


#4

We do, havent touched this for at least 3-4 years.

[*PROD* root@puppet 0 /etc/cron.d]# cat foreman
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

RAILS_ENV=production
FOREMAN_HOME=/usr/share/foreman

# Clean up the session entries in the database
15 23 * * *     foreman    /usr/sbin/foreman-rake db:sessions:clear >>/var/log/foreman/cron.log 2>&1

# Send out recurring notifications
0 7 * * *       foreman    /usr/sbin/foreman-rake reports:daily >>/var/log/foreman/cron.log 2>&1
0 5 * * 0       foreman    /usr/sbin/foreman-rake reports:weekly >>/var/log/foreman/cron.log 2>&1
0 3 1 * *       foreman    /usr/sbin/foreman-rake reports:monthly >>/var/log/foreman/cron.log 2>&1

# Expire old reports
30 7 * * *      foreman    /usr/sbin/foreman-rake reports:expire >>/var/log/foreman/cron.log 2>&1

# Collects trends data
*/30 * * * *    foreman    /usr/sbin/foreman-rake trends:counter >>/var/log/foreman/cron.log 2>&1

# Refreshes ldap usergroups. Can be disabled if you're not using LDAP authentication.
10 * * * * foreman /usr/sbin/foreman-rake ldap:refresh_usergroups >>/var/log/foreman/cron.log 2>&1

# Clean expired notifications
0 6 * * 0       foreman    /usr/sbin/foreman-rake notifications:clean >>/var/log/foreman/cron.log 2>&1

# Only use the following cronjob if you're not using the ENC or ActiveRecord-based storeconfigs
# Get the node.rb / ENC script and store at /etc/puppet/node.rb:
#   https://github.com/theforeman/puppet-foreman/blob/master/templates/external_node.rb.erb
# Send facts to Foreman, using the ENC script in a fact pushing only mode
#*/2 * * * *     puppet    /usr/bin/tfm-ruby /etc/puppet/node.rb --push-facts >>/var/log/foreman/cron.log 2>&1

# Warning: ActiveRecord-based storeconfigs is deprecated from Foreman 1.1 and Puppet 3.0
#   see http://projects.theforeman.org/wiki/foreman/ReleaseNotes#11-stable
# Only use the following cronjob if you're using ActiveRecord storeconfigs!
#*/30 * * * *    foreman    /usr/sbin/foreman-rake puppet:migrate:populate_hosts >>/var/log/foreman/cron.log 2>&1

#5

Anything in /var/log/foreman/cron.log?

Can you run /usr/sbin/foreman-rake reports:expire manually? How long it takes to finish? Any errors?


#6

/var/log/foreman/cron.log doesnt show much, but it does show an error:

[*PROD* root@puppet 0 /var/log/foreman]# cat cron.log
/usr/share/foreman/lib/core_extensions.rb:182: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here

Same response on manual expiration:

[*PROD* root@puppet 0 /var/log/foreman]# /usr/sbin/foreman-rake reports:expire days=60
/usr/share/foreman/lib/core_extensions.rb:182: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here

#7

Found the issue, here is the patch.


#8

Thank you! Fine work as always @lzap