@lzap, there is currently no package for - foreman_host_reports 1.0.3 - in the yum-repositories for el7 / el8 , maybe not compiled currently.
just informational.
@Ron_Lavi can you investigate the Host detail - Puppet tab problem please? I know that some code got into 3.2 and some not, letâs backport whatâs needed?
Please use the Monitor - Host Reports in the meantime. The team is still working on the UI.
Hello,
looks like a bug, what do you mean by itâs stuck? The âHost Reports Chartâ is supposed to show three counters: unchaged, changed, failed across all hosts (their last reports).
Are you getting new host reports imported? Note there is a new API now, you need to reconfigure Ansible or Puppet.
@adiabramovitch can you investigate?
@lzap reports are coming in and Iâm able to browse them just fine. The issue is only with widgets (that I use to detect failing and out-of-sync Puppet hosts).
They say the screenshot is worth a thousand words 
1 - âHost configuration status for allâ widget reports only 1 host with alerts disabled. But the other 221 hosts should be recognized as âgood host reports in the last 1 hourâ. The 100% graph reflects only the single âdisabledâ host. Before reports migration, the percentage was calculated against all 222 hosts and their actual state (disabled, out-of-sync, failed, good hostsâŚ)
2 and 3 - widgets donât contain any data
4 - puppet run distribution graph contained data before reports migration
5 - widget doesnât contain any information
Oh sorry I forgot to mention this in the upgrade info (edited). The following charts are deprecated and will no longer work, after Host Report plugin is installed and reports are coming in, you may remove these from your dashboard by clicking the X button:
- Host Configuration Status for All/Puppet/Ansible/âŚ
- Host Configuration Chart for All/Puppet/Ansible/âŚ
- Run Distribution Chart for Puppet/Ansible/âŚ
- Latest Events
Instead, use the following new widgets added by the Host Report plugin:
- All Host Reports
- Puppet Host Reports
- Ansible Host Reports
- All Host Reports Chart
- Puppet Host Reports Chart
- Ansible Host Reports Chart
- Latest Host Reports
Those widgets primarily show information based on the changed, unchanged and failed counters. That is more limiting that in the previous format, however, reports are much more effective (time and storage). We are also looking for feedback - let us know if you are missing some particular data. It could be possible for example create a widgets that would show particular keywords.
@lzap thank you for clarification regarding deprecated charts. Iâve removed them.
The new charts and tables are rather confusing. Some feedback⌠
EDIT: I just realized that âhosts configuration status fo allâ widget is deprecated. Unfortunately, Iâm unable to find âAll host reportsâ widget. I have only these options available:
What Iâd like to have is an overview of hosts by the state of their last report (with changes, without changes, failed). If the host didnât report in a while, that host should be categorized as âout-of-syncâ.
I believe I had this behavior prior the reports migration.
I also relied on hosts status information to get (prometheus) monitoring alerts when foreman_exporter_dashboard_bad_hosts_enabled > 0 or foreman_exporter_dashboard_out_of_sync_hosts_enabled > 0. I guess these metrics are deprecated as well, because now both metrics return 0 (even when some hosts are out-of-sync or have failing reports).
Is there an internative way to get the same information with foreman prometheus exporter?
Hello, on my development machine it works just fine,
but I also wanted to ensure that a foreman 3.2 instance is working well - created, added 2 reports and everything looks fine.
in /host_reports:
and in /new/hosts/<my-host>#/Reports:
Agreed, we will update the dashboard to reflect that, this is confusing indeed.
Not possible at the moment.
I have finished a patch that vastly improves the dashboard, numbers do match now (there is one explanation when this might not be correct - a technical thing). I explain it in this video:
I hope this is usable now, we will push out a minor update.
Report email notifications are in the works as well and expected in the minor update as well.
Hello, thank you for this guide. The below is the error we received:
foreman-rake host_reports:migrate
Starting, 115043 report(s) left
Processing report 1151 out of 115043 reports
Error when processing report ID=17333044
#<ConfigReport id: 17333044, host_id: 1632, reported_at: [FILTERED], created_at: "2022-03-24 01:34:59", updated_at: "2022-03-24 01:34:59", status: 1048577, metrics: "---\nresources: !ruby/hash:ActiveSupport::HashWithI...", type: "ConfigReport", origin: nil>
To resume the process:
***
foreman-rake host_reports:migrate from_id=17333044 from_date=1980-01-15
***
rake aborted!
NoMethodError: undefined method `downcase' for nil:NilClass
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:192:in `block (3 levels) in <top (required)>'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `block in find_each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:222:in `loop'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:222:in `in_batches'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:69:in `find_each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:165:in `block (2 levels) in <top (required)>'
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Please forgive me if someone has already noticed and mentioned what I believe may be a minor typo in script output at the completion of running foreman-rake host_reports:migrate. After the âALL Done!â it is suggested that the next step, when ready, is to run rake reports:expire report_type=config_report days=0. I noticed in this thread it is recommended correctly to run foreman-rake reports:expire report_type=config_report days=0. Running rake as opposed to foreman-rake doesnât work. I by no means mean this as criticism. I only wish to point it out to be helpful. Thanks.
1 Like
Iâm seeing a lot of Puppet reports without any applied changes that are 2 weeks old. Ideally, Iâd like to keep those reports for maybe 1-2 days.
Puppet reports that had modifications are, in my case, more important. Iâd like to keep those for a month.
Whatâs the reports retention policy after migration to the new format?
Is it possible to configure custom reports retention policy?
for me, everything was fine but
foreman-rake host_reports:refresh
becomes error like
rake aborted!
can't add a new key into hash during iteration
/usr/share/foreman/app/models/host_status.rb:7:in `add'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_host_reports-1.0.2/app/models/host_status/host_report_status.rb:185:in `<top (required)>'
/usr/share/foreman/vendor/ruby/2.7.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/usr/share/foreman/vendor/ruby/2.7.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:411:in `block in require_or_load'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:40:in `block in load_interlock'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:13:in `loading'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:40:in `load_interlock'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:389:in `require_or_load'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:544:in `load_missing_constant'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:214:in `const_missing'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:284:in `const_get'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:284:in `block in constantize'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:280:in `each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:280:in `inject'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:280:in `constantize'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:622:in `get'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:653:in `constantize'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/inheritance.rb:229:in `find_sti_class'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/inheritance.rb:215:in `discriminate_class_for_record'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:257:in `instantiate'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:58:in `block (2 levels) in find_by_sql'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/result.rb:62:in `block in each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/result.rb:62:in `each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/result.rb:62:in `each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:58:in `map'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:58:in `block in find_by_sql'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:56:in `find_by_sql'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:821:in `block in exec_queries'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:839:in `skip_query_cache_if_necessary'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:808:in `exec_queries'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/association_relation.rb:36:in `exec_queries'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:626:in `load'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:250:in `records'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:245:in `to_ary'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations/association.rb:205:in `find_target'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_association.rb:274:in `load_target'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_proxy.rb:43:in `load_target'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_proxy.rb:1001:in `records'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in `each'
/usr/share/foreman/app/models/host/managed.rb:805:in `detect'
/usr/share/foreman/app/models/host/managed.rb:805:in `get_status'
/usr/share/foreman/app/models/host/managed.rb:792:in `block in refresh_statuses'
/usr/share/foreman/app/models/host/managed.rb:791:in `refresh_statuses'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:229:in `block (5 levels) in <top (required)>'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `block in find_each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:222:in `loop'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:222:in `in_batches'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:69:in `find_each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:228:in `block (4 levels) in <top (required)>'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:227:in `block (3 levels) in <top (required)>'
/usr/share/foreman/vendor/ruby/2.7.0/gems/audited-4.10.0/lib/audited/auditor.rb:395:in `without_auditing'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:226:in `block (2 levels) in <top (required)>'
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Tasks: TOP => host_reports:refresh
(See full trace by running task with --trace)
and my dashboard shows not all widgets in the menu.
the next problem is, that all hosts are green in the list, also hosts which are not in sync, but widget shows
what exactly the issue is and how can I fix it?
Br. Torsten
Hello,
widgets were fixed yesterday and update is not yet in the repositories, but we are missing an important feature (email notifications) therefore what I suggest for not is to postpone the migration until further notice.
If you have already migrated data and not deleted old reports, rolling back can be done just by uninstalling the plugin:
# foreman-installer --no-enable-foreman-plugin-host-reports --no-enable-foreman-proxy-plugin-reports
To reset Ansible report back to the original endpoint, run:
# foreman-installer --foreman-proxy-plugin-ansible-report-type foreman
Puppet configuration was done manually via puppet config set so it should have been overwritten by foreman-installer.
Delete all Host Report Status and Widget objects:
# foreman-rake
> HostStatus::Status.where(type: "HostStatus::HostReportStatus").delete_all
> Widget.where("template LIKE 'host_reports%'").delete_all
> Host.unscoped.all.find_each { |h| h.refresh_statuses; h.refresh_global_status }
It will leave few database tables behind, just truncate them for now via psql:
truncate host_reports, report_keywords
If you deleted the old reports already, you need to wait for new to come. Apologies for the trouble.
Update: We have decided to cease development of this plugin, we will remove the upgrade migration warning from the original Reports page as an update for 3.1. Please uninstall the plugin and reconfigure Foreman back to its original reports API endpoint using the instructions in this comment. If you have any problems, let us know in the comments below.
1 Like
Hi,
now I did rollback as you described. Reports are not working:
Dashboard show like
and it is not possible to delete failure widgets.
do I need to do some forgotten task?
foreman-installer --no-enable-foreman-plugin-host-reports --no-enable-foreman-proxy-plugin-reports
truncate host_reports, report_keywords
ansible is not installed.
BR. Torsten
I have an interesting problem now. Running foreman 3.2 / katello 4.4 AIO with the following features. {SSH, Pulpcore, Logs, Puppet CA, Puppet, Openscap, Dynflow, and Reports} After uninstalling the new reports plugin, ensuring that puppet is set to reports = foreman, and truncating host_reports and report_keywords, the /hosts and /content_hosts pages will not load unless I clear out HostStatus::ConfigurationStatus and HostStatus::HostReportStatus with .delete_all as I saw in some recent code. Also, unless I clear those out, the /api/v2/hosts page doesnât load properly. On any of the three pages, the error is âundefined method `failureâ for #ConfigReport:0x0000000000000000\nDid you mean? failedâ where the 16-digit ConfigReport number varies. I have restarted services with foreman-maintain service restart with success and a health check shows no errors or warnings, but still have the problem. Please forgive me if I have omitted any pertinent information or have provided any extraneous, unnecessary info. I will supply any information that is requested. Will you please point me in the direction of what I may do to correct the problem?
1 Like
Update: It appears that rubygem-smart_proxy_reports and rubygem-foreman_host_reports did not uninstall in my case when going through the steps for reverting, maybe because of something Iâve done wrong in the past. In manually removing those packages, restarting services, and clicking OK on a prompt about cleaning up some unused tables or something along that line, I think I am working for /hosts /content_hosts /api/v2/hosts etc⌠reliably again with foreman 3.2 and katello 4.4. I apologize for not seeing that sooner and for taking up anyoneâs time with it.
Forgot this one step, updated the post:
Delete all Host Report Status and Widget objects:
# foreman-rake
> HostStatus::Status.where(type: "HostStatus::HostReportStatus").delete_all
> Widget.where("template LIKE 'host_reports%'").delete_all
That was exactly what I forgot to mention, added that to the instructions. To recalculate host statuses, either wait until new reports come in, or perform that manually in the foreman-rake CLI:
Host.unscoped.all.find_each { |h| h.refresh_statuses; h.refresh_global_status }
1 Like
One correction, to drop the plugins tables either perform:
foreman-rake db:migrate:down VERSION=20220113064436
foreman-rake db:migrate:down VERSION=20211011141813
foreman-rake db:migrate:down VERSION=20210616133601
foreman-rake db:migrate:down VERSION=20210112183526
That needs to be done before plugin is uninstalled, or just do drop table host_reports, report_keywords in postgres cli.
1 Like