Migrating Reports to the new format in Foreman 3.2+

Please forgive me if someone has already noticed and mentioned what I believe may be a minor typo in script output at the completion of running foreman-rake host_reports:migrate. After the “ALL Done!” it is suggested that the next step, when ready, is to run rake reports:expire report_type=config_report days=0. I noticed in this thread it is recommended correctly to run foreman-rake reports:expire report_type=config_report days=0. Running rake as opposed to foreman-rake doesn’t work. I by no means mean this as criticism. I only wish to point it out to be helpful. Thanks.

1 Like

I’m seeing a lot of Puppet reports without any applied changes that are 2 weeks old. Ideally, I’d like to keep those reports for maybe 1-2 days.
Puppet reports that had modifications are, in my case, more important. I’d like to keep those for a month.

What’s the reports retention policy after migration to the new format?
Is it possible to configure custom reports retention policy?

for me, everything was fine but

foreman-rake host_reports:refresh

becomes error like

rake aborted!
can't add a new key into hash during iteration
/usr/share/foreman/app/models/host_status.rb:7:in `add'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_host_reports-1.0.2/app/models/host_status/host_report_status.rb:185:in `<top (required)>'
/usr/share/foreman/vendor/ruby/2.7.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/usr/share/foreman/vendor/ruby/2.7.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:411:in `block in require_or_load'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:40:in `block in load_interlock'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:13:in `loading'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:40:in `load_interlock'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:389:in `require_or_load'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:544:in `load_missing_constant'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:214:in `const_missing'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:284:in `const_get'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:284:in `block in constantize'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:280:in `each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:280:in `inject'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:280:in `constantize'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:622:in `get'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:653:in `constantize'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/inheritance.rb:229:in `find_sti_class'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/inheritance.rb:215:in `discriminate_class_for_record'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:257:in `instantiate'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:58:in `block (2 levels) in find_by_sql'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/result.rb:62:in `block in each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/result.rb:62:in `each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/result.rb:62:in `each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:58:in `map'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:58:in `block in find_by_sql'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:56:in `find_by_sql'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:821:in `block in exec_queries'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:839:in `skip_query_cache_if_necessary'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:808:in `exec_queries'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/association_relation.rb:36:in `exec_queries'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:626:in `load'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:250:in `records'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:245:in `to_ary'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations/association.rb:205:in `find_target'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_association.rb:274:in `load_target'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_proxy.rb:43:in `load_target'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_proxy.rb:1001:in `records'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in `each'
/usr/share/foreman/app/models/host/managed.rb:805:in `detect'
/usr/share/foreman/app/models/host/managed.rb:805:in `get_status'
/usr/share/foreman/app/models/host/managed.rb:792:in `block in refresh_statuses'
/usr/share/foreman/app/models/host/managed.rb:791:in `refresh_statuses'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:229:in `block (5 levels) in <top (required)>'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `block in find_each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:222:in `loop'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:222:in `in_batches'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:69:in `find_each'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:228:in `block (4 levels) in <top (required)>'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:227:in `block (3 levels) in <top (required)>'
/usr/share/foreman/vendor/ruby/2.7.0/gems/audited-4.10.0/lib/audited/auditor.rb:395:in `without_auditing'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:226:in `block (2 levels) in <top (required)>'
/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Tasks: TOP => host_reports:refresh
(See full trace by running task with --trace)

and my dashboard shows not all widgets in the menu.

the next problem is, that all hosts are green in the list, also hosts which are not in sync, but widget shows

what exactly the issue is and how can I fix it?

Br. Torsten

Hello,

widgets were fixed yesterday and update is not yet in the repositories, but we are missing an important feature (email notifications) therefore what I suggest for not is to postpone the migration until further notice.

If you have already migrated data and not deleted old reports, rolling back can be done just by uninstalling the plugin:

# foreman-installer --no-enable-foreman-plugin-host-reports --no-enable-foreman-proxy-plugin-reports

To reset Ansible report back to the original endpoint, run:

# foreman-installer --foreman-proxy-plugin-ansible-report-type foreman

Puppet configuration was done manually via puppet config set so it should have been overwritten by foreman-installer.

Delete all Host Report Status and Widget objects:

# foreman-rake
> HostStatus::Status.where(type: "HostStatus::HostReportStatus").delete_all
> Widget.where("template LIKE 'host_reports%'").delete_all
> Host.unscoped.all.find_each { |h| h.refresh_statuses; h.refresh_global_status }

It will leave few database tables behind, just truncate them for now via psql:

truncate host_reports, report_keywords

If you deleted the old reports already, you need to wait for new to come. Apologies for the trouble.

Update: We have decided to cease development of this plugin, we will remove the upgrade migration warning from the original Reports page as an update for 3.1. Please uninstall the plugin and reconfigure Foreman back to its original reports API endpoint using the instructions in this comment. If you have any problems, let us know in the comments below.

1 Like

Hi,

now I did rollback as you described. Reports are not working:

Dashboard show like

and it is not possible to delete failure widgets.

do I need to do some forgotten task?

foreman-installer --no-enable-foreman-plugin-host-reports --no-enable-foreman-proxy-plugin-reports
truncate host_reports, report_keywords

ansible is not installed.
BR. Torsten

I have an interesting problem now. Running foreman 3.2 / katello 4.4 AIO with the following features. {SSH, Pulpcore, Logs, Puppet CA, Puppet, Openscap, Dynflow, and Reports} After uninstalling the new reports plugin, ensuring that puppet is set to reports = foreman, and truncating host_reports and report_keywords, the /hosts and /content_hosts pages will not load unless I clear out HostStatus::ConfigurationStatus and HostStatus::HostReportStatus with .delete_all as I saw in some recent code. Also, unless I clear those out, the /api/v2/hosts page doesn’t load properly. On any of the three pages, the error is “undefined method `failure’ for #ConfigReport:0x0000000000000000\nDid you mean? failed” where the 16-digit ConfigReport number varies. I have restarted services with foreman-maintain service restart with success and a health check shows no errors or warnings, but still have the problem. Please forgive me if I have omitted any pertinent information or have provided any extraneous, unnecessary info. I will supply any information that is requested. Will you please point me in the direction of what I may do to correct the problem?

1 Like

Update: It appears that rubygem-smart_proxy_reports and rubygem-foreman_host_reports did not uninstall in my case when going through the steps for reverting, maybe because of something I’ve done wrong in the past. In manually removing those packages, restarting services, and clicking OK on a prompt about cleaning up some unused tables or something along that line, I think I am working for /hosts /content_hosts /api/v2/hosts etc… reliably again with foreman 3.2 and katello 4.4. I apologize for not seeing that sooner and for taking up anyone’s time with it.

Forgot this one step, updated the post:

Delete all Host Report Status and Widget objects:

# foreman-rake
> HostStatus::Status.where(type: "HostStatus::HostReportStatus").delete_all
> Widget.where("template LIKE 'host_reports%'").delete_all

That was exactly what I forgot to mention, added that to the instructions. To recalculate host statuses, either wait until new reports come in, or perform that manually in the foreman-rake CLI:

Host.unscoped.all.find_each { |h| h.refresh_statuses; h.refresh_global_status }
1 Like

One correction, to drop the plugins tables either perform:

foreman-rake db:migrate:down VERSION=20220113064436
foreman-rake db:migrate:down VERSION=20211011141813
foreman-rake db:migrate:down VERSION=20210616133601
foreman-rake db:migrate:down VERSION=20210112183526

That needs to be done before plugin is uninstalled, or just do drop table host_reports, report_keywords in postgres cli.

1 Like

Do we have any update on email notification in Foreman 3.2?

Just another heads up, the original post got updated,
currently due to capacity issues we stopped the new Host Reports plugin development:

Hi,
when will the new Foreman-plugin-host-reports version 1.0.3 available for yum-repositories ?
I installed Foreman 3.3, but the newer host-report-package was not released yet.

Looking in the top of the article there is the fixed code linked, but no rpm was release for CentOS 7 / 8.
Any news about that ?

The plugin was discontinued at this point, hard to tell if someone will find a capacity to finish it in foreseeable future.

@Marek_Hulan, problem is here that the issue that was solved with the “patch” (changing locally code), is not feasible for the users. Normally this should be solved via a new RPM-package.

every user who is trying to install foreman will run into the issue that he has to manually change the code.

I suggested this would be solved with foreman 3.3, so therefore migration is still not available ^^

@Ron_Lavi @lzap I’ve just seen the rollback notification about this now and have already upgraded to Foreman 3.4 before attempting the steps to rollback. I’m unable to uninstall using the foreman-installer command mentioned in the notification:

# foreman-installer --no-enable-foreman-plugin-host-reports --no-enable-foreman-proxy-plugin-reports
2022-12-05 11:30:58 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-12-05 11:31:03 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-12-05 11:31:03 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
ERROR: Unrecognised option '--no-enable-foreman-plugin-host-reports'

Is there a workaround for my particular situation?

The installer support has been dropped (as well as all packaging). That means the default is essentially --no-enable-foreman-plugin-host-reports without an option to enable it.

Note that this was poor phrasing. It really didn’t uninstall, it stopped managing the plugin. There never was an uninstall command and you should manually run yum/apt/dnf remove on it.

Thanks, @ekohl.

Are these the rpms that should be uninstalled?

rubygem-foreman_host_reports.noarch 1.0.2-3.fm3_3.el8
rubygem-smart_proxy_reports.noarch 1.0.0-1.fm3_3.el8

OK, I had to perform the instructions in a slightly different order on 3.4 (and console argument was missing from the foreman-rake command). foreman-installer failed if hostreports weren’t removed.

  1. Remove database tables:
foreman-rake db:migrate:down VERSION=20220113064436
foreman-rake db:migrate:down VERSION=20211011141813
foreman-rake db:migrate:down VERSION=20210616133601
foreman-rake db:migrate:down VERSION=20210112183526
  1. Uninstall the plugin:
dnf remove rubygem-foreman_host_reports.noarch rubygem-smart_proxy_reports.noarch
  1. Delete host report status and widget objects:
# foreman-rake console
> HostStatus::Status.where(type: "HostStatus::HostReportStatus").delete_all
> Widget.where("template LIKE 'host_reports%'").delete_all
> Host.unscoped.all.find_each { |h| h.refresh_statuses; h.refresh_global_status }
  1. Reset Ansible report back to the original endpoint:
# foreman-installer --foreman-proxy-plugin-ansible-report-type foreman
  1. Reboot.
1 Like

That looks correct to me for Foreman 3.4 on EL8. The Debian package names start with ruby- and use dashes. So ruby-foreman-host-reports and ruby-smart-proxy-reports. Users on older versions should also include --no-enable-foreman-plugin-host-reports --no-enable-foreman-proxy-plugin-reports to the installer command.