Migrating Reports to the new format in Foreman 3.2+

Attention: Development of the new Host Reports plugin was ceased, do not perform the migration. The migration note will be removed from Foreman 3.2 as a minor update. Sorry for inconvenience. If you have already migrated, perform the following steps to rollback:

Remove database tables of the plugin:

foreman-rake db:migrate:down VERSION=20220113064436
foreman-rake db:migrate:down VERSION=20211011141813
foreman-rake db:migrate:down VERSION=20210616133601
foreman-rake db:migrate:down VERSION=20210112183526

Uninstall the plugin:

# foreman-installer --no-enable-foreman-plugin-host-reports --no-enable-foreman-proxy-plugin-reports

To reset Ansible report back to the original endpoint, run:

# foreman-installer --foreman-proxy-plugin-ansible-report-type foreman

Puppet configuration was done manually via puppet config set so it should have been overwritten by foreman-installer automatically by each run. Delete all Host Report Status and Widget objects:

# foreman-rake
> HostStatus::Status.where(type: "HostStatus::HostReportStatus").delete_all
> Widget.where("template LIKE 'host_reports%'").delete_all
> Host.unscoped.all.find_each { |h| h.refresh_statuses; h.refresh_global_status }

If you deleted the old reports already, you need to wait for new to come. Apologies for the trouble.

Update: We have decided to cease development of this plugin, we will remove the upgrade migration warning from the original Reports page as an update for 3.1. Please uninstall the plugin and reconfigure Foreman back to its original reports API endpoint using the instructions in this comment. If you have any problems, let us know in the comments below.


The way Foreman handles configuration management reports is changing, they are now stored in more efficient way. If you upgraded to 3.2 and noticed there is a warning when you visit Configuration Reports page, you need to reconfigure your Ansible and Puppet callback to send reports to the new API. The old API and Configuration Reports screen will continue working in 3.2 normally, but it is planned to be removed in 3.3 so we recommend to migrate early.

The migration plan

After installation of Foreman Host Reports plugin, new menu item will appear: Monitor - Host Reports. Also some links will be overriden to land on the new page, e.g. in the Host Detail page - Report. And new API and Smart Proxy Reports plugin will be available for report importing.

Migration is done via a rake task, it makes copies of selected configuration reports (older then particular date) in new tables so they appear in the new Monitor - Host Reports screens. Migration can be done while Foreman is running, can be paused at any point with Ctrl-C and resumed from the particular host ID.

New Smart Proxy Reports plugin must be installed, it is responsible for parsing incoming reports in native formats (Puppet YAML, Ansible JSON), converting it, creating keywords and sending them to Foreman. Configuration of both Ansible and Puppet is in this document.

After some testing period, we recommend to expire (or truncate) old (configuration) reports in the database. During the testing period, we recommend to test all your workflows, search queries and operation tasks. Note that the new report format is vastly different from the original which was Puppet-oriented. It is now more generic but also much more faster and efficient, this will help a lot on deployments with many reports.

Please report back to this thread with your remarks, experience, bugreports or suggestions for the next version. Also see “known issues” part down below for some known limitations or bugs we are working on.

Upgrading

Perform the upgrade to Foreman 3.2 as usual. Then enable the new Host Reports Foreman plugin and Smart Proxy Reports plugin:

# foreman-installer --enable-foreman-plugin-host-reports --enable-foreman-proxy-plugin-reports

When using Ansible, set the report type to “proxy” which will configure ansible report callback to send reports to smart proxy in the new format.

# foreman-installer --foreman-proxy-plugin-ansible-report-type proxy

To revert back to the old report format (which will be removed in Foreman 3.3) set the setting back to “foreman”. Note this API will stop working in Foreman 3.3.

Puppet don’t have installer support for the new report format yet, you need to perform the configuration of puppet server manually:

# source /etc/profile.d/puppet-agent.sh
# puppet config set reports store,http
# puppet config set reporturl https://HOSTNAME:8443/reports/puppet
# systemctl restart puppetserver

When running the Smart Proxy with the Report plugin on the same host as puppetserver, no additional configuration is needed. Otherwise, ensure that the puppet server client SSL common name (typically FQDN of the puppet server) is in the trusted hosts:

# grep trusted /etc/foreman-proxy/settings.yml -A2
:trusted_hosts:
  - HOSTNAME

Puppet and Ansible can send reports without a valid SSL client certificate if it’s sent from the same host when smart-proxy is running on. Trusted host entry does automatically match.

Migration

Nothing happened to the old Configuration Reports, in fact, they are still operating normally and you can keep using them in Foreman 3.2. There will be a migration task available in the Host Reports plugin later so reports can be migrated to the new format before 3.3. is out. It is expected that if you choose to migrate all reports, it will take some time and this can be done while Foreman is running:

foreman-rake host_reports:migrate

The rake task accepts from_date option (older reports will be ignored) and from_id, primary key (ID) to start migration from which can be used to resume previously stopped migration. Examples:

foreman-rake host_reports:migrate from_date=2021-01-01
foreman-rake host_reports:migrate from_date=2021-01-01 from_id=1234567

When you encounter an issue during migration, please reply in this thread. After we release a fix, you can resume the migration at the very same position.

After migration is complete, check the new reports in Monitor - Host Reports. All migrated reports have a special keyword Migrated assigned so it is easy to search them later. Now, it is the time to expire all old reports:

foreman-rake reports:expire report_type=config_report days=0

The old report expiration can be very slow process and it is one of the reasons why reports are now stored in a different way. There are plugins which still use the old report format, these include: OpenSCAP, Salt or Chef. When these plugins are not installed or not used on your Foreman instance, deletion can be achieved much faster directly in the PosgreSQL database:

truncate logs, messages, resources, reports

After expiration or truncate operation, make sure to reclaim DB space by issuing VACUM maintenance command. Finally, after the migration process, some hosts may have its host status calculated incorrectly. In that case, refresh all host statuses (Global Status, Configuration Status):

foreman-rake host_reports:refresh

This operation can be also done when Foreman is running.

Dashboard

The following charts are deprecated and will no longer work, after Host Report plugin is installed and reports are coming in, you may remove these from your dashboard by clicking the X button:

  • Host Configuration Status for All/Puppet/Ansible/…
  • Host Configuration Chart for All/Puppet/Ansible/…
  • Run Distribution Chart for Puppet/Ansible/…
  • Latest Events

Instead, use the following new widgets added by the Host Report plugin:

  • All Host Reports
  • Puppet Host Reports
  • Ansible Host Reports
  • All Host Reports Chart
  • Puppet Host Reports Chart
  • Ansible Host Reports Chart
  • Latest Host Reports

Those widgets primarily show information based on the changed, unchanged and failed counters. That is more limiting that in the previous format, however, reports are much more effective (time and storage). We are also looking for feedback - let us know if you are missing some particular data. It could be possible for example create a widgets that would show particular keywords.

New installs

New installation is the same as described in the upgrade steps above, except no migration is needed. In this case, simply ensure the Host Reports and Smart Proxy Reports plugins are installed (see above) and ensure report type is set to proxy (see above) for Ansible. For Puppet, configure puppet server (see above).

Troubleshooting and known issues

Unable to delete an existing host due to Foreign Key error

There is a bug in foreign key association in Foreman 3.2.0, please upgrade to 3.2.1 once it is out. In the meantime, you can apply the following patch to solve the problem.

Can’t add a new key into hash during iteration

If you see this error during foreman-rake host_reports:migrate please upgrade to foreman_host_reports 1.0.3 or apply the following patch to solve it.

Undefined method origin for HostReport

If you see this error during foreman-rake host_reports:migrate please upgrade to foreman_host_reports 1.0.3 or apply the following patch to solve it.

If you run into any issues, please drop a comment below.

Email notifications do not work

This is a missing feature we are currently working on and will be released as a minor update for Foreman 3.2.

3 Likes

I’m on the RC and have a small amount of data I can run through the migration process, so I’d be happy to test it!

1 Like

3.1 is out, so the tutorial would be awesome :slight_smile:

Just tried enabling the plugin, but that failed unfortunately:

2021-12-10 23:39:39 [NOTICE] [configure] Starting system configuration.
2021-12-10 23:40:02 [NOTICE] [configure] 250 configuration steps out of 953 steps complete.
2021-12-10 23:40:02 [NOTICE] [configure] 500 configuration steps out of 1039 steps complete.
2021-12-10 23:40:05 [ERROR ] [configure] Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install ruby-foreman-host-reports' returned 100: Reading package lists...
2021-12-10 23:40:05 [ERROR ] [configure] Building dependency tree...
2021-12-10 23:40:05 [ERROR ] [configure] Reading state information...
2021-12-10 23:40:05 [ERROR ] [configure] E: Unable to locate package ruby-foreman-host-reports
2021-12-10 23:40:05 [ERROR ] [configure] /Stage[main]/Foreman::Plugin::Host_reports/Foreman::Plugin[host_reports]/Package[ruby-foreman-host-reports]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install ruby-foreman-host-reports' returned 100: Reading package lists...
2021-12-10 23:40:05 [ERROR ] [configure] Building dependency tree...
2021-12-10 23:40:05 [ERROR ] [configure] Reading state information...
2021-12-10 23:40:05 [ERROR ] [configure] E: Unable to locate package ruby-foreman-host-reports
2021-12-10 23:40:05 [NOTICE] [configure] 750 configuration steps out of 1044 steps complete.
2021-12-10 23:40:06 [NOTICE] [configure] 1000 configuration steps out of 1151 steps complete.
2021-12-10 23:40:06 [ERROR ] [configure] Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install ruby-smart-proxy-reports' returned 100: Reading package lists...
2021-12-10 23:40:06 [ERROR ] [configure] Building dependency tree...
2021-12-10 23:40:06 [ERROR ] [configure] Reading state information...
2021-12-10 23:40:06 [ERROR ] [configure] E: Unable to locate package ruby-smart-proxy-reports
2021-12-10 23:40:06 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Plugin::Reports/Foreman_proxy::Plugin::Module[reports]/Foreman_proxy::Plugin[reports]/Package[ruby-smart-proxy-reports]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install ruby-smart-proxy-reports' returned 100: Reading package lists...
2021-12-10 23:40:06 [ERROR ] [configure] Building dependency tree...
2021-12-10 23:40:06 [ERROR ] [configure] Reading state information...
2021-12-10 23:40:06 [ERROR ] [configure] E: Unable to locate package ruby-smart-proxy-reports

Apologies, this warning is a leftover, we did not make it in time and the plugin is not yet ready. We will remove the warning from the 3.1 release. You need to wait for 3.2, thank you!

https://projects.theforeman.org/issues/34140

1 Like

I have updated the OP with new information for 3.2 and we will keep updating it as we approach 3.2 GA.

1 Like

followed the guide (foreman, puppetdb, puppetserver installed on same system CentOS 7).

issue appeared after migration (if you try to check the hosts in UI via All Hosts - Hosts):

@lzap - please advice how to solve this issue.

2022-03-21T22:47:01 [I|app|a91d2e6c]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-3.0.5/app/views/hosts/foreman_puppet/_form_list_env_field_header.html.erb (Duration: 1.0ms | Allocatio
ns: 420)
2022-03-21T22:47:01 [I|app|a91d2e6c]   Rendered hosts/_list.html.erb (Duration: 17.9ms | Allocations: 6597)
2022-03-21T22:47:01 [I|app|a91d2e6c]   Rendered hosts/index.html.erb within layouts/application (Duration: 137.3ms | Allocations: 45838)
2022-03-21T22:47:01 [W|app|a91d2e6c] undefined method `origin' for #<HostReport:0x00007f9d31c4d0a8>
2022-03-21T22:47:01 [I|app|a91d2e6c] Backtrace for 'undefined method `origin' for #<HostReport:0x00007f9d31c4d0a8>' error (ActionView::Template::Error): undefined method `origin' for #<HostReport:0x00007f9d31c4
d0a8>
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-6.0.3.7/lib/active_model/attribute_methods.rb:432:in `method_missing'
 a91d2e6c | /usr/share/foreman/app/models/host_status/configuration_status.rb:143:in `out_of_sync_disabled?'
 a91d2e6c | /usr/share/foreman/app/models/host_status/configuration_status.rb:17:in `out_of_sync?'
 a91d2e6c | /usr/share/foreman/app/models/host_status/configuration_status.rb:49:in `to_global'
 a91d2e6c | /usr/share/foreman/app/models/host_status/global.rb:20:in `block in build'
 a91d2e6c | /usr/share/foreman/app/models/host_status/global.rb:20:in `map'
 a91d2e6c | /usr/share/foreman/app/models/host_status/global.rb:20:in `build'
 a91d2e6c | /usr/share/foreman/app/models/host/managed.rb:814:in `build_global_status'
 a91d2e6c | /usr/share/foreman/app/helpers/hosts_helper.rb:102:in `host_global_status_icon_class_for_host'
 a91d2e6c | /usr/share/foreman/app/helpers/hosts_helper.rb:90:in `name_column'
 a91d2e6c | /usr/share/foreman/app/views/hosts/_list.html.erb:32:in `block in _daaf57a39eceb6378962661950b3879f'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in `each'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in `each'
 a91d2e6c | /usr/share/foreman/app/views/hosts/_list.html.erb:21:in `_daaf57a39eceb6378962661950b3879f'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/base.rb:274:in `_run'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/template.rb:185:in `block in render'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:182:in `instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/template.rb:385:in `instrument_render_template'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/template.rb:183:in `render'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:43:in `render'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/partial_renderer.rb:357:in `block in render_partial'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/abstract_renderer.rb:88:in `block in instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `block in instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/abstract_renderer.rb:87:in `instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/partial_renderer.rb:346:in `render_partial'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/partial_renderer.rb:317:in `render'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/renderer.rb:65:in `render_partial_to_object'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/renderer.rb:53:in `render_partial'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/helpers/rendering_helper.rb:38:in `render'
 a91d2e6c | /usr/share/foreman/app/views/hosts/index.html.erb:12:in `_8f55825959ec86185307c63706c59a23'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/base.rb:274:in `_run'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/template.rb:185:in `block in render'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:182:in `instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/template.rb:385:in `instrument_render_template'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/template.rb:183:in `render'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:43:in `render'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/template_renderer.rb:58:in `block (2 levels) in render_template'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/abstract_renderer.rb:88:in `block in instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `block in instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/abstract_renderer.rb:87:in `instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/template_renderer.rb:57:in `block in render_template'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/template_renderer.rb:65:in `render_with_layout'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/template_renderer.rb:56:in `render_template'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/template_renderer.rb:13:in `render'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/renderer.rb:61:in `render_template_to_object'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/renderer/renderer.rb:29:in `render_to_object'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:117:in `block in _render_template'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/base.rb:304:in `in_rendering_context'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:116:in `_render_template'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/streaming.rb:218:in `_render_template'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:103:in `render_to_body'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/rendering.rb:25:in `render'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rendering.rb:36:in `render'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:44:in `block (2 levels) in render'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
 a91d2e6c | /opt/rh/rh-ruby27/root/usr/share/ruby/benchmark.rb:308:in `realtime'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/core_ext/benchmark.rb:14:in `ms'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:44:in `block in render'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:84:in `cleanup_view_runtime'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/controller_runtime.rb:34:in `cleanup_view_runtime'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:43:in `render'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/implicit_render.rb:35:in `default_render'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `block in send_action'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `tap'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:195:in `process_action'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rendering.rb:30:in `process_action'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
 a91d2e6c | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 a91d2e6c | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 a91d2e6c | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:139:in `run_callbacks'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:41:in `process_action'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rescue.rb:22:in `process_action'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `block in instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `instrument'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:136:in `process'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:39:in `process'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:190:in `dispatch'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:254:in `dispatch'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in `each'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in `serve'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:834:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.4.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 a91d2e6c | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 a91d2e6c | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 a91d2e6c | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 a91d2e6c | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/cookies.rb:648:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:101:in `run_callbacks'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:37:in `call_app'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:28:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 a91d2e6c | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/executor.rb:14:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-cors-1.0.2/lib/rack/cors.rb:97:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:527:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `public_send'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `method_missing'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.2/lib/puma/configuration.rb:249:in `call'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.2/lib/puma/request.rb:77:in `block in handle_request'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.2/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.2/lib/puma/request.rb:76:in `handle_request'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.2/lib/puma/server.rb:447:in `process_client'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.5.2/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
 a91d2e6c | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2022-03-21T22:47:01 [I|app|a91d2e6c]   Rendering common/500.html.erb within layouts/application
2022-03-21T22:47:01 [I|app|a91d2e6c]   Rendered common/500.html.erb within layouts/application (Duration: 1.7ms | Allocations: 789)
2022-03-21T22:47:01 [I|app|a91d2e6c]   Rendered layouts/_application_content.html.erb (Duration: 0.7ms | Allocations: 397)
2022-03-21T22:47:01 [I|app|a91d2e6c]   Rendering layouts/base.html.erb
2022-03-21T22:47:01 [I|app|a91d2e6c]   Rendered layouts/base.html.erb (Duration: 5.2ms | Allocations: 3417)
2022-03-21T22:47:01 [I|app|a91d2e6c] Completed 500 Internal Server Error in 262ms (Views: 10.3ms | ActiveRecord: 53.3ms | Allocations: 68884)
[root@pp-master-test-02 ~]# foreman-rake host_reports:refresh
rake aborted!
can't add a new key into hash during iteration
/usr/share/foreman/app/models/host_status.rb:7:in `add'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_host_reports-1.0.2/app/models/host_status/host_report_status.rb:185:in `<top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:411:in `block in require_or_load'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:40:in `block in load_interlock'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:13:in `loading'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:40:in `load_interlock'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:389:in `require_or_load'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:544:in `load_missing_constant'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_hooks-0.3.17/lib/foreman_hooks/as_dependencies_hook.rb:4:in `load_missing_constant'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:214:in `const_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:284:in `const_get'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:284:in `block in constantize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:280:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:280:in `inject'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:280:in `constantize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:622:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:653:in `constantize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/inheritance.rb:229:in `find_sti_class'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/inheritance.rb:215:in `discriminate_class_for_record'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:257:in `instantiate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:58:in `block (2 levels) in find_by_sql'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/result.rb:62:in `block in each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/result.rb:62:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/result.rb:62:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:58:in `map'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:58:in `block in find_by_sql'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:56:in `find_by_sql'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:821:in `block in exec_queries'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:839:in `skip_query_cache_if_necessary'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:808:in `exec_queries'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/association_relation.rb:36:in `exec_queries'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:626:in `load'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:250:in `records'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:245:in `to_ary'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/association.rb:205:in `find_target'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_association.rb:274:in `load_target'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_proxy.rb:43:in `load_target'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_proxy.rb:1001:in `records'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in `each'
/usr/share/foreman/app/models/host/managed.rb:805:in `detect'
/usr/share/foreman/app/models/host/managed.rb:805:in `get_status'
/usr/share/foreman/app/models/host/managed.rb:792:in `block in refresh_statuses'
/usr/share/foreman/app/models/host/managed.rb:791:in `refresh_statuses'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:229:in `block (5 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `block in find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:222:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:222:in `in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:69:in `find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:228:in `block (4 levels) in <top (required)>'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:227:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:226:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
Tasks: TOP => host_reports:refresh

@lzap @ezr-ondrej @ekohl,

would be good if you could advice here at this the issue. appeared after migration of reports.
host_reports_plugin and smart_proxy_reports_plugin is also installed.

Hello,

please continue using old endpoints for now, legacy reports still do work fine in 3.2. I am looking into the issue now.

Okay it looks like there are still old host status objects for legacy configuration in your database and the recalculation of statuses does not remove them. Please remove them manually:

foreman-rake console
> HostStatus::Status.where(type: "HostStatus::ConfigurationStatus").delete_all

This is safe to do, you can always rebuild host statuses manually back if you choose to go back with the following command after you remove the plugin:

foreman-rake host_reports:refresh

Let us know how it went. Here is the PR to fix the migration step: Delete config statuses prior refreshing by lzap · Pull Request #35 · theforeman/foreman_host_reports · GitHub

Edit: Corrected the delete statement.

1 Like

Hi,
the first command did the trick to repaid the “Hosts - All Hosts” visiblilty issue :slight_smile:

currently i can´t se a report under the hosts in the new format, the old reports are still visible in the other view with the hint for the migration.

executing the refresh runs into the following:

[root@pp-master-test-02 ~]# foreman-rake host_reports:refresh
rake aborted!
can't add a new key into hash during iteration
/usr/share/foreman/app/models/host_status.rb:7:in `add'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_host_reports-1.0.2/app/models/host_status/host_report_status.rb:185:in `<top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:411:in `block in require_or_load'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:40:in `block in load_interlock'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:13:in `loading'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:40:in `load_interlock'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:389:in `require_or_load'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:544:in `load_missing_constant'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_hooks-0.3.17/lib/foreman_hooks/as_dependencies_hook.rb:4:in `load_missing_constant'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:214:in `const_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:284:in `const_get'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:284:in `block in constantize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:280:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:280:in `inject'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/inflector/methods.rb:280:in `constantize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:622:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:653:in `constantize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/inheritance.rb:229:in `find_sti_class'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/inheritance.rb:215:in `discriminate_class_for_record'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:257:in `instantiate'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:58:in `block (2 levels) in find_by_sql'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/result.rb:62:in `block in each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/result.rb:62:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/result.rb:62:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:58:in `map'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:58:in `block in find_by_sql'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:56:in `find_by_sql'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:821:in `block in exec_queries'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:839:in `skip_query_cache_if_necessary'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:808:in `exec_queries'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/association_relation.rb:36:in `exec_queries'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:626:in `load'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:250:in `records'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:245:in `to_ary'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/association.rb:205:in `find_target'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_association.rb:274:in `load_target'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_proxy.rb:43:in `load_target'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/associations/collection_proxy.rb:1001:in `records'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in `each'
/usr/share/foreman/app/models/host/managed.rb:805:in `detect'
/usr/share/foreman/app/models/host/managed.rb:805:in `get_status'
/usr/share/foreman/app/models/host/managed.rb:792:in `block in refresh_statuses'
/usr/share/foreman/app/models/host/managed.rb:791:in `refresh_statuses'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:229:in `block (5 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:70:in `block in find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:222:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:222:in `in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/batches.rb:69:in `find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:228:in `block (4 levels) in <top (required)>'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:227:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake:226:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
Tasks: TOP => host_reports:refresh
(See full trace by running task with --trace)

I just noticed the “can’t add a new key into hash during iteration” message, that is a different problem which stems from the Ruby loading of classes. I hope I solved it, I don’t see this on my development setup. Can you try this new version of the migrate script:

Either patch your instance or use View file and copy paste the script contents and overwrite the existing file. You will find the file somewhere in /opt directory on Red Hats, use GNU find to locate it :slight_smile:

@lzap, i can confirm that the migrate and the refresh are now working with your fix,
i copied it into “/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_host_reports-1.0.2/lib/tasks/migrate.rake” and there are no stacktraces even more appearing.

this update should be included in the next release.

the hostoverview works again, but if you check the new report view the reports are not shown, they are still in the old view shown and also not visible after migrating them.

i configured on my puppetmaster following:

    reports = log,store,http,puppetdb
    reporturl = https://<fqdn>:8443/reports/puppet

Seems to be a long way :wink:

1 Like

Thanks for the update.

Absolutely, will be.

Can you elaborate? What you mean by “new report view” there are two places, do you mean Monitor - Host Reports menu item? What do you mean by “in the old view” and “not visible after migrating” (where)? Can you perhaps attach screenshots?

Let me try to elaborate how this is supposed to work:

  • You install the plugin and perform migration.
  • New screen appears in Monitor - Host Reports with migrated reports.
  • You configure your puppet to point to the smart proxy where you installed the plugin.
  • New Host Reports should be coming in.
  • The old screen Monitor - Configuration Reports will still work as well as the API.
  • You can fallback to the old format at any time if you find a blocker bug. Just uninstall the Foreman plugin and perform the “refresh”.
  • If you are happy, you can expire old reports, the screen will remain in Foreman until we remove it sometimes in Foreman 3.3.

You appear to be the first one to migrate, no money-back warranties :slight_smile:

Edit: I am able to connect with you via Google Meet or similar so you can share your screen so we can get this resolved quickly, let me know.

@lzap,

i meant this

There you can see that in the host-view - tab reports no new report appears.

in the old view

there are now also no new reports appearing. This could be because the reportsurl is now not even more configured in the old way.

i checked something out:

[root@pp-master-test-02 ~]# curl https://<fqdn>:8443/reports/puppet -k
could not read client cert from environment

[root@pp-master-test-02 ~]# curl https://<fqdn>:8443/reports/puppet
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

[root@pp-master-test-02 ~]# curl -v https://<fqdn>:8443/reports/puppet
* About to connect() to <fqdn> port 8443 (#0)
*   Trying <ip>...
* Connected to <fqdn> (<ip>) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* Server certificate:
*       subject: CN=<fqdn>
*       start date: Aug 12 21:05:04 2021 GMT
*       expire date: Aug 12 21:05:04 2026 GMT
*       common name: <fqdn>
*       issuer: CN=Puppet CA: <fqdn>
* NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER)
* Peer's certificate issuer has been marked as not trusted by the user.
* Closing connection 0
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

i seems there are still some certificate issues ?
could i missed some configuration steps ?

@lzap i am currently in vacation, so call is currently not possible, i check here what i can do from time to time over the day :slight_smile: → but maybe that certificate issue is the last what could cause problems ?

Do you see any reports via Monitor - Host Reports?

Can you find any reports directly via Rails Console for this particular host?

foreman-rake console

[7] pry(main)> host_id = Host.find_by_name("report.example.com").id
=> 45
[8] pry(main)> HostReport.where(host_id: host_id).pluck(:id, :reported_at)
=> [[682, Fri, 11 Mar 2022 09:48:58 UTC +00:00],
 [683, Fri, 11 Mar 2022 09:48:58 UTC +00:00]]
[9] pry(main)> HostReport.count
=> 1341

The SSL error is expected, smart proxy requires all HTTPS requests, including the one that creates a report, to have its CN (Common Name) from the list from settings.yml entry trusted_hosts. See the original post for details. Typically this is FQDN of your puppet server. If your puppet server was deployed with our installer and smart proxy as well, then this will work out of box (they are on the same host).

Who performs upgrades and migrations on vacations? :slight_smile:

There are two problems. First, you should see all those migrated reports, maybe there were errors during migration so they were not created?

Second, SSL is not correct, yeah, this needs to be fixed.

Hi @lzap,

its for me a playground system so i am not really working on something and i am interested in the systems i support ^^

anyway, executing the commands works:

[root@pp-master-test-02 ~]# foreman-rake console
Loading production environment (Rails 6.0.3.7)
irb(main):001:0> host_id = Host.find_by_name("<fqdn>").id
=> 1

irb(main):002:0> HostReport.where(host_id: host_id).pluck(:id, :reported_at)
=> [[2966, Wed, 23 Mar 2022 19:57:50 UTC +00:00], [2960, Wed, 23 Mar 2022 19:27:49 UTC +00:00], [2954, Wed, 23 Mar 2022 18:57:49 UTC +00:00], [2948, Wed, 23 Mar 2022 18:27:49 UTC +00:00], [2942, Wed, 23 Mar 2022 17:57:48 UTC +00:00], .....

irb(main):003:0> HostReport.count
=> 2968

and i also can see under “Monitoring - Host Reports” the reports, but not as you saw in the picture before under “Hosts - All hosts - “selected host” - tab Reports” :smiley:

but maybe i understand here something wrong :slight_smile:

1 Like

Hello,

I’m running Foreman 3.2.0 and I migrated to new reports format. The migration went smoothly, however I noticed that dashboard widgets stopped working (in a sense that they don’t contain any data or show obsolete data).

Is this expected behavior or a bug?

Hey, @bobbytables, what version of Foreman Host Reports plugin do you have? Issues with the widgets should be fixed in 1.0.2 version of the plugin.

1 Like

@ofedoren I’m using version 1.0.2 (tfm-rubygem-foreman_host_reports-1.0.2-1.fm3_2.el7.noarch).