Attention: Development of the new Host Reports plugin was ceased, do not perform the migration. The migration note will be removed from Foreman 3.2 as a minor update. Sorry for inconvenience. If you have already migrated, perform the following steps to rollback:
Remove database tables of the plugin:
foreman-rake db:migrate:down VERSION=20220113064436
foreman-rake db:migrate:down VERSION=20211011141813
foreman-rake db:migrate:down VERSION=20210616133601
foreman-rake db:migrate:down VERSION=20210112183526
Uninstall the plugin:
# foreman-installer --no-enable-foreman-plugin-host-reports --no-enable-foreman-proxy-plugin-reports
To reset Ansible report back to the original endpoint, run:
# foreman-installer --foreman-proxy-plugin-ansible-report-type foreman
Puppet configuration was done manually via puppet config set so it should have been overwritten by foreman-installer automatically by each run. Delete all Host Report Status and Widget objects:
# foreman-rake
> HostStatus::Status.where(type: "HostStatus::HostReportStatus").delete_all
> Widget.where("template LIKE 'host_reports%'").delete_all
> Host.unscoped.all.find_each { |h| h.refresh_statuses; h.refresh_global_status }
If you deleted the old reports already, you need to wait for new to come. Apologies for the trouble.
Update: We have decided to cease development of this plugin, we will remove the upgrade migration warning from the original Reports page as an update for 3.1. Please uninstall the plugin and reconfigure Foreman back to its original reports API endpoint using the instructions in this comment. If you have any problems, let us know in the comments below.
The way Foreman handles configuration management reports is changing, they are now stored in more efficient way. If you upgraded to 3.2 and noticed there is a warning when you visit Configuration Reports page, you need to reconfigure your Ansible and Puppet callback to send reports to the new API. The old API and Configuration Reports screen will continue working in 3.2 normally, but it is planned to be removed in 3.3 so we recommend to migrate early.
The migration plan
After installation of Foreman Host Reports plugin, new menu item will appear: Monitor - Host Reports. Also some links will be overriden to land on the new page, e.g. in the Host Detail page - Report. And new API and Smart Proxy Reports plugin will be available for report importing.
Migration is done via a rake task, it makes copies of selected configuration reports (older then particular date) in new tables so they appear in the new Monitor - Host Reports screens. Migration can be done while Foreman is running, can be paused at any point with Ctrl-C and resumed from the particular host ID.
New Smart Proxy Reports plugin must be installed, it is responsible for parsing incoming reports in native formats (Puppet YAML, Ansible JSON), converting it, creating keywords and sending them to Foreman. Configuration of both Ansible and Puppet is in this document.
After some testing period, we recommend to expire (or truncate) old (configuration) reports in the database. During the testing period, we recommend to test all your workflows, search queries and operation tasks. Note that the new report format is vastly different from the original which was Puppet-oriented. It is now more generic but also much more faster and efficient, this will help a lot on deployments with many reports.
Please report back to this thread with your remarks, experience, bugreports or suggestions for the next version. Also see âknown issuesâ part down below for some known limitations or bugs we are working on.
Upgrading
Perform the upgrade to Foreman 3.2 as usual. Then enable the new Host Reports Foreman plugin and Smart Proxy Reports plugin:
# foreman-installer --enable-foreman-plugin-host-reports --enable-foreman-proxy-plugin-reports
When using Ansible, set the report type to âproxyâ which will configure ansible report callback to send reports to smart proxy in the new format.
# foreman-installer --foreman-proxy-plugin-ansible-report-type proxy
To revert back to the old report format (which will be removed in Foreman 3.3) set the setting back to âforemanâ. Note this API will stop working in Foreman 3.3.
Puppet donât have installer support for the new report format yet, you need to perform the configuration of puppet server manually:
# source /etc/profile.d/puppet-agent.sh
# puppet config set reports store,http
# puppet config set reporturl https://HOSTNAME:8443/reports/puppet
# systemctl restart puppetserver
When running the Smart Proxy with the Report plugin on the same host as puppetserver, no additional configuration is needed. Otherwise, ensure that the puppet server client SSL common name (typically FQDN of the puppet server) is in the trusted hosts:
# grep trusted /etc/foreman-proxy/settings.yml -A2
:trusted_hosts:
- HOSTNAME
Puppet and Ansible can send reports without a valid SSL client certificate if itâs sent from the same host when smart-proxy is running on. Trusted host entry does automatically match.
Migration
Nothing happened to the old Configuration Reports, in fact, they are still operating normally and you can keep using them in Foreman 3.2. There will be a migration task available in the Host Reports plugin later so reports can be migrated to the new format before 3.3. is out. It is expected that if you choose to migrate all reports, it will take some time and this can be done while Foreman is running:
foreman-rake host_reports:migrate
The rake task accepts from_date option (older reports will be ignored) and from_id, primary key (ID) to start migration from which can be used to resume previously stopped migration. Examples:
foreman-rake host_reports:migrate from_date=2021-01-01
foreman-rake host_reports:migrate from_date=2021-01-01 from_id=1234567
When you encounter an issue during migration, please reply in this thread. After we release a fix, you can resume the migration at the very same position.
After migration is complete, check the new reports in Monitor - Host Reports. All migrated reports have a special keyword Migrated assigned so it is easy to search them later. Now, it is the time to expire all old reports:
foreman-rake reports:expire report_type=config_report days=0
The old report expiration can be very slow process and it is one of the reasons why reports are now stored in a different way. There are plugins which still use the old report format, these include: OpenSCAP, Salt or Chef. When these plugins are not installed or not used on your Foreman instance, deletion can be achieved much faster directly in the PosgreSQL database:
truncate logs, messages, resources, reports
After expiration or truncate operation, make sure to reclaim DB space by issuing VACUM maintenance command. Finally, after the migration process, some hosts may have its host status calculated incorrectly. In that case, refresh all host statuses (Global Status, Configuration Status):
foreman-rake host_reports:refresh
This operation can be also done when Foreman is running.
Dashboard
The following charts are deprecated and will no longer work, after Host Report plugin is installed and reports are coming in, you may remove these from your dashboard by clicking the X button:
- Host Configuration Status for All/Puppet/Ansible/âŚ
- Host Configuration Chart for All/Puppet/Ansible/âŚ
- Run Distribution Chart for Puppet/Ansible/âŚ
- Latest Events
Instead, use the following new widgets added by the Host Report plugin:
- All Host Reports
- Puppet Host Reports
- Ansible Host Reports
- All Host Reports Chart
- Puppet Host Reports Chart
- Ansible Host Reports Chart
- Latest Host Reports
Those widgets primarily show information based on the changed, unchanged and failed counters. That is more limiting that in the previous format, however, reports are much more effective (time and storage). We are also looking for feedback - let us know if you are missing some particular data. It could be possible for example create a widgets that would show particular keywords.
New installs
New installation is the same as described in the upgrade steps above, except no migration is needed. In this case, simply ensure the Host Reports and Smart Proxy Reports plugins are installed (see above) and ensure report type is set to proxy (see above) for Ansible. For Puppet, configure puppet server (see above).
Troubleshooting and known issues
Unable to delete an existing host due to Foreign Key error
There is a bug in foreign key association in Foreman 3.2.0, please upgrade to 3.2.1 once it is out. In the meantime, you can apply the following patch to solve the problem.
Canât add a new key into hash during iteration
If you see this error during foreman-rake host_reports:migrate please upgrade to foreman_host_reports 1.0.3 or apply the following patch to solve it.
Undefined method origin for HostReport
If you see this error during foreman-rake host_reports:migrate please upgrade to foreman_host_reports 1.0.3 or apply the following patch to solve it.
If you run into any issues, please drop a comment below.
Email notifications do not work
This is a missing feature we are currently working on and will be released as a minor update for Foreman 3.2.
