Hello,
since Foreman 3.2 is approaching, I want to share some news about Reports refactoring. A feature we have been working on with Oleh, Anna and Ewoud with help from Maria (UX) and Ondrej (QA).
Host Reports?
There is a new plugin called host_reports which provide new models, controllers, screens and API for storing configuration management reports. It is a complete refactoring of reports which were architected around Puppet and other tools need to map their data to it and it is also extremely inefficient when storing. Because users can have many reports and may want to migrate them all, we have decided to implement the new reports feature as a plugin installable alongside the old core feature, users can install the plugin at any point on 3.1 (it will be opt-in) and then migrate the data.
Another reason why we need to keep the core feature for another one or two releases is OpenSCAP plugin, which utilizes the data model to store its own data. Again, it maps and extends many fields and since the base data is not stored efficiently, OpenSCAP suffers too. The plan is to change OpenSCAP and migrate data to its very own table and store the data in more insert/update-friendly manner. This is not happening in 3.2 tho as it is much more work that I anticipated.
In the new design, reports are sent to a new smart proxy plugin first where they are parsed, transformed, some key data extracted as “keywords” and then sent over to Foreman where they are stored as JSON blobs instead broken up into lines like previously. Previous measurements showed 20 times better performance. Each (configuration) management tool therefore need to implement its own transforming smart proxy class, then add a new report type and implement its own “show screen” for the UI and CLI. Key advantage is that it can be unique and tailor-made for the particular tool which will lead to much better user experience. Also searching can be vastly improved thanks to dropping bitfields and introducing keywords which is essentially a report tagging.
The plugins are named as Host Reports because we wanted to find a name for such plugin and differentiate the model classes from core’s Report (reports). In the future, we might choose to merge the plugin back into Foreman, or keep it this way. This is open question.
We are finishing our MVP, packages are in nightly and we opened PRs for installer too. The plan is to have this as opt-in - if the new plugin is not installed, users will not see any difference. When a new plugin is installed tho, the following changes will appear:
- new API endpoint for sending reports
- new page Monitor - Host reports
- configuration status will show data from the plugin
- there will be new set of dashboard widgets for new data
- Reports button on the (old) host detail page will take users to the new page
- there will be new tab on the new host detail page with new reports
You can look on how the new screens will look like at: Puppet tab in Host details page - Proposal
Because of tight schedule, some screens in the initial version of the plugin might use the old Foreman look, we aim to upgrade them after 3.2.
We plan no support for the installer for Puppet, it needs to be configured manually and we will document that. We would like to provide installer support for Ansible callback configuration tho, again, as a opt-in.
Migration
The plugin will provide a rake task that can be executed during upgrade or even after 3.1 is running on the background to migrate all reports from the old tables to the new one. We have seen instances with 50TB of reports data, so this may take a while. Users can use migration to test how new reports look like and give us feedback so we can further optimize things like searching, search keywords or UI/CLI before we remove the reports table from core. Or they can decide to stay on the new reports, then they will be instructed to truncate old tables and vacuuming the database as the new format takes also significantly less database space.
What is planned after 3.2
- Improve UI
- Improve installer for end-to-end experience for both Puppet and Ansible
- Refactor OpenSCAP to use own table and format
- Drop old reports from Foreman (data, screens, models)
Links