While we do not guarantee that all shipped plugins are updated to be compatible with a release on day 0, we still try to achieve that and it would be nice if we could at least include a list of known issues in the release announcement.
Today, we have no way to know this. The plugins pipeline is responsible for publishing the packages, but does not test them at all.
While talking about this issue during a recent release meeting we came up with the idea that we could have a “plugins” test pipeline run directly after the normal Foreman release pipeline happens. This would not block the Foreman release (contrary to the failures during the Foreman pipeline itself), but at least give us an indicator whether the plugins that we ship work (well, are installable and don’t break anything).
We have something similar in the luna pipeline (see Testing more plugins together regularly for the old discussion around this), but it currently only runs on nightly (this is easy to change) and depends on Katello (not so easy to change) which tends to lag a few days behind Foreman releases, which makes it less helpful for inclusion in Foreman release announcements.
- create a forklift pipeline that can test installation of plugins (that don’t require Katello)
- trigger this pipeline when
- this won’t block plugin publishing (to not block good plugins)
- the results can be used to update a list of known issues in the release announcement
I’m happy to own the implementation of this.