I was suggesting it would dogfood the testing capabilities Foreman can provide for standalone repositories which makes those repositories easier to use with foreman_templates.
I’d argue if we’re not using foreman_templates and even stopping rsyncing of templates from that repo, by starting consuming them from a gem, we make the situation even worse. The packaging process can change the repo structure.
I like your argument of maintaining an example repo of templates for foreman_templates repo. But at the same time, it’s even better to create that example yourself by extracting the template using the same plugin. That extracts up to date metadata. We don’t always have correct metadata in the community-templates repo today. We’re not updating them when we update templates. So I’d recommend everyone who is interested in this use case, export first and build on top of that.
1 Like
Good point.
How are we going to deal with templates for plugins? Do we move all REX templates into that plugin?
I see the same problems Ewoud mentioned.
On the one hand I see people syncing the templates via git, having their own branch or just wanting the newest additions and additional templates not available in Foreman itself. On the other hand I fear contributions in core are more difficult than in community templates discouraging possible contributors.
But I have no solution for this, so if it is a way to improve quality by merging them to core, I am perhaps not happy with it, but can life with it.
I’d personally prefer to move them to core too with proper metadata specifying requirements. Just to be consistent with other templates. Also I’m dreaming about a day REX is merged to core, but this wouldn’t make it any harder in future, so it’s not a strong opinion.
If we merge job templates, we may need to add some rescue logic for unknown template models. But it’s trivial and could be implemented as another seed prerequisity.
How about this:
- Let’s merge community templates into core in a way that folders structure is preserved.
- Let’s add a script back to community-templates (or job) that will sync the templates on a daily basis back from core
- Let’s lock issues and PRs on github and modify the README to explain what just happened
- Users can carry on syncing templates over git
2 Likes
I wonder what the benefit of this is. Is it as a base so they can fork the repo? Wouldn’t it be easier to only link to the directory in foreman.git in the README and provide instructions on how to export the current templates in their Foreman install?
You can disable issues, but not PRs. That’s a Github limitation.
Good call, that’s bad.
You could only add a fat warning in the pr-template…
I guess, this discussion wasn’t finished and we don’t have a nice solution yet. Therefore, I want to bring this up again.
if I remember correctly, the general opinion was that we no longer wanted a community template, correct?
My opinion is, that we should move all community-templates to foreman repository (or foreman plugin repo) and get rid of the community-template repository. We should write a README in community templates to explain what happend and tell users in which new repository a template is now located.
2 Likes
Hey, well I don’t think so. I do not like moving templates back. Foreman is a huge git repo, we have a plugin that regularly sync templates and also we do not want to scare users who would like to do small contribution. Finally, the solution to this problem is easy and we do not need to extract our rendering into a separate gem I think.
The good solution would be to simply modify our tests in core to load templates not only from app/views/unattended/provisioning_templates but also from any directory it’s given via ENV variable. Then we can simply rewrite our community-templates task to:
- checkout community-templates and foreman core repos
- run all necessary tests against community-templates directory
The only drawback is I think that if we want to also run snapshot tests which are quite useful too, those would need to be stored in community-repos too. But I don’t believe this is a blocker - we would provide instructions how to run tests against these or how to generate them via a rake task.
TBH I still think we should just merge all the community templates into core.
We have a way already of defining which plugins are required for templates to work and we don’t show templates that have unmet requirements. Alternatively, we can move plugin-specific templates to the actual plugin that uses them, like we currently sync job templates to rex instead of to core.
it’s confusing to users when they are sent to a different repo for contributing to a template, and to a contributor who is properly directed it doesn’t matter if the repo includes only the templates or a whole bunch of other code. on the contrary, if the contribution is done in core it also gets tested right there in the PR and we can tell right away if a snapshot needs updating. PRs in community templates also have a much slower response rate since less people are going over them.
As the one doing the release work for the past year and half, it’s a pain every release to sync them, make sure they work and track which fix landed in what release (and what needs to be backported to fix bugs). There is a good reason we use redmine for tracking issues.
There has been talk for years now on how we can keep community templates and better sync them to core and test them but no one has actually implemented anything other than suggesting that we can improve.
Lets move them to core and get this over with.
I just think it’s too much burden for users. For example from now on, a redmine ticket will be required for small one liners. Previously, templates did not have even RM project. Looking in the git history, we do get quite many contributions from our users. Maybe we can find easier workflow for these small fixes.
However, I do not insist on this. If you think syncing is the problem, then let’s do it
- would it be possible to use a git submodule? does this make sense?
- would it be possible to change the rule that a redmine ticket is necessary for certain directories within core?
for me, it would also be OK to move the templates OUT of foreman and package them in a separate RPM. Opinions?
I suggest to stay away from gitmodules at all costs. Let’s simply merge it into core then.
Not against, however not a goal I think.
With the 2.1 cycle under way, this came up as a pain point again in a few cases:
- Some templates were renamed, causing 2.1.0-RC1 to actually ship with duplicate templates.
- A fix that was merged into community templates had a bug that was only caught when syncing the template (due to snapshot tests). As a result several commits had to be cherry-picked post branching in both repos and required an extra cycle of template syncing (https://github.com/theforeman/foreman/pull/7644#issuecomment-628010147)
- missing mapping of commits in community templates to redmine made it difficult to tell which changes were needed to be pulled into the 2.1-stable branch for the next RCs. Each change to the community templates at this stage requires extra work to get into the 2.1 branch of core instead of only cherry-picking a single commit, and brings risk of snapshot or other template testing breaking after syncing.
My proposal to move forward (which also addresses some of the previously discussed concern):
- provisioning, partition and report templates will be synced and live in core. Metadata can be used to make sure templates are only seeded if a dependent plugin is installed.
- job templates will live either in the REX repo (for the ssh templates) or the respective plugin adding a REX provider (e.g. ansible/salt). We might want to look into template metadata as well in only seeding templates matching installed plugins.
- The community templates repo will be archived and users will be asked to submit PRs to either Core or REX/providers.
- Core maintainers will assist any community contributor in creating a redmine issue for pull requests or with fixing tests/snapshots if needed.
2 Likes
Onboard! I’ve wanted this simplification for a while with respect to cherry picking and releases.
Two ideas to help with this:
- Add a new label to specifically call out template PRs on the repository for easy filtering and review
- Have our prprocessor provide a link to issue creation URL (e.g. https://projects.theforeman.org/projects/foreman/issues/new for Foreman) if a ‘Fixes/Refs’ is not detected in a comment on the PR
2 Likes
One more thing is how to handle open PRs, this will be quite disturbing change we should think it thorugh.
There are just 20 open PRs right now, many of them inactive for a long time. Lets see what can be merged and merge them before the last sync, and ask the remaining contributors to reopen the PRs against the relevant repo or help them to do so if needed.
We discussed this at a meeting of core maintainers last week, and came to agreement that we will be doing the final sync and archiving the repo right after 2.1 GA - which is currently planned for June 16th. In the meantime we will try to get as many PRS as possible merged into the community-templates repo to reduce the effort of having to reopen them against core or plugin repos. Expect the repo to be officially retired by June 21st.
If you have any open PRs that you don’t expect will be merged by then, feel free to close them and open against the new home of the template, but if you do so before the final sync make sure to mark your PR as draft for now so it isn’t lost by mistake when syncing.
1 Like