Provisioning Templates / Testing & Separate Repository

how does that solve the problem of having to manually make sure the repos are in sync all the time?

I am not going to re-read the whole thread, but this was created as ā€œtesting & separate repositoryā€ so I am trying to figure out the best approach for tests. If this is your major concern too, than I think we should ship community-templates as a gem dependency and remove copies in core git. Thatā€™s whatā€™s best for users. Or to merge everything into single core git, thatā€™s whatā€™s best for developers.

My proposal when bringing this thread back after 6 months of inactivity, following the issues we hit during this branching and release cycle, was to merge it into core.
I still believe this is the simplest solution and am still not convinced there is any significant benefit to be gained from the effort required to extract it into a separate gem.
We still want to have snapshots in foreman core to make sure changes in core code donā€™t break template rendering, and that means we would need to update the snapshots in two repos for every change in the templates.

I agree however I can feel the pain for users. Recently, I had to checkout 2GB linux kernel git just to update few lines in the documentation/ directory.

As I mentioned above, some of the users already do submit prs directly to foreman only to be told to go to community templates. Foreman is just about 100MB repo, and shouldnā€™t take more than a few minutes to clone even on a slow connection. Some small contributions can even be done using the github UI.

Iā€™ve changed my vote to YES. I think benefits are above cons.

2 Likes

The discussion here kind of died down and the vote didnā€™t come out with a strong outcome on either side. I would like to get this resolved before 2.0 so we donā€™t keep hitting these issues.
@ekohl, @Dirk and @ezr-ondrej - since you voted against the merge, could you please suggest a different resolution that would solve the issues we are now hitting and be easy to implement? or would you be willing to accept the merge into core despite your concerns and preferences for a different resolution?
@core - does anyone else have a strong preference either way?

I think that itā€™s common for organizations to have their internal repository and then use foreman_templates to sync. Having community-templates as an example of how to do that, including testing would be best. Weā€™d be dogfooding the tools we recommend to users meaning better coverage.

That means we should:

  • Convert community-templates into a gem that we package nightly
  • Add testing
  • Consume templates via the gem
  • Drop all templates from core

Other reasons to do this is that we can easily give people commit access on community-templates without core access. We donā€™t do this now, but IMHO we should because itā€™s the best way to change (provisoning) users into maintainers.

1 Like

How does this suggestion help with dogfooding foreman_templates? Or do you suggest people stop using foreman_templates and build their own gems? Also Iā€™d be interested in amount of effort for this and if itā€™s easily feasible, who would implement this solution for 1.25?

I was suggesting it would dogfood the testing capabilities Foreman can provide for standalone repositories which makes those repositories easier to use with foreman_templates.

Iā€™d argue if weā€™re not using foreman_templates and even stopping rsyncing of templates from that repo, by starting consuming them from a gem, we make the situation even worse. The packaging process can change the repo structure.

I like your argument of maintaining an example repo of templates for foreman_templates repo. But at the same time, itā€™s even better to create that example yourself by extracting the template using the same plugin. That extracts up to date metadata. We donā€™t always have correct metadata in the community-templates repo today. Weā€™re not updating them when we update templates. So Iā€™d recommend everyone who is interested in this use case, export first and build on top of that.

1 Like

Good point.

How are we going to deal with templates for plugins? Do we move all REX templates into that plugin?

I see the same problems Ewoud mentioned.

On the one hand I see people syncing the templates via git, having their own branch or just wanting the newest additions and additional templates not available in Foreman itself. On the other hand I fear contributions in core are more difficult than in community templates discouraging possible contributors.

But I have no solution for this, so if it is a way to improve quality by merging them to core, I am perhaps not happy with it, but can life with it.

Iā€™d personally prefer to move them to core too with proper metadata specifying requirements. Just to be consistent with other templates. Also Iā€™m dreaming about a day REX is merged to core, but this wouldnā€™t make it any harder in future, so itā€™s not a strong opinion.

If we merge job templates, we may need to add some rescue logic for unknown template models. But itā€™s trivial and could be implemented as another seed prerequisity.

How about this:

  1. Letā€™s merge community templates into core in a way that folders structure is preserved.
  2. Letā€™s add a script back to community-templates (or job) that will sync the templates on a daily basis back from core
  3. Letā€™s lock issues and PRs on github and modify the README to explain what just happened
  4. Users can carry on syncing templates over git
2 Likes

I wonder what the benefit of this is. Is it as a base so they can fork the repo? Wouldnā€™t it be easier to only link to the directory in foreman.git in the README and provide instructions on how to export the current templates in their Foreman install?

You can disable issues, but not PRs. Thatā€™s a Github limitation.

Good call, thatā€™s bad.

You could only add a fat warning in the pr-templateā€¦

I guess, this discussion wasnā€™t finished and we donā€™t have a nice solution yet. Therefore, I want to bring this up again.

if I remember correctly, the general opinion was that we no longer wanted a community template, correct?

My opinion is, that we should move all community-templates to foreman repository (or foreman plugin repo) and get rid of the community-template repository. We should write a README in community templates to explain what happend and tell users in which new repository a template is now located.

2 Likes

Hey, well I donā€™t think so. I do not like moving templates back. Foreman is a huge git repo, we have a plugin that regularly sync templates and also we do not want to scare users who would like to do small contribution. Finally, the solution to this problem is easy and we do not need to extract our rendering into a separate gem I think.

The good solution would be to simply modify our tests in core to load templates not only from app/views/unattended/provisioning_templates but also from any directory itā€™s given via ENV variable. Then we can simply rewrite our community-templates task to:

  • checkout community-templates and foreman core repos
  • run all necessary tests against community-templates directory

The only drawback is I think that if we want to also run snapshot tests which are quite useful too, those would need to be stored in community-repos too. But I donā€™t believe this is a blocker - we would provide instructions how to run tests against these or how to generate them via a rake task.