One of the recent features added to Katello was support for puppet content. It allows devops
to tie the puppet manifests together with the software packages in specific versions inside content
view and use promotion path to get updates to test/production environments.
Currently, there are two ways how to get puppet content into Katello:
syncing remote locations that act as puppetforge
uploading the modules into katello in a form of tars
From what I've seen so far, non of these ways corresponds to how it's usually done in wild.
Therefore, it seems to me it might be really worth looking into ways how to support the gitish way
of doing things.
I was thinking about using git as the interface between the devops and Katello.
Let's consider this scenario:
I have a git repository containing all the modules needed for setting up some piece of infrastructure
(almost, but not quite, entirely unlike our foreman-installer repo)
Katello would provide a git repository (tied either to repo or maybe better to a content view definition)
I add this Katello's repo as remote to my git repository
When I push the repository to Katello, it would perform everything to publish new content view (or refresh
new content view version?) taking the content form the git repository (ideally including
creating the packages that Pulp accepts). Also, Katello would create a tag into the repository,
so that the connection between content view version and the git repository is really visible)
So in ideal case, the journey of my repository with all the modules to Katello/Foreman managed infrastructure
would be a question of one git push, something like PaaS for puppet modules And form the git history, I would
see instantly what versions are running in what environments.
The mapping also could be as follows:
git repository represents a content view definition
branches represent content views
tags represent content view versions
What other forks thing about this approach?
– Ivan
···
From my understanding, most of bigger deployments utilize git to hold and version the puppet modules.
I like the idea. My understading is that most of puppet users handles modules
in git so this would allow them to use existing and well known tool to use
katello.
···
On Tuesday 05 of November 2013 09:13:14 Ivan Necas wrote:
> Hi,
>
> One of the recent features added to Katello was support for puppet content.
> It allows devops to tie the puppet manifests together with the software
> packages in specific versions inside content view and use promotion path to
> get updates to test/production environments.
>
> Currently, there are two ways how to get puppet content into Katello:
>
> 1. syncing remote locations that act as puppetforge
> 2. uploading the modules into katello in a form of tars
>
> From what I've seen so far, non of these ways corresponds to how it's
> usually done in wild.
>
> From my understanding, most of bigger deployments utilize git to hold and
> version the puppet modules. Therefore, it seems to me it might be really
> worth looking into ways how to support the gitish way of doing things.
>
> I was thinking about using git as the interface between the devops and
> Katello.
>
> Let's consider this scenario:
>
> 1. I have a git repository containing all the modules needed for setting up
> some piece of infrastructure (almost, but not quite, entirely unlike our
> foreman-installer repo)
>
> 2. Katello would provide a git repository (tied either to repo or maybe
> better to a content view definition)
>
> 3. I add this Katello's repo as remote to my git repository
>
> 4. When I push the repository to Katello, it would perform everything to
> publish new content view (or refresh new content view version?) taking the
> content form the git repository (ideally including creating the packages
> that Pulp accepts). Also, Katello would create a tag into the repository,
> so that the connection between content view version and the git repository
> is really visible)
>
> So in ideal case, the journey of my repository with all the modules to
> Katello/Foreman managed infrastructure would be a question of one git push,
> something like PaaS for puppet modules :) And form the git history, I would
> see instantly what versions are running in what environments.
>
> The mapping also could be as follows:
>
> * git repository represents a content view definition
> * branches represent content views
> * tags represent content view versions
>
> What other forks thing about this approach?
Interesting idea indeed, but I think it only makes sense if we put
everything in the git - including repos/packages/filters. At least in
the form of references/pointers to the real content (not putting the
real repos/RPMs in there).
Just an idea of a fork…
···
On Tue, Nov 05, 2013 at 09:13:14AM -0500, Ivan Necas wrote:
> What other forks thing about this approach?
···
On Tue, Nov 05, 2013 at 09:13:14AM -0500, Ivan Necas wrote:
> Hi,
>
> One of the recent features added to Katello was support for puppet content. It allows devops
> to tie the puppet manifests together with the software packages in specific versions inside content
> view and use promotion path to get updates to test/production environments.
>
> Currently, there are two ways how to get puppet content into Katello:
>
> 1. syncing remote locations that act as puppetforge
> 2. uploading the modules into katello in a form of tars
>
> From what I've seen so far, non of these ways corresponds to how it's usually done in wild.
>
> From my understanding, most of bigger deployments utilize git to hold and version the puppet modules.
> Therefore, it seems to me it might be really worth looking into ways how to support the gitish way
> of doing things.
>
> I was thinking about using git as the interface between the devops and Katello.
>
> Let's consider this scenario:
>
> 1. I have a git repository containing all the modules needed for setting up some piece of infrastructure
> (almost, but not quite, entirely unlike our foreman-installer repo)
>
> 2. Katello would provide a git repository (tied either to repo or maybe better to a content view definition)
>
> 3. I add this Katello's repo as remote to my git repository
>
> 4. When I push the repository to Katello, it would perform everything to publish new content view (or refresh
> new content view version?) taking the content form the git repository (ideally including
> creating the packages that Pulp accepts). Also, Katello would create a tag into the repository,
> so that the connection between content view version and the git repository is really visible)
>
> So in ideal case, the journey of my repository with all the modules to Katello/Foreman managed infrastructure
> would be a question of one git push, something like PaaS for puppet modules :) And form the git history, I would
> see instantly what versions are running in what environments.
>
> The mapping also could be as follows:
>
> * git repository represents a content view definition
> * branches represent content views
> * tags represent content view versions
>
> What other forks thing about this approach?
Are you suggesting the library is sourced from git, or a content view
points to a branch in a git repo?
If the latter… how would items like the content browser work?
– bk
···
On 11/05/2013 10:07 AM, Marek Hulan wrote:
> On Tuesday 05 of November 2013 09:13:14 Ivan Necas wrote:
>> Hi,
>>
>> One of the recent features added to Katello was support for puppet content.
>> It allows devops to tie the puppet manifests together with the software
>> packages in specific versions inside content view and use promotion path to
>> get updates to test/production environments.
>>
>> Currently, there are two ways how to get puppet content into Katello:
>>
>> 1. syncing remote locations that act as puppetforge
>> 2. uploading the modules into katello in a form of tars
>>
>> From what I've seen so far, non of these ways corresponds to how it's
>> usually done in wild.
>>
>> From my understanding, most of bigger deployments utilize git to hold and
>> version the puppet modules. Therefore, it seems to me it might be really
>> worth looking into ways how to support the gitish way of doing things.
>>
>> I was thinking about using git as the interface between the devops and
>> Katello.
>>
>> Let's consider this scenario:
>>
>> 1. I have a git repository containing all the modules needed for setting up
>> some piece of infrastructure (almost, but not quite, entirely unlike our
>> foreman-installer repo)
>>
>> 2. Katello would provide a git repository (tied either to repo or maybe
>> better to a content view definition)
>>
>> 3. I add this Katello's repo as remote to my git repository
>>
>> 4. When I push the repository to Katello, it would perform everything to
>> publish new content view (or refresh new content view version?) taking the
>> content form the git repository (ideally including creating the packages
>> that Pulp accepts). Also, Katello would create a tag into the repository,
>> so that the connection between content view version and the git repository
>> is really visible)
>>
>> So in ideal case, the journey of my repository with all the modules to
>> Katello/Foreman managed infrastructure would be a question of one git push,
>> something like PaaS for puppet modules :) And form the git history, I would
>> see instantly what versions are running in what environments.
>>
>> The mapping also could be as follows:
>>
>> * git repository represents a content view definition
>> * branches represent content views
>> * tags represent content view versions
>>
>> What other forks thing about this approach?
>
> I like the idea. My understading is that most of puppet users handles modules
> in git so this would allow them to use existing and well known tool to use
> katello.
>
> > Hi,
> >
> > One of the recent features added to Katello was support for puppet content.
> > It allows devops
> > to tie the puppet manifests together with the software packages in specific
> > versions inside content
> > view and use promotion path to get updates to test/production environments.
> >
> > Currently, there are two ways how to get puppet content into Katello:
> >
> > 1. syncing remote locations that act as puppetforge
> > 2. uploading the modules into katello in a form of tars
> >
> > From what I've seen so far, non of these ways corresponds to how it's
> > usually done in wild.
> >
> > From my understanding, most of bigger deployments utilize git to hold and
> > version the puppet modules.
> > Therefore, it seems to me it might be really worth looking into ways how to
> > support the gitish way
> > of doing things.
> >
> > I was thinking about using git as the interface between the devops and
> > Katello.
> >
> > Let's consider this scenario:
> >
> > 1. I have a git repository containing all the modules needed for setting up
> > some piece of infrastructure
> > (almost, but not quite, entirely unlike our foreman-installer repo)
> >
> > 2. Katello would provide a git repository (tied either to repo or maybe
> > better to a content view definition)
> >
> > 3. I add this Katello's repo as remote to my git repository
> >
> > 4. When I push the repository to Katello, it would perform everything to
> > publish new content view (or refresh
> > new content view version?) taking the content form the git repository
> > (ideally including
> > creating the packages that Pulp accepts). Also, Katello would create a tag
> > into the repository,
> > so that the connection between content view version and the git repository
> > is really visible)
> >
> > So in ideal case, the journey of my repository with all the modules to
> > Katello/Foreman managed infrastructure
> > would be a question of one git push, something like PaaS for puppet modules
> > And form the git history, I would
> > see instantly what versions are running in what environments.
> >
> > The mapping also could be as follows:
> >
> > * git repository represents a content view definition
> > * branches represent content views
> > * tags represent content view versions
> >
> > What other forks thing about this approach?
>
> I am wondering how you think of dealing with external modules. Some
> commonly used options can be divided into two groups:
>
> * git based:
> ** submodules
> ** subtree
> * Puppetfile based:
> ** librarian-puppet https://github.com/rodjek/librarian-puppet
> ** r10k https://github.com/adrienthebo/r10k
I was thinking about resolving this when some commits were pushed
into this git repo: for every module it would need to create a puppet
tar (in order to be able to send it into pulp). The way how to get
the external packages could be configured, and probably all the listed
techniques could be supported.
So to be clear, it wouldn't use git directly to push the Puppet modules
into puppet master, but it would act as another user interface to get
the content into Katello/Pulp.
The advantage of packaging every module into a tar is having searchable
metadata, which could help with a mess that managing puppet modules with
git submodules often brings, that every puppet environment uses slightly
different version of some puppet module. It wouldn't be hard to compare
the modules used in environments against what's available in upstream.
>
> It could be solved by making it hookable. Either through a git hook,
> bin/make-release, rake make-release or something entirely different that
> I didn't think of. Then you can give the user full control and provide
> some examples. This is based on what foreman does:
> https://github.com/theforeman/puppet-puppet/blob/master/templates/server/post-receive.erb.
Yeah, we probably don't have to make this as part of Katello core code base.
For start, we need just a git repo with some hook that would perform the
right REST API calls.
– Ivan
···
----- Original Message -----
> On Tue, Nov 05, 2013 at 09:13:14AM -0500, Ivan Necas wrote:
> > What other forks thing about this approach?
>
> Interesting idea indeed, but I think it only makes sense if we put
> everything in the git - including repos/packages/filters. At least in
> the form of references/pointers to the real content (not putting the
> real repos/RPMs in there).
Interesting. Ideally, when the filters could be defined in plain text
of the git repo. It would be basically the source code for the packages
content So some data would be read/write (for example the filters),
some data would be read-only and Katello would be the authoritative
source there (the list of packages in the content view).
I bet for many folks no web UI could beat this user interface:)
E.g. what are the package differences between the content view in production
and evelopment. git diff CV-dev CV-prod katello/packages! Boom.
I think however, that even just with puppet content it would help a lot.
– Ivan
···
----- Original Message -----
> On Tue, Nov 05, 2013 at 09:13:14AM -0500, Ivan Necas wrote:
> >> Hi,
> >>
> >> One of the recent features added to Katello was support for puppet
> >> content.
> >> It allows devops to tie the puppet manifests together with the software
> >> packages in specific versions inside content view and use promotion path
> >> to
> >> get updates to test/production environments.
> >>
> >> Currently, there are two ways how to get puppet content into Katello:
> >>
> >> 1. syncing remote locations that act as puppetforge
> >> 2. uploading the modules into katello in a form of tars
> >>
> >> From what I've seen so far, non of these ways corresponds to how it's
> >> usually done in wild.
> >>
> >> From my understanding, most of bigger deployments utilize git to hold and
> >> version the puppet modules. Therefore, it seems to me it might be really
> >> worth looking into ways how to support the gitish way of doing things.
> >>
> >> I was thinking about using git as the interface between the devops and
> >> Katello.
> >>
> >> Let's consider this scenario:
> >>
> >> 1. I have a git repository containing all the modules needed for setting
> >> up
> >> some piece of infrastructure (almost, but not quite, entirely unlike our
> >> foreman-installer repo)
> >>
> >> 2. Katello would provide a git repository (tied either to repo or maybe
> >> better to a content view definition)
> >>
> >> 3. I add this Katello's repo as remote to my git repository
> >>
> >> 4. When I push the repository to Katello, it would perform everything to
> >> publish new content view (or refresh new content view version?) taking the
> >> content form the git repository (ideally including creating the packages
> >> that Pulp accepts). Also, Katello would create a tag into the repository,
> >> so that the connection between content view version and the git repository
> >> is really visible)
> >>
> >> So in ideal case, the journey of my repository with all the modules to
> >> Katello/Foreman managed infrastructure would be a question of one git
> >> push,
> >> something like PaaS for puppet modules And form the git history, I
> >> would
> >> see instantly what versions are running in what environments.
> >>
> >> The mapping also could be as follows:
> >>
> >> * git repository represents a content view definition
> >> * branches represent content views
> >> * tags represent content view versions
> >>
> >> What other forks thing about this approach?
> >
> > I like the idea. My understading is that most of puppet users handles
> > modules
> > in git so this would allow them to use existing and well known tool to use
> > katello.
> >
> Are you suggesting the library is sourced from git, or a content view
> points to a branch in a git repo?
>
> If the latter… how would items like the content browser work?
Non of them The suggestion is about having something that would be able
to keep the git repository in sync with the katello-managed content.
So for example with the foreman-installer repository, lets say I want to publish new
content view version based on the changes in that repo. Therefore I push the changes
into Katello's git repo. It would fetch third-party packages (either from submodules,
using librarian, r10k or whatever, based on configuration), package the rest of modules
and push all to the corresponding Pulp repo. Then it would publish/refresh new content
view version, setting the filters in a way that the result is the same as I have in
the foreman-installer repository.
In fact, there could be only one Pulp repository per Katello organization, to hold
all the puppet modules. The git repo would push the packages to the pulp repo (if
they're not already there) and then publish the content view through applying a right
filter on that repo.
Git would not be involved in the process of getting the pupppet content to puppetmaster
and we would use the standard mechanism that Pulp already has.
Is it cleaner now?
– Ivan
···
----- Original Message -----
> On 11/05/2013 10:07 AM, Marek Hulan wrote:
> > On Tuesday 05 of November 2013 09:13:14 Ivan Necas wrote:
>
>
> > > Hi,
> > >
> > > One of the recent features added to Katello was support for puppet content.
> > > It allows devops
> > > to tie the puppet manifests together with the software packages in specific
> > > versions inside content
> > > view and use promotion path to get updates to test/production environments.
> > >
> > > Currently, there are two ways how to get puppet content into Katello:
> > >
> > > 1. syncing remote locations that act as puppetforge
> > > 2. uploading the modules into katello in a form of tars
> > >
> > > From what I've seen so far, non of these ways corresponds to how it's
> > > usually done in wild.
> > >
> > > From my understanding, most of bigger deployments utilize git to hold and
> > > version the puppet modules.
> > > Therefore, it seems to me it might be really worth looking into ways how to
> > > support the gitish way
> > > of doing things.
> > >
> > > I was thinking about using git as the interface between the devops and
> > > Katello.
> > >
> > > Let's consider this scenario:
> > >
> > > 1. I have a git repository containing all the modules needed for setting up
> > > some piece of infrastructure
> > > (almost, but not quite, entirely unlike our foreman-installer repo)
> > >
> > > 2. Katello would provide a git repository (tied either to repo or maybe
> > > better to a content view definition)
> > >
> > > 3. I add this Katello's repo as remote to my git repository
> > >
> > > 4. When I push the repository to Katello, it would perform everything to
> > > publish new content view (or refresh
> > > new content view version?) taking the content form the git repository
> > > (ideally including
> > > creating the packages that Pulp accepts). Also, Katello would create a tag
> > > into the repository,
> > > so that the connection between content view version and the git repository
> > > is really visible)
> > >
> > > So in ideal case, the journey of my repository with all the modules to
> > > Katello/Foreman managed infrastructure
> > > would be a question of one git push, something like PaaS for puppet modules
> > > And form the git history, I would
> > > see instantly what versions are running in what environments.
> > >
> > > The mapping also could be as follows:
> > >
> > > * git repository represents a content view definition
> > > * branches represent content views
> > > * tags represent content view versions
GPG signed tags could please some people as well.
> > >
> > > What other forks thing about this approach?
> >
> > I am wondering how you think of dealing with external modules. Some
> > commonly used options can be divided into two groups:
> >
> > * git based:
> > ** submodules
> > ** subtree
> > * Puppetfile based:
> > ** librarian-puppet https://github.com/rodjek/librarian-puppet
> > ** r10k https://github.com/adrienthebo/r10k
>
> I was thinking about resolving this when some commits were pushed
> into this git repo: for every module it would need to create a puppet
> tar (in order to be able to send it into pulp). The way how to get
> the external packages could be configured, and probably all the listed
> techniques could be supported.
>
> So to be clear, it wouldn't use git directly to push the Puppet modules
> into puppet master, but it would act as another user interface to get
> the content into Katello/Pulp.
>
> The advantage of packaging every module into a tar is having searchable
> metadata, which could help with a mess that managing puppet modules with
> git submodules often brings, that every puppet environment uses slightly
> different version of some puppet module. It wouldn't be hard to compare
> the modules used in environments against what's available in upstream.
I think we're thinking of the same thing, so let's verify we are
There's a git repository a user can push to.
Katello will check on push if it needs to update content views. This
could be it through branches, tags or some other mechanism. In the
end it will resolve to commits that need to be translated into
content views.
For each commit it will iterate through the modules.
For each module it will try to get a tarball. The source could be
plain files from the git checkout (this includes subtree), git
submodules and/or Puppetfile based.
This is then somehow composed into content views, most likely REST
API calls.
>
> >
> > It could be solved by making it hookable. Either through a git hook,
> > bin/make-release, rake make-release or something entirely different that
> > I didn't think of. Then you can give the user full control and provide
> > some examples. This is based on what foreman does:
> > https://github.com/theforeman/puppet-puppet/blob/master/templates/server/post-receive.erb.
>
> Yeah, we probably don't have to make this as part of Katello core code base.
> For start, we need just a git repo with some hook that would perform the
> right REST API calls.
Sounds good.
···
On Wed, Nov 06, 2013 at 04:18:17AM -0500, Ivan Necas wrote:
> ----- Original Message -----
> > On Tue, Nov 05, 2013 at 09:13:14AM -0500, Ivan Necas wrote:
> >
> >
> > > > Hi,
> > > >
> > > > One of the recent features added to Katello was support for puppet
> > > > content.
> > > > It allows devops
> > > > to tie the puppet manifests together with the software packages in
> > > > specific
> > > > versions inside content
> > > > view and use promotion path to get updates to test/production
> > > > environments.
> > > >
> > > > Currently, there are two ways how to get puppet content into Katello:
> > > >
> > > > 1. syncing remote locations that act as puppetforge
> > > > 2. uploading the modules into katello in a form of tars
> > > >
> > > > From what I've seen so far, non of these ways corresponds to how it's
> > > > usually done in wild.
> > > >
> > > > From my understanding, most of bigger deployments utilize git to hold
> > > > and
> > > > version the puppet modules.
> > > > Therefore, it seems to me it might be really worth looking into ways
> > > > how to
> > > > support the gitish way
> > > > of doing things.
> > > >
> > > > I was thinking about using git as the interface between the devops and
> > > > Katello.
> > > >
> > > > Let's consider this scenario:
> > > >
> > > > 1. I have a git repository containing all the modules needed for
> > > > setting up
> > > > some piece of infrastructure
> > > > (almost, but not quite, entirely unlike our foreman-installer repo)
> > > >
> > > > 2. Katello would provide a git repository (tied either to repo or maybe
> > > > better to a content view definition)
> > > >
> > > > 3. I add this Katello's repo as remote to my git repository
> > > >
> > > > 4. When I push the repository to Katello, it would perform everything
> > > > to
> > > > publish new content view (or refresh
> > > > new content view version?) taking the content form the git repository
> > > > (ideally including
> > > > creating the packages that Pulp accepts). Also, Katello would create a
> > > > tag
> > > > into the repository,
> > > > so that the connection between content view version and the git
> > > > repository
> > > > is really visible)
> > > >
> > > > So in ideal case, the journey of my repository with all the modules to
> > > > Katello/Foreman managed infrastructure
> > > > would be a question of one git push, something like PaaS for puppet
> > > > modules
> > > > And form the git history, I would
> > > > see instantly what versions are running in what environments.
> > > >
> > > > The mapping also could be as follows:
> > > >
> > > > * git repository represents a content view definition
> > > > * branches represent content views
> > > > * tags represent content view versions
>
> GPG signed tags could please some people as well.
>
> > > >
> > > > What other forks thing about this approach?
> > >
> > > I am wondering how you think of dealing with external modules. Some
> > > commonly used options can be divided into two groups:
> > >
> > > * git based:
> > > ** submodules
> > > ** subtree
> > > * Puppetfile based:
> > > ** librarian-puppet https://github.com/rodjek/librarian-puppet
> > > ** r10k https://github.com/adrienthebo/r10k
> >
> > I was thinking about resolving this when some commits were pushed
> > into this git repo: for every module it would need to create a puppet
> > tar (in order to be able to send it into pulp). The way how to get
> > the external packages could be configured, and probably all the listed
> > techniques could be supported.
> >
> > So to be clear, it wouldn't use git directly to push the Puppet modules
> > into puppet master, but it would act as another user interface to get
> > the content into Katello/Pulp.
> >
> > The advantage of packaging every module into a tar is having searchable
> > metadata, which could help with a mess that managing puppet modules with
> > git submodules often brings, that every puppet environment uses slightly
> > different version of some puppet module. It wouldn't be hard to compare
> > the modules used in environments against what's available in upstream.
>
> I think we're thinking of the same thing, so let's verify we are
>
> 1. There's a git repository a user can push to.
> 2. Katello will check on push if it needs to update content views. This
> could be it through branches, tags or some other mechanism. In the
> end it will resolve to commits that need to be translated into
> content views.
> 3. For each commit it will iterate through the modules.
> 4. For each module it will try to get a tarball. The source could be
> plain files from the git checkout (this includes subtree), git
> submodules and/or Puppetfile based.
> 5. This is then somehow composed into content views, most likely REST
> API calls.
EXACTLY!
For the step 5, it would probably mean push the new modules into the Pulp
repo, that would act as a Library (so all the modules used in the infrastructure
would be there), generate a content view using a filter on top of this
repo.
The library Pulp repo could be then used as way how to unify modules
used across the organization: when creating new configuration,
instead of searching on github or puppetforge, I would first check
if there is not some module already used for this somewhere in my org
and prefer that.
– Ivan
···
----- Original Message -----
> On Wed, Nov 06, 2013 at 04:18:17AM -0500, Ivan Necas wrote:
> > ----- Original Message -----
> > > On Tue, Nov 05, 2013 at 09:13:14AM -0500, Ivan Necas wrote:
Yeah, we probably don’t have to make this as part of Katello core code
base.
For start, we need just a git repo with some hook that would perform the
right REST API calls.
>
>>>
>>>>> Hi,
>>>>>
>>>>> One of the recent features added to Katello was support for puppet
>>>>> content.
>>>>> It allows devops
>>>>> to tie the puppet manifests together with the software packages in
>>>>> specific
>>>>> versions inside content
>>>>> view and use promotion path to get updates to test/production
>>>>> environments.
>>>>>
>>>>> Currently, there are two ways how to get puppet content into Katello:
>>>>>
>>>>> 1. syncing remote locations that act as puppetforge
>>>>> 2. uploading the modules into katello in a form of tars
>>>>>
>>>>> From what I've seen so far, non of these ways corresponds to how it's
>>>>> usually done in wild.
>>>>>
>>>>> From my understanding, most of bigger deployments utilize git to hold
>>>>> and
>>>>> version the puppet modules.
>>>>> Therefore, it seems to me it might be really worth looking into ways
>>>>> how to
>>>>> support the gitish way
>>>>> of doing things.
>>>>>
>>>>> I was thinking about using git as the interface between the devops and
>>>>> Katello.
>>>>>
>>>>> Let's consider this scenario:
>>>>>
>>>>> 1. I have a git repository containing all the modules needed for
>>>>> setting up
>>>>> some piece of infrastructure
>>>>> (almost, but not quite, entirely unlike our foreman-installer repo)
>>>>>
>>>>> 2. Katello would provide a git repository (tied either to repo or maybe
>>>>> better to a content view definition)
>>>>>
>>>>> 3. I add this Katello's repo as remote to my git repository
>>>>>
>>>>> 4. When I push the repository to Katello, it would perform everything
>>>>> to
>>>>> publish new content view (or refresh
>>>>> new content view version?) taking the content form the git repository
>>>>> (ideally including
>>>>> creating the packages that Pulp accepts). Also, Katello would create a
>>>>> tag
>>>>> into the repository,
>>>>> so that the connection between content view version and the git
>>>>> repository
>>>>> is really visible)
>>>>>
>>>>> So in ideal case, the journey of my repository with all the modules to
>>>>> Katello/Foreman managed infrastructure
>>>>> would be a question of one git push, something like PaaS for puppet
>>>>> modules
>>>>> And form the git history, I would
>>>>> see instantly what versions are running in what environments.
>>>>>
>>>>> The mapping also could be as follows:
>>>>>
>>>>> * git repository represents a content view definition
>>>>> * branches represent content views
>>>>> * tags represent content view versions
>> GPG signed tags could please some people as well.
>>
>>>>> What other forks thing about this approach?
>>>> I am wondering how you think of dealing with external modules. Some
>>>> commonly used options can be divided into two groups:
>>>>
>>>> * git based:
>>>> ** submodules
>>>> ** subtree
>>>> * Puppetfile based:
>>>> ** librarian-puppet https://github.com/rodjek/librarian-puppet
>>>> ** r10k https://github.com/adrienthebo/r10k
>>> I was thinking about resolving this when some commits were pushed
>>> into this git repo: for every module it would need to create a puppet
>>> tar (in order to be able to send it into pulp). The way how to get
>>> the external packages could be configured, and probably all the listed
>>> techniques could be supported.
>>>
>>> So to be clear, it wouldn't use git directly to push the Puppet modules
>>> into puppet master, but it would act as another user interface to get
>>> the content into Katello/Pulp.
>>>
>>> The advantage of packaging every module into a tar is having searchable
>>> metadata, which could help with a mess that managing puppet modules with
>>> git submodules often brings, that every puppet environment uses slightly
>>> different version of some puppet module. It wouldn't be hard to compare
>>> the modules used in environments against what's available in upstream.
>> I think we're thinking of the same thing, so let's verify we are
>>
>> 1. There's a git repository a user can push to.
>> 2. Katello will check on push if it needs to update content views. This
>> could be it through branches, tags or some other mechanism. In the
>> end it will resolve to commits that need to be translated into
>> content views.
>> 3. For each commit it will iterate through the modules.
>> 4. For each module it will try to get a tarball. The source could be
>> plain files from the git checkout (this includes subtree), git
>> submodules and/or Puppetfile based.
>> 5. This is then somehow composed into content views, most likely REST
>> API calls.
> EXACTLY!
>
> For the step 5, it would probably mean push the new modules into the Pulp
> repo, that would act as a Library (so all the modules used in the infrastructure
> would be there), generate a content view using a filter on top of this
> repo.
>
> The library Pulp repo could be then used as way how to unify modules
> used across the organization: when creating new configuration,
> instead of searching on github or puppetforge, I would first check
> if there is not some module already used for this somewhere in my org
> and prefer that.
+1 to this, It should follow the workflow of the rest of content IMHO.
I do wonder about the difference between:
a) Katello hosting a git repo that a user would push too
vs
b) A git repo configured in katello that katello would pull from , and
can be triggered easily via a a git hook.
To me b) sounds easier to implement vs a), and things like authorization
might be easier as well.
Otherwise +1 from me.
-Justin
···
On 11/06/2013 05:00 AM, Ivan Necas wrote:
> ----- Original Message -----
>> On Wed, Nov 06, 2013 at 04:18:17AM -0500, Ivan Necas wrote:
>>> ----- Original Message -----
>>>> On Tue, Nov 05, 2013 at 09:13:14AM -0500, Ivan Necas wrote:
– Ivan
It could be solved by making it hookable. Either through a git hook,
bin/make-release, rake make-release or something entirely different that
I didn’t think of. Then you can give the user full control and provide
some examples. This is based on what foreman does: https://github.com/theforeman/puppet-puppet/blob/master/templates/server/post-receive.erb.
Yeah, we probably don’t have to make this as part of Katello core code
base.
For start, we need just a git repo with some hook that would perform the
right REST API calls.
Sounds good.
I think so… could this be a pulp importer… which watches git and does
the needful when it sees commits?
– bk
···
On 11/06/2013 04:31 AM, Ivan Necas wrote:
>
>
> ----- Original Message -----
>> On 11/05/2013 10:07 AM, Marek Hulan wrote:
>>> On Tuesday 05 of November 2013 09:13:14 Ivan Necas wrote:
>>>> Hi,
>>>>
>>>> One of the recent features added to Katello was support for puppet
>>>> content.
>>>> It allows devops to tie the puppet manifests together with the software
>>>> packages in specific versions inside content view and use promotion path
>>>> to
>>>> get updates to test/production environments.
>>>>
>>>> Currently, there are two ways how to get puppet content into Katello:
>>>>
>>>> 1. syncing remote locations that act as puppetforge
>>>> 2. uploading the modules into katello in a form of tars
>>>>
>>>> From what I've seen so far, non of these ways corresponds to how it's
>>>> usually done in wild.
>>>>
>>>> From my understanding, most of bigger deployments utilize git to hold and
>>>> version the puppet modules. Therefore, it seems to me it might be really
>>>> worth looking into ways how to support the gitish way of doing things.
>>>>
>>>> I was thinking about using git as the interface between the devops and
>>>> Katello.
>>>>
>>>> Let's consider this scenario:
>>>>
>>>> 1. I have a git repository containing all the modules needed for setting
>>>> up
>>>> some piece of infrastructure (almost, but not quite, entirely unlike our
>>>> foreman-installer repo)
>>>>
>>>> 2. Katello would provide a git repository (tied either to repo or maybe
>>>> better to a content view definition)
>>>>
>>>> 3. I add this Katello's repo as remote to my git repository
>>>>
>>>> 4. When I push the repository to Katello, it would perform everything to
>>>> publish new content view (or refresh new content view version?) taking the
>>>> content form the git repository (ideally including creating the packages
>>>> that Pulp accepts). Also, Katello would create a tag into the repository,
>>>> so that the connection between content view version and the git repository
>>>> is really visible)
>>>>
>>>> So in ideal case, the journey of my repository with all the modules to
>>>> Katello/Foreman managed infrastructure would be a question of one git
>>>> push,
>>>> something like PaaS for puppet modules :) And form the git history, I
>>>> would
>>>> see instantly what versions are running in what environments.
>>>>
>>>> The mapping also could be as follows:
>>>>
>>>> * git repository represents a content view definition
>>>> * branches represent content views
>>>> * tags represent content view versions
>>>>
>>>> What other forks thing about this approach?
>>>
>>> I like the idea. My understading is that most of puppet users handles
>>> modules
>>> in git so this would allow them to use existing and well known tool to use
>>> katello.
>>>
>> Are you suggesting the library is sourced from git, or a content view
>> points to a branch in a git repo?
>>
>> If the latter.. how would items like the content browser work?
>
> Non of them :) The suggestion is about having something that would be able
> to keep the git repository in sync with the katello-managed content.
>
> So for example with the foreman-installer repository, lets say I want to publish new
> content view version based on the changes in that repo. Therefore I push the changes
> into Katello's git repo. It would fetch third-party packages (either from submodules,
> using librarian, r10k or whatever, based on configuration), package the rest of modules
> and push all to the corresponding Pulp repo. Then it would publish/refresh new content
> view version, setting the filters in a way that the result is the same as I have in
> the foreman-installer repository.
>
> In fact, there could be only one Pulp repository per Katello organization, to hold
> all the puppet modules. The git repo would push the packages to the pulp repo (if
> they're not already there) and then publish the content view through applying a right
> filter on that repo.
>
> Git would not be involved in the process of getting the pupppet content to puppetmaster
> and we would use the standard mechanism that Pulp already has.
>
> Is it cleaner now?
>
···
On 11/06/2013 08:37 AM, Justin Sherrill wrote:
> On 11/06/2013 05:00 AM, Ivan Necas wrote:
>>
>> ----- Original Message -----
>>> On Wed, Nov 06, 2013 at 04:18:17AM -0500, Ivan Necas wrote:
>>>>
>>>> ----- Original Message -----
>>>>> On Tue, Nov 05, 2013 at 09:13:14AM -0500, Ivan Necas wrote:
>>>>>> Hi,
>>>>>>
>>>>>> One of the recent features added to Katello was support for puppet
>>>>>> content.
>>>>>> It allows devops
>>>>>> to tie the puppet manifests together with the software packages in
>>>>>> specific
>>>>>> versions inside content
>>>>>> view and use promotion path to get updates to test/production
>>>>>> environments.
>>>>>>
>>>>>> Currently, there are two ways how to get puppet content into Katello:
>>>>>>
>>>>>> 1. syncing remote locations that act as puppetforge
>>>>>> 2. uploading the modules into katello in a form of tars
>>>>>>
>>>>>> From what I've seen so far, non of these ways corresponds to how
>>>>>> it's
>>>>>> usually done in wild.
>>>>>>
>>>>>> From my understanding, most of bigger deployments utilize git to
>>>>>> hold
>>>>>> and
>>>>>> version the puppet modules.
>>>>>> Therefore, it seems to me it might be really worth looking into ways
>>>>>> how to
>>>>>> support the gitish way
>>>>>> of doing things.
>>>>>>
>>>>>> I was thinking about using git as the interface between the devops
>>>>>> and
>>>>>> Katello.
>>>>>>
>>>>>> Let's consider this scenario:
>>>>>>
>>>>>> 1. I have a git repository containing all the modules needed for
>>>>>> setting up
>>>>>> some piece of infrastructure
>>>>>> (almost, but not quite, entirely unlike our foreman-installer
>>>>>> repo)
>>>>>>
>>>>>> 2. Katello would provide a git repository (tied either to repo or
>>>>>> maybe
>>>>>> better to a content view definition)
>>>>>>
>>>>>> 3. I add this Katello's repo as remote to my git repository
>>>>>>
>>>>>> 4. When I push the repository to Katello, it would perform everything
>>>>>> to
>>>>>> publish new content view (or refresh
>>>>>> new content view version?) taking the content form the git repository
>>>>>> (ideally including
>>>>>> creating the packages that Pulp accepts). Also, Katello would
>>>>>> create a
>>>>>> tag
>>>>>> into the repository,
>>>>>> so that the connection between content view version and the git
>>>>>> repository
>>>>>> is really visible)
>>>>>>
>>>>>> So in ideal case, the journey of my repository with all the
>>>>>> modules to
>>>>>> Katello/Foreman managed infrastructure
>>>>>> would be a question of one git push, something like PaaS for puppet
>>>>>> modules
>>>>>> :) And form the git history, I would
>>>>>> see instantly what versions are running in what environments.
>>>>>>
>>>>>> The mapping also could be as follows:
>>>>>>
>>>>>> * git repository represents a content view definition
>>>>>> * branches represent content views
>>>>>> * tags represent content view versions
>>> GPG signed tags could please some people as well.
>>>
>>>>>> What other forks thing about this approach?
>>>>> I am wondering how you think of dealing with external modules. Some
>>>>> commonly used options can be divided into two groups:
>>>>>
>>>>> * git based:
>>>>> ** submodules
>>>>> ** subtree
>>>>> * Puppetfile based:
>>>>> ** librarian-puppet https://github.com/rodjek/librarian-puppet
>>>>> ** r10k https://github.com/adrienthebo/r10k
>>>> I was thinking about resolving this when some commits were pushed
>>>> into this git repo: for every module it would need to create a puppet
>>>> tar (in order to be able to send it into pulp). The way how to get
>>>> the external packages could be configured, and probably all the listed
>>>> techniques could be supported.
>>>>
>>>> So to be clear, it wouldn't use git directly to push the Puppet modules
>>>> into puppet master, but it would act as another user interface to get
>>>> the content into Katello/Pulp.
>>>>
>>>> The advantage of packaging every module into a tar is having searchable
>>>> metadata, which could help with a mess that managing puppet modules
>>>> with
>>>> git submodules often brings, that every puppet environment uses
>>>> slightly
>>>> different version of some puppet module. It wouldn't be hard to compare
>>>> the modules used in environments against what's available in upstream.
>>> I think we're thinking of the same thing, so let's verify we are :)
>>>
>>> 1. There's a git repository a user can push to.
>>> 2. Katello will check on push if it needs to update content views. This
>>> could be it through branches, tags or some other mechanism. In the
>>> end it will resolve to commits that need to be translated into
>>> content views.
>>> 3. For each commit it will iterate through the modules.
>>> 4. For each module it will try to get a tarball. The source could be
>>> plain files from the git checkout (this includes subtree), git
>>> submodules and/or Puppetfile based.
>>> 5. This is then somehow composed into content views, most likely REST
>>> API calls.
>> EXACTLY!
>>
>> For the step 5, it would probably mean push the new modules into the Pulp
>> repo, that would act as a Library (so all the modules used in the
>> infrastructure
>> would be there), generate a content view using a filter on top of this
>> repo.
>>
>> The library Pulp repo could be then used as way how to unify modules
>> used across the organization: when creating new configuration,
>> instead of searching on github or puppetforge, I would first check
>> if there is not some module already used for this somewhere in my org
>> and prefer that.
> +1 to this, It should follow the workflow of the rest of content IMHO. I
> do wonder about the difference between:
>
> a) Katello hosting a git repo that a user would push too
>
> vs
>
> b) A git repo configured in katello that katello would pull from , and
> can be triggered easily via a a git hook.
>
> To me b) sounds easier to implement vs a), and things like authorization
> might be easier as well.
>
> Otherwise +1 from me.
>
> -Justin
> >
> >>>
> >>>>> Hi,
> >>>>>
> >>>>> One of the recent features added to Katello was support for puppet
> >>>>> content.
> >>>>> It allows devops
> >>>>> to tie the puppet manifests together with the software packages in
> >>>>> specific
> >>>>> versions inside content
> >>>>> view and use promotion path to get updates to test/production
> >>>>> environments.
> >>>>>
> >>>>> Currently, there are two ways how to get puppet content into Katello:
> >>>>>
> >>>>> 1. syncing remote locations that act as puppetforge
> >>>>> 2. uploading the modules into katello in a form of tars
> >>>>>
> >>>>> From what I've seen so far, non of these ways corresponds to how it's
> >>>>> usually done in wild.
> >>>>>
> >>>>> From my understanding, most of bigger deployments utilize git to hold
> >>>>> and
> >>>>> version the puppet modules.
> >>>>> Therefore, it seems to me it might be really worth looking into ways
> >>>>> how to
> >>>>> support the gitish way
> >>>>> of doing things.
> >>>>>
> >>>>> I was thinking about using git as the interface between the devops and
> >>>>> Katello.
> >>>>>
> >>>>> Let's consider this scenario:
> >>>>>
> >>>>> 1. I have a git repository containing all the modules needed for
> >>>>> setting up
> >>>>> some piece of infrastructure
> >>>>> (almost, but not quite, entirely unlike our foreman-installer repo)
> >>>>>
> >>>>> 2. Katello would provide a git repository (tied either to repo or maybe
> >>>>> better to a content view definition)
> >>>>>
> >>>>> 3. I add this Katello's repo as remote to my git repository
> >>>>>
> >>>>> 4. When I push the repository to Katello, it would perform everything
> >>>>> to
> >>>>> publish new content view (or refresh
> >>>>> new content view version?) taking the content form the git repository
> >>>>> (ideally including
> >>>>> creating the packages that Pulp accepts). Also, Katello would create a
> >>>>> tag
> >>>>> into the repository,
> >>>>> so that the connection between content view version and the git
> >>>>> repository
> >>>>> is really visible)
> >>>>>
> >>>>> So in ideal case, the journey of my repository with all the modules to
> >>>>> Katello/Foreman managed infrastructure
> >>>>> would be a question of one git push, something like PaaS for puppet
> >>>>> modules
> >>>>> And form the git history, I would
> >>>>> see instantly what versions are running in what environments.
> >>>>>
> >>>>> The mapping also could be as follows:
> >>>>>
> >>>>> * git repository represents a content view definition
> >>>>> * branches represent content views
> >>>>> * tags represent content view versions
> >> GPG signed tags could please some people as well.
> >>
> >>>>> What other forks thing about this approach?
> >>>> I am wondering how you think of dealing with external modules. Some
> >>>> commonly used options can be divided into two groups:
> >>>>
> >>>> * git based:
> >>>> ** submodules
> >>>> ** subtree
> >>>> * Puppetfile based:
> >>>> ** librarian-puppet https://github.com/rodjek/librarian-puppet
> >>>> ** r10k https://github.com/adrienthebo/r10k
> >>> I was thinking about resolving this when some commits were pushed
> >>> into this git repo: for every module it would need to create a puppet
> >>> tar (in order to be able to send it into pulp). The way how to get
> >>> the external packages could be configured, and probably all the listed
> >>> techniques could be supported.
> >>>
> >>> So to be clear, it wouldn't use git directly to push the Puppet modules
> >>> into puppet master, but it would act as another user interface to get
> >>> the content into Katello/Pulp.
> >>>
> >>> The advantage of packaging every module into a tar is having searchable
> >>> metadata, which could help with a mess that managing puppet modules with
> >>> git submodules often brings, that every puppet environment uses slightly
> >>> different version of some puppet module. It wouldn't be hard to compare
> >>> the modules used in environments against what's available in upstream.
> >> I think we're thinking of the same thing, so let's verify we are
> >>
> >> 1. There's a git repository a user can push to.
> >> 2. Katello will check on push if it needs to update content views. This
> >> could be it through branches, tags or some other mechanism. In the
> >> end it will resolve to commits that need to be translated into
> >> content views.
> >> 3. For each commit it will iterate through the modules.
> >> 4. For each module it will try to get a tarball. The source could be
> >> plain files from the git checkout (this includes subtree), git
> >> submodules and/or Puppetfile based.
> >> 5. This is then somehow composed into content views, most likely REST
> >> API calls.
> > EXACTLY!
> >
> > For the step 5, it would probably mean push the new modules into the Pulp
> > repo, that would act as a Library (so all the modules used in the
> > infrastructure
> > would be there), generate a content view using a filter on top of this
> > repo.
> >
> > The library Pulp repo could be then used as way how to unify modules
> > used across the organization: when creating new configuration,
> > instead of searching on github or puppetforge, I would first check
> > if there is not some module already used for this somewhere in my org
> > and prefer that.
> +1 to this, It should follow the workflow of the rest of content IMHO.
> I do wonder about the difference between:
>
> a) Katello hosting a git repo that a user would push too
>
> vs
>
> b) A git repo configured in katello that katello would pull from , and
> can be triggered easily via a a git hook.
>
> To me b) sounds easier to implement vs a), and things like authorization
> might be easier as well.
It could start as b) with transition to a) as Katello first-class citizen later.
– Ivan
···
----- Original Message -----
> On 11/06/2013 05:00 AM, Ivan Necas wrote:
> > ----- Original Message -----
> >> On Wed, Nov 06, 2013 at 04:18:17AM -0500, Ivan Necas wrote:
> >>> ----- Original Message -----
> >>>> On Tue, Nov 05, 2013 at 09:13:14AM -0500, Ivan Necas wrote:
Otherwise +1 from me.
-Justin
– Ivan
It could be solved by making it hookable. Either through a git hook,
bin/make-release, rake make-release or something entirely different that
I didn’t think of. Then you can give the user full control and provide
some examples. This is based on what foreman does: https://github.com/theforeman/puppet-puppet/blob/master/templates/server/post-receive.erb.
Yeah, we probably don’t have to make this as part of Katello core code
base.
For start, we need just a git repo with some hook that would perform the
right REST API calls.
Sounds good.
> >
> >
> >>>> Hi,
> >>>>
> >>>> One of the recent features added to Katello was support for puppet
> >>>> content.
> >>>> It allows devops to tie the puppet manifests together with the software
> >>>> packages in specific versions inside content view and use promotion path
> >>>> to
> >>>> get updates to test/production environments.
> >>>>
> >>>> Currently, there are two ways how to get puppet content into Katello:
> >>>>
> >>>> 1. syncing remote locations that act as puppetforge
> >>>> 2. uploading the modules into katello in a form of tars
> >>>>
> >>>> From what I've seen so far, non of these ways corresponds to how it's
> >>>> usually done in wild.
> >>>>
> >>>> From my understanding, most of bigger deployments utilize git to hold
> >>>> and
> >>>> version the puppet modules. Therefore, it seems to me it might be really
> >>>> worth looking into ways how to support the gitish way of doing things.
> >>>>
> >>>> I was thinking about using git as the interface between the devops and
> >>>> Katello.
> >>>>
> >>>> Let's consider this scenario:
> >>>>
> >>>> 1. I have a git repository containing all the modules needed for setting
> >>>> up
> >>>> some piece of infrastructure (almost, but not quite, entirely unlike our
> >>>> foreman-installer repo)
> >>>>
> >>>> 2. Katello would provide a git repository (tied either to repo or maybe
> >>>> better to a content view definition)
> >>>>
> >>>> 3. I add this Katello's repo as remote to my git repository
> >>>>
> >>>> 4. When I push the repository to Katello, it would perform everything to
> >>>> publish new content view (or refresh new content view version?) taking
> >>>> the
> >>>> content form the git repository (ideally including creating the packages
> >>>> that Pulp accepts). Also, Katello would create a tag into the
> >>>> repository,
> >>>> so that the connection between content view version and the git
> >>>> repository
> >>>> is really visible)
> >>>>
> >>>> So in ideal case, the journey of my repository with all the modules to
> >>>> Katello/Foreman managed infrastructure would be a question of one git
> >>>> push,
> >>>> something like PaaS for puppet modules And form the git history, I
> >>>> would
> >>>> see instantly what versions are running in what environments.
> >>>>
> >>>> The mapping also could be as follows:
> >>>>
> >>>> * git repository represents a content view definition
> >>>> * branches represent content views
> >>>> * tags represent content view versions
> >>>>
> >>>> What other forks thing about this approach?
> >>>
> >>> I like the idea. My understading is that most of puppet users handles
> >>> modules
> >>> in git so this would allow them to use existing and well known tool to
> >>> use
> >>> katello.
> >>>
> >> Are you suggesting the library is sourced from git, or a content view
> >> points to a branch in a git repo?
> >>
> >> If the latter… how would items like the content browser work?
> >
> > Non of them The suggestion is about having something that would be able
> > to keep the git repository in sync with the katello-managed content.
> >
> > So for example with the foreman-installer repository, lets say I want to
> > publish new
> > content view version based on the changes in that repo. Therefore I push
> > the changes
> > into Katello's git repo. It would fetch third-party packages (either from
> > submodules,
> > using librarian, r10k or whatever, based on configuration), package the
> > rest of modules
> > and push all to the corresponding Pulp repo. Then it would publish/refresh
> > new content
> > view version, setting the filters in a way that the result is the same as I
> > have in
> > the foreman-installer repository.
> >
> > In fact, there could be only one Pulp repository per Katello organization,
> > to hold
> > all the puppet modules. The git repo would push the packages to the pulp
> > repo (if
> > they're not already there) and then publish the content view through
> > applying a right
> > filter on that repo.
> >
> > Git would not be involved in the process of getting the pupppet content to
> > puppetmaster
> > and we would use the standard mechanism that Pulp already has.
> >
> > Is it cleaner now?
> >
>
> I think so… could this be a pulp importer… which watches git and does
> the needful when it sees commits?
I don't think it should/could go through Pulp directly, as some parts
need to happen on Katello side anyway. But it would be nice
if pulp was able to use git repos to import the modules (it would be great
addition to syncing puppetforge to be able to sync from github
or private git repo as well).
From the community perspective however, I think having the transition
implemented outside of Pulp in Ruby could be more maintainable from
the perspective of supporting different ways of referencing third-party
modules.
What we have right now in Pulp should be enough to achieve this.
– Ivan
···
----- Original Message -----
> On 11/06/2013 04:31 AM, Ivan Necas wrote:
> > ----- Original Message -----
> >> On 11/05/2013 10:07 AM, Marek Hulan wrote:
> >>> On Tuesday 05 of November 2013 09:13:14 Ivan Necas wrote:
And form the git history, I would