Module metadata file: Request for comments

Support
1

For a while now, I've been bothered with certain aspects of foreman on
how puppet classes are imported. Mainly 2 things make me twitch:

  • There are a lot of classes imported that are not meant to be used
    directly. They are included somewhere in the module rather to split
    code up and keep it neat. Removing them manually after the import can
    be tedious and time consuming, especially if you have to do it
    multiple times.

  • Setting up each parameter for a class is bothersome: time-consuming
    and the information should be defined in the puppet module rather than
    duplicated each time within foreman.

A solution to this is pretty straightforward: we define a file within
our module which contains this metadata.

The ignoring part is something that is easily implemented since all
code is on the smart-proxy side. For exposing additional parameter
information, more work will have to be done - also on the foreman
side. I doubt I'll manage this on my own :wink:

I've started coding on a solution for the first problem and came up
with something that works. The first problem is: what do we call this
meta-file. I've thought about .foreman.yaml which makes it a hidden
file by default, but I'm no longer sure this is the best approach. The
puppet forge module file isn't hidden and the meta information is
typically something we DO want to redistribute so maybe Foreman.meta
or Modulemeta or … is a better name (open for suggestions on this
part).

My example implementation meta file looks like:

  • — SNIPPET —

Ignore these classes.

Prevents the proxy from exposing them in the api.

proxy_ignore_files:

  • ignore/*

proxy_ignore_classes:

  • foo::*

  • — SNIPPET —

Alternatively, you can also use values like "!ruby/regexp '/^foo::./'".
Regular strings get 'converted' to regexpes by replacing '' by '.*?'.

There are 2 different keys that work on different levels. To make
things clearer, I would like to choose one of these and stick with that.

The ignore_files has the advantage that it actually does not parse
files that match it. So it kinda speeds things up. ignore_classes
could be useful for times where you define multiple classes in a
single file and only want to exclude some. Since this would break the
'puppet autoloader layout', I'm not sure we need this.

I could refactor to have one proxy_ignore which checks for matches
against files and classes.

So, basically, the request for comment is on the following subjects:

How do we name a file which contains the meta? Currently only to
ignore stuff, but indication what type, validation, … a parameter
takes could be added in the future.

What do we ignore. Files? or Classes? or both?
If both, do we migrate to one key with a small impact on performance
(we filter more records twice).

Future wise, what could be a good hierarchy for storing additional
parameter information?

Note:

Cheers

Jan.

2

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> For a while now, I've been bothered with certain aspects of foreman on
> how puppet classes are imported. Mainly 2 things make me twitch:
>
> * There are a lot of classes imported that are not meant to be used
> directly. They are included somewhere in the module rather to split
> code up and keep it neat. Removing them manually after the import can
> be tedious and time consuming, especially if you have to do it
> multiple times.
>
> did you see the class ignore file? see
config/ignored_environments.yml.sample for examples

  • Setting up each parameter for a class is bothersome: time-consuming
    > and the information should be defined in the puppet module rather than
    > duplicated each time within foreman.
    >
    I agree, at the time of writing the importer, we tried to fetch as much as
    possible info (e.g. type such as boolean etc), however there was no
    standard way of documenting classes - for example, I really wanted to have
    a way to generate the description and possible allowed values etc.

one thing important that should work is that foreman should detect if
variable is required or not, e.g. if it has no value, foreman assumes you
have to give it one, if it has a value, you can select to override it in
foreman.

> A solution to this is pretty straightforward: we define a file within
> our module which contains this metadata.
>
Yay :slight_smile: afaik, puppet module tool do generate such a file by default.

>
> The ignoring part is something that is easily implemented since all
> code is on the smart-proxy side. For exposing additional parameter
> information, more work will have to be done - also on the foreman
> side. I doubt I'll manage this on my own :wink:
>
>
> I've started coding on a solution for the first problem and came up
> with something that works. The first problem is: what do we call this
> meta-file. I've thought about .foreman.yaml which makes it a hidden
> file by default, but I'm no longer sure this is the best approach. The
> puppet forge module file isn't hidden and the meta information is
> typically something we DO want to redistribute so maybe Foreman.meta
> or Modulemeta or … is a better name (open for suggestions on this
> part).
>
> I do not think that this is a foreman only info, marking classes as
private or public makes sense in contexts outside of foreman too.

> My example implementation meta file looks like:
>
> - — SNIPPET —
> - —
> # Ignore these classes.
> # Prevents the proxy from exposing them in the api.
> proxy_ignore_files:
> - ignore/*
>
> proxy_ignore_classes:
> - foo::*
>
> - — SNIPPET —
>
> Alternatively, you can also use values like "!ruby/regexp '/^foo::./'".
> Regular strings get 'converted' to regexpes by replacing '' by '.*?'.
>
> There are 2 different keys that work on different levels. To make
> things clearer, I would like to choose one of these and stick with that.
>
> The ignore_files has the advantage that it actually does not parse
> files that match it. So it kinda speeds things up. ignore_classes
> could be useful for times where you define multiple classes in a
> single file and only want to exclude some. Since this would break the
> 'puppet autoloader layout', I'm not sure we need this.
>
> I could refactor to have one proxy_ignore which checks for matches
> against files and classes.
>
> - —
>
> So, basically, the request for comment is on the following subjects:
>
> How do we name a file which contains the meta? Currently only to
> ignore stuff, but indication what type, validation, … a parameter
> takes could be added in the future.
>
> What do we ignore. Files? or Classes? or both?
> If both, do we migrate to one key with a small impact on performance
> (we filter more records twice).
>
>
> Future wise, what could be a good hierarchy for storing additional
> parameter information?
>

What do you mean?

Ohad

··· On Wed, Jan 9, 2013 at 10:12 AM, Jan Vansteenkiste wrote:

Note:

Cheers

Jan.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQIcBAEBCgAGBQJQ7SZSAAoJEH9K6PHMcbUriuYP/AocYQIijKg/5Z+jB+RmzHgU
DL/2m5m/hQRVvnI9rDd8+KwaKtG6Bfqma8wCOuOBY7CVYuQ+U/eBCJ36agzAhaev
MWrdJtNQXMR6g1hhJb7vbPXA7Zvv9yCUCOGHq2hl0dWoNAj6EXpXeKdlu3VJ2EoG
UDuYOoi1rcK4W/KHQUBOKoH2tZBWCbJWHTV27yYj2PiUpcclddnuUul+Gl3oi1is
ky4NNl/5roYa1gqMOMykqmP3b/VQwXFkvHM/Rs76MtlJlyhiIvXGNtGTaP+PY/Uj
MH4n3f69TLC2sZtAeOY/tYzsXlJZj9XknXolMccTkI4IX2mWUqtTVVR6Ayfk4e9z
+BvIBwtHVOjUjD91t5ddJAYlynqcYM9hKVh0Z7GyAxF9XpJ94p3ETcZ9yf+aAEv6
dFBWZUBTuPUMF5DpW5nmbK0MBL/nrY8Obf2DpYzRjBls1mTss8b42fLNyfvGmLVz
hosBuPzrMjiNcrACFckGGDfMhJWNX0D6qtPl9JVRCSt977Opbg5cn2I4kkulYKnb
kJjHf+Jo8UXZOgA21jFUTgVfijJw5lp5LRHMY9uJJpXSS3mTHj1NB98/Yg8l5qn7
+GVYjXjWel37BnWLpTIfCxQx7xRAiITXpAuenNUJ0R97f1i7/9LW3xeVWXN1ticQ
LSUtkmbmvxhuA9Wl0GCV
=hjXQ
-----END PGP SIGNATURE-----

3

>
>
>
>
> For a while now, I've been bothered with certain aspects of foreman
> on how puppet classes are imported. Mainly 2 things make me
> twitch:
>
> * There are a lot of classes imported that are not meant to be
> used directly. They are included somewhere in the module rather to
> split code up and keep it neat. Removing them manually after the
> import can be tedious and time consuming, especially if you have to
> do it multiple times.
>
>> did you see the class ignore file? see
>> config/ignored_environments.yml.sample for examples

Yes, but this is storing the information in the wrong place imho. Its
the module developer who should decide in the first place what to
expose and what not. Obviously, foreman users can tweak it using that
file.

>
> * Setting up each parameter for a class is bothersome:
> time-consuming and the information should be defined in the puppet
> module rather than duplicated each time within foreman.
>
>> I agree, at the time of writing the importer, we tried to fetch
>> as much as possible info (e.g. type such as boolean etc), however
>> there was no standard way of documenting classes - for example, I
>> really wanted to have a way to generate the description and
>> possible allowed values etc.

I haven't seen much progress on the puppet doc/rdoc part so I doubt it
will ever be possible to grab this information from the puppet files.
Especially since the parser tends to remove/discard all comments. So
storing this information in meta file makes sense to me and will be a
lot easier than depending on rdoc or writing a custom parser.

>
>> one thing important that should work is that foreman should
>> detect if variable is required or not, e.g. if it has no value,
>> foreman assumes you have to give it one, if it has a value, you
>> can select to override it in foreman.
>
>
> A solution to this is pretty straightforward: we define a file
> within our module which contains this metadata.
>
>> Yay :slight_smile: afaik, puppet module tool do generate such a file by
>> default.

I'll have to investigate if we can add more information to this file
and how to parse it. Although, I have plenty modules that are not on
the forge :wink: (COUGH foreman-installer modules), so a new file might
also be appropriate.

>
>
>
> The ignoring part is something that is easily implemented since
> all code is on the smart-proxy side. For exposing additional
> parameter information, more work will have to be done - also on the
> foreman side. I doubt I'll manage this on my own :wink:
>
>
> I've started coding on a solution for the first problem and came
> up with something that works. The first problem is: what do we call
> this meta-file. I've thought about .foreman.yaml which makes it a
> hidden file by default, but I'm no longer sure this is the best
> approach. The puppet forge module file isn't hidden and the meta
> information is typically something we DO want to redistribute so
> maybe Foreman.meta or Modulemeta or … is a better name (open for
> suggestions on this part).
>
>> I do not think that this is a foreman only info, marking classes
>> as private or public makes sense in contexts outside of foreman
>> too.
>
>
> My example implementation meta file looks like:
>
> — SNIPPET — --- # Ignore these classes. # Prevents the proxy
> from exposing them in the api. proxy_ignore_files: - ignore/*
>
> proxy_ignore_classes: - foo::*
>
> — SNIPPET —
>
> Alternatively, you can also use values like "!ruby/regexp
> '/^foo::./'". Regular strings get 'converted' to regexpes by
> replacing '' by '.*?'.
>
> There are 2 different keys that work on different levels. To make
> things clearer, I would like to choose one of these and stick with
> that.
>
> The ignore_files has the advantage that it actually does not parse
> files that match it. So it kinda speeds things up. ignore_classes
> could be useful for times where you define multiple classes in a
> single file and only want to exclude some. Since this would break
> the 'puppet autoloader layout', I'm not sure we need this.
>
> I could refactor to have one proxy_ignore which checks for matches
> against files and classes.
>
> —
>
> So, basically, the request for comment is on the following
> subjects:
>
> How do we name a file which contains the meta? Currently only to
> ignore stuff, but indication what type, validation, … a
> parameter takes could be added in the future.
>
> What do we ignore. Files? or Classes? or both? If both, do we
> migrate to one key with a small impact on performance (we filter
> more records twice).
>
>
> Future wise, what could be a good hierarchy for storing additional
> parameter information?
>
>
>> What do you mean?

Well, we want to store that information in our yaml structure. So,
should we go with something like this:

<classname>:
something:
parameters:
<param>:
desc:
type:
type_meta: (if its a list, allowed values etc etc)
validation_rule:
…

or do we nest it

··· On 01/09/2013 09:23 AM, Ohad Levy wrote: > On Wed, Jan 9, 2013 at 10:12 AM, Jan Vansteenkiste > wrote:

parameter_information:
:
:
…

Ohad

Note:

Cheers

Jan.

4

I was discussing uploading foreman-installer to the forge with Greg this
week, it's something I'd like to see which should streamline the
installation process for 2.7+ users. The installer module itself would
just need the submodules stripping out before upload.

On IRC yesterday, DoubleMalt was commenting that the report processor
should be a standalone module too, which I agree with. It'd be nice to
create a self-contained module for that and the ENC script (not strictly
normal module material, but they're closely linked) which the installer
uses but is also available for use by itself.

··· On 09/01/13 08:56, Jan Vansteenkiste wrote: > I'll have to investigate if we can add more information to this file > and how to parse it. Although, I have plenty modules that are not on > the forge ;) (*COUGH* foreman-installer modules), so a new file might > also be appropriate.

–
Dominic Cleal
Red Hat Engineering

5

I have done some work on making the modules more re-useable for a
non-foreman-installer setup.

Since I want to keep upstream working as much as possible, I have yet
to send out pull requests. Some of the changes can be previewed on my
develop branches:

Note: none of this has been tested with the foreman-installer module
since I use my own apache/passenger modules.

https://github.com/vStone/puppet-foreman/compare/develop
https://github.com/vStone/puppet-puppet/compare/develop (commit spree
coming soon)
https://github.com/vStone/puppet-foreman_proxy/compare/develop

··· On Wed 09 Jan 2013 02:23:50 PM CET, Dominic Cleal wrote: > On 09/01/13 08:56, Jan Vansteenkiste wrote: >> I'll have to investigate if we can add more information to this file >> and how to parse it. Although, I have plenty modules that are not on >> the forge ;) (*COUGH* foreman-installer modules), so a new file might >> also be appropriate. > > I was discussing uploading foreman-installer to the forge with Greg this > week, it's something I'd like to see which should streamline the > installation process for 2.7+ users. The installer module itself would > just need the submodules stripping out before upload. > > On IRC yesterday, DoubleMalt was commenting that the report processor > should be a standalone module too, which I agree with. It'd be nice to > create a self-contained module for that and the ENC script (not strictly > normal module material, but they're closely linked) which the installer > uses but is also available for use by itself. >
6

That's great work, I think it'd be great to include those changes
upstream. I'd like to hold off until 1.1's out though, since it's
reaching a stable point now.

··· On 09/01/13 14:08, Jan Vansteenkiste wrote: > On Wed 09 Jan 2013 02:23:50 PM CET, Dominic Cleal wrote: >> On 09/01/13 08:56, Jan Vansteenkiste wrote: >>> I'll have to investigate if we can add more information to this file >>> and how to parse it. Although, I have plenty modules that are not on >>> the forge ;) (*COUGH* foreman-installer modules), so a new file might >>> also be appropriate. >> >> I was discussing uploading foreman-installer to the forge with Greg this >> week, it's something I'd like to see which should streamline the >> installation process for 2.7+ users. The installer module itself would >> just need the submodules stripping out before upload. >> >> On IRC yesterday, DoubleMalt was commenting that the report processor >> should be a standalone module too, which I agree with. It'd be nice to >> create a self-contained module for that and the ENC script (not strictly >> normal module material, but they're closely linked) which the installer >> uses but is also available for use by itself. >> > > I have done some work on making the modules more re-useable for a > non-foreman-installer setup. > > Since I want to keep upstream working as much as possible, I have yet > to send out pull requests. Some of the changes can be previewed on my > develop branches: > > Note: none of this has been tested with the foreman-installer module > since I use my own apache/passenger modules. > > https://github.com/vStone/puppet-foreman/compare/develop > https://github.com/vStone/puppet-puppet/compare/develop (commit spree > coming soon) > https://github.com/vStone/puppet-foreman_proxy/compare/develop

–
Dominic Cleal
Red Hat Engineering