Using Foreman for masterless puppet

Support
1

After much testing I've found that my deployment of Puppet + Foreman for a
HPC cluster requires the use of masterless puppet. Our plan is to have
Puppet run between jobs via SLURM Epilog and/or Prolog scripts and we
frequently have 64 node jobs which would mean 64 servers would hit the
Puppetmaster at the same time using a traditional Puppetmaster setup. I
would still like to use Foreman as it's one of the key pieces for managing
our HPC cluster. The issues I'm facing have to do with nodes getting their
ENC data, uploading their Facter facts, and submitting their report.

  • Is there a way to configure Foreman so it authorizes a server to upload
    facts/reports and pull ENC data based on the server's certificate being
    signed by the Puppetmaster? For now I've had to set "
    restrict_registered_puppetmasters" to false to allow servers to contact
    Foreman.
  • For fact uploading, would performing a "Facter.to_hash" in place of
    reading the yaml file from the Puppetmaster have the same outcome?

Would it be more efficient to write a Foreman plugin that saves a host's
ENC YAML to a file that can be simply read very basic node.rb type script?
Currently our Puppet modules and all Puppet agent config files needed to
run "puppet apply" will be on a NFS share that is exported read-only to all
compute nodes.

My goal is to reduce the runtime of these scripts as much as possible. If
I have 64 nodes contact Foreman for ENC data, that could become a bottle
neck as I don't have the resources to run Foreman on anything but a basic
oVirt VM.

If anyone has done Masterless Puppet with Foreman, or has any tips, please
let me know.

Thanks

  • Trey
2

> After much testing I've found that my deployment of Puppet + Foreman for
> a HPC cluster requires the use of masterless puppet. Our plan is to
> have Puppet run between jobs via SLURM Epilog and/or Prolog scripts and
> we frequently have 64 node jobs which would mean 64 servers would hit
> the Puppetmaster at the same time using a traditional Puppetmaster
> setup. I would still like to use Foreman as it's one of the key pieces
> for managing our HPC cluster. The issues I'm facing have to do with
> nodes getting their ENC data, uploading their Facter facts, and
> submitting their report.

Yeah, I think we need to support this better.

> * Is there a way to configure Foreman so it authorizes a server to
> upload facts/reports and pull ENC data based on the server's certificate
> being signed by the Puppetmaster? For now I've had to set
> "restrict_registered_puppetmasters" to false to allow servers to contact
> Foreman.

No, but that's an interesting idea.

> * For fact uploading, would performing a "Facter.to_hash" in place of
> reading the yaml file from the Puppetmaster have the same outcome?

Yes, that should contain the same data.

> Would it be more efficient to write a Foreman plugin that saves a host's
> ENC YAML to a file that can be simply read very basic node.rb type
> script? Currently our Puppet modules and all Puppet agent config files
> needed to run "puppet apply" will be on a NFS share that is exported
> read-only to all compute nodes.
>
> My goal is to reduce the runtime of these scripts as much as possible.
> If I have 64 nodes contact Foreman for ENC data, that could become a
> bottle neck as I don't have the resources to run Foreman on anything but
> a basic oVirt VM.

I think you'd be OK with 64 nodes just for ENC data, as it's a very
simple request, the response times are usually very quick. But if you
do need to get it responding faster, then sure, you could export the
data easily even with a script or rake task which calls @host.info to
generate the ENC hash, or calls the API and saves the result.

The downside to this is that the ENC data might be out of date if you
rely on facts for some of the data, e.g. for smart class parameter
matchers, or on a brand new host the data may not yet exist.

Usually uploading reports and facts is the largest bottleneck as it's
heavy write activity, so if you're trying to get the total runtime down
then you might want to buffer the reports and upload those
asynchronously with a custom report processor & script. Usually facts
are uploaded synchronously due to the dependency between the fact data
and ENC response noted above.

··· On 19/05/14 21:07, treydock wrote:

–
Dominic Cleal
Red Hat Engineering

3

>> After much testing I've found that my deployment of Puppet + Foreman for
>> a HPC cluster requires the use of masterless puppet. Our plan is to
>> have Puppet run between jobs via SLURM Epilog and/or Prolog scripts and
>> we frequently have 64 node jobs which would mean 64 servers would hit
>> the Puppetmaster at the same time using a traditional Puppetmaster
>> setup. I would still like to use Foreman as it's one of the key pieces
>> for managing our HPC cluster. The issues I'm facing have to do with
>> nodes getting their ENC data, uploading their Facter facts, and
>> submitting their report.
>
> Yeah, I think we need to support this better.
>

Looking at the code in Foreman::Controller::SmartProxyAuth, it appears
that a valid and authorized user can also upload facts and reports.
Is that correct?

I think a conservative approach to such a feature would be allowing
either registered Puppetmasters OR the host itself to upload
facts/reports.

I've created Feature #5914: Allow a host to upload its own facts and reports - Support masterless Puppet - Foreman for this purpose.

>> * Is there a way to configure Foreman so it authorizes a server to
>> upload facts/reports and pull ENC data based on the server's certificate
>> being signed by the Puppetmaster? For now I've had to set
>> "restrict_registered_puppetmasters" to false to allow servers to contact
>> Foreman.
>
> No, but that's an interesting idea.
>
>> * For fact uploading, would performing a "Facter.to_hash" in place of
>> reading the yaml file from the Puppetmaster have the same outcome?
>
> Yes, that should contain the same data.
>
>> Would it be more efficient to write a Foreman plugin that saves a host's
>> ENC YAML to a file that can be simply read very basic node.rb type
>> script? Currently our Puppet modules and all Puppet agent config files
>> needed to run "puppet apply" will be on a NFS share that is exported
>> read-only to all compute nodes.
>>
>> My goal is to reduce the runtime of these scripts as much as possible.
>> If I have 64 nodes contact Foreman for ENC data, that could become a
>> bottle neck as I don't have the resources to run Foreman on anything but
>> a basic oVirt VM.
>
> I think you'd be OK with 64 nodes just for ENC data, as it's a very
> simple request, the response times are usually very quick. But if you
> do need to get it responding faster, then sure, you could export the
> data easily even with a script or rake task which calls @host.info to
> generate the ENC hash, or calls the API and saves the result.
>
> The downside to this is that the ENC data might be out of date if you
> rely on facts for some of the data, e.g. for smart class parameter
> matchers, or on a brand new host the data may not yet exist.
>
> Usually uploading reports and facts is the largest bottleneck as it's
> heavy write activity, so if you're trying to get the total runtime down
> then you might want to buffer the reports and upload those
> asynchronously with a custom report processor & script. Usually facts
> are uploaded synchronously due to the dependency between the fact data
> and ENC response noted above.

The fact upload BEFORE pulling the ENC data makes sense. However in
my particular case the facts should not change too frequently.

··· On Tue, May 20, 2014 at 2:46 AM, Dominic Cleal wrote: > On 19/05/14 21:07, treydock wrote:

–
Dominic Cleal
Red Hat Engineering

–
You received this message because you are subscribed to a topic in the Google Groups “Foreman users” group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/foreman-users/XB7IwENnQyY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.