RFC: Simple & automatic host registration WF

This RFC is part of an ongoing effort to simplify the registration process of the hosts into the Foreman. It’s not about provisioning new hosts from Foreman, but about registering already existing (running) hosts to Foreman and running user defined commands from the registration templates. That is especially useful in brown-field deployments, where people don’t provision the existing infrastructure from scratch, but want to start managing existing hosts by Foreman.

This RFC further investigates a part of an earlier RFC - RFC: Client workflows and tooling, namely the “foreman-bootstrap” part.

Motivation & Goals

Make the host registration simple and easy. We should allow users to register and set up hosts with one command.

Ideal Use Case

One command rule them all:

  • User will call curl | bash command, for example something like this:
    curl --user admin:changeme https://foreman.com/templates/register-new-host | bash
  • Foreman will generate Global registration template with commands for host registration to Foreman
  • After host creation Foreman will render the host-registration template with commands for host set up.

Features

  1. Introduce Global-Registration template
    • Not scoped to host object, the template can’t rely on @host or any other host specific variables and macros
    • Contains commands for host registration to Foreman, at first using subscription-manager however this should be configurable for platforms not having this option
    • Also contains second curl | bash after the registration, that will fetch specific host-registration template (see below)
  2. API endpoint for Global-registration template render
    • Render template from Default global registration template setting
    • Could allow the user to pass parameters to template (Organization id, activation key …)
    • Authentication by login/password or by user’s API token
  3. Host-registration template (scoped to @host)
    • Already implemented in PR#6813
    • This template allows host-specific bootstrap logic to be executed, this template can be affected by parameters stored per Host group, OS, Subnet, Domain, Organization, Location or as a Global parameter.
  4. Enable built mode for unmanaged hosts (see Questions & Ideas)
  5. New Provisioning setting "Default global registration template"

Templates

What should we do in the templates?

Global registration template

  1. (Check root privileges)
  2. (RHEL only) Check if subscription-manager is installed, if not install it.
  3. Run host registration (RHEL with subman, for other systems use different tools)
  4. Get host’s id from Foreman (based on subscription UUID)
  5. Render Host-registration template for created host, we know the host id to render it for from step 4

Host-registration template
(each step should be parameterized in Foreman)

  1. Upgrade to newest subscription-manager
  2. Setting up SSH keys for REX
  3. Puppet setup
  4. Insights setup
  5. Install Katello-host tools
  6. CNAME support (bootstrap.py has it, supported since 6.8)
  7. (… another steps from bootstrap.py - see the WF)
  8. ??? Anything else, any ideas?

Questions & Ideas

Katello client bootstrap

Bootstrap.py already covers a lot of logic regarding to host registration to Foreman. Maybe we could use it, or we could move the logic from bootstrap to Foreman templates. We need to consider all cons and pros.

Unmanaged hosts vs. managed hosts

By default a host registered via subscription-manager is unmanaged, this won’t allow us to switch host to build mode during the run. It seems we can however allow building the host easily even if it’s unmanaged. Build mode traditionally meant complete (re)provisioning. Would it be acceptable to consider building also bootstrapping existing machine?

Another option is to introduce a new host state - Registration / Registering. Since build mode is used for complete reprovisioning, adding a new state could be more user friendly than changing the meaning of “build mode”.

Support for different distros
How about debian based distros? ATIX worked on porting subscription-manager for debian, is that available today? We prefer to use subscription-manager, since it already sends some basic facts we can use to determine OS and hence OS specific parameters can tailor the host-registration template. Other agents that send facts (puppet, chef, salt) need some infrastructure to exist or their installation is more complicated.

For debian-based systems without subscription-manager we can collect required data manually (hostname, ip …) and create unmanaged the host via Foreman’s api.

Template example (POC)

An example how simple Global registration could look-like: https://pastebin.com/HxJhGv3R

Links

2 Likes

Sounds good, all things considered.

While I don’t agree that sub-man is particularly easy to handle, I think this calls for a generic approach using @lzap’s ufacter. Ideally, this would use the same technique that discovery does (e.g. the same fact parser etc.).

Good, but please drop the curl --insecure immediately. The required server certificate should be shipped as part of the template. Downloading the katello-ca-consumer-latest.noarch.rpm via curl --insecure is a pretty bad idea imho.

Why do we need to switch the host to build mode in the first place?

That could be a good alternative for platforms, where subscription-manager is not available or people don’t care about its functionality, e.g. activation keys. @lzap is ufacter packaged and available in our client repo? Does this work on all supported platforms?

Yeah, we’ll need to introduce some insecure parameter for the initial curl. While I think people should use trusted certificate for their Foreman from the very first moment, sometimes they don’t and need a way to override this. We could also use a fingerprint verification of the content we download from the server, the fingerprint would be another url param for the initial curl.

We rely on the registration templates. Currently, registration template can ge rendered for unmanaged host, but when it’s rendered, it tries to switch the host to the build state. This silently fails due to the validation preventing build mode on unmanaged host. Also when the registration template calls home to inform the host is built, it returns 404, again just because we allow this endpoint for hosts in build mode. I don’t think there’s a good reason for this validation.

As a benefit, if we allow unmanaged hosts to enter the build mode, we’d get “build status” reported for them. Users could watch the progress, e.g. this host just started the registration process, that host finished already. Or see the information about the time it took to register the host.

1 Like

While I realize the RedHat Satellite template should likely download and install Subscription Manager, that is hardly a great default template for Foreman users, or users of other Operating Systems (such as but not limited to CentOS, Debian, *BSD, illumos, …).

Also having the script accept arguments does not seem to difficult, as shown in the following example, here the { … } would be replaced with the curl ${URL}:

bash$ { echo '#!/bin/bash'; echo 'echo "My Arguments:" "${@}"'; } | bash -s -- --insecure --no-submgr
My Arguments: --insecure --no-submgr
1 Like

We actually already have an endpoint which renders templates and has authentication enabled - that’s the template preview endpoint. Maybe this could be refactored instead of creating another one. Template renderers are special, we already have two and it’s good idea to try not to increase this number.

While I would like to see less tools involved, I don’t think we should base registration process on rhsm (subscription manager), it’s quite irrelevant not only for debian users, but even for centos users. Ideally, the registration process should detect the underlying OS and pick the right tool for the job.

The problem is that creating a host with and without rhsm makes a difference - the host either has content part or not. This is a design problem we have. To solve this in a consistent way, I suggest that the process is:

  • Register via rhsm if RHEL is detected or skip this steip
  • Register via foreman tool (puppet or some generic fact upload)
  • Setup client tools according to the configuration

Honestly, I’d like to get rid of unmanaged hosts in the long term. A host that don’t need provisioning info should be just a host with those fields set to null.

No and I don’t know. I only tested it on Linux so far. But my overall goal with ufacter is to create facter-like facter replacement that reports minimum set of facts plus some extra and maybe leverage that for discovery later. The idea behind this is that users can use this tool instead of facter while sticking with the traditional foreman registration process. For us, the tool is useful to report netlink information, low level network interface info.

Sticking with facter interface allows users to use ufacter as a drop-in replacement. The registration script could offer an option to use the original facter for puppet shops and ufacter as an alternative to those who don’t use puppet:

  • Register via rhsm if RHEL is detected or skip this steip
  • Upload facts via either facter or ufacter depending on configuration
  • Setup client tools according to the configuration

I like this approach in general and think that an easy way to register hosts in brown-field environment would be super useful addition to Foreman.

There is one point I would like to highlight, as was mentioned by a few others as well:

We should definitely not base the default on subscription-manager in my opinion. it is only available for very few distros, and we want to enable registrations of multiple options, as well as allow it without katello installed.
Facter seems like the tool that is available on the most platforms (but i might be wrong and there is a better option), and it does not actually require puppet infrastructure to be set up for it to run.
Running subscription-manager can be a step done in the host-registration template, if Katello is installed (similar to other steps that are only done if certain plugins are set up).

1 Like

Now having read @lzap post, why not ask the user to provide the actions required, without guessing what is best.

curl THE_URL | bash -s -- --help

curl THE_URL | bash -s -- --install-rhsm

curl THE_URL | bash -s -- --install-rhsm=6.7

curl --insecure THE_URL | bash -s -- --install-cert-anchor --use-installed=ansible

curl --insecure THE_URL | bash -s -- --install-cert-anchor --install=ansible

curl THE_URL | bash -s -- --register=basic

While some options will take time to support a given OS (like certificate installation on non-RedHat OSes or non-Linux OSes), having a simple generic script should be doable. Adding lots of complexity will make the script un-audit-able by admins who will than likely not use it, or hack on it themselves (creating a local maintenance issue).

Start the script by including a snippet with a clear and hopefully simple case "$(uname -X):$(uname -Y)" in and establish clean variables to make further code conditional code easier to maintain. Example: [Aside: these name come from a separate older project of mine, and are illustrative only] OS_FAMILY=(linux|bsd|sunos|…), OS_FAMILY_LINAGE=redhat|debian|freebsd|netbsd|solaris|illumos|…, OS_MAJOR=…, OS_MINOR=, SYSTEM_ARCH=X86|ARM|SPARC|…, SYSTEM_OS_BITS=32|64, …

With the main script having a simple ‘basic’ registration type available, which will register a host on basic uname(1) or similar derived values (from that included snippet).

Given this script would likely be a ERB template, you could include the URL (and as previously stated the certificate), and for more complex parts (like installing providers like RHSM, puppet, ansible, …) separated out into scripts (or modules) downloaded on demand (they would also be templates, but given the fetch URL can now contain uname(1) or similar derived data - some of the complexity of which implementation to use, can be handled on the Foreman server (with templates; host or other variables, fallback defaults, etc).

This would keep provider logic out of the main script template, as it could download modules (like: RHSM, ansible, puppet, …) as required (that is, only when asked for via the command line or CLI defaults), and that fetch (as stated previously) would provide the Foreman server with enough details to get the correct script/module for your OS, or fallback to the generic script/module for that feature and tell the user to develop this functionality for their OS. Remember make each module succinct, package installation is very OS linage or version specific; but using ansible(1) once it’s installed is likely common code. Always, remember the DRY (Don’t Repeat Yourself) principal.

This approach would also mean that, if a script/module for say RHSM-6.X and RHSM-7.X differ considerably; the user told you which version they want to use (or if they didn’t the default could be controlled the Foreman server) and the code to be download can be the best script/module for the task, with that logic being on the Foreman server and not in the script. Again the KISS principal (Keep It Simple Stanley).

Supporting multiple OSes and version of software is a pain! Separating out as much conditional code into modules which can be downloaded, executed and return exit values to be checked, provides a simple interface to code to. And a way to search for the correct module to provide without huge amounts of conditional logic. or masses of code that may not be relevant to the developer or reader.

This can be especially simple, if the logic which provides the script/module on the Foreman server can use search paths (think PATH, MANPATH, RUBYPATH, PERLLIB, …) and search for the first instance of a matching template. Example: Given a bunch of uname(1) based data, construct a search path and search for the file [or alternatively construct a list of filenames to look for, and return the first one found]. The search path or the search filename than just contains a most specific to least specific implementation of that module with a guaranteed fallback implementation.

While separating out this code into modules provides more files to manage, each file should be MUCH simpler and does not need a huge amount of conditional code. Testing the module implementation as a standalone script on each target platform will be easier, and common code could still be included via the ERB template expansion process.

Furthering my fictitious example, --install-rhsm=6.7, the Foreman server knows the provided uname(1) or similar data, the module (RHSM) and the requested version (if provided), and can search for a module that can do that or provide a fallback implementation that tells the user the operation is not supported. If using constructed filenames as the search mechanism (c.f. PATH searching), than such a search might start by looking for RHSM-EL${EL_VERSION}-${RHSM_MAJOR}.${RHSM_MINOR}, followed by RHSM-EL${EL_VERSION}-${RHSM_MAJOR}, followed by RHSM-${RHSM_MAJOR}.${RHSM_MINOR}, followed by RHSM-${RHSM_MAJOR}, with a fallback of RHSM, is just one possible implementation.

While some variants of module (such as support for a given OS, or given provider on an OS) may be missing or out-dated, in my opinion (and it is just that) reducing the complexity by providing the implementation with clear divisions of responsibility and delegating specifics, will make that initial script readable, maintainable, and testable; and the same will apply to delegated modules.

One IMPORTANT thing, (I just thought of, for this environment), is have that uname(1) and related derived constants/variables, should all be part of an include file in the main script template. Why? because than the OS/Product specific modules test harness (or developers on the command line) can, load the same constants, and these can be available to the modules when they are tested on their respective OSes without trying to register hosts. Again, this gives a defined environment of constants that a module can expect to be available to it, and a set of return codes to exit with, thus providing a richer and better defined API between the main script and the implementation of each module.

I hope these approaches, which I have used in many projects (related to automation and platform management), provide food for thought and a way to create something for the Foreman project (and its more diverse community) and your likely immediate goal of something for the Satellite ecosystem (which has a lot of non-Foreman requirements, like subscription management), and provide a set of templates (a code base), that is simple but flexible enough for all consumers. And don’t be afraid to factor out any common code (functions) and constants into appropriate small include files; that the main program and the sub-modules can re-use (especially with the suggested fork()/exec() model ). Note: Don’t include the constants computation in the sub-modules, have them available via environment variables, it makes the code easier to test, especially on the command line, if the constants are loaded (and exported) into the current shell like they would be in the main program, than developers can modify a value and run a changed module to see if the code would run under different conditions (example: did the conditional statement they just introduced work the way they intended)

That probably enough of brain dump for now,
Peter

PS: OK one more thing, while I’m suggesting approaches to shell script development. I can’t remember how I learnt the following (reading or own design), but I been using it for 20+ years. Try, what I call DO=echo (and DO=: ) shell script development. All non-destructive actions are the same as regular shell scripts, but destructive actions (any command which change permanent state, so mkdir(1) falls that category) are prefixed with ${DO}, so that when the script executes with DO not set, the script runs as normal, when run with DO set, as in the following examples:

DO=echo ./myScript my1stArg ....

DO=:    ./myScript my1stArg ....

The first invocation will show all the destructive commands that would have been executed, without actually doing them. The second shows what the output would look like to the user, as again the destructive commands are not executed, but this time they don’t pollute the script’s output. There is only one downside to this approach, and that is destructive commands that require pipes or IO redirection. Basically you need to repeat the destructive command inside an if block and quote all the special cases in the DO being non-empty case. Given how many scripts are written, its not all that common that the destructive actions include pipes and/or I/O redirections. Hopefully a simple (contrived) example will assist with understanding.

#!/bin/sh
TEMP_AREA="/tmp/$(basename ${0}).$$)"
${DO} rm -rf "${TEMP_AREA}"
${DO} mkdir  "${TEMP_AREA}"
if [ -n "${DO}" ]
then
  ${DO} command1 \| command2 \> ${TEMP_AREA}/pass1
else
        command1  | command2  > ${TEMP_AREA}/pass1
fi
# ...
${DO} rm -rf "${TEMP_AREA}"

Note: Technically, this introduces a huge security hole, if a user is allowed to execute the command with elevated privileges, so don’t use this for SUID/SGID shell scripts (but thankfully no-one would ever do something so silly, given the history of SUID/SGID shell scripts). If you have elevated privileges, and need to execute such a script, explicitly set DO='' in the environment to be passed to the script.

PPS: Can anyone tell I’m a “real” programmer who has had to write a lot of shell script :wink: Lets not dwell on that too much, especially when they have had to be compatible with most implementations of /bin/sh !

too bad I wasn’t aware of this… we recently wrote a very similar project at https://github.com/ori-amizur/introspector - perhaps can join efforts…

Without wishing to cast any shade on the ufactor project and others like it, there are a lot of less mainstream OSes out there, and an increasing number of appliances using less common OSes or hardware platforms. Adding a dependency on a program that may have never been ported to that OS (or an older version of an OS) or hardware platform, seems to restrict the use case of “simple and automatic host registration”. Knowing there are older boxes/VMs and odd boxes/VMs, because they registered in Foreman seems more desirable to me, than having the most complete information on a registered host.

In a very long post (earlier in this topic), I suggested that a basic registration based of say uname(1) or similar simple data might be the best approach, as it get the host in Foreman and thus visible, even it it has to be tagged as “Unmanaged”, or “ItsScaryButWeStillNeedThisSystem” :wink:

1 Like

As a late postscript, I’ll note both projects mentioned are written in Go, which is in no way a bad thing, but Go’s support for older OSes (or non-mainstream hardware) is going to be very patchy.

1 Like

How will you handle authentication in the generated bash script? Do you create some one time use key on the Foreman side to prevent reuse?

As someone who has never used subscription-manager, I don’t really see the benefit of this. If the only goal is facts, then it’s overkill for most. As others mentioned, it’s also unavailable in many cases.

At least for modern Linux you can source os-release to easily get most information. It’d be great if we could have a fact parser for that since we can assume its presence. All you need is any shell. Any additional tools may not be present and may not work.

How much more information do we need besides the OS name, OS release and hostname? Network information is complicated and often wrong in complex setups. It may be unrealistic to rely on DNS in many environments though that would be my preference.

Step 4 is not needed and step 3 can do this: host registration can return the data. Either in a HTTP header, the HTTP body or both. API documentation and API documentation both return an ID.

What’s the difference between curl --insecure and having an insecure parameter in the request? Isn’t the result in the end same?

Yes, sub-man would be only for RHEL systems, for other systems we would choose different tools. I’ll update RFC to mention that, somehow I didn’t realize it sounds like we want to use subman everywhere.

The question is if we want to pass arguments to the bash script or we want to send them as request parameters with CURL and handle the logic in the Foreman templates.

The idea was that this endpoint would:

  • Generate template based on value from Provisioning Settings parameter “Global registration template”, so the user doesn’t have to know the id of the template.
  • Allow to pass (permitted params from request to the template, for example:

/template/global?organization=default&activation_key=meh
(Which I believe you can’t do with current endpoints - please correct me if I’m wrong.)

Hmm, Ansible didn’t come to my mind, to be honest. Not sure if this will go with the “minimum-dependencies” approach we try to keep.

Yes, that is the plan, but I don’t have any final solution yet.
I’m thinking about two ways how to do it:

Yeah, the idea is to use subman for RHEL only, for other systems we will do it without subman. I didn’t describe it clearly in RFC, my bad, now it’s updated.

Good point, for registering host & running REX we just need FQDN (… and organization & location).

Sorry, what I was trying to say was: when developing modules (scripts), don’t repeat code in multiple places. If the user asked to install a module (like RHSM, ansible, puppet, …) that code is OS dependent (e.g. package manager, repo into install from and package name), but once installed, the tool installed (subscription-manager, ansible, puppet, …) could be run from another module (script), so code related to running that tool is not part of the code related to installing the tool. So it was a comment on development approach to my suggested example usages and not a statement that we need to use ansible (or any other specific tool).

As I stated, I actually think there should be a basic register based on tools available in most OSes like uname(1) and as mentioned for Linux, the os-release file.

Sorry for the confusion, the post was very long and more about development approaches to the problem (based on Foreman as a multi-platform tool), and the need to support different user/customer environments by taking a modular approach to the development of a registration script.

1 Like

Yep, we need to get rid of both. :slight_smile:

These could be a JWT. I guess we already have most infrastructure for that, we just need to extend it to a not-host-specific token.

I believe we need to provide a way for the user to accept the risk. We should default to trusted certificate. But as in every other software using SSL, there needs to be a way to skip the server authentication. That’s why we can offer that to user via parameter. The difference is, user explicitly accepts the risk by saying ?insecure=true.

The other option would be ?trust_certificate=CertificateFingerprintValue, where the script would first fetch the server cert and verify it’s the one, that was expected. I don’t expect people to type the fingerprint manually, but Foreman UI could generate this command with the right fingerprint value. It’s then up to the user to transfer this generated curl command from Foreman UI to the registered machine without it being tampered.

I’d still say though, at lesat for testing purposes, ?insecure=true is good to have.

What is the benefit of JWT token vs user API tokens we have today? Is that because they allow full API access and are limited only by user permissions?

That’s basically what I had in mind. Foreman should render the fingerprint or the certificate directly into the script.

A personal access token makes sense to allow a user to authenticate e.g. in a script or in hammer without having to expose the password with the user permission.
The JWT could be limited to allow only host registration and Foreman UI could simply render the One-Time credentials as part of the curl command. That’s a lot more user friendly (the user has to come the command and paste it to a terminal) than having to enter username and password in the shell after executing the command.

but the script is also fetched through SSL

the idea was, the curl command is generated by the Foreman, username and token would be in the URL and they would copy it as a whole. The same way like JWT token. But, I like the token being limited to the fetching of the global template only. While API tokens would work out of the box, this is something we should look at. Do you see that as a part of minimal viable version?

I would consider the personal access token burned then. Afaic we try to only show it once to the user and actually just store a hash in the db.

I do.

Yes, the idea behind uFacter is to create facter-like compatible tool. Meaning that it should be possible for users to do drop-in replacement with the original facter for other OSes. The goal is not to start yet another inventory tool, the goal is to find a minimum set of facts which are worth implementing in a small tool that can be used by users who do not want to use puppet/facter or do not care - they just want to have a host registered with something that comes “out of box” with Foreman.

I wrote the tool for a different reason, the original facter lacks low-level networking information (netlink in Linux) so in Foreman we cannot build good representation of bonds, vlans and bridges. The ufacter can actually be used from facter to provide this information (on Linux only). Since Facter is written in Ruby and there was no good netlink Ruby wrapper, I took a small “yet another facter” project in Go and wrote it. As a sideeffect, uFactor can actually provide the core facts so it is possible to use it as drop-in replacement. That’s why we are discussing this as an option.

I agree. And I want to make a similar point. As someone who don’t use Puppet/Facter, I don’t see benefit of installing facter (from a 3rd party repositoiry) just to upload few facts to register a system into Foreman for users who are not planning to use Puppet at all. That’s why I think uFacter could be a nice default for Linux users to provide nice out-of-box experience. Granted, for other operating systems facter would be the only option.

Well, yes and no. We do not indeed need much information to register a system. In fact, we don’t even need architecture or operating system. The absolute minimum set of information is something like:

  • hostname
  • authorization (optionally)
  • organization and location
  • hostgroup (if provided)
  • activation key (if provided)

All the rest is “nice to have”, these can be fetched after host is bootstrapped and one of the fact upload tool checks in (rhsm, facter, ansible, salt):

  • architecture and platform
  • hardware vendor and model
  • cpu model and speed
  • memory
  • disks (total space)
  • serial id (when present)
  • operating system and version

Now, having those facts reported alongside registration gives Foreman a good opportunity to auto assign some things like organization, location, activation key or hostgroup based on the incoming data. It’s a nice feature, something that discovery already does.

Which brings another idea, we were discussing merging discovery into core for some time. If we do that, we could base the registration process on discovery meaning that Foreman would be able to associate organization, location, hostgroup and activation key according to discovery rules. For this feature, we’d actually need to have the registration to be based on facter from the day one because discovery is based on that today.

While this looks great on paper and I am huge fan of shell scripting and Perl and all the old good stuff :slight_smile: I have to say that just running uname or other UNIX tools does not give you portability at all. Doing this will ultimately lead to us maintaining a bunch of shell code (in our templates or repositories) for all supported OSes.

That’s why I like having uFacter as out-of-box tool (shipped in our client repo, zero dependencies) with alternative (and fully supported) way of using the original facter using a single switch (option). We can go much futher and also delegate all the “other OS” handling to the facter team.

Yes, but I think having two stage initial bootstrap compared to just a single insecure curl is better for one particular reason - users could deploy CA certs in a different way (embedded to firmware, TPM, download from external system, SSH copy or inject it into the OS in a different way).

Neither of these points mean that we could not reuse the current unattended/preview controller. We’d need to pay attention when adding those “override” parameters so it does not introduce security issues with provisioning templates but it’s doable - just by making sure these parameters are read via a special macro instead of directly using them as @variable.

Yes I came across it mentioned on the community site a few weeks back IIRC, and had a quick look at the github page.
Please don’t interpret the support of a uname(1) based fallback as anything against ufactor, or even GoLang, but a comment on the likely diversity and age of the OSes that Foreman users might have tucked away in there server rooms, home labs, …
BTW: Good Post !