[RFC] Replace /pub/katello-rhsm-consumer static script with a template

Development RFCs
,
1

With the deprecation of /pub/katello-rhsm-consumer script we need to find new ways to expose the functionality from Foreman server.

What is this script?
This script has two main tasks:

  1. Install Foreman’s CA certificate on the host
  2. Configure subscription-manager to use Foreman server to fetch content

When it was used?
Since the script was entirely public, we don’t really know where it was used. We identified a couple of use cases where the whole script or part of it was used though:

  1. While changing Foreman’s CA certificate (we actually need only part 1 of the script)
  2. While running Anaconda setup, where it needs to access content during the installation process
  3. After the host is provisioned it needs to access content from Foreman server.

Suggested change
Use case 3 is handled by global registration template for hosts that were not provisioned using Foreman at all.

Anaconda currently has a hard coded endpoint that is set up to query /pub/katello-rhsm-consumer endpoint on Satellite, download the script and run it once in the installer context (use case 2) and then in the installed OS context (use case 3).

I am proposing a new template that will be accessible through our /unattended/:type/:id endpoint. This template will be rendered and supply the same services as katello-rhsm-consumer.
To make the endpoint available without authentication, we will need a new template type (anonymous) that will remove authentication checks from the regular /unattended endpoint. This will also make the endpoint available through HTTP access (as opposed to HTTPS), so it will be accessible even if Foreman’s CA cert is not trusted yet.
Additionally we will need to add Remote Execution wrapper that will run the script both by SSH and Ansible playbook.

To make sure we comply with Anaconda, we will need to make sure /pub/katello-rhsm-consumer will either redirect or rewrite to /unattended/anonymous/register-katello template.

PRs:

Open discussions

  • Do we need the template to be exposed for everyone, or do we want to restrict the template to hosts that are known to Foreman?
  • Is it possible to make /unattended/anonymous/katello-register to be available through /pub/katello-rhsm-consumer endpoint.

Any other thoughts are welcome

4 Likes
2

Hi @Shimon_Shtein

Thanks for opening this RFC! AFAIK, katello-rhsm-consumer is currently used/documented for the following workflows:

  • Docs: guides/common/modules/proc_deploying-a-custom-ssl-certificate-to-hosts.adoc: To update SSL certs on hosts
  • Docs: guides/common/modules/proc_renaming-smart-proxy.adoc: As part of the “Renaming Smart Proxy Servers” procedure.
  • Docs: guides/common/modules/ref_registration-methods.adoc: As a deprecated method to register hosts.

I am currently investigating if it was also used to register non-RHEL hosts to Foreman/Katello. I will report back as soon as I know more.

1 Like
3

Investigation - part 2. We can stop using the /pub/katello-rhsm-consumer script altogether if we fix our kickstart anaconda scripts.

For RHEL 9 we already use the rhsm anaconda directive that configures subscription-manager to use Foreman’s endpoints, which configures rhsm for both the installation and post-installation environments.
For previous versions of RHEL we still use the redhat-register snippet, which is executed relatively late in the anaconda script (%post section). This configures the post-installation environment only.

To make sure kickstart works we will need to do the following:

In the %pre section:

  • run the CA certs update script (to make sure the installation environment can access Foreman’s repos)
  • For RHEL < 9 run subscription-manager configuration script.

In the %post section:

  • run the CA certs update script again (now it would be in the context of the newly installed OS - to make sure the new host can contact Foreman)
  • For RHEL < 9 run subscription-manager configuration script (this step is already present in the template)
4

Reconfiguring a host

I think reconfiguring a system should mostly be done through configuration management. There’s a reason we invented it, instead of trying to wrap shell scripts.

Foreman’s model has long been to first make sure a system is registered to be managed. Once a host is in the database, you use configuration management (Puppet, Ansible) to get it in line. We try to provide all details for a host via the ENC endpoint. RHSM and content URLs should be provided there as well: Fixes #32835 - Add RHSM and content URL as info providers by ekohl · Pull Request #9413 · Katello/katello · GitHub.

There is also a community.general.redhat_subscription Ansible module to reconfigure subscription-manager. Sadly it’s in community instead of a supported module, but ideally that would exactly be what we use: take all the info from Foreman’s DB and tell Ansible to reconfigure.

Having said that, I recognize that with Ansible pull mode the server certificate is used to pull the updates, so by changing the CA you can easily lock yourself out. Obviously we should tell users to prevent that, but Murphy’s law tells us that anything that can go wrong, will (eventually) go wrong.

I’m still not entirely sure what the best approach is, but we should steer users to the more secure ways. Retrieving some shell script from /unattended/anonymous/katello-register over plain text HTTP is not a secure way and should be thoroughly discouraged. That’s why I suggested we shouldn’t normalize the name. If it was a shell script, it should be --force. IMHO it should be insecure-register instead of katello-register, just to signal it’s not intended to be used for normal operations.

Kickstart

If you can fix subscription-manager in %pre (I never thought about that, not sure why) then I’d lean to avoiding the rhsm command altogether. We only use rhsm on RHEL because Anaconda on non-RHEL (like CentOS) doesn’t support it (this is a deliberate choice: 2211944 – Unable to use `rhsm` kickstart option on CentOS Stream 9). Using a single way gives us greater consistency between all platforms and reduces the testing matrix.

Having said that, I noticed in Add Satellite registration support by M4rtinK · Pull Request #3485 · rhinstaller/anaconda · GitHub that there are screenshots where it also shows up in the UI. I don’t have a RHEL install at hand to check, but if that’s the case then we can’t just drop it.

Other

Currently we don’t set the server_ca_cert setting in our installer. Introduce server_ca_file parameter by laugmanuel · Pull Request #910 · theforeman/puppet-foreman · GitHub was opened a long time ago, but we never got around to merging it. The proposed functionality above relies on a correct CA and I think we need to revisit that PR.

5

I would say we have two different problems here: Initial configuration and maintenance.

For initial configuration we have to do two tasks: set up the CA certificates and set up subscription-manager to work with Foreman instead of the default CDN configuration.
For maintenance, and specifically for updating CA certificates, we only need to set up the CA properly on the hosts.

So for initial config, I would suggest pushing the CA fix part into the %pre and %post section of kickstart and handle the subscription-manager configuration in accordance with the OS version - if it supports proper rhsm command, we will use it and if it does not, we will use %pre and %post script to configure it manually. In addition to global registration this should cover all types of hosts that are managed by Foreman.

For Foreman CA change we have a bit more complicated situation:
Ideally we would like to orchestrate the whole process:
Introduce a new CA certificate to hosts as an additional one, change the certificate on Foreman server, remove the old certificate from the hosts. Currently we don’t have that ability, at least not in a straight forward way. Additionally we need to have a way to fix the state of setups where Foreman server was already updated, but the hosts didn’t.
Here we have a couple of cases: if Remote Execution connection was not severed, either because the cert that is in use by Foreman host is still valid, or we have username/password auth set up for tasks, we can execute a job that will fix the CA on the host. In case REX connection is severed too, I don’t see any other choice than downloading a script insecurely and executing it.

For jobs, we already have PRs for the wrappers, and we can make them more secure, since we can add certificate validation as part of the job itself (put the cert as part of the job template, and make sure curl validates the endpoint against it).

6

P.S.
About the Anaconda: This is the exact place that insecurely downloads the katello-rhsm-consumer script. I think we can avoid this.

7

Right, but we have host registration for this as well. Why provide 2 mechanisms?

I also heard the use case of golden images, but I’d argue that there should be better mechanisms for that. For example, inserting the CA may be OK for a while, but it can expire or mismatch. For the specific URLs, they may change. For example, if you want to register some hosts to one content source (like in Amsterdam) and some hosts to another (like in New York). Do you build 2 different images?

I think it’d be preferable to use a provisioning method which can inject those values more dynamically. (Way out of scope: you could even use DNS).

There’s also the action where you change a host’s content source. Then you need to update the host’s config as well.

Did you also verify a non-automated installation? The screenshots indicated a user should be able to enter the Foreman (branded: Satellite) hostname in interactive mode. We’ll likely break that workflow if we drop the script.

8

Unless you intend to do host registration as part of the kickstart script, you need to set up subscription-manager for hosts that were provisioned using one of Foreman’s provisioning templates.

Agreed, golden images are a bit strange beast. Only one specific use case is considered: where the CA is always valid (meaning rebuilding the image on CA change) and all the hosts are connected to the same content source (more content sources means more images as well).

I would also like to have either global registration, or something similar to cloud-init for this case. But maybe we are missing something critical, like if they don’t have the ability to register the host immediately.

That’s a different use case, I am not sure I want to get it into the same scope. Anyway, this use case is only about reconfiguring subscription-manager (the second part of the initial configuration).

The parameter that you are entering is the base URI for Foreman server. If we find a way to not use this option, I think we can document that it will break the installation and discourage users from doing it.

But what is the use case we are talking about? I think there is no proper way to provision host from Foreman using interactive mode. So if the provisioning is not done by Foreman, we will need to do global registration anyway or specify Foreman repos through boot options. I don’t think that specifying the repos directly is a supported way to provision a host.

9

I did not find any templates that rely on katello-rhsm-consumer or katello-ca-consumer.

1 Like
10

I meant 2 mechanism to do host registration. It would be 3 if you count unattended installations.

Put in another way: your proposed configuration scripts are reinventing host registration and I object to that. We need very good reasons to do such a thing.

If you don’t register immediately, then why do you need to configure it? Maybe I’m missing something, but either the machine is an image (and doesn’t run) or it’s a VM and it is running. In my mind a running VM always needs access to repositories and if you want to consume content, it should be registered to Foreman. There’s no in between stage where the machine is configured to consume content, but not registered.

In theory there’s the edge case where you can consume content anonymously from Katello, but I don’t think we need to support that use case. That’s non-standard and users can find their own ways to configure it.

But it’s exactly the same thing: you need to reconfigure subscription-manager. With configuration management it should be idempotent: you tell it exactly how it should be configured and Puppet/Ansible can fix it. If we limit ourselves to the Ansible use case over SSH then an expired CA is not a problem as long as SSH still works.

If the user boots the ISO manually and goes through the installation steps manually then they can be prompted for the Satellite hostname. The expectation is that the host ends up registered to Katello. That ends up calling the rhsm code path we’ve been talking about.

11

I have gotten lost in the conversation, and since this may be a me thing, I am going to re-state what I think are the key elements and how I am thinking about this problem.

Definitions

Global registration (GR) provides two primary mechanisms that are coupled: configuration and connection.

Configuration encompasses everything that needs to be done to a client in order for connection to Foreman to occur. For example with subscription-manager, this lays down CA certificates in the right locations and changes the configuration of subscription-manager to point at Foreman (or a proxy server).

Connection initiates the registering into and either creation of or finalization of a host record in the inventory. For example with subscription-manager, this calls register to create the host record in Foreman and Candlepin and retrieves the client certificates that will be used for secure communication there after.

Use Cases

Changing CA

A sysadmin may need to change the CA certificate on every host ahead of time if it is set to expire.

If we assume the certificate has not yet expired, the sysadmin can opt to use existing tooling to perform the mentioned three step process:

  1. update the Foreman server with a bundle containing the old and new CA certificates
  2. update all clients
  3. deploy only the new certificate

The sysadmin may opt to use Remote Execution (REX), script or Ansible, if they have it connected. If the sysadmin uses a different system, for example, home grown or AWX or Puppet, then they need a reliable endpoint to get the CA certificate such that they can integrate those methods.

Anaconda

Anaconda needs to be able to perform configuration and connection independently at different stages. Thus, global registration fails for that use case as it couples the two actions together.

This could manifest as a method to acquire the CA certificate and manual configuration of subscription-manager or a script that combines the two together based off data stored in Foreman.

Changing content source

This use case is needing to modify one or more hosts subscription-manager configurations to point at a different proxy server/Capsule.

Disaster recovery of CA

Assume the CA certificate has already expired and every client needs to be updated. In this use case, Remote Execution or any other system for updating a client will not work. The sysadmin will need to touch every host in some other way to update the CA certificates. They do not need to change the sub-man configuration, for example, just update and have access to the new CA certificate.

Images

A sysadmin who creates golden images will need access to content during image creation. The sysadmin may also want to pre-configure that image to point to a particular Foreman or proxy server and may want the CA to be present on the image. During deployment, they may want to change the configuration prior to connection.

Concerns

I’d sum this up into two key points:

  • How do we create fundamental and re-usable solutions?
  • How do we avoid one-off solutions but ensure we solve the actual workflows?
12

We can look at it from another angle: I am splitting the host registration into separate parts: CA update, rhsm.conf changes and the rest of registration.

We are talking about use cases where we only want to do CA update, without rhsm and the rest of the script.
I can propose a different approach: make the global registration entirely idempotent, so for hosts that require only the CA update, the rest will become a noop. Very similar to what we do with any other configuration. The only use case that drops out of this scheme is the case where we want to update CA, but we don’t want the rhsm.conf changes nor do we want the rest of the registration.

Will it make more sense to have it as an option for global registration? Something like “prepare only, don’t actually register”?

@Marek_Hulan I would need your expertise to identify such cases.

I don’t really look at CA change as part of rhsm configuration. I would say that those are two different things. And yes, I do think that rhsm.conf changes can be done using any puppet/ansible/pure REX.

I think that if the user can download the ISO manually, it means he does not want to consume content from Katello. If the user wants Katello content, I think he should download the ISO from Katello, which he can’t do unless the CA is set up correctly on the machine (or the user bypasses the check, but it’s problematic).
Anyway, we can make it so it won’t hard-fail by publishing a no-op script instead of the registration one on the /pub endpoint. Then the registration would be done by the global registration template.

BTW, our api allows access to the rendered template, so if someone needs to yank the CA change portion out of it, he is still able to do it.

13

You got the whole flow correctly and it’s the same for my point of view.

As for the proposed solution:
I propose to split global registration into three parts:
Configuration1 - CA update
Configuration2 - sub-man redirection (hope it’s a god word) to the Foreman server
Connection - the rest of GR script as you have defined it.
Each one of the parts will be a separate snippet, and will be consumed by various tools depending on the use case:
CA update (config1) will be consumed by global registration, REX SSH and REX Ansible variants (for CA change and partially recovery [1] use cases) and by the Kickstart %pre and %post stages [2].
sub-man redirection (config2) will be consumed by GR and Kickstart (both pre and post) and may be consumed by other configuration management tools that implement the content source change use case for the user.
Connection will be consumed only by GR.

[1] For CA recovery use case we may still use REX as long as it is not using a certificate signed by the old CA to identify - it may use another identity cert, or even username/password auth.
In case none of those work, we will need an endpoint to make the life of the user a bit easier.

[2] If we use %pre and %post sections of the Kickstart, we can avoid Anaconda’s built-in solution of insecurely downloading a script from /pub/katello-rhsm-consumer endpoint on the Foreman server that was then used to configure Anaconda installation environment and the installed OS (stages config1 + config2).

So it leaves us with the question of how should we expose those snippets according to the use cases it is used.

So as far as I see it:

  • config1 must be exposed as a downloadable script (for CA recovery), as REX template and as a snippet to be consumed by kickstart and GR
  • config2 must be exposed as a snippet for GR and kickstart and may be exposed for other config tools (TBD how)
  • connection should be a snippet/template for the GR

So now it leaves us with the question of are these ways of exposing the snippets sufficient, and are they the best way to answer all the use cases.

14

Don’t forget to add:

point at Foreman (or a proxy server, or a load balancer with smart proxies behind it) :slight_smile:

15

FYI, I have always wanted to have a way to make a foreman template “unprotected” for systems to be able to access/grab it without having to be authenticated/registered, for things like an initial ipxe commands, or cloud-init etc, or even a script to download as suggested here.

1 Like
16

In the meantime I intend also adding the certificates to the ENC, so it would be available for every tool that knows how to read the ENC in Foreman.

17

Can I add a feature request on top of this RFC? :slight_smile:

Today, next to /pub/katello-rhsm-consumer script, we also have a /pub/katello-server-ca.crt file that contains the certificate(s) to make them downloadable by other means (wget, Ansible’s get_url etc).
Could we also add something like that to the new code? Essentially just another template that renders just the cert, w/o all the deployment logic. (And before someone asks: no, please don’t use that endpoint in the deployment script, but embed the certs there – this makes the script self-contained if it needs to be transferred via some out-of-band method for CA recovery).

Other than that, what I read here is good. Today we have a big script (GR) that does everything (certificate, rhsm config, actual registration) in one go. We split this up into snippets that can be also used independently when only part of the flow is required (only “certificate” when you refresh certs, for example – I could imagine “only rhsm” could be useful for the “change content source” flow).

Grepping around in Katello, I think this would largely replace katello/app/helpers/katello/content_source_helper.rb at master · Katello/katello · GitHub, right?

2 Likes
18

Makes sense, we can think about it later.

I understand that it’s really easy to consume it as a pure cert, but wouldn’t ENC be enough? It will require curl and then a bit of parsing of the results.

19

I think the parsing is what people are afraid of (even tho we know that it’s not that hard).

20

Well, I can add yet another template to expose it, no problem.