RFC: Katello repositories for internal use

Development RFCs
1

RFC: Katello repositories for internal use

Written together with @iballou

Decision Due Date: None

Context and Problem Statement

Usually, the Pulp instance provisioned on a Foreman Proxy server is assumed to be under full control of Katello.

One of the first plugins that challenges this assumption, is foreman_ansible_director.
To store Ansible content, it effectively bypasses Katello and interacts with the Pulp API directly.

While careful naming hygiene and a change to Katello’s orphan cleanup mechanism made this possible and relatively unproblematic, there are use cases - such as container image storage - where this is not an option due to limitations in the registry API.

This got us thinking about a new type of Katello repository, which is intended for internal use only by Foreman, Katello and their plugins.
In this RFC, we want to outline our inital thoughts and start the discussion.

Use cases for internal-only repositories:

Container image storage

  • Red Hat Lightspeed on-premises uses multiple containerized services to monitor hosts and generate advisories.
    Currently, users themselves are responsible for procuring these images in the correct version.
    With internal-only repositories, the foreman_rh_cloud plugin could:

    • handle the version and dependency resolution internally,
    • clone the correct images into Katello, and
    • provide the user with a simple “run this image” URL.

  • Another containerized tool is the MCP server. This service queries the Foreman API and transforms the values into something conformant with the MCP spec, which is used by various LLM-providers.
    This too currently requires the user to find/build and pull a container image, corresponding with the Foreman version.
    Here, Foreman itself could register an internal-only repository and serve the image from there.

  • In foreman_ansible_director, such a repository would be essential for storage and distribution of Ansible Execution Environments.

SCAP content

foreman_openscap can be used to ensure hosts comply with security policies using SCAP files, which contain remediation rules understood and used by openscap to re-configure hosts.
Today, these files reside in Foreman’s Postgres instance. While functional, a relational database is not the ideal tool for this job, at least when compared to pulp_file.
Here, foreman_openscap could store this scap content in an internal-only file repository and distribute it from there.

Proposal

  • Introduce an internal flag on repositories to mark them as for internal use only.
  • When this flag is set, Katello could:
    • Skip indexing,
    • Omit creation of related Candlepin entities
    • Exclude these repositories from API responses/payloads to prevent accidental user interaction

By reusing a lot of already existing functionality, users of these internal-only repositories benefit from existing Katello features, such as the synchronization between Smart Proxies by default.

Decision Outcome

TBD

Impacts

While it’s too early to be certain, we believe this can be implemented without disrupting existing workflows.

8 Likes
2

This is a great idea; there are a few things popping up here and there where internal repositories would be a reasonable solution so I’m all for this.

Since we’re skipping Candlepin and API entirely for these repos, are you envisioning these being exposed at all to an admin user or will we pretend they don’t exist? I could see some obscure error telling an admin to look into an id for an internal repository and the admin not being able to locate it. We should probably have a way for the admin to delete these if needed.

1 Like
3

Thanks for the write-up @thorbend !

Katello’s tight ownership of Pulp has halted past ideas around having Katello’s Pulp host files for it. Either we would need to instruct users to sync and distribute specific files (which is tedious), or special products/repositories would exist that users could destroy.

OpenSCAP and IoP Vulnerability both could potentially benefit from having Pulp manage and serve vulnerability related files. For these, there are numerous files, so we could take advantage of Pulp’s de-duplication and general content bookkeeping features.

I have considered if projects should just create a separate Pulp Domain (if Katello enabled Pulp Domains) and work in there, but Katello does have so much existing code for working with Pulp. Using Katello as a Ruby library in a way for managing Pulp repositories with the help of Foreman Tasks could prevent a duplication of efforts.

So, onto how these internal repositories could work. I think we should consider them being as simple as possible from the start.

What information about the Pulp repositories does foreman_ansible_director need? Does it only care that the repository is synced and distributed? Or does it want to know about content units? Since you mentioned skipping indexing I assume the content unit information isn’t needed.

Perhaps we could start by clearly laying out the actions that Katello needs to perform on these internal repositories. Then, we could map those actions to our existing repo orchestration steps.

3 Likes
4

Having some sort of view of what Katello is hosting would likely be appreciated by users who like having maximal control of their environments. They could see perhaps the repo and the plugin that created it.

Deletion of internal repositories is an interesting thought, we’d need to make sure that users can’t get their Katello into a bad state accidentally via a normal user interface like hammer.

2 Likes
5

Why does it do this?

This sounds like it has the potential to be fragile. How are you mitigating that? I have flashbacks of the Pulp 2 notes field and that pain that caused.

How would foreman_rh_cloud track these?

I don’t understand this statement. The Insights images have configuration and deployment are not as simple as pull an image.

Currently this can be done on a per-org basis. Is the idea that this would be for the whole of the application? How would user access work?

Can you explain why it’s essential for that director workflow? Are there base execution environments that apply to all organizations?

I can see the value of this.

I am missing from the proposal how does a Foreman administrator manage these internal repositories?

@iballou Can you expand on why domains are not the right answer for how to organize and ensure a clear split between these repositories and organization-level ones?

3 Likes
6

I think domains would be good to have, but their implementation into Katello is added overhead. I was considering Pulp domains more for the case where pulp_ansible_director would not use Katello at all, and would rather be completely on its own. Katello would operate in its domain, and any other plugin could work in its own domain. Thus, plugins would not be so easily able to touch each others’ repositories.

There are multiple ideas to consider with domains:

  1. pulp_ansible_director continues with its own implementation of how to talk to Pulp, uses none of Katello’s code, and works in its own Pulp domain.
  2. pulp_ansible_director determines it wants to take advantage of repository tracking in Katello’s database and use our existing code as a library for talking to Pulp.
    • a) pulp_ansible_director’s repositories live in the shared default domain, and the internal repos feature ensures that they are not deleted by orphan cleanup.
    • b) pulp_ansible_director’s repositories live in a different domain. Katello tracks their existence but “physically” separates its repositories from the pulp_ansible_director’s repos via domains.
  3. pulp_ansible_director uses Katello’s code to help manage repositories, but we skip the internal repositories idea and just use domains

Domains do change paths where the API and content are both accessed, so it enabling them would not be a simple task. See the following for more info: Enable and create Domains - Pulp Project

The pro of the internal repositories idea is that Katello would remain the “broker” for Pulp in Foreman’s context. It reinforces the idea that plugins go through Katello to use Pulp, and thus can also expect help in managing repositories through Katello’s Pulp management code.

Now, if we don’t want Katello to be the Foreman mafia kingpin of Pulp, then ideas (1) and (3) above are better. As much as I like the internal repositories idea, I don’t want my bias there to get in the way.

Ideally the admin doesn’t need to do anything, but since we’re not perfect it would be good at least to be able to see and possibly delete what is created like mentioned by @qcjames53 . @ehelms did you have other ideas for how an admin might want to interact with these internal-use repos?

2 Likes
7

My understanding of this proposal was that it was a broad idea beyond the ansible_director use cases. I read:

  • Storing and managing container images that are used in deployment of services (e.g. iop images, MCP) and are beyond organizational boundaries.
  • Storing and distributing Ansible Execution Environments that are applicable across organizations.
  • Storing SCAP content

An admin would need to be able to see, understand and manage this content. Additionally, these use cases start to transcend Katello’s use cases as they put Pulp in a position similar to the database. Take the SCAP content idea, this would require Pulp with the openscap plugin but not necessarily Katello.

Today, as I understand it, we scope content based upon Organization. The repos for internal use idea is to have content that is scoped to the entirety of the Foreman deployment itself. I interpreted domains as a clear way to segment content. Which sounded like it applied here where we would want internal-use repositories segmented from any content created and used within an Organization boundary.

3 Likes
8

Enabling Pulp Domains in Katello is generally a good idea. Today content is not even perfectly scoped between organizations. Pulp de-duplicates content, so technically there is some sharing going on between organizations.

I was thinking that plugins, to take advantage of this, would need to go through Katello, but perhaps that’s an unfair requirement. Katello has a lot of good code for managing Pulp, but there is a decent amount of business logic mixed in. If Katello is not a required dependency to use Pulp, and we enable domains then plugins would have the freedom to do what they want with Pulp. They would lose the benefit of Katello’s code but perhaps that’s not a huge loss.

My next question then…

If the plugins have freedom to use Pulp as an internal repository source without the use of Katello, the architecture of the management side of this feature/concept would need to be determined. Plugins could expose their own repo management bits, or some shared internal repo management capability could be created.

Picture a single page to manage all of the internal Pulp repositories among plugins in your Foreman. Now then we’re talking about a shared dependency again, which could possibly be some lightweight plugin to connect other plugins to Pulp.

If we went that route, some of the Pulp managing code that Katello has could be separated from Katello business logic and included with the management capabilities. However, I think it would be good to nail down what an admin should be able to do with content that the plugins are supposed to be automatically managing. If it’s just a few simple things, great, but if it balloons, then it might start competing with what Katello does today.

1 Like
9

Thank you both very much for your comments!

One of the core features of foreman_ansible_director is that users don’t have to manage Ansible content manually anymore. In the UI, users can tell it to import a collection and it pretty much handles the rest automatically. The content is stored in Pulp because Pulp makes it quite easy to do this. Especially distribution of content to the Smart Proxies would not be trivial, as content is consumed through ansible-galaxy, which only understands so many source types.
I have not put much thought into this, but a “Pulp-less” solution would probably be something along the lines of “Clone collection on Foreman Server, make it available through some kind of webserver and have Smart Proxies consume it from there”. Not impossible, but compared to pulp_ansible, which just provides an Ansible Galaxy API compatible URL, quite a bit more difficult.

This actually works rather well. Repositories, Remotes and Distributions are named in such a way that they do not interfere with objects created by Katello. For cleanup purposes, these objects are “tagged” with labels, which the plugin also uses to perform a “Consistency Check”, which is its rendition of orphan cleanup.

Full disclosure: I do not have much experience with Insights, so I may have oversimplified a little here. If I was way off target, feel free to disregard that point.

Everything, including Execution Environments is scoped to an organization. The plugin uses two “types” of Execution Environment images (the difference is in how they are managed). The first kind is defined by users, giving them control over Ansible version and base image. Those Execution Environments are then used as bases for the second kind of image, which the plugin generates itself. The second image is needed to make the Ansible content required for the run available in the image. This isn’t a perfect solution, so this may change in the future.
All kinds of Execution Environments are built by the corresponding Smart Proxy plugin. Being able to push those images into Pulp would allow deduplication of Execution Environments between Smart Proxies.

That’s a good question. In the case of foreman_ansible_director, I consider Pulp to be an implementation detail, which also applies to Katello to some extent. The closest admins get to Pulp objects is through either the consistency check (read: orphan cleanup) or a rake task that nukes all Pulp content created by foreman_ansible_director.
I’m not opposed to this, but I also don’t quite see the use-case for giving admins control over, say, the concrete CollectionRemote used to procure some content.

Domains are a very good point. I too thought about those a lot. In the case of foreman_ansible_director, domains would definitely be good enough, if not better in some cases, as the plugin also makes use of the Role- and GitRemotes, which Katello (AFAIK) doesn’t.
However, I too worry about the amount of work that would be needed to make Katello play nice with domains. Even if Katello used the default domain, there would still be a need to change some things. Especially regarding container images, which IMO are the most interesting content type at the moment, Katello’s API would probably need to be changed to include the “default”, “katello” or some other domain’s name. That change would be user facing and I’m not sure how big the impact would be.

10

It was my understanding that Pulp 4 will enable domains by default. Even if it’s painful now, perhaps it’s worth accepting that pain now when it’s already on the roadmap.

1 Like
11

This makes sense. I would find it helpful to this discussion to see a medium-high level breakdown of what would be in the shared library and what then would live within Katello. The layering between the two (or three if you want to include how another plugin would then take advantage).

What does “tagged” translate to since you used quotes? Are labels an actual field in Pulp or is this a meta-tagging that’s been invented? Again, I am reminded of Pulp 2 and it’s notes field which worked rather well until it didn’t.

Given this is storing content, and that content is being used to execute remote actions, I can see admins asking these kinds of questions:

  • Why is X amount of storage being used? What is using that storage? How do I clean it up? Do I need to add more storage?
  • How do I see these execution environments? Do they have vulnerabilities inside them? How do I update them?
2 Likes
12

Labels are a native Pulp 3 feature. In a nutshell, you can add a list of arbitrary key value pairs stored on the pulp_labels field on most Pulp objects. Then you can filter those objects by those labels. To my knowledge Katello has not made any use of Pulp labels in the past.

3 Likes
13

I’m starting to think that domains might be the easier and more impactful beast to chase down to improve foreman_ansible_director’s use of Pulp. Domains are something that are on our more immediate backlog to enable, and I don’t believe their initial enablement has to have as much Katello impact as the “internal repos” idea. Domains may even be considered a prerequisite technology if we continue down the path of having these internal repos be outside of Katello’s grasp.

Katello should make better use of domains anyway, for example with organizations. Today, organizations share artifacts, which could be considered problematic for some multi-tenant Katello installations, I imagine.

As long as Katello has its default domain and remains within its confines, other domains could exist that Katello doesn’t know anything about. I imagine we should be able to have Katello’s Pulp calls work against a single “Katello domain” locally and via a similar “Katello domain” on smart proxies.

From Enable and create Domains - Pulp Project

Once domains are enabled all URLs in Pulp will require the domain name in the path after the API_ROOT for the Pulp API, e.g. /pulp/my_domain_name/api/v3/, or after the CONTENT_PATH_PREFIX for the Content App, e.g. /pulp/content/my_domain_name/. Objects present in Pulp before enabling domains will now be present under the default domain. To work in a domain you must specify the domain’s name in the URL.

Once they are generally enabled, plugins then could create their own domains during deployment or during runtime.

Domains aside, I’d like to put some thought into pulling core Pulp-management code out of Katello and into a separate library. It would be one step towards enabling other plugins to manage files in Pulp, and could potentially morph into a lightweight Pulp management tool with the admin access that @ehelms was discussing above.

2 Likes