Hello Foreman community,
I would like to get some feedback around a planned Katello feature called “Alternate Content Sources”. This is a feature that was available with through behind-the-scenes Pulp 2 configuration changes, but never directly in Katello itself. While it will once again be configurable outside of Katello through a later Pulp 3 release, we would like to make it a fully-fledged feature in Katello for a friendlier user experience.
What are alternate content sources (ACSs)?
An alternate content source is a filesystem or network path where Pulp will look to first when downloading content during a sync or on-demand content fetch. A repository with a slower connection might need to be the authoritative figure who determines the repository structure, but a faster ACS could be defined so that the actual content downloads (which is potentially much larger in size that the repodata) over a faster connection. With Katello in mind, this could be helpful for speeding up smart proxy synchronizations. Perhaps your smart proxy on the other side of the world has a slow connection to your Katello server. Instead of syncing CentOS 8 AppStream from this slow connection, an ACS could be set up to download it from your smart proxy’s local CentOS mirror.
Pulp 3 will be handling all of the backend ACS logic. Katello will be simply created and orchestrating them. In Katello, an ACS needs the associated smart proxies, optional content credentials, a base url (e.g. http://192.168.1.11/pub/content/), and subpaths (e.g. [beta/rhel/server/7/x86_64/satellite/6/os/, eus/rhel/server/7/7.3/x86_64/os/, dist/rhel/client/6/6.8/i386/kickstart/Client/, …]).
In Pulp, to create an ACS, Katello will first create a remote with the base path, auth credentials, and any necessary headers. Then, Katello will create one or more ACSs using that remote as an argument. The backend ACS itself will hold the subpaths. Then, to actually use the ACS, Katello will “refresh” it so that Pulp can index in all of the content metadata.
For now, every ACS will be global per smart proxy. That means, in the background, Pulp will consider every ACS for every repo sync. There is no need to associate repositories to ACSs.
How will ACSs be used in Katello?
Firstly, assume ACSs will be yum-only for now.
Here is the proposed workflow from the UI’s standpoint:
- The user creates or enables some repositories.
- The user optionally creates content credentials for protected alternate content sources.
- The user goes to the “Alternate Content Sources” page and clicks a button to create a new one.
- The user chooses the type of ACS: Custom, Red Hat CDN, or Red Hat Update Infrastructure (RHUI).
→ For custom ACSs, the user enters the ACS’s smart proxies, content credentials or basic auth info, base URL, and a list of subpaths. Katello may also suggest base paths based on repositories that are already synced, but this needs to be investigated more.
→ For Red Hat CDN ACSs, the user will likely only need to select the product and repositories that should be included in the ACS. Everything else can come from the Katello database as long as there is a valid subscription manifest. One product per ACS seems like the best balance between usability and fit with Pulp’s backend objects.
→ RHUI ACSs are still being planned out, but we hope to include a shortcut for creating them, much like the CDN ACSs.
- The user submits the ACS form. Katello tells Pulp to refresh the ACS in the background.
→ Refreshing for all ACSs will also happen on a cron schedule to ensure the metadata is up to date. For now, Katello will likely create a cron entry and run an ACS refresh rake task.
→ Existing cron entry: foreman-packaging/katello.cron at rpm/develop · theforeman/foreman-packaging · GitHub
- The user syncs a repository and enjoys the benefits of the ACS in the background.
The user will be able to update ACSs by selecting one from the list of ACSs and going to its information page, much like other Foreman or Katello constructs.
We are also going to have Hammer, API, and Foreman Ansible Module support for ACSs.
Planned DB tables:
- belongs_to :gpg_key, :ssl_ca_cert, :ssl_client_cert, :ssl_client_key (optional)
- belongs_to :product (optional, for CDN ACS)
- has_many :contents (optional, for CDN ACS)
- Must belong to the same product
- subpaths (array of strings)
- type (custom, cdn, rhui)
Backend DB steps after the create form is submitted for a custom ACS:
- Create ACS in Katello DB
- Create ACSs in Pulp for each Smart Proxy associated to the Katello ACS:
- Create Pulp remote using Katello ACS base url & auth credentials
- Create Pulp ACS using the created remote and the Katello ACS’s subpaths
- Create Smart Proxy ACSs with a remote href, ACS href, Smart Proxy ID, and ACS ID from the ACS creation tasks
- Necessary since smart proxies can share Katello ACSs but will have separate ACSs and ACS remotes in Pulp
This feature is still in the very early planning phases, so please let us know your concerns, ideas, and questions. Thanks for reading!