Using katello for many repos in complex environment

[areyus]

Hi everyone,

first off, a lot of these questions and the related topics themself are what (to my experience) are the things that trubble nearly every new user the most. After several years of using Katello, I would do a lot of things differently myself if I was starting from scratch now. The genral problem is that everything content related is really complex and there is no “best way to do it”. The correct answer to most of your question would be “it depends”, since it will be a little different depending on what you want to achieve. There are some general guidelines though what I would consider at least “good practice”, but keep in mind that everything I write here is little tainted since I’m mainly looking at these things through the goggles of my company setup.

@ColinD

First, products:

• Should a product be something like “ Centos ”, or “ Centos 7 ”, or “ Centos 7 x86_64 ”? It seems like “ Centos 6 ” and “ Centos 7 ” are different products. If at some point I’m requested to mirror the 32bit repos for one of those, is that a different product, “ Centos 6 i386 ”?

The Redhat way of doing this is seperating this on a license/subscription basis, so if you need a specialized subscription for it, it will be it’s own product. This works somewhat well for RedHat stuff, but since you technically do not need any licenses for anything else with Katello, this is only transferable to some extent for custom products.
My general advise is: Look at these like they are called, products. If you think something is a different product, create a new one. If it is part of some product you already have, put it there. So I would put everything CentOS in a product called “CentOS” and e.g. everything Foreman/Katello related in a different product (called either “Foreman” or “Katello”) and EPEL into a product called EPEL.
What you always have to keep in mind designing products: If you want to use sync plans (so, scheduled automatic repo syncs), these can only be assigned to products, not individual repos. So if you want to sync some repos using different schedules, these can never go into the same product.

@ColinD

Repos inside a product:

• If the repo is “ Centos 7 x86_64 ”, then it’s clear to add the os/updates/extras to it, but if it’s the more generic such as “ Centos 7 ” or “ Centos ”, should I add multiple arch and versions (6,7) into a product?

Basically, I would recommend putting these in the same product if you do not have strong reasons to do otherwise.

@ColinD

Content Views:

• The idea of creating a “locked version” of a product and promoting it is clear. However, what would be an example of a name for a content view? Same as product, should it be “ Centos ”, or “ Centos 7 ”, or “ Centos 7 x86_64 ”.

Here is where the “it depends” part really starts. Generally speaking, the idea of one CV per product is a good guideline. What you should never do is seperate things into different CVs when updating one will probably breake the other if that’s not updated too. If you have the need to update different major releases in different schedules, promote those to different Lifecycleenvironments, or something like that, it might be a good idea to seperate those. If you want all versions updated simultaniously, I would just go with putting all versions in one CV.

@ColinD

• I’ve seen some people make a content view called “ Centos ”, did they put both Centos 6 and 7 repos inside one content view?

They probably did.

@ColinD

• If that’s not correct, then I’m assuming I’ll basically have a 1:1 mapping of content views to products. I’m not sure what hosts will overlap for different repos (db hosts, vmware hosts, etc all have different repos on them). Then we have multiple internal repos for our app, I guess each of these should be it’s own content view?

Some of this is already answered in my previous answers. Depending on your app architecture, like with products I would recommend creating one CV per app and not per app-component or repo. Building on that, I would go on and bundle as many of those CVs as possible (and reasonable) into one composit content view and let activation keys handle the rest. The only scenario I could think of where you want to split into different CCVs without a technical need to do so is when you have to make sure certain hosts are unable to see certain repos no matter what.
We currently have one CCV set up per operatingsystem (RHEL7, OL7, SLES12) and additional CCVs for things that need to be updated seperately (like our openshift infrastructure and SAP systems) but they all use the same base CVs where possible. This is a setup I am quite happy with and reduces CV maintenence quite a bit.

@ColinD

Yum.repos.d:

• Right now we are using repos the traditional way, using yum.repos.d files. Is this even possible for us anymore? I manage all those files using puppet to add the correct repos to the correct hosts, and I can have a single file for all hosts using the variables $releasever and $basearch , for example:
  – baseurl=http://yum.example.com/repos/puppet5/centos/$releasever/$basearch/
• Is it still possible to access the yum repos using that same system, or is Host subscription required?

Basically, this is still possible. Katello repo URLs are somewhat more complicated than this and you will lose some of the benefits of using Katello this way if you do not invest some time in crafting a decent erb template for that. If you want to do http, you have to explicitly enable publishing via http for each repo you create. For https, authentication via clientcert is required and to my knowledge, this requires client registration.

@ColinD

• When I subscribe a host, no files are added to yum.repos.d. I guess the only way to see which repos a host has is to check foreman?

There is one file where subscription-manager places all repos known to it. It is /etc/yum.repos.d/redhat.repo (at least on EL based distros). You can still alway use something like “yum repolist” or “subscription-manager repos --list”.

@ColinD

Content Hosts:

• In an attempt to understand the subscription model, I subscribed a host as a “content host”. There was a lot of manual clicking for assigning a host to an environment and which content view the host should use, etc, which wouldn’t be ideal for setting up thousands of hosts. I’d need to figure out a way to totally automate this process of subscribing hosts to the correct repos based on their host type and environment during the imaging process. Is this possible?

This is what activation keys are for. I would suggest setting up one activation per Lifecycle Environment + (composit) contentview combination you want to use that enables all the needed subsctiptions and repos for all hosts of that contentview. Then I would set up additional keys for each additional set of products/repos you want to always enable at once. These should not have CV or LCE set since using more than one activation key with these settings will most likely result in a lot of trubble. These activation keys can then be assigned to hosts via hostgroups. The automatic assigning of activation keys can be done differently via parameters in other places as well, but using hostgroups is usually the most convenient way.
For newly provisiond hosts, this can then be setup easily through provisioning templates (take a look at the community templates for some examples). For importing existing hosts with activation keys you will most likely have to come up with some automation around that yourself, wether it is some sort of SSH bulk command or something completely different. I would recommend not trying to use Puppet for this, since to my experience using subscription-manager through Puppet is only useable with a bulk of exec statements and is basically always more pain than its worth.
A word of warning though, since you asked about the ability to use Puppet for managing repo files: If you go this way with subscription-manager handling the repo setup, I would strongly recommend agains using Puppet for any repository already managed by subscription-manager. That will most likely result in a huge mess since the Puppet yumrepo provider does not understand the concept of more than one repo per file and you will probably end up with a lot of duplicate repo declarations in your yum config.

@ColinD

• Also, it looks like I can only assign a single Content View (epel for example), to a host. I can’t figure out how to assign many content views for all my repos to a host. Maybe I’m supposed to use Composite Content View, but this would become very complex and is much less flexible than adding multiple Content Views to hosts depending on their needs. Otherwise, I would have many different composite content views for every variation of repos.

I partly already answered that beforehand, but summed up shortly: Yes, you can only assign one Content View per host and you are supposed to use composit content views for that usecase. I suggest creating the minimum required amount of CCVs to make this somewhat sane and let activation keys handle the rest of the work.

@ColinD

Provisioning using kickstart:

• Normally when I image a host using PXE/kickstart, I point to a local centos repo URL in the .ks file. Is this still possible or am I required to use the Foreman provisioning system if I’m using Katello repos?

This is still possible but you will need to write your own kickstart and provisioning template for that (or customize the community or default ones). The Foreman provisioning method does nothing differently on the technical side, it just tries to hide some of the work for for you with builtin features usable by the provisioning and PXE templates.

I hope this can help you in grasping these topics a little more.
A lot of these things come down to a point of personal preference and depend on your situation quite a lot. In specific scenarios it is for sure reasonable to some things differently than I recommended, but this is what I experienced to work quite well and is a good tradeoff between complexity and added benefits for our setup.