[Katello] 3.3 Activation keys not working as expected

In a previous thread I brought up naming/organizing. I thought I had all
this worked out but now my design and activation keys aren't working as
expected.

  1. I don't see that there is any scenario where using multiple keys works
  2. Only one content-view is ever selected subscription-manager
    –activationkey=centos7,product-foo,env-dev doesn't do anything useful
    2. Product content overrides don't seem to apply, if I make a
    composite containing OS, and other content views all of the repos seem to
    be selected all of the time regardless of "product content" yes/no settings.
  3. If multiple keys don't work instead of having a few keys I will
    instead have to have many combinations of keys
    1. Instead of the number of potential activation keys being:#env +
      #os + #products + #facetX + #facetY
      It is:
      #env * #os * #products * #facetX * #facetY
    2. The effect is similar for number of content-views

Consider:

Products:
CentOS
Repo: CentOS 6 OS
Repo: CentOS 6 Updates
Repo: CentOS 6 Extras
Repo: CentOS 6 Plus
Repo: CentOS 7 OS
Repo: CentOS 7 Updates
Repo: CentOS 7 Extras
Repo: CentOS 7 Plus

EPEL
Repo: EPEL 6
Repo: EPEL 7

Fedora COPR
Repo: Subscription Manager (dgoodwin) el6

Katello
Client el6
Client el7
Pulp el7
Candlepin el7

Content Views:
CentOS 6
Repo: CentOS 6 OS
Repo: CentOS 6 Updates
CentOS 7
Repo: CentOS 7 OS
Repo: CentOS 7 Updates
EPEL 6
Repo: EPEL 6
EPEL 7
Repo: EPEL 7
Katello el6
Repo Katello el6 Client
Katello el7
Repo Katello el7 Client
Subscription Manager el6
Repo: Subscription Manager (dgoodwin) el6

CentOS 6 Base System (Composite)
Content View: CentOS 6
Content View: EPEL 6
Content View: Katello el6
Content View: Subscription Manager el6

Activation Keys:
ak-centos-6-base
Auto Attach: False
Lifecycle Environment: None
Content View: CentOS 6 Base System (Composite)
Subscriptions: All relevant selected
Product Content:
Product: CentOS 6
Repo: CentOS 6 OS: Override Yes
Repo: CentOS 6 Updates: Override Yes
Repo: CentOS 6 Extras: Default No
Repo: CentOS 6 Plus: Default No
Repo: CentOS 7 OS: Default No
Repo: CentOS 7 Updates: Default No
Repo: CentOS 7 Extras: Default No
Repo: CentOS 7 Plus: Default No
Product: EPEL
Repo: EPEL 6: Override Yes
Repo: EPEL 7: Default No
Product: Fedora COPR
Repo: Subscription Manager (dgoodwin) el6: Override Yes
Product: Katello
Client el6: Override Yes
Client el7: Default No
Pulp el7: Default No
Candlepin el7: Default No

ak-centos-6-base-dev
Auto Attach: False
Lifecycle Environment: Dev
Content View: CentOS 6 Base System (Composite)
Subscriptions: All relevant selected
Product Content:
Product: CentOS 6
Repo: CentOS 6 OS: Override Yes
Repo: CentOS 6 Updates: Override Yes
Repo: CentOS 6 Extras: Default No
Repo: CentOS 6 Plus: Default No
Repo: CentOS 7 OS: Default No
Repo: CentOS 7 Updates: Default No
Repo: CentOS 7 Extras: Default No
Repo: CentOS 7 Plus: Default No
Product: EPEL
Repo: EPEL 6: Override Yes
Repo: EPEL 7: Default No
Product: Fedora COPR
Repo: Subscription Manager (dgoodwin) el6: Override Yes
Product: Katello
Client el6: Override Yes
Client el7: Default No
Pulp el7: Default No
Candlepin el7: Default No

ak-env-dev:
Auto Attach: False
Lifecycle Environment: Dev
Content View: None

If I register a host with something like:

subscription-manager register --org "My Org" --activationkey=ak-env-dev,ak-
centos-6-base

I would expect to get the following repos:
Repo: Dev/CentOS 6 OS
Repo: Dev/CentOS 6 Updates
Repo: Dev/EPEL 6
Repo: Dev/Subscription Manager (dgoodwin) el6
Repo: Dev/Katello Client el6

Instead I get Library/* and CentOS 6 Extras and Plus are also included
despite being excluded in the product content.

Even using the activation key w/ lifecycle environment isn't correct. I do
get the correct lifecycle environment, but I still get the de-selected
product content repos.

There was so much to configure! Producs, Repos, (Under the hood those
become subscriptions and pools and lions and tigers and bears), content
views and activation keys and I'm still not where I want to be.

  1. What am I missing?
  2. Is there a concise explanation of the relationship between
    Product/Repo/CV/Subscription/AK/Product Content that I've not found?
  3. Also is there a way to simulate activation keys instead of having to
    try to unsubscribe/subscribe hosts?
  4. Or more easily debug them, I *think *it's candlepin where that
    debugging might be but I don't know if it's there or how to turn it on.
  5. How do you set it up fellow #foreman-users

Thank you,
-Alan