Using Redhat registry in Foreman/Katello 1.19.1/3.8

FlatOut · December 13, 2018, 4:13pm

1. Background & problem

**Problem: Adding redhat registry, documentation missing.

Background: I want to use foreman + katello to create baselines for redhat openstack.

The documentation for using docker registry - see katello install chapter 5.8: (Foreman :: Plugin Manuals) for redhat docker registry using manifest leads to a broken link: (https://theforeman.org/plugins/katello/3.8/user_guide/red_hat_content/content.html).

I did check all versions available from 2.4 -> 3.10 but all have a broken link for this.

I have added a redhat manifest and I use that to download rhel7 + rhosp repos I need, but I cant figure out how to add the registry.

2. What I have tried so far

General information:
Using http proxy to get to internet. I believe I have this working as it should based on that I can download yum repos without issues and I have a wildcard for all redhat domains in the proxy. I installed katello with the necessary parameters for this to work.

This is what I have tried so far:

2.1. Adding the registry via Containers -> Registry -> Create Registry

Name: rhosp
Url: https://registry.access.redhat.com

Here I also tried to add the namespaces I need without any luck, but I didnt really figure out what this feature really does.

2.2. I added a new product and in there I manually created repos of type docker.

For registry URL and upstream repository I have tried a few different setups, these are a few of them:
1.
Registry URL: https://registry.access.redhat.com/
Upstream repo: rhosp13

Registry URL: https://registry.access.redhat.com/v2
Upstream repo: rhosp13

I had to disable SSL verification, otherwise I got an error, I suppose this is due to the proxy:

Dec 13 17:04:37 foreman pulp: requests.packages.urllib3.connectionpool:INFO: [37149198] Starting new HTTPS connection (1): registry.access.redhat.com
Dec 13 17:04:37 foreman pulp: nectar.downloaders.threaded:ERROR: [37149198] Skipping requests to registry.access.redhat.com due to repeated connection failures: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
Dec 13 17:04:37 foreman pulp: pulp.server.async.tasks:INFO: [37149198] Task failed : [37149198-f3a4-4ca0-844b-c42ef298e35a] : Could not find registry API at https://registry.access.redhat.com/v2
Dec 13 17:04:37 foreman pulp: celery.app.trace:INFO: [37149198] Task pulp.server.managers.repo.sync.sync[37149198-f3a4-4ca0-844b-c42ef298e35a] raised expected: PulpCodedException()
Dec 13 17:04:37 foreman pulp: celery.app.trace:INFO: [5dadbe94] Task pulp.server.async.tasks._release_resource[5dadbe94-48b8-4a71-bbbc-bd5517db920f] succeeded in 0.00131727400003s: None

Sync logs for 1

Dec 13 17:07:39 foreman pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Dec 13 17:07:39 foreman pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[f80bb610-dc95-4470-8df5-dc6768cb2f51]
Dec 13 17:07:39 foreman pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[580b125f-56e1-4edb-b3b1-b8bc1ece0c29]
Dec 13 17:07:39 foreman pulp: py.warnings:WARNING: [580b125f] (9247-17792) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Dec 13 17:07:39 foreman pulp: py.warnings:WARNING: [580b125f] (9247-17792)   "MongoClient opened before fork. Create MongoClient "
Dec 13 17:07:39 foreman pulp: py.warnings:WARNING: [580b125f] (9247-17792)
Dec 13 17:07:39 foreman pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[7645df45-c804-433b-9aa9-124bd76d25ee]
Dec 13 17:07:39 foreman pulp: celery.app.trace:INFO: [f80bb610] Task pulp.server.async.tasks._queue_reserved_task[f80bb610-dc95-4470-8df5-dc6768cb2f51] succeeded in 0.01387285901s: None
Dec 13 17:07:39 foreman pulp: requests.packages.urllib3.connectionpool:INFO: [580b125f] Starting new HTTPS connection (1): registry.access.redhat.com
Dec 13 17:07:39 foreman pulp: py.warnings:WARNING: [580b125f] (9247-17792) /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
Dec 13 17:07:39 foreman pulp: py.warnings:WARNING: [580b125f] (9247-17792)   InsecureRequestWarning)
Dec 13 17:07:39 foreman pulp: py.warnings:WARNING: [580b125f] (9247-17792)
Dec 13 17:07:40 foreman pulp: nectar.downloaders.threaded:INFO: [580b125f] Download succeeded: https://registry.access.redhat.com/v2/.
Dec 13 17:07:40 foreman pulp: nectar.downloaders.threaded:INFO: [580b125f] Download failed: Download of https://registry.access.redhat.com/v2/rhosp13/tags/list failed with code 404: Not Found
Dec 13 17:07:40 foreman pulp: pulp.server.async.tasks:INFO: [580b125f] Task failed : [580b125f-56e1-4edb-b3b1-b8bc1ece0c29] : Could not fetch repository rhosp13 from registry https://registry.access.redhat.com/ - Not Found
Dec 13 17:07:40 foreman pulp: celery.app.trace:INFO: [580b125f] Task pulp.server.managers.repo.sync.sync[580b125f-56e1-4edb-b3b1-b8bc1ece0c29] raised expected: PulpCodedException()
Dec 13 17:07:40 foreman pulp: celery.app.trace:INFO: [7645df45] Task pulp.server.async.tasks._release_resource[7645df45-c804-433b-9aa9-124bd76d25ee] succeeded in 0.00141810602508s: None
Dec 13 17:07:40 foreman pulp: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Dec 13 17:07:40 foreman pulp: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Dec 13 17:07:41 foreman pulp: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}

Sync logs for 2

Dec 13 17:09:42 foreman pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Dec 13 17:09:43 foreman pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[dee62ac2-2d17-49b4-abc5-aca65789917d]
Dec 13 17:09:43 foreman pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[480f748e-2ece-4ea4-aa1c-956fc76a5f2e]
Dec 13 17:09:43 foreman pulp: celery.app.trace:INFO: [dee62ac2] Task pulp.server.async.tasks._queue_reserved_task[dee62ac2-2d17-49b4-abc5-aca65789917d] succeeded in 0.0128797629732s: None
Dec 13 17:09:43 foreman pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[df5c447a-29f7-4622-8a90-840f7e4f79d1]
Dec 13 17:09:43 foreman pulp: py.warnings:WARNING: [480f748e] (9532-17792) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Dec 13 17:09:43 foreman pulp: py.warnings:WARNING: [480f748e] (9532-17792)   "MongoClient opened before fork. Create MongoClient "
Dec 13 17:09:43 foreman pulp: py.warnings:WARNING: [480f748e] (9532-17792)
Dec 13 17:09:43 foreman pulp: requests.packages.urllib3.connectionpool:INFO: [480f748e] Starting new HTTPS connection (1): registry.access.redhat.com
Dec 13 17:09:43 foreman pulp: py.warnings:WARNING: [480f748e] (9532-17792) /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
Dec 13 17:09:43 foreman pulp: py.warnings:WARNING: [480f748e] (9532-17792)   InsecureRequestWarning)
Dec 13 17:09:43 foreman pulp: py.warnings:WARNING: [480f748e] (9532-17792)
Dec 13 17:09:44 foreman pulp: nectar.downloaders.threaded:INFO: [480f748e] Download succeeded: https://registry.access.redhat.com/v2/.
Dec 13 17:09:44 foreman pulp: nectar.downloaders.threaded:INFO: [480f748e] Download failed: Download of https://registry.access.redhat.com/v2/rhosp13/tags/list failed with code 404: Not Found
Dec 13 17:09:44 foreman pulp: pulp.server.async.tasks:INFO: [480f748e] Task failed : [480f748e-2ece-4ea4-aa1c-956fc76a5f2e] : Could not fetch repository rhosp13 from registry https://registry.access.redhat.com/v2/ - Not Found
Dec 13 17:09:44 foreman pulp: celery.app.trace:INFO: [480f748e] Task pulp.server.managers.repo.sync.sync[480f748e-2ece-4ea4-aa1c-956fc76a5f2e] raised expected: PulpCodedException()
Dec 13 17:09:44 foreman pulp: celery.app.trace:INFO: [df5c447a] Task pulp.server.async.tasks._release_resource[df5c447a-29f7-4622-8a90-840f7e4f79d1] succeeded in 0.00135567400139s: None
Dec 13 17:09:45 foreman pulp: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Dec 13 17:09:45 foreman pulp: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Dec 13 17:09:46 foreman pulp: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}

I thing 1 is the closest to correct but that I am lacking something or this is completely the wrong track based on that I should use the manifest… Any help is highly appreciated.

3. Versions etc

**Expected outcome: Get it working and hopefully get the documentation corrected

**Foreman and Proxy versions: Foreman 1.19.1, Katello 3.8

akofink · December 13, 2018, 6:23pm

Thanks for pointing out this broken link! It should link to https://theforeman.org/plugins/katello/3.8/user_guide/red_hat_content/index.html. I’ve submitted a PR to theforeman.org to fix this (feel free to review :).

Could you explain a bit more what you mean by this? I assume you want to sync RHEL container images to Katello from https://access.redhat.com/containers/, is that right?

The 2.1 method is using foreman_docker, a foreman plugin. It’s documentation is under plugins.

FlatOut · December 17, 2018, 12:49pm

Hi,

Thanks for your reply and posting the PR. What I’m trying to achieve is to be able to download RPMs and docker containers for Redhat Openstack 13 nightly and create a baseline of which RPMs and containers are used that particular night.

This will then be used in a CI loop to verify that the new RPMs and containers dont break anything for our application. Exactly how I can handle the later part of creating a baseline I havnt figured out yet, that is the next step in this, but please tell me if I’m heading full speed into a wall

Currently I have the YUM repos added, but I cant find a way to add the container registries to download the images.

akofink · December 17, 2018, 2:27pm

Hmm… I just looked back at your original post and tested the registry upstream repository name you have (rhosp13). I don’t think such an image exists. Though, I see images of the format rhosp13/something like rhosp13/openstack-redis-base. If I try to pull just rhosp13, it’s not there:

$ docker pull registry.access.redhat.com/rhosp13
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhosp13 ... 
unknown: Not Found

But openstack-redis-base, for example, works:

$ docker pull registry.access.redhat.com/rhosp13/openstack-redis-base
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhosp13/openstack-redis-base ... 
sha256:94ccbf7f20f3820861b05daf40ec298d321d3544a8ed18f2d13683a7922a36ad: Pulling from registry.access.redhat.com/rhosp13/openstack-redis-base
9a1bea865f79: Pull complete 
602125c154e3: Pull complete 
67a69bb7b406: Pull complete 
4d167b5c582b: Pull complete 
Digest: sha256:94ccbf7f20f3820861b05daf40ec298d321d3544a8ed18f2d13683a7922a36ad
Status: Downloaded newer image for registry.access.redhat.com/rhosp13/openstack-redis-base:latest

FlatOut · December 18, 2018, 10:21am

Yeah, but I want to download all images in that path of the registry and preferably not a single one each time. So I basically want to clone the repo down to my host, all images in all versions.

akofink · December 18, 2018, 2:17pm

Oh, I see. Hmm… I don’t think Katello supports that. A single repository represents a single container image (and all it’s versions and tags). You’d only need to set up the repositories once, and you could set it up in a script using hammer with sync plans to keep them up to date. I think that might be the best solution for what you want.

FlatOut · December 18, 2018, 2:35pm

I think you are right, I cant find anything pointing towards being able to do what I want

Thanks for all your answers!