Foreman 2.2 and katello 3.16 docker pull

masadrasheed · October 19, 2020, 1:22am

I am trying to pull docker image from foreman but showing error , although all images are showing on foreman UI with tags

docker pull fooreman2.test.com:5000/test-airo-ko:v6

Error response from daemon: unknown: Not Found

Should I need to install some plugin ? I just need foreman to handle all my images and tags.

masadrasheed · October 19, 2020, 5:08pm

10.32.7.210 - - [19/Oct/2020:12:59:38 -0400] “GET /v2/ HTTP/1.1” 200 2 “-” “docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/3.10.0-1127.18.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(linux\))”

10.32.7.210 - - [19/Oct/2020:12:59:38 -0400] “GET /v2/abc-production-airo-airo-keycloak/manifests/latest HTTP/1.1” 404 53 “-” “docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/3.10.0-1127.18.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(linux\))”

I think katello not publishing v2 docker registry as I can’t access via browser also.

masadrasheed · October 19, 2020, 5:35pm

docker pull abc-production-airo-airo-keycloak

Using default tag: latest

Error response from daemon: pull access denied for abc-production-airo-airo-keycloak, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied

[root@V6TestDevops devops]#

masadrasheed · October 19, 2020, 5:38pm

==> /var/log/httpd/foreman-ssl_access_ssl.log <==

10.32.9.206 - - [19/Oct/2020:13:35:28 -0400] “GET /notification_recipients HTTP/1.1” 200 40 “https://host2.xyz.com/content_views/3/versions/4/docker?page=1&per_page=20” “Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36”

10.32.9.206 - - [19/Oct/2020:13:35:38 -0400] “GET /notification_recipients HTTP/1.1” 200 40 “https://host2.xyz.com/content_views/3/versions/4/docker?page=1&per_page=20” “Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36”

10.32.9.206 - - [19/Oct/2020:13:35:48 -0400] “GET /notification_recipients HTTP/1.1” 200 40 “https://fountain2.xyz.com/content_views/3/versions/4/docker?page=1&per_page=20” “Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36”

==> /var/log/httpd/error_log <==

[Mon Oct 19 17:35:56.990633 2020] [:error] [pid 22824] [INFO] crane.data: loading metadata from /var/lib/pulp/published/docker/v2/app

[Mon Oct 19 17:35:56.991311 2020] [:error] [pid 22824] [INFO] crane.data: finished loading metadata

masadrasheed · October 20, 2020, 1:18am

iballou · October 22, 2020, 2:47pm

Hi @masadrasheed,

Is your setup a fresh install since 3.15? If not, have you run the content migration steps?

Since Katello 3.15, fresh installs use Pulp 3 rather than Pulp 2 to manage container content. This means that docker/podman workflows are a bit different:

You no longer need to access the registry on port 5000. We use the default now, 443, so you can omit the port field altogether.
You must docker/podman login first. docker login <hostname> is all you need. You’ll be asked to enter your Foreman username and password.

masadrasheed · October 22, 2020, 3:53pm

its a new installation and thank you it’s really helpful only problem is now its only working if I add full url instead of just image name ?

[root@V6TestDevops devops]# docker login fountain2.abc.com

Authenticating with existing credentials…

WARNING! Your password will be stored unencrypted in /root/.docker/config.json.

Configure a credential helper to remove this warning. See

Login Succeeded

[root@V6TestDevops devops]#

First example , its working

######################
[root@V6TestDevops devops]# docker pull fountain2.abc.com/abc-production-airo-airo-keycloak:v8

v8: Pulling from abc-production-airo-airo-keycloak

Digest: sha256:698f2d0a39fb8eb95d69e4c9d731ffadbefa4e9d50bd8e3963c47e1a8787a3de

Status: Downloaded newer image for fountain2.abc.com/abc-production-airo-airo-keycloak:v8

fountain2.abc.com/abc-production-airo-airo-keycloak:v8

######################################

Its not working ::

[root@V6TestDevops devops]# docker pull abc-production-airo-airo-keycloak:v8

Error response from daemon: pull access denied for abc-production-airo-airo-keycloak, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied

[root@V6TestDevops devops]#

masadrasheed · November 18, 2020, 10:02pm

Hello,

@iballou

I think I upgrade foreman and it stop working

docker pull fountain2.abc.com/abc-production-airo-airo-keycloak:v8

Error response from daemon: error parsing HTTP 404 response body: invalid character ‘:’ after top-level value: “404: Not Found”

masadrasheed · November 18, 2020, 10:15pm

Nov 18 17:11:41 fountain2 pulpcore-content: 127.0.0.1 [18/Nov/2020:22:11:41 +0000] “GET /v2/abc-production-airo-airo-keycloak/manifests/v8 HTTP/1.1” 404 172 “-” “docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/3.10.0-1127.18.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 (linux))”

qq^C

iballou · November 19, 2020, 3:45pm

Hi @masadrasheed,

Apologies for the delay in responding. It’s expected that you should put the hostname in your docker pull command. docker pull fountain2.abc.com/abc-production-airo-airo-keycloak:v8 was correct.

Can you see your docker image if you do docker search fountain2.abc.com/ ?

Can you show me your Pulp 3 versions? sudo pip3 list | grep pulp

You mentioned you upgraded foreman, how did you do so?

masadrasheed · November 19, 2020, 9:48pm

Apologies for the delay in responding. It’s expected that you should put the hostname in your docker pull command. docker pull fountain2.abc.com/abc-production-airo-airo-keycloak:v8 was correct.

Yes it was correct and as you see in my previous post it was working.

[root@V6TestDevops devops]# docker search fountain2.abc.com/
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
abc-airo-2_0-airo-keycloak 0
abc-airo-2_0-airo-kong 0
abc-airo-2_0-airo-postgres 0
abc-airo-keycloak 0
abc-airo-kong 0
abc-airo-mega_v6 0
abc-airo-postgres 0
abc-library-airo-airo-keycloak 0
abc-library-airo-airo-kong 0
abc-library-airo-airo-postgres 0
abc-production-airo-airo-keycloak 0
abc-production-airo-airo-kong 0
abc-production-airo-airo-postgres 0

[root@fountain2 fountain]# sudo pip3 list | grep pulp

DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.

pulp-2to3-migration (0.5.0)

pulp-certguard (1.0.2)

pulp-container (2.0.1)

pulp-file (1.2.0)

pulp-rpm (3.6.2)

pulpcore (3.6.3)

You mentioned you upgraded foreman, how did you do so?

yum update
foreman-installer

its latest to 2.2.1

masadrasheed · November 19, 2020, 10:09pm

also if image is not there it give Error response from daemon: error parsing HTTP 404 response body: invalid character ‘:’ after top-level value: “404: Not Found” instead unknown not found.

[root@V6TestDevops devops]# docker pull fountain2.abc.com/abc-library-airo-airo-keycloak:89

Error response from daemon: unknown: 89 was not found!

[root@V6TestDevops devops]#

iballou · November 20, 2020, 6:53pm

@masadrasheed,

I was able to reproduce your error parsing HTTP 404 response body: invalid character ':' after top-level value: "404: Not Found" issue. I’ll be looking into it.

masadrasheed · November 20, 2020, 8:33pm

thank you appreciated.

iballou · November 20, 2020, 9:44pm

I’ve created a bug for it: Bug #31362: podman pull 404 error only on Katello 3.17 - Katello - Foreman

masadrasheed · November 20, 2020, 10:22pm

what you think when they look into it ? I mean how much it will take to get it fixed.

secondly any recommendation or work around for me ?

masadrasheed · November 25, 2020, 2:52pm

@iballou

I think we need to enable relativeurls: true instead of false to make it work.
form where I can edit that

environment:

REGISTRY_HTTP_RELATIVEURLS=true

iballou · November 30, 2020, 3:55pm

Good to know about REGISTRY_HTTP_RELATIVEURLS, I hadn’t seen that before now.

I can’t say for sure yet what’s going on with the docker pull issue. I wasn’t able to reproduce the problem on the latest Katello 3.18 RC. This may be an issue with Pulp 3 rather than Katello.

If you are really stuck and need a workaround ASAP, I’d suggest that you revert back to Pulp 2 for Docker content. It requires some resetting of your content, but here it is:

Delete all docker repos and delete all content view versions that have docker content in them.
- Alternatively, you could also do a full foreman-installer reset.
Run foreman-installer --katello-use-pulp-2-for-docker 1

I was able to fix the issue as well by upgrading to the latest Katello 3.18 RC, but that comes with its own risks since it’s a release candidate.

masadrasheed · December 1, 2020, 7:08pm

what if I delete docker repos and remove docker repo from content view instead of delete the whole view ?

Also how to revert below change

foreman-installer --katello-use-pulp-2-for-docker 1

Last thing , is foreman 2.2.1 and katello 3.18 RC supported ? and what’s the procedure ? how I can upgrade ?

iballou · December 2, 2020, 6:26pm

If you want to avoid deleting entire content views, the minimum would be to delete all docker repos and delete all content view versions that contain docker content.

If you want to revert it, you’d run

foreman-installer --katello-use-pulp-2-for-docker 0

Are you reverting because you decided you do want to be on Pulp 3 after all?

To upgrade, you follow these instructions: Foreman :: Plugin Manuals
If you upgrade using the instructions there, your Katello server will be on Foreman 2.3 and Katello 3.18. Both will be updated.

Note that upgrading to an RC is always a risk because it is considered “unstable”. The least “risky” route would be to revert to Pulp 2 instead. I would recommend you try using Docker on Pulp 2 before upgrading Katello to 3.18. Upgrading is easy, but downgrading is not in case you hit a new bug.