Foreman 2.2 and katello 3.16 docker pull

I am trying to pull docker image from foreman but showing error , although all images are showing on foreman UI with tags

docker pull fooreman2.test.com:5000/test-airo-ko:v6

Error response from daemon: unknown: Not Found

Should I need to install some plugin ? I just need foreman to handle all my images and tags.

10.32.7.210 - - [19/Oct/2020:12:59:38 -0400] “GET /v2/ HTTP/1.1” 200 2 “-” “docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/3.10.0-1127.18.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(linux\))”

10.32.7.210 - - [19/Oct/2020:12:59:38 -0400] “GET /v2/abc-production-airo-airo-keycloak/manifests/latest HTTP/1.1” 404 53 “-” “docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/3.10.0-1127.18.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(linux\))”

I think katello not publishing v2 docker registry as I can’t access via browser also.

docker pull abc-production-airo-airo-keycloak

Using default tag: latest

Error response from daemon: pull access denied for abc-production-airo-airo-keycloak, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied

[root@V6TestDevops devops]#

==> /var/log/httpd/foreman-ssl_access_ssl.log <==

10.32.9.206 - - [19/Oct/2020:13:35:28 -0400] “GET /notification_recipients HTTP/1.1” 200 40 “https://host2.xyz.com/content_views/3/versions/4/docker?page=1&per_page=20” “Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36”

10.32.9.206 - - [19/Oct/2020:13:35:38 -0400] “GET /notification_recipients HTTP/1.1” 200 40 “https://host2.xyz.com/content_views/3/versions/4/docker?page=1&per_page=20” “Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36”

10.32.9.206 - - [19/Oct/2020:13:35:48 -0400] “GET /notification_recipients HTTP/1.1” 200 40 “https://fountain2.xyz.com/content_views/3/versions/4/docker?page=1&per_page=20” “Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36”

==> /var/log/httpd/error_log <==

[Mon Oct 19 17:35:56.990633 2020] [:error] [pid 22824] [INFO] crane.data: loading metadata from /var/lib/pulp/published/docker/v2/app

[Mon Oct 19 17:35:56.991311 2020] [:error] [pid 22824] [INFO] crane.data: finished loading metadata

Hi @masadrasheed,

Is your setup a fresh install since 3.15? If not, have you run the content migration steps?

Since Katello 3.15, fresh installs use Pulp 3 rather than Pulp 2 to manage container content. This means that docker/podman workflows are a bit different:

  • You no longer need to access the registry on port 5000. We use the default now, 443, so you can omit the port field altogether.

  • You must docker/podman login first. docker login <hostname> is all you need. You’ll be asked to enter your Foreman username and password.

its a new installation and thank you it’s really helpful only problem is now its only working if I add full url instead of just image name ?

[root@V6TestDevops devops]# docker login fountain2.abc.com

Authenticating with existing credentials…

WARNING! Your password will be stored unencrypted in /root/.docker/config.json.

Configure a credential helper to remove this warning. See

Login Succeeded

[root@V6TestDevops devops]#

First example , its working

######################
[root@V6TestDevops devops]# docker pull fountain2.abc.com/abc-production-airo-airo-keycloak:v8

v8: Pulling from abc-production-airo-airo-keycloak

Digest: sha256:698f2d0a39fb8eb95d69e4c9d731ffadbefa4e9d50bd8e3963c47e1a8787a3de

Status: Downloaded newer image for fountain2.abc.com/abc-production-airo-airo-keycloak:v8

fountain2.abc.com/abc-production-airo-airo-keycloak:v8

######################################

Its not working ::

[root@V6TestDevops devops]# docker pull abc-production-airo-airo-keycloak:v8

Error response from daemon: pull access denied for abc-production-airo-airo-keycloak, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied

[root@V6TestDevops devops]#

Hello,

@iballou

I think I upgrade foreman and it stop working

docker pull fountain2.abc.com/abc-production-airo-airo-keycloak:v8

Error response from daemon: error parsing HTTP 404 response body: invalid character ‘:’ after top-level value: “404: Not Found”

Nov 18 17:11:41 fountain2 pulpcore-content: 127.0.0.1 [18/Nov/2020:22:11:41 +0000] “GET /v2/abc-production-airo-airo-keycloak/manifests/v8 HTTP/1.1” 404 172 “-” “docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/3.10.0-1127.18.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 (linux))”

qq^C

Hi @masadrasheed,

Apologies for the delay in responding. It’s expected that you should put the hostname in your docker pull command. docker pull fountain2.abc.com/abc-production-airo-airo-keycloak:v8 was correct.

Can you see your docker image if you do docker search fountain2.abc.com/ ?

Can you show me your Pulp 3 versions? sudo pip3 list | grep pulp

You mentioned you upgraded foreman, how did you do so?

1 Like

Apologies for the delay in responding. It’s expected that you should put the hostname in your docker pull command. docker pull fountain2.abc.com/abc-production-airo-airo-keycloak:v8 was correct.

Yes it was correct and as you see in my previous post it was working.

[root@V6TestDevops devops]# docker search fountain2.abc.com/
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
abc-airo-2_0-airo-keycloak 0
abc-airo-2_0-airo-kong 0
abc-airo-2_0-airo-postgres 0
abc-airo-keycloak 0
abc-airo-kong 0
abc-airo-mega_v6 0
abc-airo-postgres 0
abc-library-airo-airo-keycloak 0
abc-library-airo-airo-kong 0
abc-library-airo-airo-postgres 0
abc-production-airo-airo-keycloak 0
abc-production-airo-airo-kong 0
abc-production-airo-airo-postgres 0

[root@fountain2 fountain]# sudo pip3 list | grep pulp

DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.

pulp-2to3-migration (0.5.0)

pulp-certguard (1.0.2)

pulp-container (2.0.1)

pulp-file (1.2.0)

pulp-rpm (3.6.2)

pulpcore (3.6.3)

You mentioned you upgraded foreman, how did you do so?

yum update
foreman-installer

its latest to 2.2.1

also if image is not there it give Error response from daemon: error parsing HTTP 404 response body: invalid character ‘:’ after top-level value: “404: Not Found” instead unknown not found.

[root@V6TestDevops devops]# docker pull fountain2.abc.com/abc-library-airo-airo-keycloak:89

Error response from daemon: unknown: 89 was not found!

[root@V6TestDevops devops]#

@masadrasheed,

I was able to reproduce your error parsing HTTP 404 response body: invalid character ':' after top-level value: "404: Not Found" issue. I’ll be looking into it.

1 Like

thank you appreciated.

I’ve created a bug for it: Bug #31362: podman pull 404 error only on Katello 3.17 - Katello - Foreman

what you think when they look into it ? I mean how much it will take to get it fixed.

secondly any recommendation or work around for me ?