I am trying to pull docker image from foreman but showing error , although all images are showing on foreman UI with tags
docker pull fooreman2.test.com:5000/test-airo-ko:v6
Error response from daemon: unknown: Not Found
Should I need to install some plugin ? I just need foreman to handle all my images and tags.
10.32.7.210 - - [19/Oct/2020:12:59:38 -0400] “GET /v2/ HTTP/1.1” 200 2 “-” “docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/3.10.0-1127.18.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(linux\))”
10.32.7.210 - - [19/Oct/2020:12:59:38 -0400] “GET /v2/abc-production-airo-airo-keycloak/manifests/latest HTTP/1.1” 404 53 “-” “docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/3.10.0-1127.18.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(linux\))”
I think katello not publishing v2 docker registry as I can’t access via browser also.
docker pull abc-production-airo-airo-keycloak
Using default tag: latest
Error response from daemon: pull access denied for abc-production-airo-airo-keycloak, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied
[root@V6TestDevops devops]#
==> /var/log/httpd/foreman-ssl_access_ssl.log <==
10.32.9.206 - - [19/Oct/2020:13:35:28 -0400] “GET /notification_recipients HTTP/1.1” 200 40 “https://host2.xyz.com/content_views/3/versions/4/docker?page=1&per_page=20” “Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36”
10.32.9.206 - - [19/Oct/2020:13:35:38 -0400] “GET /notification_recipients HTTP/1.1” 200 40 “https://host2.xyz.com/content_views/3/versions/4/docker?page=1&per_page=20” “Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36”
10.32.9.206 - - [19/Oct/2020:13:35:48 -0400] “GET /notification_recipients HTTP/1.1” 200 40 “https://fountain2.xyz.com/content_views/3/versions/4/docker?page=1&per_page=20” “Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36”
==> /var/log/httpd/error_log <==
[Mon Oct 19 17:35:56.990633 2020] [:error] [pid 22824] [INFO] crane.data: loading metadata from /var/lib/pulp/published/docker/v2/app
[Mon Oct 19 17:35:56.991311 2020] [:error] [pid 22824] [INFO] crane.data: finished loading metadata
Hi @masadrasheed,
Is your setup a fresh install since 3.15? If not, have you run the content migration steps?
Since Katello 3.15, fresh installs use Pulp 3 rather than Pulp 2 to manage container content. This means that docker/podman workflows are a bit different:
You no longer need to access the registry on port 5000. We use the default now, 443, so you can omit the port field altogether.
You must docker/podman login first. docker login <hostname> is all you need. You’ll be asked to enter your Foreman username and password.
its a new installation and thank you it’s really helpful only problem is now its only working if I add full url instead of just image name ?
[root@V6TestDevops devops]# docker login fountain2.abc.com
Authenticating with existing credentials…
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
Login Succeeded
[root@V6TestDevops devops]#
First example , its working
######################
[root@V6TestDevops devops]# docker pull fountain2.abc.com/abc-production-airo-airo-keycloak:v8
v8: Pulling from abc-production-airo-airo-keycloak
Digest: sha256:698f2d0a39fb8eb95d69e4c9d731ffadbefa4e9d50bd8e3963c47e1a8787a3de
Status: Downloaded newer image for fountain2.abc.com/abc-production-airo-airo-keycloak:v8
fountain2.abc.com/abc-production-airo-airo-keycloak:v8
######################################
Its not working ::
[root@V6TestDevops devops]# docker pull abc-production-airo-airo-keycloak:v8
Error response from daemon: pull access denied for abc-production-airo-airo-keycloak, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied
[root@V6TestDevops devops]#
Hello,
I think I upgrade foreman and it stop working
docker pull fountain2.abc.com/abc-production-airo-airo-keycloak:v8
Error response from daemon: error parsing HTTP 404 response body: invalid character ‘:’ after top-level value: “404: Not Found”
Nov 18 17:11:41 fountain2 pulpcore-content: 127.0.0.1 [18/Nov/2020:22:11:41 +0000] “GET /v2/abc-production-airo-airo-keycloak/manifests/v8 HTTP/1.1” 404 172 “-” “docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/3.10.0-1127.18.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 (linux))”
qq^C
Hi @masadrasheed,
Apologies for the delay in responding. It’s expected that you should put the hostname in your docker pull command. docker pull fountain2.abc.com/abc-production-airo-airo-keycloak:v8 was correct.
Can you see your docker image if you do docker search fountain2.abc.com/ ?
Can you show me your Pulp 3 versions? sudo pip3 list | grep pulp
You mentioned you upgraded foreman, how did you do so?
Apologies for the delay in responding. It’s expected that you should put the hostname in your docker pull command. docker pull fountain2.abc.com/abc-production-airo-airo-keycloak:v8 was correct.
Yes it was correct and as you see in my previous post it was working.
[root@V6TestDevops devops]# docker search fountain2.abc.com/
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
abc-airo-2_0-airo-keycloak 0
abc-airo-2_0-airo-kong 0
abc-airo-2_0-airo-postgres 0
abc-airo-keycloak 0
abc-airo-kong 0
abc-airo-mega_v6 0
abc-airo-postgres 0
abc-library-airo-airo-keycloak 0
abc-library-airo-airo-kong 0
abc-library-airo-airo-postgres 0
abc-production-airo-airo-keycloak 0
abc-production-airo-airo-kong 0
abc-production-airo-airo-postgres 0
[root@fountain2 fountain]# sudo pip3 list | grep pulp
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pulp-2to3-migration (0.5.0)
pulp-certguard (1.0.2)
pulp-container (2.0.1)
pulp-file (1.2.0)
pulp-rpm (3.6.2)
pulpcore (3.6.3)
You mentioned you upgraded foreman, how did you do so?
yum update
foreman-installer
its latest to 2.2.1
also if image is not there it give Error response from daemon: error parsing HTTP 404 response body: invalid character ‘:’ after top-level value: “404: Not Found” instead unknown not found.
[root@V6TestDevops devops]# docker pull fountain2.abc.com/abc-library-airo-airo-keycloak:89
Error response from daemon: unknown: 89 was not found!
[root@V6TestDevops devops]#
I was able to reproduce your error parsing HTTP 404 response body: invalid character ':' after top-level value: "404: Not Found" issue. I’ll be looking into it.
thank you appreciated.
I’ve created a bug for it: Bug #31362: podman pull 404 error only on Katello 3.17 - Katello - Foreman
what you think when they look into it ? I mean how much it will take to get it fixed.
secondly any recommendation or work around for me ?
I think we need to enable relativeurls: true instead of false to make it work.
form where I can edit that
environment:
Good to know about REGISTRY_HTTP_RELATIVEURLS, I hadn’t seen that before now.
I can’t say for sure yet what’s going on with the docker pull issue. I wasn’t able to reproduce the problem on the latest Katello 3.18 RC. This may be an issue with Pulp 3 rather than Katello.
If you are really stuck and need a workaround ASAP, I’d suggest that you revert back to Pulp 2 for Docker content. It requires some resetting of your content, but here it is:
foreman-installer --katello-use-pulp-2-for-docker 1
I was able to fix the issue as well by upgrading to the latest Katello 3.18 RC, but that comes with its own risks since it’s a release candidate.
what if I delete docker repos and remove docker repo from content view instead of delete the whole view ?
Also how to revert below change
foreman-installer --katello-use-pulp-2-for-docker 1
Last thing , is foreman 2.2.1 and katello 3.18 RC supported ? and what’s the procedure ? how I can upgrade ?
If you want to avoid deleting entire content views, the minimum would be to delete all docker repos and delete all content view versions that contain docker content.
If you want to revert it, you’d run
foreman-installer --katello-use-pulp-2-for-docker 0
Are you reverting because you decided you do want to be on Pulp 3 after all?
To upgrade, you follow these instructions: Foreman :: Plugin Manuals
If you upgrade using the instructions there, your Katello server will be on Foreman 2.3 and Katello 3.18. Both will be updated.
Note that upgrading to an RC is always a risk because it is considered “unstable”. The least “risky” route would be to revert to Pulp 2 instead. I would recommend you try using Docker on Pulp 2 before upgrading Katello to 3.18. Upgrading is easy, but downgrading is not in case you hit a new bug.