Problem:
error message when performing a docker pull command
missing or empty Content-Type header
Expected outcome:
image is pulled from the foreman
Foreman and Proxy versions:
Foreman 2.3.4
Foreman and Proxy plugin versions:
foreman-tasks 3.0.5
foreman-remote-execution 4.2.2
katello 3.18.2.1
Distribution and version:
CentOS 7.9
ruby: 2.0.0p648
puppet: 6.22.1
DENIALS: 1653
Other relevant data:
Docker-CE 20.10.7
I created a product of type docker and added the docker hub url to pull a public image and I was able to sync the repo successfully. Once its published the link generated is not accessible from foreman as nothing is displayed, I get a message the page you are looking for does not exists.
Do I need to have account details specified in order perform a pull from docker hub or quay.io? I will not be doing push but I thought the images were available publicly and can be pulled without login.
Hi,
You should be able to pull from Docker Hub without auth. Was there a traceback in /var/log/messages from Pulp?
1 Like
I dont have tracing enabled but I see GET request in the /var/log/messages. here is what is in the logs.
Jun 15 18:04:26 {hostname_removed} pulpcore-content: [15/Jun/2021:18:04:26 +0000] “GET /v2/{product_repo_url_folder_path}/manifests/latest HTTP/1.1” 302 0 “-” “docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/3.10.0-1160.15.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 (linux))”
Jun 15 18:04:26 {hostname_removed} pulpcore-content: [15/Jun/2021:18:04:26 +0000] “GET /pulp/container/{product_repo_url_folder_path}/manifests/latest?validate_token={token_id} HTTP/1.1” 200 3538 “-” “docker/20.10.7 go/go1.13.15 git-commit/b0f5bc3 kernel/3.10.0-1160.15.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 (linux))”
@iballou
Was there any issues with Katello 3.18.2 and docker with Pulp? I thought there was a fix for something similar that went into 4.0?
Hi @user202133,
Have you tried docker/podman login <katello-server> to your Katello server and listing the images with docker/podman search <katello-server>/? You will have to log in to your Katello server to pull images unless you’ve enabled unauthenticated pull in your lifecycle environment (other than Library).
Hi @iballou
Yes, I am able to perform a login successfully, its basic auth but I am just trying to set it up and working before making it hardened solution. I am also able to perform a search to the repo and I get a listing of 1 item
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
{repo_name} 0
Thanks,
Are you able to pull your repo_name docker image okay? If you can do that, then it looks like everything is working from Katello’s standpoint. Let me know if I’m missing the issue.
Also, from above, you shouldn’t need any account details to sync from dockerhub or quay.io. If they’re public, you’re all set.
So, thats the problem. I have a client host on which i have docker-ce installed and I am able to run docker login and docker search commands successfully, however when I perform a docker pull I get this message:
Error response from daemon: missing or empty Content-Length header
Since you’re on Katello 3.18, I’d like to find out if you’re using Pulp 2 or Pulp 3 for docker content. Can you tell me what you have under “Pulpcore” on your smart proxy’s “Services” tab? That would be your primary/master smart proxy, which typically is running on the same machine as your Katello server.
Under pulpcore, I have following details:
Version: 2.1.0
Supported Content Types: deb, docker, file, yum
Here is some additional data, in reviewing the logs tab under smart proxies I see below two WARN lines being repeated several times.
File at /var/lib/pulp/content defined in pulp_content_dir parameter doesn't exist or is unreadable
File at /var/lib/mangodb defined in mongodb_dir parameter doesn't exist or is unreadable
Do you see any errors in /var/log/messages when you try to pull? It looks like you’re on Pulp 3 (Pulpcore).
Nothing in the foreman server, I have shared what’s in the messages here - Error pulling docker image from foreman - #3 by user202133
Do you know how I can turn on debug or something to get more verbose logs to see what’s happening and causing the failure?
Apologies, I didn’t see the other comment about logs above. The logs that you showed are first giving a 302 and then a 200, so it looks like that is working at least.
Here’s how I have my Foreman settings.yaml at the moment:
# Individual logging types can be toggled on/off here. See settings.yaml.example for more options
:loggers:
:audit:
:enabled: true
:level: error
:taxonomy:
:enabled: true
:level: error
:dynflow:
:enabled: true
:level: info
logging:
:level: debug
When I do a docker pull ..., I see lines in my foreman server logs that look like:
Started GET "/v2/default_organization-product-container-image/blobs/sha256:b8dfde127a2919ff59ad3fd4a0776de178a555a76fff77a506e128aea3ed41e3" for 192.168.122.110 at 2021-06-17 19:03:29 +0000
That’s Docker reaching out to Foreman’s container registry. I’d first check if there are any errors around those GET requests.
As for more in-depth Pulp 3 logs, I’ll have to ask their team for some input.
Can you tell us which docker repo you synced? We can try on our side too just to see if it’s a repo-specific error.
@user202133 I just installed docker-ce myself and got the same error. There must be an incompatibility between docker-ce and Katello. I’ll make an issue for it. In the meantime, are you able to try using podman?
1 Like
Thanks for the help, in reviewing /var/log/messages…i see only those 2 GET statements and don’t see any errors or other data. I was doing search online and did notice this issue with harbor tool here - docker 20.10 pull proxy cache image failed: Error response from daemon: missing or empty Content-Type header · Issue #13740 · goharbor/harbor · GitHub
Just sharing the link with hope it might help troubleshoot the issue.
I am trying to pull newrelic/synthetics-minion (docker.com)
2 Likes
