I have troubles to understand the outcome of it, but the discussion is also about pushes that take longer than the lifetime of the token from the registry. For Gitlab, the issue seemed to be Cloudflare I understood. I do not use Cloudflare.
I found the code for the token lifetime in Katello:
The lifetime is hardcoded 3 or 6 minutes.
The upload of my 9GiB image takes roughly 30minutes.
I tried to call podman push --log-level=debug ... to get more debug logs, but this is not giving more insights.
....
Copying blob 13b8522a89d2 done |
Copying blob 596d46a1191d done |
Copying blob 14583fc917a5 done |
DEBU[1954] Looking up image "registry.gitlab.com/eu-os/workspace-images/eu-os-base-demo/eu-os-demo:latest" in local containers storage
DEBU[1954] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[1954] Trying "registry.gitlab.com/eu-os/workspace-images/eu-os-base-demo/eu-os-demo:latest" ...
DEBU[1954] parsed reference into "[overlay@/home/rriemann/.local/share/containers/storage+/run/user/1000/containers]@024fe129075da23167b7040c43caeedce2490d52d2467b2b0f61769494873e5a"
DEBU[1954] Found image "registry.gitlab.com/eu-os/workspace-images/eu-os-base-demo/eu-os-demo:latest" as "registry.gitlab.com/eu-os/workspace-images/eu-os-base-demo/eu-os-demo:latest" in local containers storage
Error: writing blob: uploading layer chunked: authentication required
DEBU[1954] Shutting down engines
INFO[1954] Received shutdown.Stop(), terminating! PID=3467450
Thanks for the detailed report, I think you might be spot-on about the token expiration being an issue. I would’ve expected a new token to be provisioned.
We should deal with this soon, I’m a little surprised more people haven’t hit it. I supposed multiple upload attempts might solve the issue since the blobs uploaded remain in Pulp, maybe they’re just dealing with it that way.
I managed to upload the image today. I copied the image with skopeo to an oci-archive on the same host that runs foreman. Then I copied from the oci-archive to foreman with success. The blobs were already on the server and not reuploaded again and it took maybe 5min. Podman was always reuploading the blobs again and it was always taking 30min.
User João from Gitlab made a script for testing their issues. Maybe this also helps debugging here.
Debugging Image Script
#!/bin/bash
repo="jdrpereira/registry-test/issue-361279"
tmp=$(mktemp -d)
echo "Using temporary dir $tmp"
cd $tmp
gcping_bin="gcping_darwin_amd64_latest"
dd_bs="64m"
dd_iflag=""
if [[ $(uname) == 'Linux' ]]; then
gcping_bin="gcping_linux_amd64_latest"
dd_bs="64M"
dd_iflag="iflag=fullblock"
fi
echo "Measuring GCP ping latency with https://github.com/GoogleCloudPlatform/gcping ..."
curl -s https://storage.googleapis.com/gcping-release/$gcping_bin > gcping
chmod +x gcping
./gcping -r us-east1
echo "Measuring internet bandwidth with https://github.com/sivel/speedtest-cli ..."
curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py > speedtest-cli
chmod +x speedtest-cli
./speedtest-cli --simple --bytes
echo "Generating random layer files..."
dd if=/dev/urandom of=1GB bs=$dd_bs count=16 $dd_iflag
du -h $tmp/1GB
dd if=/dev/urandom of=2GB bs=$dd_bs count=32 $dd_iflag
du -h $tmp/2GB
dd if=/dev/urandom of=3GB bs=$dd_bs count=48 $dd_iflag
du -h $tmp/3GB
echo "Building image..."
cat <<EOT >> Dockerfile
FROM scratch
ADD 1GB /
ADD 2GB /
ADD 3GB /
EOT
docker build -t registry.gitlab.com/$repo:latest .
echo "Pushing image..."
time docker push registry.gitlab.com/$repo:latest
echo "Cleaning up..."
rm -rf $tmp
docker rmi registry.gitlab.com/$repo:latest
