Ubuntu repo sync not working

Problem:

Following what others have done for a Ubuntu repo but I can’t get it working.

**Errors:**

No valid Release file found for 'focal'.

Repo config:

Sync Settings
Upstream URL: http://archive.ubuntu.com/ubuntu/
Releases/Distributions: focal
Components: main
Architectures: amd64
Verify SSL: Yes
Upstream Authorization: -
 
HTTP Proxy: Global Default (None)
Unprotected: Yes
Published At: https://foreman.domain.com/pulp/content/COMPANY/Library/custom/Ubuntu_20_04_LTS/archive_ubuntu_com_focal/
GPG Key: archive.ubuntu.com_focal

Expected outcome:

Foreman and Proxy versions:

Foreman 3.3
Katello 4.5

Distribution and version:

CentOS 8 stream (EL8)

Other relevant data:

New deployment, just finding my way round the various bits and pieces.

Does it work if you leave Components blank? Or not selecting a GPG key?

1 Like

Taking the GPG key off did the trick. Which is odd, and makes me wonder if the GPG key that I have is either wrong or configured wrong.

Any advice on that?

The way a sync with GPG key verification works here, is as follows:

  1. Your “Upstream URL” and your “Releases/Distributions” are combined to the following URL:
    http://archive.ubuntu.com/ubuntu/dists/focal/
    
  2. The files InRelease, Release, and Release.gpg are downloaded from the URL from step 1.
  3. pulp_deb tries to verify the signatures using the GPG key you provided. Any downloaded files that cannot be verified are simply discarded.
  4. If not at least one of InRelease, or Release is left, you get the error you got.

So yes, most likely pulp_deb decided that the key you provided does not work for verifying the signatures. Perhaps you provided the wrong public key, or maybe pulp_deb did not like the format you provided it in. There are some docs on how to retrieve the right keys here: Importing Content - orcharhino documentation

Independently of what exactly you did, I think we need to improve the error message you got. From this error message it is impossible to tell, if the sync failed because you entered for example an incorrect “Upstream URL”, or if it failed because the downloaded files failed GPG key verification. If you dig through the logs you might be able to spot the difference, but this is not a good user experience! Turns out we already have an issue for this, but never worked on it: https://github.com/pulp/pulp_deb/issues/399 (I will try to revive this issue).

1 Like

Thank you, good to know, and great that the documentation will get some love.

As for the link to the gpg key export command, it’s not working as the detailed key servers have been decommissioned. Having found this issue I encountered another, and I don’t seem to be the only one, but found this Getting GPG key for ubuntu repository? - #3 by saltyollpheist

This link, point #3 no longer works:

https://docs.theforeman.org/nightly/Content_Management_Guide/index-katello.html#Extracting_GPG_Public_Key_Fingerprints_from_a_Release_Files_content-management

Thanks for the info, we will try to look into it tomorrow.

Hi @dmgeurts

Please have a look at Extracting GPG Public Key Fingerprints from a Release Files. The link you’ve posted is old but sadly not broken. Unfortunately, we don’t delete old guides on docs.theforeman.org, but I’ve opened an issue to do some maintenance.

I’ve tested this just now and it should work. Please let me know if you spot any other issues.

Hi @maximilian

Thank you, but I tried that and step 3 fails:

user@foreman:~$ wget http://archive.ubuntu.com/ubuntu/dists/focal/Release
--2022-08-17 13:38:26--  http://archive.ubuntu.com/ubuntu/dists/focal/Release
Resolving archive.ubuntu.com (archive.ubuntu.com)... 185.125.190.36, 91.189.91.39, 91.189.91.38, ...
Connecting to archive.ubuntu.com (archive.ubuntu.com)|185.125.190.36|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 263289 (257K)
Saving to: ‘Release’

Release                                                    100%[======================================================================================================================================>] 257.12K  --.-KB/s    in 0.08s

2022-08-17 13:38:27 (3.30 MB/s) - ‘Release’ saved [263289/263289]

dgeurts@fm01:~$ wget http://archive.ubuntu.com/ubuntu/dists/focal/Release.gpg
--2022-08-17 13:38:31--  http://archive.ubuntu.com/ubuntu/dists/focal/Release.gpg
Resolving archive.ubuntu.com (archive.ubuntu.com)... 185.125.190.39, 185.125.190.36, 91.189.91.39, ...
Connecting to archive.ubuntu.com (archive.ubuntu.com)|185.125.190.39|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1554 (1.5K) [application/pgp-signature]
Saving to: ‘Release.gpg’

Release.gpg                                                100%[======================================================================================================================================>]   1.52K  --.-KB/s    in 0s

2022-08-17 13:38:32 (286 MB/s) - ‘Release.gpg’ saved [1554/1554]

user@foreman:~$ gpg --verify Release.gpg Release
gpg: directory '/home/user/.gnupg' created
gpg: keybox '/home/user/.gnupg/pubring.kbx' created
gpg: Signature made Thu 23 Apr 2020 19:34:16 CEST
gpg:                using RSA key 3B4FE6ACC0B21F32
gpg: Can't check signature: No public key
gpg: Signature made Thu 23 Apr 2020 19:34:16 CEST
gpg:                using RSA key 871920D1991BC93C
gpg: Can't check signature: No public key
2 user@foreman:~$ gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys 3B4FE6ACC0B21F32
gpg: keyserver receive failed: No keyserver available
2 user@foreman:~$ gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3B4FE6ACC0B21F32
gpg: keyserver receive failed: No keyserver available

Am I doing something blatantly wrong or am I missing something important?

I’ve just retested this in a Rocky Linux 8 container to ensure I actually download the keys. It works for me:

[root@d5a9327a5c64 /]# gpg --verify Release.gpg Release
gpg: Signature made Thu Apr 23 17:34:16 2020 UTC
gpg:                using RSA key 3B4FE6ACC0B21F32
gpg: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 790B C727 7767 219C 42C8  6F93 3B4F E6AC C0B2 1F32
gpg: Signature made Thu Apr 23 17:34:16 2020 UTC
gpg:                using RSA key 871920D1991BC93C
gpg: Good signature from "Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: F6EC B376 2474 EDA9 D21B  7022 8719 20D1 991B C93C

[root@d5a9327a5c64 /]# history
    1  dnf install -y wget
    2  wget http://archive.ubuntu.com/ubuntu/dists/focal/Release
    3  wget http://archive.ubuntu.com/ubuntu/dists/focal/Release.gpg
    4  gpg --verify Release.gpg Release
    5  gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys 3B4FE6ACC0B21F32
    6  gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys 871920D1991BC93C
    7  gpg --verify Release.gpg Release
    8  history

It works even without protocol (in a fresh container):

[root@f913e2a7683e /]# gpg --keyserver keyserver.ubuntu.com --recv-keys 3B4FE6ACC0B21F32
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 3B4FE6ACC0B21F32: public key "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1
[root@f913e2a7683e /]# gpg --keyserver keyserver.ubuntu.com --recv-keys 871920D1991BC93C
gpg: key 871920D1991BC93C: public key "Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1

Could you try running this on another machine, like your local host? Foreman only needs the gpg export.

I’m stumped. As I can pull a key on an external machine, but not one in the corporate environment here. But when I do a tcpdump I see no outbound connections attempted at all.

Further searching is pointing to DNS, but I don’t see any DNS requests made so how on earth does the verification happen?

Apologies, this is clearly an issue with the environment my servers are in.

Unless you need an automated way, it’s very easy to just provide the ascii-armored GPG pub keys and upload them to your Foreman instance. Good luck digging through your corp. environment. :v:

1 Like

Yeah, I’m just using my external node to grab the keys now, they don’t change very often. But heck I’ve got no idea why it’s failing… Just grateful I noticed this in the syslog on the Ubuntu host:

Aug 17 16:01:42 server dirmngr[165778]: no alive host found in pool 'keyserver.ubuntu.com'
Aug 17 16:01:42 server dirmngr[165778]: command 'KS_GET' failed: No keyserver available

I just got tripped by this. Upstream had changed the GPG key, and I was getting the same error. Updating the key resolved it, but it only once @quba42 pointed it out. :slight_smile:

I plan to provide a clearer error message in cases where sync failed due to signatures failing to verify. This is on our board, and slated for the next release.

1 Like