Add grafana repo to Foreman

I have problem adding grafana repo

I would like to sync grafana repo:

name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

Errors:

401, message=‘Unauthorized’, url=URL(‘https://packages.grafana.com/oss/rpm/.treeinfo’)

Problem is that it require an authentication vi ssl certificate. If I add ca-bundle.crt in credential as SSL Certification then I get this error:

An error occurred while creating the Content Credential: Validation failed: Content cannot be a binary file.

Thanks

Seems Katello is having problems reading your .crt file. It should be human-readable with cat or in a text editor.

You can try manually pasting the file contents in the web UI instead of uploading; maybe that will work better?

I have try copy/paste but I got the same error. It is human-readable. This cert is on linux filesystem on RHEL 7

(base) [jost@jostr ~]$ ll /etc/pki/tls/certs/ca-bundle.crt
lrwxrwxrwx. 1 root root 49 Jun 17 2020 /etc/pki/tls/certs/ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
(base) [jost@jostr ~]$

It does not require a certificate for authentication, options sslverify and sslcacert are only for validating the source.

The problem is they use some repository software which does not display the files and so it is hard to find out if they have some error in their repository.

We had some similar issues with Gitlab: Unable to add Gitlab-ee repo
But Grafana has always worked in the past, when I have tried!

1 Like

Please post the exact repository configuration you have used. Or use “hammer repository list” to find the id of the repository and then “hammer repository info --id=” to print the config.

this is repo config:

[root@f-1 ~]# hammer repository info --id=86
ID: 86
Name: Grafana
Label: Grafana
Description:
Organization: XXXX
Red Hat Repository: no
Content Type: yum
Mirror on Sync: yes
URL: https://packages.grafana.com/oss/rpm
Publish Via HTTP: yes
Published At: http://xxxx/pulp/repos/xxx/Library/custom/Grafana/Grafana/
Relative Path: xxx/Library/custom/Grafana/Grafana
Download Policy: immediate
HTTP Proxy:
ID: 1
Name: fac-1
HTTP Proxy Policy: use_selected_http_proxy
Product:
ID: 25
Name: Grafana
GPG Key:
ID: 9
Name: Grafana.key
Sync:
Status: Warning
Last Sync Date: about 7 hours
Created: 2021/03/31 10:48:56
Updated: 2021/03/31 10:48:57
Content Counts:
Packages: 0
Source RPMS: 0
Package Groups: 0
Errata: 0
Module Streams: 0

[root@f-1 ~]#

Do you have something configured for upstream authorization on the repository?

When I check with curl on the failed URL I only get an 404:

$ curl -v 'https://packages.grafana.com/oss/rpm/.treeinfo'
* About to connect() to packages.grafana.com port 443 (#0)
...
< HTTP/1.1 404 Not Found
...

You have configured the repository to use an http proxy. My first guess would be that your http proxy requires authentication but you don’t provide it.

What do you have configured as content credential?

# hammer content-credentials info --id 9 --organization=<ORG>

Maybe that’s a binary?

I didn’t configure any authorization. And proxy doesn’t requires authentication. If you point your browser to this address

https://packages.grafana.com/oss/rpm

you cant see any rpms. They have something strange in there repo…

This is content credential → pgp key from Grafana
[root@f-1 ~]# hammer content-credentials info --id 9 --organization 1
ID: 9
Name: Grafana.key
Organization: xxxx
Content:
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1

mQENBFiHXVIBCADr3VDEAGpq9Sg/xrPVu1GGqWGXdbnTbbNKeveCtFHZz7/GSATW
iwiY1skvlAOBiIKCqJEji0rZZgd8WxuhdfugiCBk1hDTMWCpjI0P+YymV77jHjYB
jHrKNlhb+aLjEd9Gf2EtbKUT1fvGUkzlVrcRGSX/XR9MBZlgja7NIyuVbn3uwZQ4
jflWSNSlvMpohNxTFkrBFTRrCJXhbDLfCS46+so22CP3+1VQyqJ7/6RWK9v9KYdS
AVNgILXMggSrMqha4WA1a/ktczVQXNtP8IuPxTdp9pNYsklOTmrFVeq3mXsvWh9Q
lIhpYHIZlTZ5wVBq4wTRchsXC5MubIhz+ASDABEBAAG0GkdyYWZhbmEgPGluZm9A
Z3JhZmFuYS5jb20+iQE4BBMBAgAiBQJYh11SAhsDBgsJCAcDAgYVCAIJCgsEFgID
AQIeAQIXgAAKCRCMjDTFJAmMthxJB/9Id6JrwqRkJW+eSBb71FGQmRsJvNFR8J+3
NPVhJNkTFFOM7TnjAMUIv+LYEURqGcceTNAN1aHq/7n/8ybXucCS0CnDYyNYpyVs
tWJ3FOQK3jPrmziDCWPQATqMM/Z2auXVFWrDFqfh2xKZNjuix0w2nyuWB8U0CG2U
89w+ksPJblGGU5xLPPzDQoAqyZXY3gpGGTkCuohMq2RWYbp/QJSQagYhQkKZoJhr
XJlnw4At6R1A5UUPzDw6WJqMRkGrkieE6ApIgf1vZSmnLRpXkqquRTAEyGT8Pugg
ee6YkD19/LK6ED6gn32StY770U9ti560U7oRjrOPK/Kjp4+qBtkQuQENBFiHXVIB
CACz4hO1g/4fKO9QWLcbSWpB75lbNgt1kHXP0UcW8TE0DIgqrifod09lC85adIz0
zdhs+00lLqckM5wNbp2r+pd5rRaxOsMw2V+c/y1Pt3qZxupmPc5l5lL6jzbEVR9g
ygPaE+iabTk9Np2OZQ7Qv5gIDzivqK2mRHXaHTzoQn2dA/3xpFcxnen9dvu7LCpA
CdScSj9/UIRKk9PHIgr2RJhcjzLx0u1PxN9MEqfIsIJUUgZOoDsr8oCs44PGGIMm
cK1CKALLLiC4ZM58B56jRyXo18MqB6VYsC1X9wkcIs72thL3tThXO70oDGcoXzoo
ywAHBH63EzEyduInOhecDIKlABEBAAGJAR8EGAECAAkFAliHXVICGwwACgkQjIw0
xSQJjLbWSwf/VIM5wEFBY4QLGUAfqfjDyfGXpcha58Y24Vv3n6MwJqnCIbTAaeWf
30CZ/wHg3NNIMB7I31vgmMOEbHQdv0LPTi9TG205VQeehcpNtZRZQ0D8TIetbxyi
Emmn9osig9U3/7jaAWBabE/9bGx4TF3eLlEH9wmFrNYeXvgRqmyqVoqhIMCNAAOY
REYyHyy9mzr9ywkwl0aroBqhzKIPyFlatZy9oRKllY/CCKO9RJy4DZidLphuwzqU
ymdQ1sqe5nKvwG5GvcncPc3O7LMevDBWnpNNkgERnVxCqpm90TuE3ONbirnU4+/S
tUsVU1DERc1fjOCnAm4pKIlNYphISIE7OQ==
=0pMC
-----END PGP PUBLIC KEY BLOCK-----

[root@fore-1 ~]#

[quote=“jost, post:9, topic:22959”]
I didn’t configure any authorization
[/quote] Did you check, it’s really empty in the configuration page. The hammer command above doesn’t show it and I am not sure which one does. My browser always wants to autofill the username and password if I create a new repo and I always have to remove/clean both text fields before I submit…

What do you have set as SSL CA Cert, SSL Client Cert and SSL Client Key?

[quote=“jost, post:9, topic:22959”]
If you point your browser to this address

https://packages.grafana.com/oss/rpm

you cant see any rpms.
[/quote] Yes. I know. You cannot browse the repository. You I asked what curl prints out for ‘https://packages.grafana.com/oss/rpm/.treeinfo’ on that server, not in your browser…

If you want to see, what’s in the repository you have to start with the repomd file:

$ curl -v 'https://packages.grafana.com/oss/rpm/repodata/repomd.xml'

I didn’t set SSL CA Cert, SSL Client Cert and SSL Client Key, because if I try to import this CA bundle from /etc/pki/tls/certs/ca-bundle.crt (link to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem) into foreman then I get error:

Could not create GPG Key:
Validation failed: Content cannot be a binary file.

On my foreman server I can reach this page via curl,

[root@f-1 ~]# curl -v https://packages.grafana.com/oss/rpm/.treeinfo

  • About to connect() to packages.grafana.com port 443 (#0)
  • Trying 199.232.194.217…
  • Connected to packages.grafana.com (199.232.194.217) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  • SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  • Server certificate:
  •   subject: CN=packages.grafana.com
    
  •   start date: Mar 22 19:44:23 2021 GMT
    
  •   expire date: Apr 23 19:44:22 2022 GMT
    
  •   common name: packages.grafana.com
    
  •   issuer: CN=GlobalSign Atlas R3 DV TLS CA 2020,O=GlobalSign nv-sa,C=BE
    

GET /oss/rpm/.treeinfo HTTP/1.1
User-Agent: curl/7.29.0
Host: packages.grafana.com
Accept: /

< HTTP/1.1 404 Not Found
< Connection: keep-alive
< Content-Length: 192
< X-GUploader-UploadID: ABg5-Uy04-ctaMxNBvGRcMhugCKGK2-81Tue-ds3atBCZHq6Xvlej1RwqOa1nBPKFMNiTfMeXXB00FHUMCK8RW3gF-k
< Content-Type: application/xml; charset=UTF-8
< Expires: Thu, 01 Apr 2021 06:49:04 GMT
< Cache-Control: private, max-age=0
< Server: UploadServer
< Accept-Ranges: bytes
< Date: Thu, 01 Apr 2021 06:49:04 GMT
< Via: 1.1 varnish
< X-Served-By: cache-dub4333-DUB
< X-Cache: MISS
< X-Cache-Hits: 0
< X-Timer: S1617259745.851595,VS0,VE143
<

<?xml version='1.0' encoding='UTF-8'?>NoSuchKeyThe specified key does not exist.
No such object: grafana-repo/oss/rpm/.treeinfo
[root@f-1 ~]#

Well, yes, that’s pointless anyway. That’s the collection of trusted root certificates on the system. That should be used by default, anyway, if you enable ssl verification on the repository.

Check /var/log/foreman/production.log file for errors. It should show an error at the time the sync fails.

Where did you see your error messages you have posted? I only see “Katello::Errors::PulpError: RPM1004: Error retrieving metadata: Authentication required” if I try to access a repository for which I am not authenticated…

Why do you use a http proxy if you don’t need to? Your foreman server can access the repository directly…

1 Like

/var/log/foreman/production.log
2021-04-01T08:59:05 [E|bac|] 401, message=‘Unauthorized’, url=URL(‘https://packages.grafana.com/oss/rpm/.treeinfo’) (Katello::Errors::Pulp3Error)
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors’

/var/log/messages
Apr 1 08:59:05 fore-1 pulpcore-worker-4: aiohttp.client_exceptions.ClientResponseError: 401, message=‘Unauthorized’, url=URL(‘https://packages.grafana.com/oss/rpm/.treeinfo’)

I’m using proxy because in the future foreman server will be allowed only to use proxy to get to the internet.

O.K. You are using pulp3. That’s probably why the message is different.

Have you tried it without proxy to rule that out? As curl can connect directly without problems (i.e. without a 401 error) I still think it must be your proxy which denies the access…

1 Like

Yes, I have try without proxy and It doesn’t work. I got he same error.

2021-04-01T09:48:07 [E|bac|] 401, message=‘Unauthorized’, url=URL(‘https://packages.grafana.com/oss/rpm/.treeinfo’) (Katello::Errors::Pulp3Error)

any other idea or solution for that?

I am out of ideas and my environment with pulp2 and no proxy shows no problems. I guess a developer or someone with deeper insights has to tag in here. Sorry.

foreman-rake console

cfile = File.read(‘/etc/pki/tls/certs/ca-bundle.crt’)
puts cfile.encode(“UTF-8”, ‘binary’)

I can see all certificates:


So, on your Foreman, you can configure Grafana repo and rpm’s are staring downloaded?
I have try curl with proxy and it also works. So there is no proxy problem.
I’m using:
katello-3.18.2-1.el7.noarch
foreman-2.3.3-1.el7.noarch
pulp-server-2.21.5-1.el7.noarch

Yes. But the file is in UTF-8 because it has some UTF-8 in the comments. A PEM certificate itself is simple pure 7 bit ASCII thus I guess it doesn’t like the UTF-8 in the file. If you make a copy of the file and remove the comments you’ll probably get it imported.

But again: you don’t need the ca-bundle. I am pretty sure, that simply enabling “Verify SSL” on a repository effectively does exactly the same. Unless you specify a specific CA certificate to use, the ssl libraries will use the system default, i.e. the ca-bundle to verify the https connection. So don’t set it. Don’t use it. If you want to be sure you can try to setup a repository pointing to a self-signed webserver to check if it is accepted or not when “Verify SSL” is enabled.

Importing ca-bundle is pointless and it also won’t accept any changes anymore which are automatically added to ca-bundle in the system.

Yes. It’s happily syncing every day and getting new updates every couple of days. But again, I am still on pulp2.

I don’t think that shows you whether you use pulp2 or pulp3. Your error messaged show pulp3 thus I guess you have made the pulp3 migration.

yes, you are right regarding importing ca-bundle.
This is a fresh installation of Foreman with version that I mention before.
So it’s look like, I will not be able to sync grafana repo which it’s pretty annoying ?

I don’t think that shows you whether you use pulp2 or pulp3. Your error messaged show pulp3 thus I guess you have made the pulp3 migration.

To verify you’re using pulp3, you can check Infrastructure > Smart Proxies > (your Foreman server) > Services tab. Look for ‘yum’ under Supported content types - if it’s listed under Pulpcore it means pulp3, ‘Pulp’ means pulp 2.