Add grafana repo to Foreman

I have problem adding grafana repo

I would like to sync grafana repo:



401, message=‘Unauthorized’, url=URL(‘’)

Problem is that it require an authentication vi ssl certificate. If I add ca-bundle.crt in credential as SSL Certification then I get this error:

An error occurred while creating the Content Credential: Validation failed: Content cannot be a binary file.


Seems Katello is having problems reading your .crt file. It should be human-readable with cat or in a text editor.

You can try manually pasting the file contents in the web UI instead of uploading; maybe that will work better?

I have try copy/paste but I got the same error. It is human-readable. This cert is on linux filesystem on RHEL 7

(base) [jost@jostr ~]$ ll /etc/pki/tls/certs/ca-bundle.crt
lrwxrwxrwx. 1 root root 49 Jun 17 2020 /etc/pki/tls/certs/ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
(base) [jost@jostr ~]$

It does not require a certificate for authentication, options sslverify and sslcacert are only for validating the source.

The problem is they use some repository software which does not display the files and so it is hard to find out if they have some error in their repository.

We had some similar issues with Gitlab: Unable to add Gitlab-ee repo
But Grafana has always worked in the past, when I have tried!

1 Like

Please post the exact repository configuration you have used. Or use “hammer repository list” to find the id of the repository and then “hammer repository info --id=” to print the config.

this is repo config:

[root@f-1 ~]# hammer repository info --id=86
ID: 86
Name: Grafana
Label: Grafana
Organization: XXXX
Red Hat Repository: no
Content Type: yum
Mirror on Sync: yes
Publish Via HTTP: yes
Published At: http://xxxx/pulp/repos/xxx/Library/custom/Grafana/Grafana/
Relative Path: xxx/Library/custom/Grafana/Grafana
Download Policy: immediate
HTTP Proxy:
ID: 1
Name: fac-1
HTTP Proxy Policy: use_selected_http_proxy
ID: 25
Name: Grafana
GPG Key:
ID: 9
Name: Grafana.key
Status: Warning
Last Sync Date: about 7 hours
Created: 2021/03/31 10:48:56
Updated: 2021/03/31 10:48:57
Content Counts:
Packages: 0
Source RPMS: 0
Package Groups: 0
Errata: 0
Module Streams: 0

[root@f-1 ~]#

Do you have something configured for upstream authorization on the repository?

When I check with curl on the failed URL I only get an 404:

$ curl -v ''
* About to connect() to port 443 (#0)
< HTTP/1.1 404 Not Found

You have configured the repository to use an http proxy. My first guess would be that your http proxy requires authentication but you don’t provide it.

What do you have configured as content credential?

# hammer content-credentials info --id 9 --organization=<ORG>

Maybe that’s a binary?

I didn’t configure any authorization. And proxy doesn’t requires authentication. If you point your browser to this address

you cant see any rpms. They have something strange in there repo…

This is content credential → pgp key from Grafana
[root@f-1 ~]# hammer content-credentials info --id 9 --organization 1
ID: 9
Name: Grafana.key
Organization: xxxx
Version: GnuPG v1


[root@fore-1 ~]#

[quote=“jost, post:9, topic:22959”]
I didn’t configure any authorization
[/quote] Did you check, it’s really empty in the configuration page. The hammer command above doesn’t show it and I am not sure which one does. My browser always wants to autofill the username and password if I create a new repo and I always have to remove/clean both text fields before I submit…

What do you have set as SSL CA Cert, SSL Client Cert and SSL Client Key?

[quote=“jost, post:9, topic:22959”]
If you point your browser to this address

you cant see any rpms.
[/quote] Yes. I know. You cannot browse the repository. You I asked what curl prints out for ‘’ on that server, not in your browser…

If you want to see, what’s in the repository you have to start with the repomd file:

$ curl -v ''

I didn’t set SSL CA Cert, SSL Client Cert and SSL Client Key, because if I try to import this CA bundle from /etc/pki/tls/certs/ca-bundle.crt (link to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem) into foreman then I get error:

Could not create GPG Key:
Validation failed: Content cannot be a binary file.

On my foreman server I can reach this page via curl,

[root@f-1 ~]# curl -v

  • About to connect() to port 443 (#0)
  • Trying…
  • Connected to ( port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  • SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  • Server certificate:
  •   subject:
  •   start date: Mar 22 19:44:23 2021 GMT
  •   expire date: Apr 23 19:44:22 2022 GMT
  •   common name:
  •   issuer: CN=GlobalSign Atlas R3 DV TLS CA 2020,O=GlobalSign nv-sa,C=BE

GET /oss/rpm/.treeinfo HTTP/1.1
User-Agent: curl/7.29.0
Accept: /

< HTTP/1.1 404 Not Found
< Connection: keep-alive
< Content-Length: 192
< X-GUploader-UploadID: ABg5-Uy04-ctaMxNBvGRcMhugCKGK2-81Tue-ds3atBCZHq6Xvlej1RwqOa1nBPKFMNiTfMeXXB00FHUMCK8RW3gF-k
< Content-Type: application/xml; charset=UTF-8
< Expires: Thu, 01 Apr 2021 06:49:04 GMT
< Cache-Control: private, max-age=0
< Server: UploadServer
< Accept-Ranges: bytes
< Date: Thu, 01 Apr 2021 06:49:04 GMT
< Via: 1.1 varnish
< X-Served-By: cache-dub4333-DUB
< X-Cache: MISS
< X-Cache-Hits: 0
< X-Timer: S1617259745.851595,VS0,VE143

<?xml version='1.0' encoding='UTF-8'?>NoSuchKeyThe specified key does not exist.
No such object: grafana-repo/oss/rpm/.treeinfo
[root@f-1 ~]#

Well, yes, that’s pointless anyway. That’s the collection of trusted root certificates on the system. That should be used by default, anyway, if you enable ssl verification on the repository.

Check /var/log/foreman/production.log file for errors. It should show an error at the time the sync fails.

Where did you see your error messages you have posted? I only see “Katello::Errors::PulpError: RPM1004: Error retrieving metadata: Authentication required” if I try to access a repository for which I am not authenticated…

Why do you use a http proxy if you don’t need to? Your foreman server can access the repository directly…

1 Like

2021-04-01T08:59:05 [E|bac|] 401, message=‘Unauthorized’, url=URL(‘’) (Katello::Errors::Pulp3Error)
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello- `block in check_for_errors’

Apr 1 08:59:05 fore-1 pulpcore-worker-4: aiohttp.client_exceptions.ClientResponseError: 401, message=‘Unauthorized’, url=URL(‘’)

I’m using proxy because in the future foreman server will be allowed only to use proxy to get to the internet.

O.K. You are using pulp3. That’s probably why the message is different.

Have you tried it without proxy to rule that out? As curl can connect directly without problems (i.e. without a 401 error) I still think it must be your proxy which denies the access…

1 Like

Yes, I have try without proxy and It doesn’t work. I got he same error.

2021-04-01T09:48:07 [E|bac|] 401, message=‘Unauthorized’, url=URL(‘’) (Katello::Errors::Pulp3Error)

any other idea or solution for that?

I am out of ideas and my environment with pulp2 and no proxy shows no problems. I guess a developer or someone with deeper insights has to tag in here. Sorry.

foreman-rake console

cfile =’/etc/pki/tls/certs/ca-bundle.crt’)
puts cfile.encode(“UTF-8”, ‘binary’)

I can see all certificates:

So, on your Foreman, you can configure Grafana repo and rpm’s are staring downloaded?
I have try curl with proxy and it also works. So there is no proxy problem.
I’m using:

Yes. But the file is in UTF-8 because it has some UTF-8 in the comments. A PEM certificate itself is simple pure 7 bit ASCII thus I guess it doesn’t like the UTF-8 in the file. If you make a copy of the file and remove the comments you’ll probably get it imported.

But again: you don’t need the ca-bundle. I am pretty sure, that simply enabling “Verify SSL” on a repository effectively does exactly the same. Unless you specify a specific CA certificate to use, the ssl libraries will use the system default, i.e. the ca-bundle to verify the https connection. So don’t set it. Don’t use it. If you want to be sure you can try to setup a repository pointing to a self-signed webserver to check if it is accepted or not when “Verify SSL” is enabled.

Importing ca-bundle is pointless and it also won’t accept any changes anymore which are automatically added to ca-bundle in the system.

Yes. It’s happily syncing every day and getting new updates every couple of days. But again, I am still on pulp2.

I don’t think that shows you whether you use pulp2 or pulp3. Your error messaged show pulp3 thus I guess you have made the pulp3 migration.

yes, you are right regarding importing ca-bundle.
This is a fresh installation of Foreman with version that I mention before.
So it’s look like, I will not be able to sync grafana repo which it’s pretty annoying ?

I don’t think that shows you whether you use pulp2 or pulp3. Your error messaged show pulp3 thus I guess you have made the pulp3 migration.

To verify you’re using pulp3, you can check Infrastructure > Smart Proxies > (your Foreman server) > Services tab. Look for ‘yum’ under Supported content types - if it’s listed under Pulpcore it means pulp3, ‘Pulp’ means pulp 2.