Problem:
When trying to create a yum repository from the Content > Products section, I’m given the error: «Authentication credentials were not provided.»
Expected outcome:
Repository should be created
Foreman and Proxy versions:
Foreman 3.3
Foreman and Proxy plugin versions:
Katello 4.5
Distribution and version:
RockyLinux 8
Other relevant data:
I’ve created the servers entirely from puppet, using Foreman and Katello classes available on the puppet forge. Everything works as expected until I got this error…
Pulp is on its own server (vdgl-pulp-a01), with foreman-proxy configured. The primary foreman server is named vdgl-foreman-a01, and use a proxy for loadbalancing with the url https://foreman.DOMAIN. I use my own TLS Certificate Authority (FreeIPA).
I can communicate with the API successfully with curl:
root@vdgl-foreman-a01:~# curl https://vdgl-pulp-a01.vagrant.teluq.ca/pulp/api/v3/ --cert /etc/pki/tls/certs/testuser.crt --key /etc/pki/tls/private/testuser.key
{"access_policies":"http://pulpcore-api/pulp/api/v3/access_policies/","artifacts":"http://pulpcore-api/pulp/api/v3/artifacts/","contentguards":"http://pulpcore-api/pulp/api/v3/contentguards/","contentguards/core/content_redirect":"http://pulpcore-api/pulp/api/v3/contentguards/core/content_redirect/","content":"http://pulpcore-api/pulp/api/v3/content/","exporters/core/filesystem":"http://pulpcore-api/pulp/api/v3/exporters/core/filesystem/","users":"http://pulpcore-api/pulp/api/v3/users/","groups":"http://pulpcore-api/pulp/api/v3/groups/","repositories":"http://pulpcore-api/pulp/api/v3/repositories/","publications":"http://pulpcore-api/pulp/api/v3/publications/","exporters/core/pulp":"http://pulpcore-api/pulp/api/v3/exporters/core/pulp/","importers/core/pulp":"http://pulpcore-api/pulp/api/v3/importers/core/pulp/","contentguards/core/rbac":"http://pulpcore-api/pulp/api/v3/contentguards/core/rbac/","roles":"http://pulpcore-api/pulp/api/v3/roles/","signing-services":"http://pulpcore-api/pulp/api/v3/signing-services/","task-groups":"http://pulpcore-api/pulp/api/v3/task-groups/","task-schedules":"http://pulpcore-api/pulp/api/v3/task-schedules/","tasks":"http://pulpcore-api/pulp/api/v3/tasks/","uploads":"http://pulpcore-api/pulp/api/v3/uploads/","workers":"http://pulpcore-api/pulp/api/v3/workers/","content/rpm/distribution_trees":"http://pulpcore-api/pulp/api/v3/content/rpm/distribution_trees/","content/rpm/modulemd_defaults":"http://pulpcore-api/pulp/api/v3/content/rpm/modulemd_defaults/","content/rpm/modulemds":"http://pulpcore-api/pulp/api/v3/content/rpm/modulemds/","content/rpm/packagecategories":"http://pulpcore-api/pulp/api/v3/content/rpm/packagecategories/","content/rpm/packageenvironments":"http://pulpcore-api/pulp/api/v3/content/rpm/packageenvironments/","content/rpm/packagegroups":"http://pulpcore-api/pulp/api/v3/content/rpm/packagegroups/","content/rpm/packagelangpacks":"http://pulpcore-api/pulp/api/v3/content/rpm/packagelangpacks/","content/rpm/packages":"http://pulpcore-api/pulp/api/v3/content/rpm/packages/","content/rpm/repo_metadata_files":"http://pulpcore-api/pulp/api/v3/content/rpm/repo_metadata_files/","acs/rpm/rpm":"http://pulpcore-api/pulp/api/v3/acs/rpm/rpm/","distributions/rpm/rpm":"http://pulpcore-api/pulp/api/v3/distributions/rpm/rpm/","publications/rpm/rpm":"http://pulpcore-api/pulp/api/v3/publications/rpm/rpm/","remotes/rpm/rpm":"http://pulpcore-api/pulp/api/v3/remotes/rpm/rpm/","repositories/rpm/rpm":"http://pulpcore-api/pulp/api/v3/repositories/rpm/rpm/","remotes/rpm/uln":"http://pulpcore-api/pulp/api/v3/remotes/rpm/uln/","content/rpm/advisories":"http://pulpcore-api/pulp/api/v3/content/rpm/advisories/"}
Not providing the certificates fail as expected:
root@vdgl-foreman-a01:~# curl https://vdgl-pulp-a01.vagrant.teluq.ca/pulp/api/v3/
{"detail":"Authentication credentials were not provided."}
Here is an excerpt from my puppet profile:
$_certfile = "/etc/pki/tls/certs/${facts['networking']['fqdn']}.crt"
$_keyfile = "/etc/pki/tls/private/${facts['networking']['fqdn']}.key"
$_cafile = '/etc/ipa/ca.crt'
##########
# Foreman-proxy
class { 'foreman_proxy' :
foreman_base_url => "https://foreman.${facts['networking']['domain']}",
puppet => false,
puppetca => false,
ssl_ca => $_cafile,
ssl_cert => $_certfile,
ssl_key => $_keyfile,
manage_puppet_group => false,
trusted_hosts => [
$facts['networking']['fqdn'],
"vdgl-foreman-a01.${facts['networking']['domain']}",
],
}
package { 'foreman-proxy-content':
ensure => 'present',
}
class { 'foreman_proxy::plugin::pulp' :
require => Class['pulpcore'],
}
##########
# Pulpcore
class { 'pulpcore' :
[...]
apache_https_cert => $_certfile,
apache_https_ca => $_cafile,
apache_https_key => $_keyfile,
}
include pulpcore::plugin::rpm
I created a new user (named testuser), because I could not create a certificate with the admin user:
root@vdgl-pulp-a01:~# PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager createsuperuser --username testuser
[...]
Email address: [...]
Password:
Password (again):
Superuser created successfully.
I manually created a certificate to replace the built-in pulp-client.crt, and configured foreman to use these to talk to pulp:
root@vdgl-foreman-a01:~# foreman-rake config -- -k 'pulp_client_key' -v '/etc/pki/tls/private/testuser.key'
pulp_client_key: /etc/pki/tls/private/testuser.key
root@vdgl-foreman-a01:~# foreman-rake config -- -k 'pulp_client_cert' -v '/etc/pki/tls/certs/testuser.crt'
pulp_client_cert: /etc/pki/tls/certs/testuser.crt
When I inspect the TLS communication between Foreman and Pulp in wireshark, I can clearly see the difference between my manual curl request (which provides the client certificate) and from the Foreman server (which does not provides the client certificate to authenticate to the Pulp api).
Is there another configuration I am missing to tell Foreman to use the provided certificates in the pulp_client_cert configuration? The next step for me would be to check the source code, but I’m a system administrator, not a developper, so digging deep in the foreman source code would require a lot of work from me.
Also, I’ve tried to keep the config pulp_client_key and pulp_client_cert to its default and replacing my custom certificate in place of /etc/pki/katello/certs/pulp-client.crt (followed by a service foreman restart for good mesure) without success.
The trace in the production.log is quite big, but essentially it is the same error:
Authentication credentials were not provided. (Katello::Errors::Pulp3Error)
Any help would be appreciated!
Here is the partial log, because I could not paste the entire log, limited by the community website:
2022-08-10T14:02:04 [I|app|1122a014] Started POST "/katello/api/v2/repositories?organization_id=1" for 172.16.194.140 at 2022-08-10 14:02:04 -0400
2022-08-10T14:02:04 [I|app|1122a014] Processing by Katello::Api::V2::RepositoriesController#create as HTML
2022-08-10T14:02:04 [I|app|1122a014] Parameters: {"product_id"=>"3", "unprotected"=>true, "checksum_type"=>nil, "verify_ssl_on_sync"=>true, "download_policy"=>"immediate", "arch"=>"x86_64", "mirroring_policy"=>"mirror_content_only", "include_tags"=>[], "exclude_tags"=>["*-source"], "http_proxy_policy"=>"global_default_http_proxy", "os_versions"=>["rhel-8"], "name"=>"Foreman 3.3", "label"=>"Foreman_3_3", "content_type"=>"yum", "url"=>"https://yum.theforeman.org/releases/3.3/el8/x86_64/", "ignorable_content"=>[], "organization_id"=>"1", "api_version"=>"v2", "repository"=>{"name"=>"Foreman 3.3", "arch"=>"x86_64", "label"=>"Foreman_3_3", "url"=>"https://yum.theforeman.org/releases/3.3/el8/x86_64/", "unprotected"=>true, "content_type"=>"yum", "product_id"=>"3", "checksum_type"=>nil, "download_policy"=>"immediate", "verify_ssl_on_sync"=>true, "ignorable_content"=>[], "include_tags"=>[], "http_proxy_policy"=>"global_default_http_proxy", "os_versions"=>["rhel-8"], "mirroring_policy"=>"mirror_content_only", "exclude_tags"=>["*-source"]}}
2022-08-10T14:02:04 [I|bac|1122a014] Task {label: , execution_plan_id: 3a64601a-a2f0-4d9c-91dc-469d5fdef80b} state changed: pending
2022-08-10T14:02:04 [I|bac|1122a014] Task {label: Actions::Katello::Repository::CreateRoot, id: fa933c32-f297-4a7b-9b79-9b3d1e943c5b, execution_plan_id: 3a64601a-a2f0-4d9c-91dc-469d5fdef80b} state changed: planning
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on name Foreman 3.3
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on major
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on minor
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on gpg_key_id
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on arch x86_64
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on label Foreman_3_3
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on url https://yum.theforeman.org/releases/3.3/el8/x86_64/
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on unprotected true
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on content_type yum
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on product_id 3
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on checksum_type
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on pulp_scratchpad_checksum_type
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on docker_upstream_name
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on download_policy immediate
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on verify_ssl_on_sync true
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on upstream_username
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on upstream_password
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on deb_releases
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on deb_components
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on deb_architectures
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on ssl_ca_cert_id
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on ssl_client_cert_id
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on ssl_client_key_id
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on ignorable_content []
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on include_tags
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on description
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on ansible_collection_requirements
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on http_proxy_policy global_default_http_proxy
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on http_proxy_id
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on download_concurrency
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on os_versions ["rhel-8"]
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on ansible_collection_auth_url
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on ansible_collection_auth_token
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on retain_package_versions_count
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on generic_remote_options
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on upstream_authentication_token
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on mirroring_policy mirror_content_only
2022-08-10T14:02:04 [I|aud|1122a014] Katello::RootRepository (33) create event on exclude_tags
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on pulp_id c8e25012-b9a2-4846-86bd-b4be64c8ba62
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on library_instance_id
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on content_view_version_id 1
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on relative_path Default_Organization/Library/custom/Foreman/Foreman_3_3
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on environment_id 1
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on saved_checksum_type
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on distribution_version
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on distribution_arch
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on distribution_bootable
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on distribution_family
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on distribution_variant
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on container_repository_name
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on root_id 33
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on remote_href
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on publication_href
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on version_href
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on last_contents_changed 1969-12-31 19:00:00 -0500
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on last_applicability_regen 1969-12-31 19:00:00 -0500
2022-08-10T14:02:04 [I|aud|1122a014] Katello::Repository (33) create event on last_indexed 1969-12-31 19:00:00 -0500
2022-08-10T14:02:04 [E|bac|1122a014] Authentication credentials were not provided. (Katello::Errors::Pulp3Error)
1122a014 | /usr/share/gems/gems/katello-4.5.0/app/services/katello/pulp3/service_common.rb:62:in `rescue in reformat_api_exception'
1122a014 | /usr/share/gems/gems/katello-4.5.0/app/services/katello/pulp3/service_common.rb:57:in `reformat_api_exception'
1122a014 | /usr/share/gems/gems/katello-4.5.0/app/services/katello/pulp3/service_common.rb:11:in `create_remote'
1122a014 | /usr/share/gems/gems/katello-4.5.0/app/services/katello/pulp3/repository.rb:71:in `create_remote'
1122a014 | /usr/share/gems/gems/katello-4.5.0/app/lib/actions/pulp3/repository/create_remote.rb:6:in `plan'
1122a014 | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:525:in `block (3 levels) in execute_plan'
1122a014 | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
1122a014 | /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
1122a014 | /usr/share/gems/gems/katello-4.5.0/app/lib/actions/middleware/backend_services_check.rb:23:in `plan'
[...]