External Private Yum Repository on JFrog.io not synchronizing in Foreman/Katello

Support
1

Problem:
Intro: JFrog.io yum repository protected by username and “api_key”(this is how our vendor referred to it) fails with error “XML file(s): filelists not found”.
Details:
We had an external yum repository syncing with our Spacewalk server from a 3rd party vendor, this is a repository specifically for our company. It’s formatted like:
https://username:<api_key>@vendor.jfrog.io/artifactory/product-company/
This repository is syncing succesfully like that.

I have been able to configure this repository in Foreman, given the same URL. But when trying to sync the task fails with “XML file(s): filelists not found”.
When I try to configure it to use the upstream username and password without the username and “api_key” in the URL, I get the same error. (if I keep the Username and “api_key” in the url, it fails with “Cannot combine AUTH argument with credentials encoded in URL”).

I have tried to manually request the repomd.xml and the mentioned xml.gz files in there using wget, which succeeded.

So I am really curious if someone has “private” jfrog.io repositories in their Foreman/Katello setup and have had this issue and know how to solve it.
Otherwise if someone could advice me on how to troubleshoot this, as with my common knowledge about yum repositories I feel stuck here.

ps. Asking here since Jfrog.io seems rather standard and my contact at the vendor is currently out on vacation.

Expected outcome:
Repository succesfully synchronising.

Foreman and Proxy versions:
foreman-2.5.2-1.el7.noarch
foreman-proxy-2.5.2-1.el7.noarch
katello-4.1.2.1-1.el7.noarch

Distribution and version:
RHEL 7.9

Other relevant data:
The (to me) relevant section in production.log

2021-08-19T09:38:15 [I|bac|5783a777] Task {label: Actions::Katello::Repository::Sync, id: 756b3e6e-5604-4ea3-9609-b050c9efd10c, execution_plan_id: 1cbe7939-eb1a-4c20-95fb-c6ee1fcf1bc6} state changed: running
2021-08-19T09:38:15 [I|app|d95d4c16] Started POST "/foreman_tasks/api/tasks/bulk_search" for 172.30.1.105 at 2021-08-19 09:38:15 +0200
2021-08-19T09:38:15 [I|app|d95d4c16] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2021-08-19T09:38:15 [I|app|d95d4c16]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"de4cad41-dcdc-4a63-93bf-ddaa53353b23", "search_id"=>"1"}], "task"=>{}}
2021-08-19T09:38:15 [I|app|d95d4c16] Completed 200 OK in 113ms (Views: 0.8ms | ActiveRecord: 4.6ms | Allocations: 76036)
2021-08-19T09:38:17 [I|app|33661cb3] Started POST "/foreman_tasks/api/tasks/bulk_search" for 172.30.1.105 at 2021-08-19 09:38:17 +0200
2021-08-19T09:38:17 [I|app|33661cb3] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML
2021-08-19T09:38:17 [I|app|33661cb3]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"de4cad41-dcdc-4a63-93bf-ddaa53353b23", "search_id"=>"1"}], "task"=>{}}
2021-08-19T09:38:17 [E|bac|5783a777] XML file(s): filelists not found (Katello::Errors::Pulp3Error)
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:22:in `run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/cancellable.rb:14:in `run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.2.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.2.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in `run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `catch'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `block in execute_run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in `execute_run'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 5783a777 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
2

Well, I guess the obvious reason for this would be that there is no filelists entry in repomd.xml.

3

Thank you for your reply,
the repomd.xml contains entries

<?xml version="1.0"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
  <data type="other">
    <location href="repodata/aef9db49b559d3069d52e435dd18f785a0cdd659-other.xml.gz"/>
    <checksum type="sha" pkgid="YES">aef9db49b559d3069d52e435dd18f785a0cdd659</checksum>
    <size>278995</size>
    <timestamp>1623074289</timestamp>
    <open-checksum type="sha" pkgid="YES">9b93f449975def6bf3538d3d7cf4c10423165288</open-checksum>
    <open-size>1567783</open-size>
    <revision/>
  </data>
  <data type="primary">
    <location href="repodata/f16a0655f6b7a3693d66d12018a432ad6c6afdc4-primary.xml.gz"/>
    <checksum type="sha" pkgid="YES">f16a0655f6b7a3693d66d12018a432ad6c6afdc4</checksum>
    <size>46726</size>
    <timestamp>1623074289</timestamp>
    <open-checksum type="sha" pkgid="YES">f03c9761124f4e1409e303287cb8351a777fb593</open-checksum>
    <open-size>470918</open-size>
    <revision/>
  </data>
</repomd>

both files in location are also available and could be succesfully downloaded using wget.

4

So there is no

<data type="filelists">
...
</data>

That’s missing. It’s an essential part of a repository and contains the files contained in each package. It seems the repodata was not created by createrepo…

5

Thank you!!! I was not aware of that.
That would explain a lot indeed.
I will see if this can be fixed and give an update on this thread as soon as I know more

6

Can’t you just mirror the files somewhere else and then do a createrepo?

I have a similar problem with the Zimbra repository that lives on AWS and needs s3-cmd to be downloaded.

I do that somewhere, export it via nginx and sync from that location.

I know it’s a bit silly, and I wish all this crazy crap would just stop and they’d just build “normal” repos - but this might be one of the cases where going the path of least resistance is just the best…

7

Thank you for your response, I appreciate you thinking with me for pragmatic solutions.
I have currently reached out to our vendor. Even though my contact is on vacation, I presume there must be at least 1 of his colleagues that could recreate the repository(metadata).
So I will first try that path, before making all sorts of constructions for which we become responsible.

8

I just spoke to our vendor and their DevOps engineer that set it up(our contact) will be back monday, so agreed to wait for him to return. Will keep you posted.

9

Update, this morning I have had contact with the vendor and they have been able to fix the repomd.xml after which Foreman has been able to succesfully synchronise.