CentOS repository sync frustrations

Support
1

I’m having a problem synchronizing repositories for CentOS, whether remote or locally hosted.

I’d like to see the repositories fully loaded

foreman-2.4.0-1.el7.noarch
katello-4.0.0-1.el7.noarch
This is running on a very recent install of CentOS 7, fully patched, in an attempt to start from scratch.

$ hammer repository synchronize --id 8
[.......................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: An invalid remote URL was provided.
ahdcos019:<jkalc>:(/home/jkalc/katello)
$ wget -O - -q http://localhost/pub/repos/centos7/base
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /pub/repos/centos7/base</title>
 </head>
 <body>
<h1>Index of /pub/repos/centos7/base</h1>
  <table>
   <tr><th valign="top"><img src="/icons/blank.gif" alt="[ICO]"></th><th><a href="?C=N;O=D">Name</a></th><th><a href="?C=M;O=A">Last modified</a></th><th><a href="?C=S;O=A">Size</a></th><th><a href="?C=D;O=A">Description</a></th></tr>
   <tr><th colspan="5"><hr></th></tr>
<tr><td valign="top"><img src="/icons/back.gif" alt="[PARENTDIR]"></td><td><a href="/pub/repos/centos7/">Parent Directory</a></td><td>&nbsp;</td><td align="right">  - </td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/folder.gif" alt="[DIR]"></td><td><a href="Packages/">Packages/</a></td><td align="right">2021-05-19 16:41  </td><td align="right">  - </td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/compressed.gif" alt="[   ]"></td><td><a href="a4e2b46586aa556c3b6f814dad5b16db5a669984d66b68e873586cd7c7253301-c7-x86_64-comps.xml.gz">a4e2b46586aa556c3b6f814dad5b16db5a669984d66b68e873586cd7c7253301-c7-x86_64-comps.xml.gz</a></td><td align="right">2021-05-19 16:28  </td><td align="right">153K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/text.gif" alt="[TXT]"></td><td><a href="comps.xml">comps.xml</a></td><td align="right">2021-05-19 16:28  </td><td align="right">727K</td><td>&nbsp;</td></tr>
   <tr><th colspan="5"><hr></th></tr>
</table>
</body></html>

I get the “Error: An invalid remote URL was provided” generally when I use the CentOS mirrorlist URL, an element of the mirrorlist, or even when I host a reposync’ed copy locally. I’ve gone so far as to even run the foreman-installer script with a --reset-data option to completely reset my environment, load the content-credentials GPG keys, recreate products and supporting repositories as clean as I can (without reprovisioning the base OS.) EPEL, Oracle Linux, etc. all appear to be synchronizing properly, it’s only the CentOS repositories that are showing this issue.

Am I missing a support code module of some sort? python-urlgrabber is installed and apparently getting called.

2

It’s difficult to say if you don’t post the exact URL you have tried or or the more detailed error from the Tasks page or production.log.

# hammer repository info --id 8

would show the repository configuration.

3

Whoops…

$ hammer repository info --id 8
Error writing to temporary file
Id:                 8
Name:               CentOS 7
Label:              centos7
Description:
Organization:       Land O' Lakes
Red Hat Repository: no
Content Type:       yum
Mirror on Sync:     no
Url:                http://localhost/pub/repos/centos7/base
Publish Via HTTP:   yes
Published At:       https://localhost/pulp/content/Land_O_Lakes/Library/custom/centos7/centos7/
Relative Path:      Land_O_Lakes/Library/custom/centos7/centos7
Download Policy:    immediate
HTTP Proxy:
    HTTP Proxy Policy: global_default_http_proxy
Product:
    Id:   2
    Name: CentOS 7
GPG Key:
    Id:   14
    Name: RPM-GPG-KEY-CentOS-7
Sync:
    Status:         Warning
    Last Sync Date: about 1 hour
Created:            2021/05/20 12:53:41
Updated:            2021/05/20 12:59:01
Content Counts:
    Packages:       0
    Source RPMS:    0
    Package Groups: 0
    Errata:         0
    Module Streams: 0

I’ve also tried “http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os” as well as some random elements from what a wget against the mirrorlist URL returns.

4

Well, that random element you have tried would be the interesting thing, in particular really set and synced for that repository.

I am using http://mirror.centos.org/centos/7/os/x86_64/ without issues.

Do you have to use a proxy for internet access?

What about the error messages from Tasks or production.log?

5

That’s interesting… you’re using a slightly different mirror URL than I am.

I’m probably going to end up mirroring all of the CentOS stuff locally anyway, as I want to get the errata loaded. Given what’s been happening w/ Pulp2 -> Pulp3, getting the updateinfo.xml files in the repositories may be my best bet.

6

I cannot tell if it is interesting or maybe even the reason for your problems if you don’t post the exact URL you have used…

7

Uh… I have posted that mirror url.

Regardless, looks like part of my local reposync didn’t complete. That’s been addressed, and at least a local sync appears to be in process.

Stay tuned…

8

You have posted your localhost URL and the mirrorlist URL but not the other mirror URL (“random element”/“element of the mirrorlist”) you have tried.

9

Okay, back to it… on another freshly provisioned CentOS 7 host, fully patched, and up to date Foreman & Katello installs (with the work-around for the current qpid issue.) This system is at home, no proxies in use (nor at my client, direct internet access works.)

Working on a smaller repository, for the CentOS 7 Extras repo. Using http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=extras as the remote URL fails, with the same invalid URL.

I’ve executed the following:

for url in $(wget -qO - 'http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=extras')
do  
        echo hammer repository update --id 9 --url $url
        echo hammer repository synchronize --id 9
        hammer repository update --id 9 --url $url
        hammer repository synchronize --id 9
        if [[ $? -eq 0 ]]
        then
                break
        fi
done
hammer repository update --id 9 --url http://mirror.grid.uchicago.edu/pub/linux/centos/7.9.2009/extras/x86_64/
hammer repository synchronize --id 9
Repository updated.
[...........................................................................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: An invalid remote URL was provided.
hammer repository update --id 9 --url http://mirror.us.oneandone.net/linux/distributions/centos/7.9.2009/extras/x86_64/
hammer repository synchronize --id 9
Repository updated.
[...........................................................................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: An invalid remote URL was provided.
hammer repository update --id 9 --url http://mirror.sjc02.svwh.net/centos/7.9.2009/extras/x86_64/
hammer repository synchronize --id 9
Repository updated.
[...........................................................................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: An invalid remote URL was provided.
hammer repository update --id 9 --url http://mirror.vtti.vt.edu/centos/7.9.2009/extras/x86_64/
hammer repository synchronize --id 9
Repository updated.
[...........................................................................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: An invalid remote URL was provided.
hammer repository update --id 9 --url http://centos.mirror.shastacoe.net/centos/7.9.2009/extras/x86_64/
hammer repository synchronize --id 9
Repository updated.
[...........................................................................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: An invalid remote URL was provided.
hammer repository update --id 9 --url http://mirror.twinlakes.net/centos/7.9.2009/extras/x86_64/
hammer repository synchronize --id 9
Repository updated.
[...........................................................................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: An invalid remote URL was provided.
hammer repository update --id 9 --url http://repos.hou.layerhost.com/centos/7.9.2009/extras/x86_64/
hammer repository synchronize --id 9
Repository updated.
[...........................................................................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: An invalid remote URL was provided.
hammer repository update --id 9 --url http://linux.cc.lehigh.edu/centos/7.9.2009/extras/x86_64/
hammer repository synchronize --id 9
Repository updated.
[...........................................................................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: An invalid remote URL was provided.
hammer repository update --id 9 --url http://mirror.phx1.us.spryservers.net/centos/7.9.2009/extras/x86_64/
hammer repository synchronize --id 9
Repository updated.
[...........................................................................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: An invalid remote URL was provided.
hammer repository update --id 9 --url http://mirrors.maine.edu/CentOS/7.9.2009/extras/x86_64/
hammer repository synchronize --id 9
Repository updated.
[...........................................................................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: An invalid remote URL was provided.

An example from the final sync run, from /var/log/foreman/production.log:

2021-05-21T10:40:11 [I|app|8aa05b75] Completed 200 OK in 178ms (Views: 53.8ms | ActiveRecord: 13.1ms | Allocations: 51848)
2021-05-21T10:40:12 [I|app|bc0bfdba] Started PUT "/katello/api/repositories/9" for 10.24.68.145 at 2021-05-21 10:40:12 -0500
2021-05-21T10:40:12 [I|app|bc0bfdba] Processing by Katello::Api::V2::RepositoriesController#update as JSON
2021-05-21T10:40:12 [I|app|bc0bfdba]   Parameters: {"url"=>"http://mirrors.maine.edu/CentOS/7.9.2009/extras/x86_64/", "api_version"=>"v2", "id"=>"9", "repository"=>{"url"=>"http://mirrors.maine.edu/CentOS/7.9.2009/extras/x86_64/"}}
2021-05-21T10:40:13 [I|app|bc0bfdba] Authorized user admin(Admin User)
2021-05-21T10:40:13 [I|bac|bc0bfdba] Task {label: Actions::Katello::Repository::Update, id: fbf2ce6f-4da8-4b45-b836-33232a9123cd, execution_plan_id: 2acecf7d-f83b-45ae-b0ef-728ad8c54915} state changed: planning 
2021-05-21T10:40:13 [I|aud|bc0bfdba] Katello::RootRepository (9) update event on url http://mirror.phx1.us.spryservers.net/centos/7.9.2009/extras/x86_64/, http://mirrors.maine.edu/CentOS/7.9.2009/extras/x86_64/
2021-05-21T10:40:13 [I|bac|bc0bfdba] Task {label: Actions::Katello::Repository::Update, id: fbf2ce6f-4da8-4b45-b836-33232a9123cd, execution_plan_id: 2acecf7d-f83b-45ae-b0ef-728ad8c54915} state changed: planned 
2021-05-21T10:40:13 [I|bac|bc0bfdba] Task {label: Actions::Katello::Repository::Update, id: fbf2ce6f-4da8-4b45-b836-33232a9123cd, execution_plan_id: 2acecf7d-f83b-45ae-b0ef-728ad8c54915} state changed: running 
2021-05-21T10:40:14 [I|bac|bc0bfdba] Task {label: Actions::Katello::Repository::MetadataGenerate, id: 7e62154b-9578-4621-90dc-3f732e99787e, execution_plan_id: 80577701-1e17-4e51-a283-1e9a9ebb5140} state changed: planning 
2021-05-21T10:40:14 [I|bac|bc0bfdba] Task {label: Actions::Katello::Repository::MetadataGenerate, id: 7e62154b-9578-4621-90dc-3f732e99787e, execution_plan_id: 80577701-1e17-4e51-a283-1e9a9ebb5140} state changed: planned 
2021-05-21T10:40:14 [I|bac|bc0bfdba] Task {label: Actions::Katello::Repository::MetadataGenerate, id: 7e62154b-9578-4621-90dc-3f732e99787e, execution_plan_id: 80577701-1e17-4e51-a283-1e9a9ebb5140} state changed: running 
2021-05-21T10:40:14 [I|bac|bc0bfdba] Task {label: Actions::Katello::Repository::Update, id: fbf2ce6f-4da8-4b45-b836-33232a9123cd, execution_plan_id: 2acecf7d-f83b-45ae-b0ef-728ad8c54915} state changed: stopped  result: success
2021-05-21T10:40:14 [I|bac|bc0bfdba] Task {label: Actions::Katello::Repository::Update, id: fbf2ce6f-4da8-4b45-b836-33232a9123cd, execution_plan_id: 2acecf7d-f83b-45ae-b0ef-728ad8c54915} state changed: stopped  result: success
2021-05-21T10:40:14 [I|app|bc0bfdba]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/views/katello/api/v2/repositories/show.json.rabl within katello/api/v2/layouts/resource
2021-05-21T10:40:14 [I|app|bc0bfdba]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/views/katello/api/v2/repositories/show.json.rabl within katello/api/v2/layouts/resource (Duration: 55.0ms | Allocations: 23371)
2021-05-21T10:40:14 [I|app|bc0bfdba] Completed 200 OK in 1923ms (Views: 40.1ms | ActiveRecord: 54.7ms | Allocations: 115551)
2021-05-21T10:40:15 [I|aud|bc0bfdba] Katello::Repository (9) update event on publication_href /pulp/api/v3/publications/rpm/rpm/b2642ba3-a3c3-47ce-b525-91f9e21c1430/, /pulp/api/v3/publications/rpm/rpm/5b00b1e2-7aef-4ddd-b200-17add02fd04e/
2021-05-21T10:40:16 [I|bac|bc0bfdba] Task {label: Actions::Katello::Repository::MetadataGenerate, id: 7e62154b-9578-4621-90dc-3f732e99787e, execution_plan_id: 80577701-1e17-4e51-a283-1e9a9ebb5140} state changed: stopped  result: success
2021-05-21T10:40:16 [I|bac|bc0bfdba] Task {label: Actions::Katello::Repository::MetadataGenerate, id: 7e62154b-9578-4621-90dc-3f732e99787e, execution_plan_id: 80577701-1e17-4e51-a283-1e9a9ebb5140} state changed: stopped  result: success
2021-05-21T10:40:16 [I|app|b773c911] Started POST "/katello/api/repositories/9/sync" for 10.24.68.145 at 2021-05-21 10:40:16 -0500
2021-05-21T10:40:16 [I|app|b773c911] Processing by Katello::Api::V2::RepositoriesController#sync as JSON
2021-05-21T10:40:16 [I|app|b773c911]   Parameters: {"api_version"=>"v2", "id"=>"9", "repository"=>{}}
2021-05-21T10:40:16 [I|app|b773c911] Authorized user admin(Admin User)
2021-05-21T10:40:16 [I|bac|b773c911] Task {label: Actions::Katello::Repository::Sync, id: 7383e18f-3fc0-4952-bf8a-ecd9ce50e29b, execution_plan_id: 5b0f579a-7509-470a-839f-8b3f7b019f97} state changed: planning 
2021-05-21T10:40:16 [I|bac|b773c911] Task {label: Actions::Katello::Repository::Sync, id: 7383e18f-3fc0-4952-bf8a-ecd9ce50e29b, execution_plan_id: 5b0f579a-7509-470a-839f-8b3f7b019f97} state changed: planned 
2021-05-21T10:40:16 [I|app|b773c911]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource
2021-05-21T10:40:16 [I|app|b773c911]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource (Duration: 31.6ms | Allocations: 36673)
2021-05-21T10:40:16 [I|app|b773c911] Completed 202 Accepted in 464ms (Views: 34.3ms | ActiveRecord: 27.5ms | Allocations: 157905)
2021-05-21T10:40:16 [I|app|060f0b4f] Started GET "/foreman_tasks/api/tasks/7383e18f-3fc0-4952-bf8a-ecd9ce50e29b" for 10.24.68.145 at 2021-05-21 10:40:16 -0500
2021-05-21T10:40:16 [I|app|060f0b4f] Processing by ForemanTasks::Api::TasksController#show as JSON
2021-05-21T10:40:16 [I|app|060f0b4f]   Parameters: {"id"=>"7383e18f-3fc0-4952-bf8a-ecd9ce50e29b", "task"=>{}}
2021-05-21T10:40:16 [I|bac|b773c911] Task {label: Actions::Katello::Repository::Sync, id: 7383e18f-3fc0-4952-bf8a-ecd9ce50e29b, execution_plan_id: 5b0f579a-7509-470a-839f-8b3f7b019f97} state changed: running 
2021-05-21T10:40:17 [I|app|060f0b4f] Authorized user admin(Admin User)
2021-05-21T10:40:17 [I|app|060f0b4f]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/views/foreman_tasks/api/tasks/show.json.rabl
2021-05-21T10:40:17 [I|app|060f0b4f]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/views/foreman_tasks/api/tasks/show.json.rabl (Duration: 45.1ms | Allocations: 40774)
2021-05-21T10:40:17 [I|app|060f0b4f] Completed 200 OK in 157ms (Views: 45.6ms | ActiveRecord: 10.8ms | Allocations: 50368)
2021-05-21T10:40:17 [E|bac|b773c911] An invalid remote URL was provided. (Katello::Errors::Pulp3Error)
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/pulp3/abstract_async_task.rb:100:in `each'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/pulp3/abstract_async_task.rb:100:in `check_for_errors'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/pulp3/abstract_async_task.rb:133:in `poll_external_task'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 b773c911 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 b773c911 | [ sidekiq ]
 b773c911 | [ concurrent-ruby ]

Looking into the Foreman/Katello GUI console, in Monitor/Tasks, the tasks are in a stopped state with a warning result. The contents of the Errors tab appear to be what I see in the production.log file. The exception reads: Katello::Errors::Pulp3Error: An invalid remote URL was provided.

None of the URLs appear to work, from the initial mirrorlist URL, to the contents of the mirrorlist URLs.

Is it possible that I’m simply missing a package (python? ruby?) to successfully parse the contents of the remote repositories?

10

I do not think Pulp and thus Katello handles mirrorlists. @Justin_Sherrill can you confirm?

1 Like
11

Uh oh… I’m pretty darned sure that they used to… but even so, if I burn through the mirrors (yes I can script that to handle updates,) the individual elements thereof still fail. :frowning:

12

I hope I am wrong too :slight_smile: I do agree the URL looks to be valid but the code does not think it is. The error appears to be coming from Pulp itself and how it is validating repo URLs.

13

I’d agree that it appears to be a Pulp3 issue, I have no idea how to solve this. This is completely a trial server, I can do anything necessary to diagnose & fix this.

14

The error message seems to be coming from here:

which uses this function

Looks to me as if it tries to access http://mirrors.maine.edu/CentOS/7.9.2009/extras/x86_64/repodata/repomd.xml and it fails.

Try

  curl http://mirrors.maine.edu/CentOS/7.9.2009/extras/x86_64/repodata/repomd.xml

on the server to see if you can download the file.

If it works, my first assumption for mysterious issues likes this would be SeLinux. Check with

# ausearch -m avc

and check if selinux denied access at the time of the error message.

15

The curl retrieval of the repomd.xml file succeeds.

This is interesting… output from the ‘ausearch -m avc’ command:

----
time->Thu May 20 21:02:07 2021
type=PROCTITLE msg=audit(1621562527.046:1147): proctitle=2F7573722F62696E2F707974686F6E33002F7573722F62696E2F67756E69636F726E0070756
C70636F72652E6170702E777367693A6170706C69636174696F6E002D770031002D2D6163636573732D6C6F6766696C65002D
type=SYSCALL msg=audit(1621562527.046:1147): arch=c000003e syscall=41 success=yes exit=3 a0=1 a1=80002 a2=0 a3=0 items=0 ppid=1 pid=
4457 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295 comm=
"gunicorn" exe="/usr/bin/python3.6" subj=system_u:system_r:pulpcore_server_t:s0 key=(null)
type=AVC msg=audit(1621562527.046:1147): avc:  denied  { create } for  pid=4457 comm="gunicorn" scontext=system_u:system_r:pulpcore_
server_t:s0 tcontext=system_u:system_r:pulpcore_server_t:s0 tclass=unix_dgram_socket permissive=1
----
time->Thu May 20 21:02:07 2021
type=PROCTITLE msg=audit(1621562527.046:1148): proctitle=2F7573722F62696E2F707974686F6E33002F7573722F62696E2F67756E69636F726E0070756
C70636F72652E6170702E777367693A6170706C69636174696F6E002D770031002D2D6163636573732D6C6F6766696C65002D
type=SYSCALL msg=audit(1621562527.046:1148): arch=c000003e syscall=42 success=yes exit=0 a0=3 a1=7ffd51ea91f0 a2=15 a3=f0 items=0 pp
id=1 pid=4457 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967
295 comm="gunicorn" exe="/usr/bin/python3.6" subj=system_u:system_r:pulpcore_server_t:s0 key=(null)
type=AVC msg=audit(1621562527.046:1148): avc:  denied  { sendto } for  pid=4457 comm="gunicorn" path="/run/systemd/notify" scontext=
system_u:system_r:pulpcore_server_t:s0 tcontext=system_u:system_r:kernel_t:s0 tclass=unix_dgram_socket permissive=1
type=AVC msg=audit(1621562527.046:1148): avc:  denied  { connect } for  pid=4457 comm="gunicorn" scontext=system_u:system_r:pulpcore
_server_t:s0 tcontext=system_u:system_r:pulpcore_server_t:s0 tclass=unix_dgram_socket permissive=1
----
time->Thu May 20 21:02:08 2021
type=PROCTITLE msg=audit(1621562528.399:1150): proctitle=2F7573722F62696E2F707974686F6E33002F7573722F62696E2F67756E69636F726E0070756
C70636F72652E636F6E74656E743A736572766572002D2D776F726B65722D636C6173730061696F687474702E47756E69636F726E576562576F726B6572002D77003
9002D2D6163636573732D6C6F6766696C65002D
type=SYSCALL msg=audit(1621562528.399:1150): arch=c000003e syscall=41 success=yes exit=3 a0=1 a1=80002 a2=0 a3=0 items=0 ppid=1 pid=
4630 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295 comm=
"gunicorn" exe="/usr/bin/python3.6" subj=system_u:system_r:pulpcore_server_t:s0 key=(null)
type=AVC msg=audit(1621562528.399:1150): avc:  denied  { create } for  pid=4630 comm="gunicorn" scontext=system_u:system_r:pulpcore_
server_t:s0 tcontext=system_u:system_r:pulpcore_server_t:s0 tclass=unix_dgram_socket permissive=1
----
time->Thu May 20 21:02:08 2021
type=PROCTITLE msg=audit(1621562528.399:1151): proctitle=2F7573722F62696E2F707974686F6E33002F7573722F62696E2F67756E69636F726E0070756
C70636F72652E636F6E74656E743A736572766572002D2D776F726B65722D636C6173730061696F687474702E47756E69636F726E576562576F726B6572002D77003
9002D2D6163636573732D6C6F6766696C65002D
type=SYSCALL msg=audit(1621562528.399:1151): arch=c000003e syscall=42 success=yes exit=0 a0=3 a1=7ffd7da14180 a2=15 a3=ae0 items=0 p
pid=1 pid=4630 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=429496
7295 comm="gunicorn" exe="/usr/bin/python3.6" subj=system_u:system_r:pulpcore_server_t:s0 key=(null)
type=AVC msg=audit(1621562528.399:1151): avc:  denied  { sendto } for  pid=4630 comm="gunicorn" path="/run/systemd/notify" scontext=
system_u:system_r:pulpcore_server_t:s0 tcontext=system_u:system_r:kernel_t:s0 tclass=unix_dgram_socket permissive=1
type=AVC msg=audit(1621562528.399:1151): avc:  denied  { connect } for  pid=4630 comm="gunicorn" scontext=system_u:system_r:pulpcore
_server_t:s0 tcontext=system_u:system_r:pulpcore_server_t:s0 tclass=unix_dgram_socket permissive=1
----
time->Fri May 21 10:26:43 2021
type=PROCTITLE msg=audit(1621610803.353:2288): proctitle=2F7573722F62696E2F707974686F6E33002F7573722F62696E2F67756E69636F726E0070756
C70636F72652E6170702E777367693A6170706C69636174696F6E002D770031002D2D6163636573732D6C6F6766696C65002D
type=SYSCALL msg=audit(1621610803.353:2288): arch=c000003e syscall=41 success=yes exit=3 a0=1 a1=80002 a2=0 a3=0 items=0 ppid=1 pid=
2873 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295 comm=
"gunicorn" exe="/usr/bin/python3.6" subj=system_u:system_r:pulpcore_server_t:s0 key=(null)
type=AVC msg=audit(1621610803.353:2288): avc:  denied  { create } for  pid=2873 comm="gunicorn" scontext=system_u:system_r:pulpcore_
server_t:s0 tcontext=system_u:system_r:pulpcore_server_t:s0 tclass=unix_dgram_socket permissive=1
----
time->Fri May 21 10:26:43 2021
type=PROCTITLE msg=audit(1621610803.353:2289): proctitle=2F7573722F62696E2F707974686F6E33002F7573722F62696E2F67756E69636F726E0070756
C70636F72652E6170702E777367693A6170706C69636174696F6E002D770031002D2D6163636573732D6C6F6766696C65002D
type=SYSCALL msg=audit(1621610803.353:2289): arch=c000003e syscall=42 success=yes exit=0 a0=3 a1=7ffd70ab34d0 a2=15 a3=f0 items=0 pp
id=1 pid=2873 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967
295 comm="gunicorn" exe="/usr/bin/python3.6" subj=system_u:system_r:pulpcore_server_t:s0 key=(null)
type=AVC msg=audit(1621610803.353:2289): avc:  denied  { sendto } for  pid=2873 comm="gunicorn" path="/run/systemd/notify" scontext=
system_u:system_r:pulpcore_server_t:s0 tcontext=system_u:system_r:kernel_t:s0 tclass=unix_dgram_socket permissive=1
type=AVC msg=audit(1621610803.353:2289): avc:  denied  { connect } for  pid=2873 comm="gunicorn" scontext=system_u:system_r:pulpcore
_server_t:s0 tcontext=system_u:system_r:pulpcore_server_t:s0 tclass=unix_dgram_socket permissive=1
----
time->Fri May 21 10:26:44 2021
type=PROCTITLE msg=audit(1621610804.195:2291): proctitle=2F7573722F62696E2F707974686F6E33002F7573722F62696E2F67756E69636F726E0070756
C70636F72652E636F6E74656E743A736572766572002D2D776F726B65722D636C6173730061696F687474702E47756E69636F726E576562576F726B6572002D77003
9002D2D6163636573732D6C6F6766696C65002D
type=SYSCALL msg=audit(1621610804.195:2291): arch=c000003e syscall=41 success=yes exit=3 a0=1 a1=80002 a2=0 a3=0 items=0 ppid=1 pid=
2970 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295 comm=
"gunicorn" exe="/usr/bin/python3.6" subj=system_u:system_r:pulpcore_server_t:s0 key=(null)
type=AVC msg=audit(1621610804.195:2291): avc:  denied  { create } for  pid=2970 comm="gunicorn" scontext=system_u:system_r:pulpcore_
server_t:s0 tcontext=system_u:system_r:pulpcore_server_t:s0 tclass=unix_dgram_socket permissive=1
----
time->Fri May 21 10:26:44 2021
type=PROCTITLE msg=audit(1621610804.195:2292): proctitle=2F7573722F62696E2F707974686F6E33002F7573722F62696E2F67756E69636F726E0070756
C70636F72652E636F6E74656E743A736572766572002D2D776F726B65722D636C6173730061696F687474702E47756E69636F726E576562576F726B6572002D77003
9002D2D6163636573732D6C6F6766696C65002D
type=SYSCALL msg=audit(1621610804.195:2292): arch=c000003e syscall=42 success=yes exit=0 a0=3 a1=7ffd6fd57280 a2=15 a3=ae0 items=0 p
pid=1 pid=2970 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=429496
7295 comm="gunicorn" exe="/usr/bin/python3.6" subj=system_u:system_r:pulpcore_server_t:s0 key=(null)
type=AVC msg=audit(1621610804.195:2292): avc:  denied  { sendto } for  pid=2970 comm="gunicorn" path="/run/systemd/notify" scontext=
system_u:system_r:pulpcore_server_t:s0 tcontext=system_u:system_r:kernel_t:s0 tclass=unix_dgram_socket permissive=1
type=AVC msg=audit(1621610804.195:2292): avc:  denied  { connect } for  pid=2970 comm="gunicorn" scontext=system_u:system_r:pulpcore
_server_t:s0 tcontext=system_u:system_r:pulpcore_server_t:s0 tclass=unix_dgram_socket permissive=1

I didn’t get foreman-installer to run successfully until this morning, after the fix for qpid was posted. To my very untrained eye, there are fewer entries there than I would expect from the multiple sync errors this morning.

16

They are all with permissive=1, so nothing has been blocked. It must be something else…

17

To be clear, the url you are syncing is http://localhost/pub/repos/centos7/base ?

You mention its a mirror list, but also that its a local mirror?

can you clarify exactly what is located at that location? From the wget you posted, it looks like it did not include ./repodata/repomd.xml

18

Good morning, Jusin.

It appears that a lot of repositories using mirror lists behind their base URL seem to be failing, even after a morning upgrade (CentOS and EPEL are the big ones for me.) I’ve had to resort to using reposync to copy down content (picked the Katello host to hold it,) to get a sync to succeed.

19

when you were trying with a mirror list, what did your mirror list look like? syncing something like this: http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os
seems to work just fine with pulp 3.11.

20

Unfortunately, not here, it doesn’t.

jkalchik@katello katello]$ hammer repository synchronize --id 8
    [...........................................................................................................................] [100%]
    Total steps: 0/0
    --------------------------------
    Error: An invalid remote URL was provided.
    [jkalchik@katello katello]$ hammer repository info --id 8
    Id:                 8
    Name:               CentOS 7
    Label:              centos7
    Description:        
    Organization:       Land O' Lakes
    Red Hat Repository: no
    Content Type:       yum
    Mirror on Sync:     no
    Url:                http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os
    Publish Via HTTP:   yes
    Published At:       https://localhost/pulp/content/Land_O_Lakes/Library/custom/centos7/centos7/
    Relative Path:      Land_O_Lakes/Library/custom/centos7/centos7
    Download Policy:    immediate
    HTTP Proxy:         
        HTTP Proxy Policy: global_default_http_proxy
    Product:            
        Id:   2
        Name: CentOS 7
    GPG Key:            
        Id:   7
        Name: RPM-GPG-KEY-CentOS-7
    Sync:               
        Status:         Warning
        Last Sync Date: less than a minute
    Created:            2021/05/21 15:30:07
    Updated:            2021/05/21 15:30:09
    Content Counts:     
        Packages:       0
        Source RPMS:    0
        Package Groups: 0
        Errata:         0
        Module Streams: 0
jkalchik@katello katello]$ rpm -qa | grep -i pulp
tfm-rubygem-pulpcore_client-3.9.0-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.9.1-1.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_2.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.7.0-1.el7.noarch
pulpcore-selinux-1.2.4-1.el7.x86_64
python3-pulp-file-1.5.0-1.el7.noarch
python3-pulp-certguard-1.1.0-1.el7.noarch
python3-pulp-rpm-3.10.0-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.6.0-1.el7.noarch
python3-pulp-container-2.2.1-1.el7.noarch
python3-pulp-deb-2.9.1-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.10.0-1.el7.noarch
pulp-client-1.0-1.noarch
tfm-rubygem-pulp_file_client-1.5.0-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.3-1.el7.noarch
python3-pulpcore-3.9.1-2.el7.noarch
tfm-rubygem-pulp_container_client-2.2.0-1.el7.noarch