CentOS repository sync frustrations

Justin_Sherrill · May 24, 2021, 4:50pm

While i am on a slightly newer version of the rpm pulp plugin, the pulp team seems to indicate this is not a recent feature.

I’m wondering if you’re just having general sync issues related to connectivity?

is there an http proxy involved at all?
can you check the journalctl -u pulpcore-worker@* logs and see if you locate the errors related to that particular sync?

jkalchik · May 24, 2021, 6:55pm

No proxies involved in either of my test environments, both can connect directly outbound without intervention.

Potentially useful output from the journalctl cmd:

May 24 13:51:05 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: pulp [f5c0e6c1c9e54f00b0873c26723405f8]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=centos7-861
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: pulp [f5c0e6c1c9e54f00b0873c26723405f8]: rq.worker:ERROR: Traceback (most recent call last):
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: rv = job.perform()
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: self._result = self._execute()
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: return self.func(*self.args, **self.kwargs)
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 219, in synchronize
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: raise ValueError(_("An invalid remote URL was provided."))
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: ValueError: An invalid remote URL was provided.
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: Traceback (most recent call last):
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: rv = job.perform()
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: self._result = self._execute()
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: return self.func(*self.args, **self.kwargs)
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 219, in synchronize
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: raise ValueError(_("An invalid remote URL was provided."))
May 24 13:51:07 katello.asymptotedevelopment.com pulpcore-worker-4[6255]: ValueError: An invalid remote URL was provided.

I’m pretty sure that I was using mirror list URLs prior to trying to get on the latest stable release (installed early January 2021.)

If I have something improper, or am missing something, the installation process will probably have to be revisited.

jkalchik · May 24, 2021, 9:53pm

Strap in, this is gonna get a little rough…

[jkalchik@katello ~]$ wget -qO - 'http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=stock'
http://mirror.pit.teraswitch.com/centos/7.9.2009/os/x86_64/
http://mirrors.tummy.com/mirrors/CentOS/7.9.2009/os/x86_64/
http://distro.ibiblio.org/centos/7.9.2009/os/x86_64/
http://repos.lax.quadranet.com/centos/7.9.2009/os/x86_64/
http://mirror.atlanticmetro.net/centos/7.9.2009/os/x86_64/
http://ewr.edge.kernel.org/centos/7.9.2009/os/x86_64/
http://mirror.phx1.us.spryservers.net/centos/7.9.2009/os/x86_64/
http://mirror.chpc.utah.edu/pub/centos/7.9.2009/os/x86_64/
http://mirror.centos.lax1.serverforge.org/7.9.2009/os/x86_64/
http://atl.mirrors.clouvider.net/CentOS/7.9.2009/os/x86_64/
[jkalchik@katello ~]$ wget -qO - 'http://distro.ibiblio.org/centos/7.9.2009/os/x86_64/'
<!DOCTYPE html>
<html>
<head>
<title>Index of /centos/7.9.2009/os/x86_64/</title>
<style type="text/css">
a, a:active {text-decoration: none; color: blue;}
a:visited {color: #48468F;}
a:hover, a:focus {text-decoration: underline; color: red;}
body {background-color: #F5F5F5;}
h2 {margin-bottom: 12px;}
table {margin-left: 12px;}
th, td { font: 90% monospace; text-align: left;}
th { font-weight: bold; padding-right: 14px; padding-bottom: 3px;}
td {padding-right: 14px;}
td.s, th.s {text-align: right;}
div.list { background-color: white; border-top: 1px solid #646464; border-bottom: 1px solid #646464; padding-top: 10px; padding-bottom: 14px;}
div.foot { font: 90% monospace; color: #787878; padding-top: 4px;}
</style>
</head>
<body>
<h2>Index of /centos/7.9.2009/os/x86_64/</h2>
<div class="list">
<table summary="Directory Listing" cellpadding="0" cellspacing="0">
<thead><tr><th class="n">Name</th><th class="m">Last Modified</th><th class="s">Size</th><th class="t">Type</th></tr></thead>
<tbody>
<tr class="d"><td class="n"><a href="../">..</a>/</td><td class="m">&nbsp;</td><td class="s">- &nbsp;</td><td class="t">Directory</td></tr>
<tr class="d"><td class="n"><a href="EFI/">EFI</a>/</td><td class="m">2020-Oct-26 12:25:28</td><td class="s">- &nbsp;</td><td class="t">Directory</td></tr>
<tr class="d"><td class="n"><a href="LiveOS/">LiveOS</a>/</td><td class="m">2020-Oct-26 12:25:27</td><td class="s">- &nbsp;</td><td class="t">Directory</td></tr>
<tr class="d"><td class="n"><a href="Packages/">Packages</a>/</td><td class="m">2020-Nov-05 05:37:01</td><td class="s">- &nbsp;</td><td class="t">Directory</td></tr>
<tr class="d"><td class="n"><a href="images/">images</a>/</td><td class="m">2020-Oct-26 12:26:06</td><td class="s">- &nbsp;</td><td class="t">Directory</td></tr>
<tr class="d"><td class="n"><a href="isolinux/">isolinux</a>/</td><td class="m">2020-Nov-05 05:23:16</td><td class="s">- &nbsp;</td><td class="t">Directory</td></tr>
<tr class="d"><td class="n"><a href="repodata/">repodata</a>/</td><td class="m">2020-Nov-12 11:20:15</td><td class="s">- &nbsp;</td><td class="t">Directory</td></tr>
<tr><td class="n"><a href="CentOS_BuildTag">CentOS_BuildTag</a></td><td class="m">2020-Oct-29 17:14:44</td><td class="s">0.1K</td><td class="t">application/octet-stream</td></tr>
<tr><td class="n"><a href="EULA">EULA</a></td><td class="m">2017-Aug-30 10:33:06</td><td class="s">0.2K</td><td class="t">application/octet-stream</td></tr>
<tr><td class="n"><a href="GPL">GPL</a></td><td class="m">2015-Dec-09 17:35:46</td><td class="s">17.6K</td><td class="t">application/octet-stream</td></tr>
<tr><td class="n"><a href="RPM-GPG-KEY-CentOS-7">RPM-GPG-KEY-CentOS-7</a></td><td class="m">2015-Dec-09 17:35:46</td><td class="s">1.6K</td><td class="t">application/octet-stream</td></tr>
<tr><td class="n"><a href="RPM-GPG-KEY-CentOS-Testing-7">RPM-GPG-KEY-CentOS-Testing-7</a></td><td class="m">2015-Dec-09 17:35:46</td><td class="s">1.6K</td><td class="t">application/octet-stream</td></tr>
</tbody>
</table>
</div>
<div class="foot">lighttpd/1.4.53</div>

<script type="text/javascript">
// <!--

var click_column;
var name_column = 0;
var date_column = 1;
var size_column = 2;
var type_column = 3;
var prev_span = null;

if (typeof(String.prototype.localeCompare) === 'undefined') {
 String.prototype.localeCompare = function(str, locale, options) {
   return ((this == str) ? 0 : ((this > str) ? 1 : -1));
 };
}

if (typeof(String.prototype.toLocaleUpperCase) === 'undefined') {
 String.prototype.toLocaleUpperCase = function() {
  return this.toUpperCase();
 };
}

function get_inner_text(el) {
 if((typeof el == 'string')||(typeof el == 'undefined'))
  return el;
 if(el.innerText)
  return el.innerText;
 else {
  var str = "";
  var cs = el.childNodes;
  var l = cs.length;
  for (i=0;i<l;i++) {
   if (cs[i].nodeType==1) str += get_inner_text(cs[i]);
   else if (cs[i].nodeType==3) str += cs[i].nodeValue;
  }
 }
 return str;
}

function isdigit(c) {
 return (c >= '0' && c <= '9');
}

function unit_multiplier(unit) {
 return (unit=='K') ? 1000
      : (unit=='M') ? 1000000
      : (unit=='G') ? 1000000000
      : (unit=='T') ? 1000000000000
      : (unit=='P') ? 1000000000000000
      : (unit=='E') ? 1000000000000000000 : 1;
}

var li_date_regex=/(\d{4})-(\w{3})-(\d{2}) (\d{2}):(\d{2}):(\d{2})/;

var li_mon = ['Jan','Feb','Mar','Apr','May','Jun',
              'Jul','Aug','Sep','Oct','Nov','Dec'];

function li_mon_num(mon) {
 var i; for (i = 0; i < 12 && mon != li_mon[i]; ++i); return i;
}

function li_date_cmp(s1, s2) {
 var dp1 = li_date_regex.exec(s1)
 var dp2 = li_date_regex.exec(s2)
 for (var i = 1; i < 7; ++i) {
  var cmp = (2 != i)
   ? parseInt(dp1[i]) - parseInt(dp2[i])
   : li_mon_num(dp1[2]) - li_mon_num(dp2[2]);
  if (0 != cmp) return cmp;
 }
 return 0;
}

function sortfn_then_by_name(a,b,sort_column) {
 if (sort_column == name_column || sort_column == type_column) {
  var ad = (a.cells[type_column].innerHTML === 'Directory');
  var bd = (b.cells[type_column].innerHTML === 'Directory');
  if (ad != bd) return (ad ? -1 : 1);
 }
 var at = get_inner_text(a.cells[sort_column]);
 var bt = get_inner_text(b.cells[sort_column]);
 var cmp;
 if (sort_column == name_column) {
  if (at == '..') return -1;
  if (bt == '..') return  1;
 }
 if (a.cells[sort_column].className == 'int') {
  cmp = parseInt(at)-parseInt(bt);
 } else if (sort_column == date_column) {
  var ad = isdigit(at.substr(0,1));
  var bd = isdigit(bt.substr(0,1));
  if (ad != bd) return (!ad ? -1 : 1);
  cmp = li_date_cmp(at,bt);
 } else if (sort_column == size_column) {
  var ai = parseInt(at, 10) * unit_multiplier(at.substr(-1,1));
  var bi = parseInt(bt, 10) * unit_multiplier(bt.substr(-1,1));
  if (at.substr(0,1) == '-') ai = -1;
  if (bt.substr(0,1) == '-') bi = -1;
  cmp = ai - bi;
 } else {
  cmp = at.toLocaleUpperCase().localeCompare(bt.toLocaleUpperCase());
  if (0 != cmp) return cmp;
  cmp = at.localeCompare(bt);
 }
 if (0 != cmp || sort_column == name_column) return cmp;
 return sortfn_then_by_name(a,b,name_column);
}

function sortfn(a,b) {
 return sortfn_then_by_name(a,b,click_column);
}

function resort(lnk) {
 var span = lnk.childNodes[1];
 var table = lnk.parentNode.parentNode.parentNode.parentNode;
 var rows = new Array();
 for (j=1;j<table.rows.length;j++)
  rows[j-1] = table.rows[j];
 click_column = lnk.parentNode.cellIndex;
 rows.sort(sortfn);

 if (prev_span != null) prev_span.innerHTML = '';
 if (span.getAttribute('sortdir')=='down') {
  span.innerHTML = '&uarr;';
  span.setAttribute('sortdir','up');
  rows.reverse();
 } else {
  span.innerHTML = '&darr;';
  span.setAttribute('sortdir','down');
 }
 for (i=0;i<rows.length;i++)
  table.tBodies[0].appendChild(rows[i]);
 prev_span = span;
}

function init_sort(init_sort_column, ascending) {
 var tables = document.getElementsByTagName("table");
 for (var i = 0; i < tables.length; i++) {
  var table = tables[i];
  //var c = table.getAttribute("class")
  //if (-1 != c.split(" ").indexOf("sort")) {
   var row = table.rows[0].cells;
   for (var j = 0; j < row.length; j++) {
    var n = row[j];
    if (n.childNodes.length == 1 && n.childNodes[0].nodeType == 3) {
     var link = document.createElement("a");
     var title = n.childNodes[0].nodeValue.replace(/:$/, "");
     link.appendChild(document.createTextNode(title));
     link.setAttribute("href", "#");
     link.setAttribute("class", "sortheader");
     link.setAttribute("onclick", "resort(this);return false;");
     var arrow = document.createElement("span");
     arrow.setAttribute("class", "sortarrow");
     arrow.appendChild(document.createTextNode(":"));
     link.appendChild(arrow)
     n.replaceChild(link, n.firstChild);
    }
   }
   var lnk = row[init_sort_column].firstChild;
   if (ascending) {
    var span = lnk.childNodes[1];
    span.setAttribute('sortdir','down');
   }
   resort(lnk);
  //}
 }
}

init_sort(0, 0);

// -->
</script>

</body>
</html>
[jkalchik@katello ~]$ hammer repository synchronize --id 8
[...........................................................................................................................] [100%]
Total steps: 3117/10185
--------------------------------
Associating Content: 0/0
Downloading Artifacts: 0/0
Downloading Metadata Files: 4/4
Parsed Comps: 109/109
Parsed Packages: 3004/10072
Error: 403, message='Forbidden', url=URL('http://distro.ibiblio.org/centos/7.9.2009/os/x86_64/LiveOS/squashfs.img/?release=7&arch=x86_64&repo=os&infra=stock')

Yah… I’m gonna be the first to admit that I definitely have some source URL issues.

I pulled the CentOS 7 base url right out of a ‘yum repolist -v’ and updated the Katello/Pulp3 repository configuration. A wget against that URL succeeds.

So… with a better mirror list URL configured in the repository w/ id=8, I ran the repository sync, and you can see the result. The distro.ibiblio.org element of the mirror list does succeed, and if I look at it either via wget or in a browser, I can see a LiveOS link. If I burn through that link, I see a single object for a squashfs.img file.

The URL that the repository sync now dumps out is correct through the squashfs.img part, but why is the ‘/?release=7&arch=x86_64&repo=os&infra=stock’ getting appended to a page analysis?

Justin_Sherrill · May 25, 2021, 4:42pm

ahh! I think this is a combination of a couple bugs:

katello treats all get params ( ?release=7&arch=x86_64&repo=os&infra=stock) as ‘auth tokens’ which are used by Suse as auth tokens to sync content. We pass this to pulp to do. This isn’t really desired in this situation and would generally be ignored except for
pulp seems to not use these auth tokens properly when downloading kickstart files and is constructing the url incorrectly by putting a / after.

I’ve opened two issues:
https://projects.theforeman.org/issues/32660
https://pulp.plan.io/issues/8816

rbremer · June 2, 2021, 1:47pm

This issue seems to be occurring in regular upgrades from Foreman 2.3 and Katello 3.18 too. After the migration some of my repositories throw the same error.

jkalchik · June 2, 2021, 2:04pm

Justin, in response to the question at the end of that bug report, none of what I’m working with are SLES repos (although I may eventually chase that for personal use at home.) These are CentOS, Oracle Linux, EPEL & the like.

jkalchik · June 2, 2021, 4:00pm

Just bringing this back around…

[jkalchik@katello ~]$ hammer repository update --id 8 --url 'http://mirrorlist.c
entos.org/?release=7&arch=x86_64&repo=os'
Repository updated.
[jkalchik@katello ~]$ hammer repository sync --id 8
[.......................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: An invalid remote URL was provided.
[jkalchik@katello ~]$ rpm -q pulp
package pulp is not installed
[jkalchik@katello ~]$ rpm -qa | grep pulp
tfm-rubygem-pulpcore_client-3.9.0-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.9.1-1.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_2.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.7.0-1.el7.noarch
pulpcore-selinux-1.2.4-1.el7.x86_64
python3-pulp-file-1.5.0-1.el7.noarch
python3-pulp-certguard-1.1.0-1.el7.noarch
python3-pulp-container-2.2.2-1.el7.noarch
python3-pulp-rpm-3.10.0-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.6.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.10.0-1.el7.noarch
pulp-client-1.0-1.noarch
tfm-rubygem-pulp_file_client-1.5.0-1.el7.noarch
python3-pulp-deb-2.9.2-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.3-1.el7.noarch
python3-pulpcore-3.9.1-2.el7.noarch
tfm-rubygem-pulp_container_client-2.2.0-1.el7.noarc

I’ve set the URL of my base CentOS 7 repository to the value you’ve been using, and it’s failing. As you’ve noted, you’re on a newer release of Pulp3. Any idea of when that release of Pulp3 will see primetime in a Katello release?

rbremer · June 3, 2021, 2:02pm

For me the error occurs with a direct URL too, not just when using a mirror list.

rbremer · June 4, 2021, 11:14am

So… something interesting happened this morning. I was able to successful sync 3 out of 4 failing repositories. Now I do have a theory, though. Is it possible that the issue is caused by the actual mirror server Pulp talks to?
I tested yesterday to take the first link of the mirror list and used it directly at the repo, and I had the same error (An invalid remote URL was provided.). Of course I though this is a permanent problem. I reverted the config back to the mirror list and over night the sync plan again had the same error message in its logs.
However, this morning I could manually run hammer sync and it came up successful. Since there is no way to tell which mirror it used (no sync logs … a bit frustrating to be honest), I can only assume that it jumped to a different mirror today and could successfully synchronize.

jkalchik · June 7, 2021, 1:51pm

I concur. I’ve been keeping up on patching my off-site test host, but repository syncs seem to be working a lot better. I have to wonder if the source pages for the mirrorlist have been updated and we’ve gotten in the middle.

Justin_Sherrill · June 8, 2021, 2:52pm

its possible that some mirrors ignore those extra characters on the request and some throw errors, you might be getting lucky sometimes with a particular mirror

jkalchik · June 8, 2021, 3:12pm

Oh, the joys of open source… I’m analyzing a pile of repositories right now, but so far, this appear to be running.

jkalchik · June 8, 2021, 4:57pm

Well, this ain’t good…

I added the CentOS products & repositories back in, initially with the default mirrorlist URLs. Prior to running the sync, I manually updated the repository URLs for the CentOS 7 stuff to a locally host set of repositories. On initial sync, it failed wth an invalid URL. But, when I checked foreman_access.log, I found that the URL that was actually being requested was:

GET /pub/repos/centos7/base/repodata/repomd.xml/?release=7&arch=x86_64&repo=os

That has ‘/?release-7&arch-x86_64&repos=os’ appended to the URL that I’d entered, and what ‘hammer repository info…’ returned. I’ve dropped the 4 repositories and recreated them with the same URLs, and the syncs now work.

@Justin_Sherrill, this is getting stranger by the minute. This is another real problem if the URL in use is not what’s entered in the repository info.

jkalchik · June 8, 2021, 5:21pm

I should point out that I am hosting the CentOS 7 repositories locally on the Katello master, via the /pub URL, hence why I was looking in the foreman_access.log file.

jkalchik · June 14, 2021, 6:52pm

@Justin_Sherrill : I see there’s an upgrade available from 4.0/2.4 to 4.0/2.5. I did upgrade my test server here at home, seems to be working better, but I’ll need a few days to run this in the office and analyze there as well.

es7x · June 22, 2021, 7:05pm

Hi, i tried create repo and add url

http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os

sync repo and get error repository synchronize Error: An invalid remote URL was provided.
Then i try changed url to

https://mirror.yandex.ru/centos/7/os/x86_64/

and i get this error again.
I deleted repo and recreate it, but added url repo
https://mirror.yandex.ru/centos/7/os/x86_64/
sync and all sync sucssesful
Tested on:

Centos8-stream
Foreman 2.5
Katello 4.1

jkalchik · June 24, 2021, 12:55am

Frustrating…

[jkalchik@katello yum.repos.d]$ hammer repository update --id 20 --url 'https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64&infra=stock&content=centos'
Repository updated.
[jkalchik@katello yum.repos.d]$ cd
[jkalchik@katello ~]$ hammer repository synchronize --id 20
[.......................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: 404, message='Not Found', url=URL('https://mirrors.fedoraproject.org/metalink/?repo=epel-7&arch=x86_64&infra=stock&content=centos')
[jkalchik@katello ~]$ wget -qO- 'https://mirrors.fedoraproject.org/metalink/?repo=epel-7&arch=x86_64&infra=stock&content=centos'
[jkalchik@katello ~]$ rpm -q katello foreman *pulp*
katello-4.1.0-1.el7.noarch
foreman-2.5.0-1.el7.noarch

Server is fully patched. I updated a repository URL, watched the sync fail, and noted that there’s an extra character listed in the error URL. There’s an extra “/” after the metalink element. With the extra slash, it fails with a 404 Not Found error. Running this in a wget taking the extra “/” out yields a what looks like an XML file.

What can I do, as a local administrator, to address this situation?

es7x · June 24, 2021, 6:54am

Hi, I wrote that if you add a url repolist, you get fail, and then update url to a regular repository (not a mirrorlist), the error persists.
You need to delete the repository and create again, then
add url repository (not a mirrorlist ).

jkalchik · June 24, 2021, 2:06pm

I noted that in post #32.

Having to drop and recreate repositories to update URLs is barely a work-around, let alone a solution.

@Justin_Sherrill I see that the Katello bug you filed has been targeted to be fixed in 4.2. Do you have any idea of what that’s slated for availability? As well, the Pulp3 issue you filed has apparently been fixed and closed, so that should be filtering it’s way through as well.

Just for the sake of completeness:

$ rpm -qa | egrep -i '^katello|^foreman|pulp'
foreman-2.5.0-1.el7.noarch
python3-pulp-rpm-3.11.0-1.el7.noarch
katello-common-4.1.0-1.el7.noarch
foreman-postgresql-2.5.0-1.el7.noarch
katello-4.1.0-1.el7.noarch
katello-repos-4.1.0-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.3-2.el7.noarch
foreman-service-2.5.0-1.el7.noarch
foreman-debug-2.5.0-1.el7.noarch
foreman-release-2.5.0-1.el7.noarch
tfm-rubygem-pulp_container_client-2.4.0-1.el7.noarch
katello-debug-4.1.0-1.el7.noarch
tfm-rubygem-pulp_file_client-1.6.1-1.el7.noarch
foreman-dynflow-sidekiq-2.5.0-1.el7.noarch
python3-pulp-certguard-1.2.0-1.el7.noarch
katello-certs-tools-2.7.3-1.el7.noarch
katello-server-ca-1.0-1.noarch
foreman-installer-katello-2.5.0-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.11.1-1.el7.noarch
foreman-vmware-2.5.0-1.el7.noarch
python3-pulp-deb-2.11.2-1.el7.noarch
foreman-cli-2.5.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.10.0-1.el7.noarch
tfm-rubygem-pulpcore_client-3.11.0-1.el7.noarch
tfm-rubygem-smart_proxy_pulp-3.0.0-1.fm2_5.el7.noarch
foreman-proxy-2.5.0-1.el7.noarch
python3-pulpcore-3.11.2-1.el7.noarch
katello-client-bootstrap-1.7.6-1.el7.noarch
pulp-client-1.0-1.noarch
foreman-installer-2.5.0-1.el7.noarch
python3-pulp-container-2.5.3-1.el7.noarch
python3-pulp-file-1.6.0-1.el7.noarch
python3-pulp-ansible-0.7.3-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.7.3-1.el7.noarch
katello-default-ca-1.0-1.noarch

jkalchik · June 24, 2021, 2:29pm

Better indication of what’s going on:

ahdcos019.ent.lolcentral.com:<jkalc>:(/home/jkalc)
$ hammer repository info --id 2
Id:                 2
Name:               EPEL 7
Label:              epel7
Description:
Organization:       Land O' Lakes
Red Hat Repository: no
Content Type:       yum
Mirror on Sync:     yes
Url:                https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64&infra=stock&content=centos
Publish Via HTTP:   yes
Published At:       https://ahdcos019.ent.lolcentral.com/pulp/content/Land_O_Lakes/Library/custom/epel/epel7/
Relative Path:      Land_O_Lakes/Library/custom/epel/epel7
Download Policy:    immediate
HTTP Proxy:
    HTTP Proxy Policy: global_default_http_proxy
Product:
    Id:   1
    Name: EPEL
GPG Key:
    Id:   4
    Name: RPM-GPG-KEY-EPEL-7
Sync:
    Status:         Warning
    Last Sync Date: about 24 hours
Created:            2021/05/21 14:03:18
Updated:            2021/06/24 14:14:19
Content Counts:
    Packages:       0
    Source RPMS:    0
    Package Groups: 0
    Errata:         0
    Module Streams: 0

ahdcos019.ent.lolcentral.com:<jkalc>:(/home/jkalc)
$ hammer repository synchronize --id 2
[.......................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: 404, message='Not Found', url=URL('https://mirrors.fedoraproject.org/metalink/?repo=epel-7&arch=x86_64&infra=stock&content=centos')
ahdcos019.ent.lolcentral.com:<jkalc>:(/home/jkalc)
$ hammer repository info --id 2
Id:                 2
Name:               EPEL 7
Label:              epel7
Description:
Organization:       Land O' Lakes
Red Hat Repository: no
Content Type:       yum
Mirror on Sync:     yes
Url:                https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64&infra=stock&content=centos
Publish Via HTTP:   yes
Published At:       https://ahdcos019.ent.lolcentral.com/pulp/content/Land_O_Lakes/Library/custom/epel/epel7/
Relative Path:      Land_O_Lakes/Library/custom/epel/epel7
Download Policy:    immediate
HTTP Proxy:
    HTTP Proxy Policy: global_default_http_proxy
Product:
    Id:   1
    Name: EPEL
GPG Key:
    Id:   4
    Name: RPM-GPG-KEY-EPEL-7
Sync:
    Status:         Warning
    Last Sync Date: less than a minute
Created:            2021/05/21 14:03:18
Updated:            2021/06/24 14:14:19
Content Counts:
    Packages:       0
    Source RPMS:    0
    Package Groups: 0
    Errata:         0
    Module Streams: 0

I update the download URL in repository ID 2, and immediately ran a synchronize. That failed. The following repository info output shows the URL that I used for update, but the task error shows an ever so slightly different URL in use (there’s a “/” inserted immediate after “metalink”.

NUTS…

$ hammer repository delete --id 2
hammer repository create --content-type "yum" --label "epel7" --name "EPEL 7" --url 'https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64&infra=stock&content=centos' --download-policy "immediate" --product "EPEL"  --gpg-key-id $(hammer --no-headers content-credentials list --name "RPM-GPG-KEY-EPEL-7" --fields id)
Repository created.
ahdcos019.ent.lolcentral.com:<jkalc>:(/home/jkalc/katello/products)
$ hammer repository list | gview -
Vim: Reading from stdin...
ahdcos019.ent.lolcentral.com:<jkalc>:(/home/jkalc/katello/products)
$ hammer repository sync --id 48
[.......................................................................] [100%]
Total steps: 0/0
--------------------------------
Error: 404, message='Not Found', url=URL('https://mirrors.fedoraproject.org/metalink/?repo=epel-7&arch=x86_64&infra=stock&content=centos')

Deleting and recreating the repository even shows the same issue.