Cannot sync CentOS 8-stream AppStream repo

Support
1

Problem:
When trying to sync AppStream repo for CentOS 8-Stream, getting
PLP0000: Importer indicated a failed response

Expected outcome:
Successful sync

Foreman and Proxy versions:
foreman-2.3.5-1
foreman-proxy-2.3.5-1

Foreman and Proxy plugin versions:

Distribution and version:
CentOS Linux release 7.9.2009

Other relevant data:
Sep 30 07:44:19 pulp: nectar.downloaders.threaded:INFO: Download succeeded: http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/repodata/8e78f2cd57ef01333e8f9033989d63e639789fb84a71365e74924897e73e63a4-filelists.xml.gz.
Sep 30 07:44:19 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [836e2656] Generating metadata databases.
Sep 30 07:45:32 pulpcore-resource-manager: pulp: rq.worker:INFO: Cleaning registries for queue: resource-manager
Sep 30 07:45:38 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [836e2656] Determining which units need to be downloaded.
Sep 30 07:48:19 auditd[892]: Audit daemon rotating log files
Sep 30 07:49:05 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [836e2656] Downloading 0 RPMs.
Sep 30 07:49:50 pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: [836e2656] The content container reported: {‘downloads’: {}, ‘total_sources’: 0} for base URL: CentOS Mirror
Sep 30 07:49:50 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [836e2656] Downloading additional units.
Sep 30 07:49:50 pulp: urllib3.connectionpool:INFO: Starting new HTTP connection (1): mirror.centos.org
Sep 30 07:49:50 pulp: nectar.downloaders.threaded:INFO: Download succeeded: http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/.treeinfo.
Sep 30 07:49:51 pulp: pulp_rpm.plugins.importers.yum.parse.treeinfo:INFO: [836e2656] upstream distribution unchanged and force_full not set; skipping
Sep 30 07:49:52 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [836e2656] (3338-80736) expected a character buffer object
Sep 30 07:49:52 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [836e2656] (3338-80736) Traceback (most recent call last):
Sep 30 07:49:52 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [836e2656] (3338-80736) File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py”, line 312, in run
Sep 30 07:49:52 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [836e2656] (3338-80736) repair=self.validate)
Sep 30 07:49:52 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [836e2656] (3338-80736) File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/modularity.py”, line 412, in synchronize
Sep 30 07:49:52 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [836e2656] (3338-80736) remainder = add_modulemds(repository, modulemds, repair=repair)
Sep 30 07:49:52 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [836e2656] (3338-80736) File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/modularity.py”, line 202, in add_modulemds
Sep 30 07:49:52 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [836e2656] (3338-80736) add_modulemd(repository, modulemd, model)
Sep 30 07:49:52 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [836e2656] (3338-80736) File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/modularity.py”, line 78, in add_modulemd
Sep 30 07:49:52 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [836e2656] (3338-80736) fp.write(document)
Sep 30 07:49:52 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [836e2656] (3338-80736) TypeError: expected a character buffer object
Sep 30 07:49:52 pulp: pulp.server.async.tasks:INFO: [836e2656] Task failed : [836e2656-c309-4ccc-83fa-85afb25f34d6]
Sep 30 07:49:52 pulp: celery.app.trace:ERROR: [836e2656] (3338-80736) Task pulp.server.managers.repo.sync.sync[836e2656-c309-4ccc-83fa-85afb25f34d6] raised unexpected: PulpExecutionException(‘Importer indicated a failed response’,)
Sep 30 07:49:52 pulp: celery.app.trace:ERROR: [836e2656] (3338-80736) Traceback (most recent call last):
Sep 30 07:49:52 pulp: celery.app.trace:ERROR: [836e2656] (3338-80736) File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 367, in trace_task
Sep 30 07:49:52 pulp: celery.app.trace:ERROR: [836e2656] (3338-80736) R = retval = fun(*args, **kwargs)
Sep 30 07:49:52 pulp: celery.app.trace:ERROR: [836e2656] (3338-80736) File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 688, in call
Sep 30 07:49:52 pulp: celery.app.trace:ERROR: [836e2656] (3338-80736) return super(Task, self).call(*args, **kwargs)
Sep 30 07:49:52 pulp: celery.app.trace:ERROR: [836e2656] (3338-80736) File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 110, in call
Sep 30 07:49:52 pulp: celery.app.trace:ERROR: [836e2656] (3338-80736) return super(PulpTask, self).call(*args, **kwargs)
Sep 30 07:49:52 pulp: celery.app.trace:ERROR: [836e2656] (3338-80736) File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 622, in protected_call
Sep 30 07:49:52 pulp: celery.app.trace:ERROR: [836e2656] (3338-80736) return self.run(*args, **kwargs)
Sep 30 07:49:52 pulp: celery.app.trace:ERROR: [836e2656] (3338-80736) File “/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py”, line 860, in sync
Sep 30 07:49:52 pulp: celery.app.trace:ERROR: [836e2656] (3338-80736) raise pulp_exceptions.PulpExecutionException(_(‘Importer indicated a failed response’

2

I experience the exact same error, same repo, same versions.

3

We’re also getting this error in RedHat Satellite. It seems like something with the metadata might be off in the repository, as Dynflow Console seems to point to a more specific error. See below. I’ve tried force verification, etc. and still get this error. Anyone have any ideas here? Or maybe found another mirror that works?

In Satelite, the end result is a completed task with “warning” status and it doesn’t update the module stream data. This is problematic because we have systems looking for updated modules, and getting locked behind dependencies because the metadata isn’t updating.

  modules:
    state: FAILED
    error: expected a character buffer object
4

This is still ongoing months later, even after applying recent updates. At this point our content views are basically unusable because module list isn’t synced up with what packages are available. I can’t believe no one else has seen similar issues here. I can’t believe there’s only two people using CentOS 8 Stream here.

5

Wait… is the whole issue here because Centos updated the modules file to use modulemd 2, but Foreman/Satellite is still using 1.7.0-1??? I’m on Satellite 6.9 and near as I can tell it’s using 1.7.0-1.pulp.el7sat version for pulp right now…

Apparently we might be forced to upgrade here? Anyone care to comment before I go down that rabbit hole.

6

This might be a version issue indeed.
We have not had any trouble syncing and module repos for a while now. On our EL7 production machine with Foreman 3.2/Katello 4.4, we have both libmodulemd 1.7 and libmodulemd2 2.14 installed.
On our EL8 testing system with the identical Foreman/Katello version, we have libmodulemd 2.13 (there might be updates pending, didn’t check). The versions we use are between Satellite 6.11 and what 6.12 will ship with.
So, if CentOS is indeed publishing there metadata in a format modulemd 1.7 can not understand, it could very well be solved by upgrading. While I can not give you a concrete answer (we are not syncing CentOS, only RHEL and Alma), I would assume upgrading to Satellite 6.10 or 6.11 should solve your issue.

7

We are using a very old version Foreman 1.24.3 with libmodulemd 1.7.0.

We have also been unable to sync CentOS Stream 8 AppStream repo with the same error as logged above. We sync weekly and the last successful sync was May 13th 2022.

However, we have no issues syncing CentOS Stream 9 AppStream so I’m wondering if it just a problem with the CentOS Stream 8 AppStream repo.

8

I can confirm that the upgrade to Satellite 6.10, which also includes Pulp 2 to 3 migration, does indeed fix the issue. There’s a ton of things that change with Pulp 3 and it automagically made this problem go away. We have been able to successfully patch multiple CentOS 8 clients now.

I’m not sure exactly where this equates in the Foreman versioning, but Satellite 6.9 hit EOL as of last month anyway. I’m guessing that upstream Foreman is already fixed, since Satellite is downstream, and they’re just not supporting it when older versions break.

9

In case you are interested, the version mapping between Satellite and the upstream world can be found here: Satellite 6 Component Versions - Red Hat Customer Portal
The versions mentioned on that page do not always workout 100% perfectly, because RH sometime does some backporting of patches to older versions (at least to my knowledge), but it is a pretty good guideline of what you can expect.

10

I have the same problem in Foreman 2.3. The installation of package libmodulemd2 didn’t help.