Unable to sync docker-ce repo for Ubuntu

Problem: I created a Product Docker, with deb content, upstream URL https://download.docker.com/linux/ubuntu, Releases: bionic, Component: stable and try to Sync the repo. The outcome is:

Errors:
‘Codename’

Other repo’s with deb content work fine.

Expected outcome: A normally synced deb-repo with docker-ce packages.

Foreman and Proxy versions: foreman-2.3.5-1

Foreman and Proxy plugin versions: pulp-deb-plugins-1.10.2-1

Distribution and version: Foreman / Katello is installed on CentOS 7.9.2009, fully patched

Other relevant data:

Hi @voorman

Do you see any errors from Pulp in /var/log/messages

Well, not sure what to look for, these might be two attempts at the same time:

Jun 14 17:14:51 foremanserver pulpcore-worker-4: pulp: rq.worker:ERROR: Traceback (most recent call last):
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
Jun 14 17:14:51 foremanserver pulpcore-worker-4: rv = job.perform()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: pulp: rq.worker:ERROR: Traceback (most recent call last):
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
Jun 14 17:14:51 foremanserver pulpcore-worker-4: rv = job.perform()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
Jun 14 17:14:51 foremanserver pulpcore-worker-4: self._result = self._execute()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
Jun 14 17:14:51 foremanserver pulpcore-worker-4: self._result = self._execute()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
Jun 14 17:14:51 foremanserver pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
Jun 14 17:14:51 foremanserver pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py”, line 106, in synchronize
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py”, line 106, in synchronize
Jun 14 17:14:51 foremanserver pulpcore-worker-4: DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Jun 14 17:14:51 foremanserver pulpcore-worker-4: loop.run_until_complete(pipeline)
Jun 14 17:14:51 foremanserver pulpcore-worker-4: loop.run_until_complete(pipeline)
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Jun 14 17:14:51 foremanserver pulpcore-worker-4: return future.result()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: return future.result()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Jun 14 17:14:51 foremanserver pulpcore-worker-4: await asyncio.gather(*futures)
Jun 14 17:14:51 foremanserver pulpcore-worker-4: await asyncio.gather(*futures)
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Jun 14 17:14:51 foremanserver pulpcore-worker-4: await self.run()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: await self.run()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py”, line 257, in run
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py”, line 257, in run
Jun 14 17:14:51 foremanserver pulpcore-worker-4: release_file.codename = release_file_dict[“Codename”]
Jun 14 17:14:51 foremanserver pulpcore-worker-4: release_file.codename = release_file_dict[“Codename”]
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/debian/deb822.py”, line 538, in getitem
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/debian/deb822.py”, line 538, in getitem
Jun 14 17:14:51 foremanserver pulpcore-worker-4: value = self.__dict[keyi]
Jun 14 17:14:51 foremanserver pulpcore-worker-4: value = self.__dict[keyi]
Jun 14 17:14:51 foremanserver pulpcore-worker-4: KeyError: ‘Codename’
Jun 14 17:14:51 foremanserver pulpcore-worker-4: KeyError: ‘Codename’
Jun 14 17:14:51 foremanserver pulpcore-worker-4: Traceback (most recent call last):
Jun 14 17:14:51 foremanserver pulpcore-worker-4: Traceback (most recent call last):
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
Jun 14 17:14:51 foremanserver pulpcore-worker-4: rv = job.perform()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: rv = job.perform()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
Jun 14 17:14:51 foremanserver pulpcore-worker-4: self._result = self._execute()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: self._result = self._execute()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
Jun 14 17:14:51 foremanserver pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
Jun 14 17:14:51 foremanserver pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py”, line 106, in synchronize
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py”, line 106, in synchronize
Jun 14 17:14:51 foremanserver pulpcore-worker-4: DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 148, in create
Jun 14 17:14:51 foremanserver pulpcore-worker-4: loop.run_until_complete(pipeline)
Jun 14 17:14:51 foremanserver pulpcore-worker-4: loop.run_until_complete(pipeline)
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib64/python3.6/asyncio/base_events.py”, line 484, in run_until_complete
Jun 14 17:14:51 foremanserver pulpcore-worker-4: return future.result()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: return future.result()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 225, in create_pipeline
Jun 14 17:14:51 foremanserver pulpcore-worker-4: await asyncio.gather(*futures)
Jun 14 17:14:51 foremanserver pulpcore-worker-4: await asyncio.gather(*futures)
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py”, line 43, in call
Jun 14 17:14:51 foremanserver pulpcore-worker-4: await self.run()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: await self.run()
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py”, line 257, in run
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py”, line 257, in run
Jun 14 17:14:51 foremanserver pulpcore-worker-4: release_file.codename = release_file_dict[“Codename”]
Jun 14 17:14:51 foremanserver pulpcore-worker-4: release_file.codename = release_file_dict[“Codename”]
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/debian/deb822.py”, line 538, in getitem
Jun 14 17:14:51 foremanserver pulpcore-worker-4: File “/usr/lib/python3.6/site-packages/debian/deb822.py”, line 538, in getitem
Jun 14 17:14:51 foremanserver pulpcore-worker-4: value = self.__dict[keyi]
Jun 14 17:14:51 foremanserver pulpcore-worker-4: value = self.__dict[keyi]
Jun 14 17:14:51 foremanserver pulpcore-worker-4: KeyError: ‘Codename’
Jun 14 17:14:51 foremanserver pulpcore-worker-4: KeyError: ‘Codename’
Jun 14 17:14:51 foremanserver pulpcore-worker-4: pulp: rq.worker:INFO: 1500@foremanserver.localdomain: 92a2f9a5-8caf-45ec-b08b-580b200a1772
Jun 14 17:14:51 foremanserver pulpcore-worker-4: pulp: rq.worker:INFO: 1500@foremanserver.localdomain: 92a2f9a5-8caf-45ec-b08b-580b200a1772
Jun 14 17:14:51 foremanserver pulpcore-worker-4: pulp: rq.worker:INFO: 1500@foremanserver.localdomain: Job OK (92a2f9a5-8caf-45ec-b08b-580b200a1772)
Jun 14 17:14:51 foremanserver pulpcore-worker-4: pulp: rq.worker:INFO: 1500@foremanserver.localdomain: Job OK (92a2f9a5-8caf-45ec-b08b-580b200a1772)
Jun 14 17:14:51 foremanserver pulpcore-api: - - [14/Jun/2021:15:14:51 +0000] “GET /pulp/api/v3/tasks/bf53c534-fc0e-42d4-bc68-b6afa01c6f65/ HTTP/1.1” 200 2491 “-” “OpenAPI-Generator/3.7.1/ruby”
Jun 14 17:14:51 foremanserver pulpcore-api: - - [14/Jun/2021:15:14:51 +0000] “GET /pulp/api/v3/tasks/bf53c534-fc0e-42d4-bc68-b6afa01c6f65/ HTTP/1.1” 200 2491 “-” “OpenAPI-Generator/3.7.1/ruby”
Jun 14 17:14:51 foremanserver pulpcore-api: - - [14/Jun/2021:15:14:51 +0000] “PATCH /pulp/api/v3/distributions/deb/apt/c4ad3c54-1d81-43c4-9c1d-f847eb3c00a5/ HTTP/1.1” 202 67 “-” “OpenAPI-Generator/2.7.0/ruby”
Jun 14 17:14:51 foremanserver pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: 726cdbc6-097c-4f6e-805b-49a9d850a57c
Jun 14 17:14:51 foremanserver pulpcore-api: - - [14/Jun/2021:15:14:51 +0000] “PATCH /pulp/api/v3/distributions/deb/apt/c4ad3c54-1d81-43c4-9c1d-f847eb3c00a5/ HTTP/1.1” 202 67 “-” “OpenAPI-Generator/2.7.0/ruby”
Jun 14 17:14:51 foremanserver pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: 726cdbc6-097c-4f6e-805b-49a9d850a57c
Jun 14 17:14:51 foremanserver pulpcore-api: - - [14/Jun/2021:15:14:51 +0000] “GET /pulp/api/v3/tasks/23e8a31b-937c-4790-89e8-4c9ad181e4bb/ HTTP/1.1” 200 401 “-” “OpenAPI-Generator/3.7.1/ruby”
Jun 14 17:14:51 foremanserver pulpcore-api: - - [14/Jun/2021:15:14:51 +0000] “GET /pulp/api/v3/tasks/23e8a31b-937c-4790-89e8-4c9ad181e4bb/ HTTP/1.1” 200 401 “-” “OpenAPI-Generator/3.7.1/ruby”
Jun 14 17:14:51 foremanserver pulpcore-worker-3: pulp: rq.worker:INFO: 1510@foremanserver.localdomain: 23e8a31b-937c-4790-89e8-4c9ad181e4bb
Jun 14 17:14:51 foremanserver pulpcore-worker-3: pulp: rq.worker:INFO: 1510@foremanserver.localdomain: 23e8a31b-937c-4790-89e8-4c9ad181e4bb
Jun 14 17:14:51 foremanserver pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (726cdbc6-097c-4f6e-805b-49a9d850a57c)
Jun 14 17:14:51 foremanserver pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (726cdbc6-097c-4f6e-805b-49a9d850a57c)
Jun 14 17:14:52 foremanserver pulpcore-worker-3: pulp: rq.worker:INFO: 1510@foremanserver.localdomain: Job OK (23e8a31b-937c-4790-89e8-4c9ad181e4bb)
Jun 14 17:14:52 foremanserver pulpcore-worker-3: pulp: rq.worker:INFO: 1510@foremanserver.localdomain: Job OK (23e8a31b-937c-4790-89e8-4c9ad181e4bb)
Jun 14 17:14:52 foremanserver pulpcore-worker-3: pulp: rq.worker:INFO: 1510@foremanserver.localdomain: ff653d7e-f8c1-4357-935a-2890b67817b5
Jun 14 17:14:52 foremanserver pulpcore-worker-3: pulp: rq.worker:INFO: 1510@foremanserver.localdomain: ff653d7e-f8c1-4357-935a-2890b67817b5
Jun 14 17:14:52 foremanserver pulpcore-worker-3: pulp: rq.worker:INFO: 1510@foremanserver.localdomain: Job OK (ff653d7e-f8c1-4357-935a-2890b67817b5)
Jun 14 17:14:52 foremanserver pulpcore-worker-3: pulp: rq.worker:INFO: 1510@foremanserver.localdomain: Job OK (ff653d7e-f8c1-4357-935a-2890b67817b5)
Jun 14 17:14:52 foremanserver pulpcore-api: - - [14/Jun/2021:15:14:52 +0000] “GET /pulp/api/v3/tasks/23e8a31b-937c-4790-89e8-4c9ad181e4bb/ HTTP/1.1” 200 509 “-” “OpenAPI-Generator/3.7.1/ruby”
Jun 14 17:14:52 foremanserver pulpcore-api: - - [14/Jun/2021:15:14:52 +0000] “GET /pulp/api/v3/tasks/23e8a31b-937c-4790-89e8-4c9ad181e4bb/ HTTP/1.1” 200 509 “-” “OpenAPI-Generator/3.7.1/ruby”

I will have a go at reproducing this.

Extra info perhaps; this is happening at work, I built the same Foreman setup with same products and repo’s at home; I got no problems there.

At work I’m behind a (double) proxy.

Other deb and rpm repos are syncing fine, both at work and at home.

I had another look and the Release file for this repo does not have a Codename field, but pulp_deb treats this as a required field. I would expect this repo to fail to sync everywhere irrespective of the proxy situation.

There is already an open pulp_deb issue to reassess the required status of various release file fields: Issue #7673: Support synchronization of upstream repos with minimal Release file fields - Debian Support - Pulp

I may just prioritize this issue.
Still don’t understand why you are not consistently getting this error for this repo everywhere…

Actually, according to this issue: Issue #7886: Make Codename field optional - Debian Support - Pulp
The codename field was already made optional in version 2.8.0 of pulp_deb.
Can you verify what version of python-pulp-deb is installed on your two foreman instances?

Version is: python-pulp-deb-common-1.10.2-1.el7.noarch (on both systems)
Or to be more specific:

pulp-deb-plugins-1.10.2-1.el7.noarch
python-pulp-deb-common-1.10.2-1.el7.noarch
python3-pulp-deb-2.7.0-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.7.0-1.el7.noarch

This is the version I was looking for, which is smaller than 2.8.0.
Katello is currently in the process of releasing 3.18.3.1 to go with your Foreman version.
Once that is released, there will be a 3.18.4 shortly after. The main difference in this version will be that it will include python3-pulp-deb-2.9.2. That should then fix your problem.

I am afraid there isn’t much to do other than to wait for this release.

Well, thanks for looking in to this anyway! Still strange the Foreman install at home is working fine with the docker-ce repo. I see Katello 3.18.3-1.el7 is currently available by the way, but as I understand it, this will not yet include 2.9.2 of python3-pulp-deb.
Could it be configuration differences between home and work that allows my home setup to sync the docker-ce repo?

Correct, the plan is to release Katello 3.18.4 soon after which will include the newer pulp_deb.

If home and work are using the same foreman version, then they should be using the same version of the pulpcore repo. If they are using the same version of this repo, they should be using the same version of the python3-pulp-deb package. What version of the pulpcore repo is used is normally controlled by the katello-repos package.

Your version of Foreman should be using version 3.7 of the pulpcore repo. Newer versions of python3-pulp-deb are available in newer versions of this repo, so in theory it is possible you somehow installed one of those (but only via manual intervention).

Versions are the same at home and work.
I see something in my /var/log/messages at home:

Jun 16 12:01:01 phobos mongod.27017[10802]: [conn58] command pulp_database.$cmd command: update { update: “units_deb_release”, ordered: true, updates: [ { q: { _id: “7e368a47-fd7d-48a0-b9c3-5903db5d935f” }, u: { _id: “7e368a47-fd7d-48a0-b9c3-5903db5d935f”, pulp_user_metadata: {}, _last_updated: 1623837661, repoid: “3a69505b-0384-4b7b-acc4-757b25fcaec3”, distribution: “focal”, codename: “foo”, suite: “focal”, _ns: “units_deb_release”, _content_type_id: “deb_release” }, multi: false, upsert: true } ], writeConcern: { w: 1 } } numYields:0 reslen:274 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, MMAPV1Journal: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 1 } }, Collection: { acquireCount: { W: 1 } } } protocol:op_query 118ms

There seems to be a ‘codename’ field added with value ‘foo’. On my work system, mongodb doesn’t seem to log in /var/log/messages (or anywhere for that matter…)

I think your home system may be using Pulp 2, while your work system is using Pulp 3. That would certainly explain the difference!

1 Like

Okay! Where can I see what the systems are using? The installed packages are exactly the same on both systems

Ah, I think I can find it in the GUI under Infrastructure, Smart Proxies, Services and I see at work Pulp and Pulpcore version 2.1.0 is used. On my home system under Pulpcore after Types, it says ‘file’, ‘deb’ and ‘yum’, with “supported, but overridden to use Pulp 2” behind ‘deb’ and ‘yum’.

On my work system, it just says ‘file’, ‘deb’ and ‘yum’ after ‘Types’.

Correct, you found it.

Don’t know how I’ve overridden my home system to use Pulp 2 however, nor whether I should try to revert that. I could ofcourse (if I can find out how) try to override pulpcore to use Pulp 2 on at work as well.
Could you advise or give directions on how to proceed?

It’s probably because I upgraded my home Foreman through the years and never did a Pulp 2->3 migration, whereas the installation at work is a fresh 2.3.5 version. Still wonder how to proceed…

I recommend moving towards Pulp 3 since that is where all the development effort is going and Katello 4.0 drops support for Pulp 2. Katello 3.18.* is the last Katello version with Pulp 2 support.

“supported, but overridden to use Pulp 2” is a configuration file setting.
If you have a look at about minute 53 of this video you can see me changing the setting. If you want to do a Pulp 2to3 content migration before you switch, you may want to start at around minute 45 of the video. :wink:

You can also read the “Pulp3 Migration” instructions here: Foreman :: Plugin Manuals

1 Like

One last recommendation: Wait with the switch to Pulp 3 and any 2to3 migration until Katello 3.18.4 is out. (At least for deb content type).