Foreman / Katello Ubuntu repository sync

jpurtell · January 8, 2021, 11:14pm

Problem:
Ubuntu 18.04 (Bionic) repository sync fails

Expected outcome:
Ubuntu sync works

Foreman and Proxy versions:
Host OS: Centos 7
Foreman version: 2.2.2
Katello version: 3.17

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:
Here are some /var/log/messages log entries to help troubleshoot.

Jan  8 17:53:24 foreman mongod.27017[3955]: [conn52] command pulp_database.$cmd command: update { update: "task_status", ordered: true, updates: [ { q: { task_id: "fc83655a-2406-4da0-8bd1-62fc14bcca32" }, u: { $set: { progress_report: { f91c9f0d-7884-4002-9e8c-a8ffd86767d3: [ { num_success: 0, description: "Publishing modules", step_type: "publish_modules", sub_steps: [ { num_success: 1, description: "", step_type: "publish_deb_releases", items_total: 1, state: "FINISHED", error_details: [], details: "", num_failures: 0, step_id: "8da9ccc8-4f96-4d32-b904-1ab612b585cc", num_processed: 1 }, { num_success: 2, description: "", step_type: "publish_deb_components", items_total: 2, state: "FINISHED", error_details: [], details: "", num_failures: 0, step_id: "84c01eca-f164-473d-a670-bbca70dceb85", num_processed: 2 }, { num_success: 5634, description: "", step_type: "publish_deb", items_total: 6441, state: "IN_PROGRESS", error_details: [], details: "", num_failures: 0, step_id: "3d074d67-5443-4699-96e5-a13218f3f031", num_processed: 5634 }, { num_success: 0, description: "", step_type: "publish_repodata", items_total: 1, state: "NOT_STARTED", error_details: [], details: "", num_failures: 0, step_id: "2efc81b7-d268-4c29-a45d-4a2eb4725bc8", num_processed: 0 } ], items_total: 1, state: "NOT_STARTED", error_details: [], details: "", num_failures: 0, step_id: "126d24a2-e87c-494c-be73-eca30a51fa34", num_processed: 0 }, { num_success: 0, description: "Publishing files to web", step_type: "publish_directory", items_total: 1, state: "NOT_STARTED", error_details: [], details: "", num_failures: 0, step_id: "9968119a-a722-4c9e-80de-64a24ac87034", num_processed: 0 }, { num_success: 0, description: "Writing Listings File", step_type: "generate_listing_files", items_total: 1, state: "NOT_STARTED", error_details: [], details: "", num_failures: 0, step_id: "6154db5f-5163-45a9-aa49-bca6046af254", num_processed: 0 }, { num_success: 0, description: "Writing Listings File", step_type: "generate_listing_files", items_total: 1, state: "NOT_STARTED", error_details: [], details: "", num_failures: 0, step_id: "25d97190-3a82-4fef-8061-969daccc5cfb", num_processed: 0 } ] } } }, multi: false, upsert: false } ] } numYields:0 reslen:59 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } protocol:op_query 170ms
Jan  8 17:53:35 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480) Exception caught from plugin during publish for repo [f91c9f0d-7884-4002-9e8c-a8ffd86767d3]
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480) Traceback (most recent call last):
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1297, in _do_publish
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 892, in wrap_f
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)     return f(*args, **kwargs)
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py", line 84, in publish_repo
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)     return self._publisher.process_lifecycle()
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)     super(PluginStep, self).process_lifecycle()
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)     step.process()
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 257, in process
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)     self._process_block()
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 304, in _process_block
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)     self.process_main()
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py", line 418, in process_main
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)     signer.sign(release_file_path)
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/debpkgr/signer.py", line 124, in sign
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480)     stdout=stdout, stderr=stderr)
Jan  8 17:53:36 foreman pulp: pulp.server.controllers.repository:ERROR: [fc83655a] (4182-00480) SignerError: Return code: 2
Jan  8 17:53:36 foreman pulp: pulp.server.async.tasks:INFO: [fc83655a] Task failed : [fc83655a-2406-4da0-8bd1-62fc14bcca32]
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480) Task pulp.server.managers.repo.publish.publish[fc83655a-2406-4da0-8bd1-62fc14bcca32] raised unexpected: SignerError(u'Return code: 2',)
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480) Traceback (most recent call last):
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     R = retval = fun(*args, **kwargs)
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in __call__
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     return super(Task, self).__call__(*args, **kwargs)
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in __call__
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     return super(PulpTask, self).__call__(*args, **kwargs)
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     return self.run(*args, **kwargs)
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1136, in publish
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1245, in check_publish
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1297, in _do_publish
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 892, in wrap_f
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     return f(*args, **kwargs)
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py", line 84, in publish_repo
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     return self._publisher.process_lifecycle()
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     super(PluginStep, self).process_lifecycle()
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     step.process()
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 257, in process
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     self._process_block()
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 304, in _process_block
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     self.process_main()
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py", line 418, in process_main
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     signer.sign(release_file_path)
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)   File "/usr/lib/python2.7/site-packages/debpkgr/signer.py", line 124, in sign
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480)     stdout=stdout, stderr=stderr)
Jan  8 17:53:36 foreman pulp: celery.app.trace:ERROR: [fc83655a] (4182-00480) SignerError: Return code: 2
Jan  8 17:53:36 foreman pulp: celery.app.trace:INFO: [9e451158] Task pulp.server.async.tasks._release_resource[9e451158-26e8-4767-b52b-b502647d8195] succeeded in 0.00254562200007s: None
Jan  8 17:53:38 foreman dynflow-sidekiq@worker: 2021-01-08T22:53:38.180Z 4841 TID-30uoy9 Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork JID-6f1a07827250551cee98a3cf INFO: start
Jan  8 17:53:38 foreman dynflow-sidekiq@worker: 2021-01-08T22:53:38.373Z 4841 TID-30uoy9 Dynflow::Executors::Sidekiq::WorkerJobs::PerformWork JID-6f1a07827250551cee98a3cf INFO: done: 0.192 sec
Jan  8 17:53:38 foreman dynflow-sidekiq@orchestrator: 2021-01-08T22:53:38.375Z 4205 TID-2un2d1 Dynflow::Executors::Sidekiq::OrchestratorJobs::WorkerDone JID-4c7910a87d8ff83abf8e9c3b INFO: start
Jan  8 17:53:38 foreman dynflow-sidekiq@orchestrator: 2021-01-08T22:53:38.377Z 4205 TID-2un2d1 Dynflow::Executors::Sidekiq::OrchestratorJobs::WorkerDone JID-4c7910a87d8ff83abf8e9c3b INFO: done: 0.003 sec

Thanks in advance for any help that’s provided.
I’d really like to get this working, and see other’s that have gotten it working. So I’m hopeful.

Jim

jpurtell · January 10, 2021, 6:27pm

Here is what my repository looks like. I’m only doing one at the moment till I get it working.

jpurtell · January 10, 2021, 6:46pm

Here’s the sync Task for Ubuntu

quba42 · January 11, 2021, 10:47am

It looks like the sync task is failing during the publish step, when it is trying to sign the published metadata:

  File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py", line 418, in process_main
    signer.sign(release_file_path)
  File "/usr/lib/python2.7/site-packages/debpkgr/signer.py", line 124, in sign
    stdout=stdout, stderr=stderr)
SignerError: Return code: 2

This most likely means that your external signing command is either not configured correctly and/or running into some error when run.

You can find some documentation on how to configure the Pulp 2 pulp_deb plugin (which is what you are using) to use metadata signing here: GitHub - pulp/pulp_deb at 2-master (See “Signing support” in the README.md file).

If you post whatever you have in your /etc/pulp/server/plugins.conf.d/deb_distributor.json file, I may be able to help further. For test purposes, you could try renaming this file, restart pulp, and then repeat the sync (it should be successful, but your metadata won’t be signed).

quba42 · January 11, 2021, 10:58am

This is a bit speculative, but assuming the Return code: 2 you are getting is from gpg, this is often the case if your GPG home folder (can be specified via environmental variable or the --homedir flag) does not have the correct permissions (which gpg is very nit-picky about). Make sure everything in the GPG home dir you are using is owned by the apache user.

jpurtell · January 11, 2021, 5:52pm

quba42,
Thanks so much for your help. I figured I’d try the easier thing first and changed the ownership of gpg home dir to apache (since everything was owned by root), and that worked!

[root@foreman pulp]# pwd
/var/lib/pulp
[root@foreman pulp]# cd gpg-home/
[root@foreman gpg-home]# ll
total 28
-rw-r–r-- 1 root root 43 Jan 8 09:36 gpg.conf
-rw------- 1 root root 2311 Jan 8 16:03 pubring.gpg
-rw------- 1 root root 2311 Jan 8 16:03 pubring.gpg~
-rw------- 1 root root 600 Jan 8 16:03 random_seed
-rw------- 1 root root 4893 Jan 8 16:03 secring.gpg
-rw------- 1 root root 1360 Jan 8 16:03 trustdb.gpg
[root@foreman gpg-home]# chown apache:apache ./*
[root@foreman gpg-home]# ll
total 28
-rw-r–r-- 1 apache apache 43 Jan 8 09:36 gpg.conf
-rw------- 1 apache apache 2311 Jan 8 16:03 pubring.gpg
-rw------- 1 apache apache 2311 Jan 8 16:03 pubring.gpg~
-rw------- 1 apache apache 600 Jan 8 16:03 random_seed
-rw------- 1 apache apache 4893 Jan 8 16:03 secring.gpg
-rw------- 1 apache apache 1360 Jan 8 16:03 trustdb.gpg