Publish CV: Katello::Errors::Pulp3Error: None

Hello Foreman community,

at first, I’d like to mention I found a similar thread here: Repo sync receiving : Katello::Errors::Pulp3Error: None
with the same exception, but during a different situation (repository sync <> publish CV).

Problem:
When publishing a Content View with a yum filter (exclude, all repositories), Pulp fails with the following exception: Katello::Errors::Pulp3Error: None

Expected outcome:
The Content View should be published successfully without the packages I have specified.

Foreman and Proxy versions:
Foreman: 2.1.2

Foreman and Proxy plugin versions:
Katello: 3.16

Distribution and version:
Oracle Linux 7.8

Other relevant data:
Specifically, I’m using the following repositories for my CV

https://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/
https://public-yum.oracle.com/repo/OracleLinux/OL7/optional/latest/x86_64/
https://public-yum.oracle.com/repo/OracleLinux/OL7/addons/x86_64/
https://yum.oracle.com/repo/OracleLinux/OL7/SoftwareCollections/x86_64/

with the following filter rules (exclude, rpm)

rhn-check
rhn-client-tools
rhnlib
rhnsd
rhn-setup
rhn-setup-gnome
yum-plugin-ulninfo
yum-rhn-plugin

to get subscription manager running later on.

Task error

{"pulp_tasks"=>
  [{"pulp_href"=>"/pulp/api/v3/tasks/1c7dd20d-2a74-4724-847a-536f2971e1cc/",
    "pulp_created"=>"2020-09-14T14:18:04.903+00:00",
    "state"=>"completed",
    "name"=>"pulpcore.app.tasks.repository.add_and_remove",
    "started_at"=>"2020-09-14T14:18:04.976+00:00",
    "finished_at"=>"2020-09-14T14:18:05.110+00:00",
    "worker"=>"/pulp/api/v3/workers/365e371a-d21d-437c-bcc6-8fe426e238c2/",
    "child_tasks"=>[],
    "progress_reports"=>[],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/138df107-c579-4a9e-abaf-e8fe14e52780/"]},
   {"pulp_href"=>"/pulp/api/v3/tasks/a4e1b9b0-9270-4b61-87cd-58e2710ea43b/",
    "pulp_created"=>"2020-09-14T14:18:04.976+00:00",
    "state"=>"completed",
    "name"=>"pulpcore.app.tasks.repository.add_and_remove",
    "started_at"=>"2020-09-14T14:18:05.057+00:00",
    "finished_at"=>"2020-09-14T14:18:05.216+00:00",
    "worker"=>"/pulp/api/v3/workers/bdc417f5-6476-460a-b116-5ae654812f85/",
    "child_tasks"=>[],
    "progress_reports"=>[],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/d70ea05e-1777-44af-aec3-ae29df1eebf2/"]},
   {"pulp_href"=>"/pulp/api/v3/tasks/f6d3a7cd-4ef4-467e-bfab-47c73694ac81/",
    "pulp_created"=>"2020-09-14T14:18:05.042+00:00",
    "state"=>"completed",
    "name"=>"pulpcore.app.tasks.repository.add_and_remove",
    "started_at"=>"2020-09-14T14:18:05.394+00:00",
    "finished_at"=>"2020-09-14T14:18:05.533+00:00",
    "worker"=>"/pulp/api/v3/workers/365e371a-d21d-437c-bcc6-8fe426e238c2/",
    "child_tasks"=>[],
    "progress_reports"=>[],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/15e6f90b-ba41-4ea1-a9c7-36b29f90124d/"]},
   {"pulp_href"=>"/pulp/api/v3/tasks/eaee8e06-f907-46b4-a841-1854b1ed8dc1/",
    "pulp_created"=>"2020-09-14T14:18:05.134+00:00",
    "state"=>"completed",
    "name"=>"pulpcore.app.tasks.repository.add_and_remove",
    "started_at"=>"2020-09-14T14:18:05.473+00:00",
    "finished_at"=>"2020-09-14T14:18:05.638+00:00",
    "worker"=>"/pulp/api/v3/workers/bdc417f5-6476-460a-b116-5ae654812f85/",
    "child_tasks"=>[],
    "progress_reports"=>[],
    "created_resources"=>
     ["/pulp/api/v3/repositories/rpm/rpm/f0d263cc-dbec-4633-8436-96289b5d1451/versions/15/"],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/f0d263cc-dbec-4633-8436-96289b5d1451/"]},
   {"pulp_href"=>"/pulp/api/v3/tasks/94e81e24-2376-4a65-9901-64a293266c3e/",
    "pulp_created"=>"2020-09-14T14:18:05.170+00:00",
    "state"=>"completed",
    "name"=>"pulpcore.app.tasks.repository.add_and_remove",
    "started_at"=>"2020-09-14T14:18:05.757+00:00",
    "finished_at"=>"2020-09-14T14:18:05.888+00:00",
    "worker"=>"/pulp/api/v3/workers/bdc417f5-6476-460a-b116-5ae654812f85/",
    "child_tasks"=>[],
    "progress_reports"=>[],
    "created_resources"=>
     ["/pulp/api/v3/repositories/rpm/rpm/f0d263cc-dbec-4633-8436-96289b5d1451/versions/16/"],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/f0d263cc-dbec-4633-8436-96289b5d1451/"]},
   {"pulp_href"=>"/pulp/api/v3/tasks/841981d4-e60f-48f0-9246-a167342ac7ea/",
    "pulp_created"=>"2020-09-14T14:18:35.057+00:00",
    "state"=>"failed",
    "name"=>"pulp_rpm.app.tasks.copy.copy_content",
    "started_at"=>"2020-09-14T14:18:35.738+00:00",
    "finished_at"=>"2020-09-14T14:18:36.274+00:00",
    "error"=>{"traceback"=>"", "description"=>"None"},
    "worker"=>"/pulp/api/v3/workers/365e371a-d21d-437c-bcc6-8fe426e238c2/",
    "child_tasks"=>[],
    "progress_reports"=>[],
    "created_resources"=>[],
    "reserved_resources_record"=>
     ["/pulp/api/v3/repositories/rpm/rpm/8b94084c-5932-4241-af20-09f79b7118f4/",
      "/pulp/api/v3/repositories/rpm/rpm/26cdc656-c48e-4a39-98b7-c864f81bc825/",
      "/pulp/api/v3/repositories/rpm/rpm/beb6f889-c754-441c-b92e-417e62be2ab0/",
      "/pulp/api/v3/repositories/rpm/rpm/ddfd5270-85cf-47d0-bf6f-baf24e4b51f1/",
      "/pulp/api/v3/repositories/rpm/rpm/138df107-c579-4a9e-abaf-e8fe14e52780/",
      "/pulp/api/v3/repositories/rpm/rpm/d70ea05e-1777-44af-aec3-ae29df1eebf2/",
      "/pulp/api/v3/repositories/rpm/rpm/15e6f90b-ba41-4ea1-a9c7-36b29f90124d/",
      "/pulp/api/v3/repositories/rpm/rpm/f0d263cc-dbec-4633-8436-96289b5d1451/"]}],
 "task_groups"=>[],
 "poll_attempts"=>{"total"=>3, "failed"=>3}}
 
Exception:
Katello::Errors::Pulp3Error: None

Journalctl

Sep 14 16:18:04 foreman pulpcore-worker-2[1326]: pulp: rq.worker:INFO: 1326@foreman.example.com: 1c7dd20d-2a74-4724-847a-536f2971e1cc
Sep 14 16:18:05 foreman pulpcore-worker-1[1331]: pulp: rq.worker:INFO: 1331@foreman.example.com: a4e1b9b0-9270-4b61-87cd-58e2710ea43b
Sep 14 16:18:05 foreman pulpcore-worker-2[1326]: pulp: rq.worker:INFO: 1326@foreman.example.com: Job OK (1c7dd20d-2a74-4724-847a-536f2971e1cc)
Sep 14 16:18:05 foreman pulpcore-worker-2[1326]: pulp: rq.worker:INFO: 1326@foreman.example.com: f0a7b2cb-ecb5-4c81-8ebb-3d4ed2cc2e21
Sep 14 16:18:05 foreman pulpcore-worker-2[1326]: pulp: rq.worker:INFO: 1326@foreman.example.com: Job OK (f0a7b2cb-ecb5-4c81-8ebb-3d4ed2cc2e21)
Sep 14 16:18:05 foreman pulpcore-worker-1[1331]: pulp: rq.worker:INFO: 1331@foreman.example.com: Job OK (a4e1b9b0-9270-4b61-87cd-58e2710ea43b)
Sep 14 16:18:05 foreman pulpcore-worker-1[1331]: pulp: rq.worker:INFO: 1331@foreman.example.com: 99c433ca-5641-46c0-8e16-e74c4c088f45
Sep 14 16:18:05 foreman pulpcore-worker-1[1331]: pulp: rq.worker:INFO: 1331@foreman.example.com: Job OK (99c433ca-5641-46c0-8e16-e74c4c088f45)
Sep 14 16:18:05 foreman pulpcore-worker-2[1326]: pulp: rq.worker:INFO: 1326@foreman.example.com: f6d3a7cd-4ef4-467e-bfab-47c73694ac81
Sep 14 16:18:05 foreman pulpcore-worker-1[1331]: pulp: rq.worker:INFO: 1331@foreman.example.com: eaee8e06-f907-46b4-a841-1854b1ed8dc1
Sep 14 16:18:05 foreman pulpcore-worker-2[1326]: pulp: rq.worker:INFO: 1326@foreman.example.com: Job OK (f6d3a7cd-4ef4-467e-bfab-47c73694ac81)
Sep 14 16:18:05 foreman pulpcore-worker-2[1326]: pulp: rq.worker:INFO: 1326@foreman.example.com: 966cd19b-4e9b-4b34-a12e-c35ad187d08b
Sep 14 16:18:05 foreman pulpcore-worker-2[1326]: pulp: rq.worker:INFO: 1326@foreman.example.com: Job OK (966cd19b-4e9b-4b34-a12e-c35ad187d08b)
Sep 14 16:18:05 foreman pulpcore-worker-1[1331]: pulp: rq.worker:INFO: 1331@foreman.example.com: Job OK (eaee8e06-f907-46b4-a841-1854b1ed8dc1)
Sep 14 16:18:05 foreman pulpcore-worker-1[1331]: pulp: rq.worker:INFO: 1331@foreman.example.com: 6c2827d9-5d84-4582-9fb1-4f8c43cbeab9
Sep 14 16:18:05 foreman pulpcore-worker-1[1331]: pulp: rq.worker:INFO: 1331@foreman.example.com: Job OK (6c2827d9-5d84-4582-9fb1-4f8c43cbeab9)
Sep 14 16:18:05 foreman pulpcore-worker-1[1331]: pulp: rq.worker:INFO: 1331@foreman.example.com: 94e81e24-2376-4a65-9901-64a293266c3e
Sep 14 16:18:05 foreman pulpcore-worker-1[1331]: pulp: rq.worker:INFO: 1331@foreman.example.com: Job OK (94e81e24-2376-4a65-9901-64a293266c3e)
Sep 14 16:18:05 foreman pulpcore-worker-1[1331]: pulp: rq.worker:INFO: 1331@foreman.example.com: e47dba61-4b38-4d4c-90ae-34dc91208d46
Sep 14 16:18:05 foreman pulpcore-worker-1[1331]: pulp: rq.worker:INFO: 1331@foreman.example.com: Job OK (e47dba61-4b38-4d4c-90ae-34dc91208d46)
Sep 14 16:18:35 foreman pulpcore-worker-2[1326]: pulp: rq.worker:INFO: 1326@foreman.example.com: 841981d4-e60f-48f0-9246-a167342ac7ea
Sep 14 16:18:36 foreman pulpcore-worker-2[1326]: python3: ../src/repodata.c:2929: repodata_add_dirstr: Assertion `dir' failed.
Sep 14 16:18:36 foreman pulpcore-worker-2[1326]: pulp: rq.worker:WARNING: Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 134)
Sep 14 16:18:36 foreman pulpcore-worker-2[1326]: pulp: rq.worker:INFO: 1326@foreman.example.com: d2b040c4-a3d7-47f8-a0c2-a58bcd1905dd
Sep 14 16:18:36 foreman pulpcore-worker-2[1326]: pulp: rq.worker:INFO: 1326@foreman.example.com: Job OK (d2b040c4-a3d7-47f8-a0c2-a58bcd1905dd)

The actual cause seems to be the filter itself, since I’ve tried to publish the CV without any filter rules, where it failed as well with the above-mentioned exception. Only by deleting the filter, I’m able to successfully publish my Oracle Linux 7 CV.

I’ll be happy to provide further information.

Any help on this would be much appreciated :slight_smile:

Hi,
i have the same problem with the same Foreman, Katello and OS. With filter i can not publish a new CV…without filter publishing is working normaly. Publishing other OS with filter is working too, only with OracleLinux not.
Is there any solution for that?

Thanks for raising this.
We have a similar issue being tracked here. Bug #30775: publishing a large content view with filters fails with error on multi-copy api - Katello - Foreman .This should help resolve the issue you’re seeing. The fix is currently in the works.

Thank you very much. I have applied the respective patches for Katello 3.16.1, completely rebooted my Foreman VM and tried to resume the failing task for publishing my CV.
Since resuming the task failed, I cancelled it by using

su - postgres
psql foreman
select id, label, started_at, state, result from foreman_tasks_tasks where state != 'stopped';
# Alternatively, you can just lookt at the Task ID contained in the URL

# Return back to root user
foreman-rake console
ForemanTasks::Task.find("<TASK ID>").destroy

Next workday, I’ve started a new task for publishing my CV with filters. At first, I was impressed by the performance increase, until I realized a very big surprise: By applying these patches, it seems that every single repository inside my products was synchronized with 0 packages, which means, everything is empty as soon as the new CV version gets published.

Could you please verify that these patches do not have a very large side effect?
Basically, the only things I have changed:
Patch: https://patch-diff.githubusercontent.com/raw/Katello/katello/pull/8935.patch
Directory: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0/

Sorry for the inconvenience, I’ve posted the wrong patch URL.
Applied patch was https://patch-diff.githubusercontent.com/raw/Katello/katello/pull/8969.patch

Short update on this:
I’ve installed everything again from scratch to get a clean installation with Foreman 2.1.3 and Katello 3.16.1.2. My repository syncs are not empty anymore, but publishing my Oracle Linux 7 CV with filters still fails with error “None”.

Oct 14 12:10:44 foreman pulpcore-worker-1[1275]: python3: ../src/repodata.c:2929: repodata_add_dirstr: Assertion `dir' failed.
Oct 14 12:10:44 foreman pulpcore-worker-1[1275]: pulp: rq.worker:WARNING: Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 134)
Oct 14 12:10:44 foreman pulpcore-worker-1[1275]: pulp: rq.worker:INFO: 1275@foreman.example.com: b8dff633-bfe4-46e0-9704-85da54eb3026
Oct 14 12:10:44 foreman pulpcore-worker-1[1275]: pulp: rq.worker:INFO: 1275@foreman.example.com: Job OK (b8dff633-bfe4-46e0-9704-85da54eb3026)
Oct 14 12:10:46 foreman pulpcore-worker-2[1318]: pulp: rq.worker:INFO: 1318@foreman.example.com: 27e39076-5171-4bd1-b0d6-cd45d1a8a1ac
Oct 14 12:10:46 foreman pulpcore-worker-2[1318]: python3: ../src/repodata.c:2929: repodata_add_dirstr: Assertion `dir' failed.
Oct 14 12:10:46 foreman pulpcore-worker-2[1318]: pulp: rq.worker:WARNING: Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 134)
Oct 14 12:10:46 foreman pulpcore-worker-2[1318]: pulp: rq.worker:INFO: 1318@foreman.example.com: 981f7797-e3d3-4fb4-b205-33ca18103d0e
Oct 14 12:10:46 foreman pulpcore-worker-2[1318]: pulp: rq.worker:INFO: 1318@foreman.example.com: Job OK (981f7797-e3d3-4fb4-b205-33ca18103d0e)
Oct 14 12:10:52 foreman pulpcore-worker-2[1318]: pulp: rq.worker:INFO: 1318@foreman.example.com: c78e3cf9-b7d2-451d-87fb-c4b636a9f6c2
Oct 14 12:10:52 foreman pulpcore-worker-2[1318]: python3: ../src/repodata.c:2929: repodata_add_dirstr: Assertion `dir' failed.
Oct 14 12:10:52 foreman pulpcore-worker-2[1318]: pulp: rq.worker:WARNING: Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 134)
Oct 14 12:10:52 foreman pulpcore-worker-2[1318]: pulp: rq.worker:INFO: 1318@foreman.example.com: e962644a-c3af-4f43-a326-cdb6a1bb1802
Oct 14 12:10:53 foreman pulpcore-worker-2[1318]: pulp: rq.worker:INFO: 1318@foreman.example.com: Job OK (e962644a-c3af-4f43-a326-cdb6a1bb1802)
Oct 14 12:10:54 foreman pulpcore-worker-1[1275]: pulp: rq.worker:INFO: 1275@foreman.example.com: 72694d1c-2d04-4640-a83a-8ea6e9321dc6
Oct 14 12:10:54 foreman pulpcore-worker-1[1275]: python3: ../src/repodata.c:2929: repodata_add_dirstr: Assertion `dir' failed.
Oct 14 12:10:55 foreman pulpcore-worker-1[1275]: pulp: rq.worker:WARNING: Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 134)
Oct 14 12:10:55 foreman pulpcore-worker-1[1275]: pulp: rq.worker:INFO: 1275@foreman.example.com: dac12d04-897f-4389-abde-1c3e2b1944dd
Oct 14 12:10:55 foreman pulpcore-worker-1[1275]: pulp: rq.worker:INFO: 1275@foreman.example.com: Job OK (dac12d04-897f-4389-abde-1c3e2b1944dd)
Oct 14 12:11:01 foreman pulpcore-worker-2[1318]: pulp: rq.worker:INFO: 1318@foreman.example.com: 406705da-b3c4-41a1-9a53-0dae3d89e425
Oct 14 12:11:01 foreman pulpcore-worker-2[1318]: python3: ../src/repodata.c:2929: repodata_add_dirstr: Assertion `dir' failed.
Oct 14 12:11:01 foreman pulpcore-worker-2[1318]: pulp: rq.worker:WARNING: Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 134)
Oct 14 12:11:01 foreman pulpcore-worker-2[1318]: pulp: rq.worker:INFO: 1318@foreman.example.com: b5043558-24b9-4ed0-9be5-1be9a4bf24c2
Oct 14 12:11:01 foreman pulpcore-worker-2[1318]: pulp: rq.worker:INFO: 1318@foreman.example.com: Job OK (b5043558-24b9-4ed0-9be5-1be9a4bf24c2)
Oct 14 12:11:02 foreman pulpcore-worker-2[1318]: pulp: rq.worker:INFO: 1318@foreman.example.com: 7bb2e5f5-8d7b-4a55-8b20-b7c1275f5f74
Oct 14 12:11:02 foreman pulpcore-worker-2[1318]: python3: ../src/repodata.c:2929: repodata_add_dirstr: Assertion `dir' failed.
Oct 14 12:11:02 foreman pulpcore-worker-2[1318]: pulp: rq.worker:WARNING: Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 134)
Oct 14 12:11:02 foreman pulpcore-worker-2[1318]: pulp: rq.worker:INFO: 1318@foreman.example.com: 2679ed72-b5b3-4b5a-b842-c286af1506b1
Oct 14 12:11:02 foreman pulpcore-worker-2[1318]: pulp: rq.worker:INFO: 1318@foreman.example.com: Job OK (2679ed72-b5b3-4b5a-b842-c286af1506b1)
Oct 14 12:12:16 foreman pulpcore-worker-2[1318]: pulp: rq.worker:INFO: 1318@foreman.example.com: e17842b9-8b90-400a-a00f-48e25820fa6b