Support importing content-views with Deb content with hammer content-import

Development
1

I am working on adding the feature to import/export content-views with the debian content.

The feature works as expected on pulp_deb side using API. The PR: https://github.com/pulp/pulp_deb/pull/454
But when I tried to add the feature on Katello side by modifying the exportable_types in repository model, the export feature works fine with hammer command but when I try to import the debian content using hammer it throws the following error:

Nov  8 09:30:47 or pulpcore-worker-2: pulp [2f2787e8-b4e0-4590-9652-f763e014d640]: pulpcore.app.tasks.importer:INFO: ...3 import-errors encountered importing ./tmpxsirwjt_/repository-Debian_Client_11-93516_1/pulp_deb.app.modelresource.ReleaseArchitectureResource.json, attempt 2, retrying
Nov  8 09:30:47 or pulpcore-worker-2: pulp [2f2787e8-b4e0-4590-9652-f763e014d640]: pulpcore.app.tasks.importer:INFO: ...3 import-errors encountered importing ./tmpxsirwjt_/repository-Debian_Client_11-93516_1/pulp_deb.app.modelresource.ReleaseArchitectureResource.json, attempt 3, retrying
Nov  8 09:30:48 or pulpcore-worker-2: pulp [2f2787e8-b4e0-4590-9652-f763e014d640]: pulpcore.app.tasks.importer:ERROR: FATAL import-failure importing ./tmpxsirwjt_/repository-Debian_Client_11-93516_1/pulp_deb.app.modelresource.ReleaseArchitectureResource.json
Nov  8 09:30:48 or pulpcore-worker-2: pulp [2f2787e8-b4e0-4590-9652-f763e014d640]: pulpcore.tasking.pulpcore_worker:INFO: Task eb3febd3-573a-40df-9431-fd9f695b8ad1 failed (Release matching query does not exist.)
Nov  8 09:30:48 or pulpcore-worker-2: pulp [2f2787e8-b4e0-4590-9652-f763e014d640]: pulpcore.tasking.pulpcore_worker:INFO:   File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 380, in _perform_task
Nov  8 09:30:48 or pulpcore-worker-2: result = func(*args, **kwargs)
Nov  8 09:30:48 or pulpcore-worker-2: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/tasks/importer.py", line 241, in import_repository_version
Nov  8 09:30:48 or pulpcore-worker-2: for a_result in _import_file(os.path.join(rv_path, filename), res_class, retry=True):
Nov  8 09:30:48 or pulpcore-worker-2: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/tasks/importer.py", line 130, in _import_file
Nov  8 09:30:48 or pulpcore-worker-2: a_result = resource.import_data(data, raise_errors=True)
Nov  8 09:30:48 or pulpcore-worker-2: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/import_export/resources.py", line 757, in import_data
Nov  8 09:30:48 or pulpcore-worker-2: return self.import_data_inner(dataset, dry_run, raise_errors, using_transactions, collect_failed_rows, **kwargs)
Nov  8 09:30:48 or pulpcore-worker-2: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/import_export/resources.py", line 805, in import_data_inner
Nov  8 09:30:48 or pulpcore-worker-2: raise row_result.errors[-1].error
Nov  8 09:30:48 or pulpcore-worker-2: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/import_export/resources.py", line 650, in import_row
Nov  8 09:30:48 or pulpcore-worker-2: instance, new = self.get_or_init_instance(instance_loader, row)
Nov  8 09:30:48 or pulpcore-worker-2: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/import_export/resources.py", line 342, in get_or_init_instance

I tried to change the order of import as well but it is still throwing the same error with different modelResource .json file.

Is there anything that needs to be done on katello side other than modifying the exportable_types ?

2

@iballou @Partha_Aji Do you have any ideas how to fix this?

3

@MSinghal, could you share the Dynflow task information for where this failed? It looks like the importer is getting fed something unexpected.

Also, do the pulp_deb.app.modelresource.ReleaseArchitectureResource.json files look like something that Pulp expects? I’m not sure how the import works from the Pulp side, or even what these files do, but I think a good starting place would be to figure out what is wrong with the specific file.

4

@MSinghal can you also share your katello changes either with a PR or pointing to your fork or something :slight_smile:

As @iballou pointed out it may be something pulp specific. Have you been able to verify pulp debian imports directly before katello steps into the picture.

6

The pulp_deb.app.modelresource.ReleaseArchitectureResource.json is architecture json file which is expected during import. If I change the order to say, import ReleaseComponentResource first then the error I get is pulp_deb.app.modelresource.ReleaseComponentResource.json failed (Release matching query does not exist.)

So the error is not related to this particular file as the import was successful when I used REST API with only pulp_deb and didn’t involve Katello.

When I added the changes on Katello side the metadata.json looks as follows:

"organization":"ÄTIX","repositories":{"Debian_Client_12-93516":{"name":"Debian Client 
11","label":"Debian_Client_11","description":null,"arch":"noarch","content_type":"deb","unprotected":true,
"checksum_type":null,"os_versions":null,"major":null,"minor":null,"download_policy":"immediate"
,"mirroring_policy":"mirror_content_only","product":{"label":"Debian_Client"},"gpg_key":{"name":"orcharhino Debian client signing key"},
"content":{"id":"1663084635540","label":"18119237-ed37-4e5e-99d7-00fe94809515_Debian_Client_Debian_Client_11"},"redhat":false}},"content_view":
{"name":"cv_deb","label":"cv_deb","description":""},"content_view_version":{"major":1,"minor":0,"description":""},"incremental":false,
"products":{"Debian_Client":{"name":"Debian Client","label":"Debian_Client","description":null,"gpg_key":{"name":"orcharhino Debian client signing key"},"redhat":false}},
"gpg_keys":{"orcharhino Debian client signing key":{"name":"orcharhino Debian client signing key","content_type":"gpg_key","content":"-----BEGIN PGP PUBLIC KEY BLOCK- ...

Note that the architecture is set to noarch although when exporting the repository it had no architecture set and release/distribution was set to stable which is not mentioned in this metadata.json.

7

I verified the pulp imports directly on pulp-deb side and they were successful. The PR is already merged in pulp_deb. The issue is only when I try to add it in Katello side.
Here is list of all logs and changes:
Katello changes

Dynflow log:
All steps are successful and only 24: Actions::Pulp3::ContentViewVersion::Import (skipped) [ 38.25s / 4.21s] step is skipped with following details:

---
pulp_tasks: []
task_groups:
- pulp_href: "/pulp/api/v3/task-groups/8de49a16-2e39-450e-84c6-35182b62914a/"
  description: Import of None
  all_tasks_dispatched: true
  waiting: 0
  skipped: 0
  running: 0
  completed: 1
  canceled: 0
  failed: 1
  canceling: 0
  group_progress_reports:
  - message: Importing repository versions
    code: import.repo.versions
    total: 1
    done: 0
  tasks:
  - pulp_href: "/pulp/api/v3/tasks/3e57b44b-bc77-4ed0-a241-7c0ef8aff139/"
    pulp_created: '2022-11-16T10:32:24.296+00:00'
    name: pulpcore.app.tasks.importer.pulp_import
    state: completed
    started_at: '2022-11-16T10:32:24.712+00:00'
    finished_at: '2022-11-16T10:32:47.370+00:00'
    worker: "/pulp/api/v3/workers/bcf149fa-7334-4820-88b8-efdf31c60258/"
  - pulp_href: "/pulp/api/v3/tasks/c76bca9c-f8bc-428d-a966-afb22c90f115/"
    pulp_created: '2022-11-16T10:32:46.916+00:00'
    name: pulpcore.app.tasks.importer.import_repository_version
    state: failed
    started_at: '2022-11-16T10:32:51.249+00:00'
    finished_at: '2022-11-16T10:32:58.665+00:00'
    worker: "/pulp/api/v3/workers/adca2a22-491a-4222-9874-64e0f357184a/"
poll_attempts:
  total: 19
  failed: 1

 Katello::Errors::Pulp3Error

1 subtask(s) failed for task group /pulp/api/v3/task-groups/8de49a16-2e39-450e-84c6-35182b62914a/.
8

@Partha_Aji @iballou Any other suggestions that I can implement on Katello side? It seems to work on Pulp side.

9

Do debian repositories have any special attributes that need to get included
Check your debian repo on console and print values for major, minor and arch attributes and figure out. If the “stable” is not there in major/minor find which repo object will give that info.

Update → katello/metadata_generator.rb at master · Katello/katello · GitHub accordingly.

10

I added this part for debian and now the previous errors are gone. But there is new error that “Release matching query does not exist” - pulp_href: "/pulp/api/v3/tasks/932cc403-0991-427c-be8f-4c180ace22a7/" pulp_created: '2022-12-15T13:13:06.254+00:00' name: pulpcore.app.tasks.importer.import_repository_version state: failed started_at: '2022-12-15T13:13:06.786+00:00' finished_at: '2022-12-15T13:14:01.515+00:00' worker: "/pulp/api/v3/workers/ec269b01-969e-4c83-92db-d3a037697527/" poll_attempts: total: 27 failed: 1

I think the error is because it expects the default distribution for published deb repository which we added directly in Katello. But the error can also be because of other missing part of the import/export code on Katello side.

11

@Partha_Aji @iballou Any other suggestions that I can implement on Katello side?

12

@MSinghal, can you show us what Pulp API you are hitting, and the data that is being sent across? It would be good to start first to see what data Pulp needs that it isn’t getting, or, if we’re hitting a Pulp bug.

13

Here is the log showing pulp APIs and data asked on pulp side.
pulp_api.log (29.2 KB)

I can see it is failing to match the Release query but it only fails if I use import/export with Katello. I directly used import on pulp-deb side and it didn’t throw any such errors.

14

I would stick a debugger where the Rails code is calling the related Pulp API to see exactly what is being sent over. To debug this, we’ll need to compare what Katello is sending Pulp vs what you are sending Pulp directly.