General Troubleshooting- hammer content-import library

Support
1

Problem:
I am unable to import my library
Expected outcome:
Import of the data should complete without errors
Foreman and Proxy versions:
3.2
Foreman and Proxy plugin versions:
Katello 4.4
Distribution and version:
Rocky 8.5
Other relevant data:
I have been reading and following the content management guide here:
https://docs.theforeman.org/3.2/Content_Management_Guide/index-katello.html#Synchronizing_Content_Between_Servers_content-management

One of my systems is air gapped. The system is a new build and getting the repos set up for the 1st time. I export my content from the connected system by running a full export of the library. The data is then transferred to the disconnected system in the imports folder. I then run: hammer content-import library --organization"<name_here>" --path=/var/lib/pulp/imports/

After about 20 mins the process fails with: Error: 1 subtask(s) failed the task group /pulp/api/v3/task-groups/894b1e7c-xxx-xxx-xxx-xxx/.

By reviewing Montior>Tasks and finding the import task that was run, I can find more info about the failure. From there I can review the Errors tab and see the Action:, Input:, Output:, Exception:, and Backtrace:

Can anyone provide some general troubleshooting hints and tips? Due to the type of system I am unable to pull logs for upload easily. It requires approval by authorities above me. This is why I am asking a general question. Sorry I am not able to be more specific at this time.

2

If you go to the subtask which failed, you may find out what goes wrong by checking the reason of the error result from the subtask’s output.

1 Like
3

Yes, thanks. The subtask Output is generally the same as what I listed above, the last three lines say this-

        "worker"=>
          " /pulp/api/v3/workers/<some_alphanumeric_number>/"}]}],
"poll_attempts:=>("total"=>111, "failed=>1)}

The Exception section says this:
Katello::Errors::Pulp3Error: Error: 1 subtask(s) failed the task group /pulp/api/v3/task-groups/894b1e7c-xxx-xxx-xxx-xxx/.

4

From the Task tab of the failed task, click on Dynflow console button.
In the popup Dynflow window, click on Run tab. You should see a list of actions.
The failed action is in red. Click on it that will show what error you run into.

5

Ok, I don’t see anything in Red in the Dynflow console. All the items are in orange/blue but no red. Also, every action listed in the console has a status of (success). Now I am a little more confused.

My understanding is that importing a library will also import all the RPMs located in the repository. Is my understanding correct? Maybe I have it wrong…

I do see the Products and Repos getting created when running the content-import job.

6

@lfu I appreciate your support greatly! Are there any other suggestions you can provide or any further comments based on my last response?

I do have approval now to upload logs if needed.

7

This issue seems very similar to BZ2032602 (maybe even the same issue)

8

Let’s find out the status of the failed task-group. Please replace with your task-group id before running the command.
curl https://`hostname`/pulp/api/v3/task-groups/<894b1e7c-xxx-xxx-xxx-xxx>/ --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key | python -m json.tool

    "tasks": [
        {
            ....
            "pulp_href": "/pulp/api/v3/tasks/f029f55e-01f7-4282-81ea-523c3c195abc/",
            ....
        }
    ],

From its result, grab pulp_href and run:

curl https://`hostname`<pulp_href> --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key | python -m json.tool

Similar to this:

curl https://`hostname`/pulp/api/v3/tasks/f029f55e-01f7-4282-81ea-523c3c195abc/   --cert /etc/pki/katello/certs/pulp-client.crt  --key /etc/pki/katello/private/pulp-client.key | python -m json.tool

That should tell us the reason why the task failed.
Thanks.

9

This was perfect. It allowed me to see what the problem really was. My humble .02 worth - pin this. Getting to the bottom of a cryptic error message was invaluable in solving the problem. By seeing the actual error message that was buried two layers deep I was able to resolve my issue.