Publishing a content view to the library environment gets stuck at 19% with no errors

Support
#1

Problem: I created a content view for the EPEL repository and entered the command to publish it to the library. It gets stuck at 19% and no errors are listed. I let it run for over 24 hours and no progress. So, I re-kickstarted the server and re-installed. Again, when I get to publishing the EPEL content view to the library, it gets stuck at 19%.

Any ideas how to troubleshoot?

Expected outcome: The content view gets published to the library.

Foreman and Proxy versions: 2.3.3-1

Katello versions: 3.18.0-1

Distribution and version: CentOS 7.9

Other relevant data: Ran the following:
IDs=$(hammer repository list --product el7_repos | grep epel | awk ‘{ print $1 }’)
hammer content-view create --name “EPEL by version”
–description “Content view for epel by version”
–repository-ids ${IDs}
–solve-dependencies true
CVID=$(hammer content-view list | grep ‘EPEL by version’ | awk ‘{ print $1 }’)
hammer content-view filter create
–content-view-id ${CVID}
–description “Filter for EPEL packages without errata”
–inclusion true
–original-packages true
–name “EPEL Filter without Errata”
–repository-ids ${IDs}
–type rpm
hammer content-view filter create
–content-view-id ${CVID}
–description “Filter for EPEL errata by date”
–inclusion true
–name “EPEL Filter Errata by Date”
–repository-ids ${IDs}
–type erratum
CVFID=$(hammer content-view filter list --content-view-id ${CVID} | grep ‘EPEL Filter Errata by Date’ | awk ‘{ print $1 }’)
hammer content-view filter rule create
–content-view-filter-id ${CVFID}
–content-view-id ${CVID}
–date-type updated
–end-date ${PDATE}
–types enhancment,bugfix,security
hammer content-view filter list -content-view-id ${CVID}
hammer content-view filter rule list --content-view-filter-id ${CVFID} --content-view-id ${CVID}
hammer content-view publish --id ${CVID} --async

#2

A little more information. From the gui task listing, I clicked on Dynflow console. It shows:

Actions::Candlepin::Environment::Create (success)
Actions::Pulp3::Repository::Create (success)
Actions::Pulp3::Repository::SaveVersion (success)
Actions::Katello::Repository:Create (success)
Actions::Pulp3::Repository::MultiCopyContent (waiting for Pulp to start the task)

I then found: [Satellite6] Various tasks are pending on "waiting for Pulp to start the task" - Red Hat Customer Portal and restarted the pulp_resource_manager pulp_workers pulp_celerybeat services, but still stuck.

#3

See if you can resume the task: in Dynflow console near the status, there may be a “resume” link. Or on the Foreman task detail page, see if resume is available.

If not I would force-cancel the task and try again.

#4

I tried resume. I tried cancel, etc. None of that worked. I’ve re-kickstarted three times in the past week and it always gets stuck here. I just re-kicked again this morning. I did make sure that the EPEL repository had finished synching before creating the content view. So, there are rpms there. It’s just whenever I try to promote to the library environment, it hangs. I did the same thing for the CentOS 7 repositories with identical content views and they worked just fine. Just very odd.

#5

In the Dynflow console “Run” tab, there should be an indication of which sub-action it got stuck on, Can you see if you can expand this sub-action to reveal the error stack?

#6

Sorry, got waylaid by a security scan of our systems. Had to answer a lot of questions.
I re-kickstarted a few days ago and started to configure the content views again. I just issued the command for the EPEL publish to library. I’m going to give it over the weekend to see if it completes? Maybe I just wasn’t being patient enough? :slight_smile:

Also, taking a step back, I’d like to make sure my satellite component versions are a good mix. I’m running on CentOS 7.9 Have katello 3.18.0-1. Have for puppet:
pulp-puppet-tools-2.21.1-1
python-pulp-puppet-common-2.21.1-1
puppet-agent-6.21.1-1
pulp-puppet-plugins-2.21.1-1
puppet-agent-oauth-0.5.1-4
puppetserver-6.15.1-1
Have foreman 2.3.3-1

If any of that looks wrong, I’d appreciate hearing about it.

Thanks!

#7

Well, so much for waiting it out. :slight_smile: Execution had stopped. I told it to resume. In the Dynflow page for the current task, the Error shows: Katello::Errors::Pulp3Error None. Then there is a lot of lines of output. Starts with:
- “/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0/app/lib/actions/pulp3/abstract_async_task.rb:102:in `block in check_for_errors’”

and ends with:
- “/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’”

#8

Thanks for the info!

Can you check

journalctl -u pulpcore-worker*

for a traceback?

#9

I think I figured out the problem. The network all of our servers are on is disconnected from the Internet. We rely on another group to bring in content and set up internal mirrors. I had --solve-dependencies true when it was getting hung. I suspect that the folks mirroring the EPEL repository must have missed something. When I changed it to false, the task finished. I don’t think we use a lot from EPEL. So, not going to worry about it for now. Thanks so much for the help!

1 Like