[Katello] 3.16.1.2 “NoMethodError: undefined method `repository_href’ for nil:NilClass” when syncing yum repos

Problem:

When syncing repositories, some fail with “NoMethodError: undefined method `repository_href’ for nil:NilClass”. Syncing worked without a problem for about 2 weeks then a handful of repos fail with this error. I’ve delete/rebuilt products in the past that had this issue but it seems to come back.

Input:
{“repo_id”=>675,
“smart_proxy_id”=>1,
“options”=>{“source_url”=>nil},
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”,
“current_request_id”=>nil,
“current_timezone”=>“America/Chicago”,
“current_user_id”=>4,
“current_organization_id”=>1,
“current_location_id”=>nil}

Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/services/katello/pulp3/repository.rb:198:in sync' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/repository/sync.rb:13:in invoke_external_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:84:in initiate_external_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:19:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/pulp3/abstract_async_task.rb:10:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:16:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/models/katello/concerns/user_extensions.rb:21:in cp_config’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.1.2/app/lib/actions/middleware/remote_action.rb:16:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in wrap’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in with_progress_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in block (2 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in block in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in execute_job' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in block (2 levels) in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in block in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in local' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in block in <module:Sidekiq>' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in stats' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in global' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in block in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in with_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in with_job_hash_context’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in process_one' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in watchdog' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in block in safe_thread’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’

Installed Package:
* candlepin-3.1.16-1.el7.noarch
* candlepin-selinux-3.1.16-1.el7.noarch
* foreman-2.1.3-1.el7.noarch
* foreman-cli-2.1.3-1.el7.noarch
* foreman-debug-2.1.3-1.el7.noarch
* foreman-dynflow-sidekiq-2.1.3-1.el7.noarch
* foreman-ec2-2.1.3-1.el7.noarch
* foreman-installer-2.1.3-1.el7.noarch
* foreman-installer-katello-2.1.3-1.el7.noarch
* foreman-postgresql-2.1.3-1.el7.noarch
* foreman-proxy-2.1.3-1.el7.noarch
* foreman-release-2.1.3-1.el7.noarch
* foreman-release-scl-7-3.el7.noarch
* foreman-service-2.1.3-1.el7.noarch
* foreman-vmware-2.1.3-1.el7.noarch
* katello-3.16.1-1.el7.noarch
* katello-certs-tools-2.7.0-1.el7.noarch
* katello-client-bootstrap-1.7.5-1.el7.noarch
* katello-common-3.16.1-1.el7.noarch
* katello-debug-3.16.1-1.el7.noarch
* katello-default-ca-1.0-1.noarch
* katello-repos-3.16.1-1.el7.noarch
* katello-server-ca-1.0-1.noarch
* pulp-client-1.0-1.noarch
* pulp-deb-plugins-1.10.1-1.el7.noarch
* pulp-docker-plugins-3.2.7-1.el7.noarch
* pulp-katello-1.0.3-1.el7.noarch
* pulp-puppet-plugins-2.21.3-1.el7.noarch
* pulp-puppet-tools-2.21.3-1.el7.noarch
* pulp-rpm-plugins-2.21.3-1.el7.noarch
* pulp-selinux-2.21.3-1.el7.noarch
* pulp-server-2.21.3-1.el7.noarch
* puppet-foreman_scap_client-0.4.0-1.el7.noarch
* python-gofer-qpid-2.12.5-3.el7.noarch
* python-pulp-common-2.21.3-1.el7.noarch
* python-pulp-deb-common-1.10.1-1.el7.noarch
* python-pulp-docker-common-3.2.7-1.el7.noarch
* python-pulp-oid_validation-2.21.3-1.el7.noarch
* python-pulp-puppet-common-2.21.3-1.el7.noarch
* python-pulp-repoauth-2.21.3-1.el7.noarch
* python-pulp-rpm-common-2.21.3-1.el7.noarch
* python-pulp-streamer-2.21.3-1.el7.noarch
* python2-qpid-1.37.0-4.el7.noarch
* python2-qpid-proton-0.31.0-3.el7.x86_64
* python2-qpid-qmf-1.39.0-1.el7.x86_64
* python3-pulp-2to3-migration-0.2.0-0.1.b7.el7.noarch
* python3-pulp-certguard-0.1.0-0.2.rc5.el7.noarch
* python3-pulp-container-1.4.2-1.el7.noarch
* python3-pulp-file-1.0.1-1.el7.noarch
* python3-pulp-rpm-3.5.1-1.el7.noarch
* python3-pulpcore-3.4.1-1.el7.noarch
* qpid-cpp-client-1.39.0-1.el7.x86_64
* qpid-cpp-client-devel-1.39.0-1.el7.x86_64
* qpid-cpp-server-1.39.0-1.el7.x86_64
* qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
* qpid-dispatch-router-1.14.0-1.el7.x86_64
* qpid-proton-c-0.31.0-3.el7.x86_64
* qpid-qmf-1.39.0-1.el7.x86_64
* qpid-tools-1.39.0-1.el7.noarch
* rubygem-foreman_maintain-0.6.13-1.el7.noarch
* rubygem-foreman_maintain-doc-0.6.13-1.el7.noarch
* stlppkgmgt03.rgare.net-foreman-client-1.0-1.noarch
* stlppkgmgt03.rgare.net-foreman-proxy-1.0-1.noarch
* stlppkgmgt03.rgare.net-foreman-proxy-client-1.0-1.noarch
* stlppkgmgt03.rgare.net-qpid-broker-1.0-1.noarch
* stlppkgmgt03.rgare.net-qpid-client-cert-1.0-1.noarch
* stlppkgmgt03.rgare.net-qpid-router-client-1.0-1.noarch
* stlppkgmgt03.rgare.net-qpid-router-server-1.0-1.noarch
* tfm-rubygem-actioncable-6.0.3.1-1.el7.noarch
* tfm-rubygem-actionmailbox-6.0.3.1-1.el7.noarch
* tfm-rubygem-actionmailer-6.0.3.1-1.el7.noarch
* tfm-rubygem-actionpack-6.0.3.1-1.el7.noarch
* tfm-rubygem-actiontext-6.0.3.1-1.el7.noarch
* tfm-rubygem-actionview-6.0.3.1-1.el7.noarch
* tfm-rubygem-activejob-6.0.3.1-1.el7.noarch
* tfm-rubygem-activemodel-6.0.3.1-1.el7.noarch
* tfm-rubygem-activerecord-6.0.3.1-1.el7.noarch
* tfm-rubygem-activerecord-import-1.0.0-2.el7.noarch
* tfm-rubygem-activerecord-session_store-1.1.1-4.el7.noarch
* tfm-rubygem-activestorage-6.0.3.1-1.el7.noarch
* tfm-rubygem-activesupport-6.0.3.1-1.el7.noarch
* tfm-rubygem-addressable-2.6.0-2.el7.noarch
* tfm-rubygem-algebrick-0.7.3-7.el7.noarch
* tfm-rubygem-amazing_print-1.1.0-1.el7.noarch
* tfm-rubygem-ancestry-3.0.7-1.el7.noarch
* tfm-rubygem-anemone-0.7.2-17.el7.noarch
* tfm-rubygem-angular-rails-templates-1.1.0-1.el7.noarch
* tfm-rubygem-ansi-1.5.0-2.el7.noarch
* tfm-rubygem-apipie-bindings-0.3.0-2.el7.noarch
* tfm-rubygem-apipie-dsl-2.2.2-2.el7.noarch
* tfm-rubygem-apipie-params-0.0.5-4.el7.noarch
* tfm-rubygem-apipie-rails-0.5.17-3.el7.noarch
* tfm-rubygem-audited-4.9.0-3.el7.noarch
* tfm-rubygem-bcrypt-3.1.12-3.el7.x86_64
* tfm-rubygem-builder-3.2.4-1.el7.noarch
* tfm-rubygem-bundler_ext-0.4.1-5.el7.noarch
* tfm-rubygem-clamp-1.1.2-6.el7.noarch
* tfm-rubygem-concurrent-ruby-1.1.6-2.el7.noarch
* tfm-rubygem-concurrent-ruby-edge-0.6.0-2.fm2_1.el7.noarch
* tfm-rubygem-connection_pool-2.2.2-2.el7.noarch
* tfm-rubygem-crass-1.0.6-1.el7.noarch
* tfm-rubygem-css_parser-1.4.7-4.el7.noarch
* tfm-rubygem-daemons-1.2.3-6.el7.noarch
* tfm-rubygem-deacon-1.0.0-4.el7.noarch
* tfm-rubygem-deep_cloneable-3.0.0-3.el7.noarch
* tfm-rubygem-deface-1.5.3-2.el7.noarch
* tfm-rubygem-diffy-3.0.1-5.el7.noarch
* tfm-rubygem-domain_name-0.5.20160310-4.el7.noarch
* tfm-rubygem-dynflow-1.4.7-1.fm2_1.el7.noarch
* tfm-rubygem-erubi-1.9.0-1.el7.noarch
* tfm-rubygem-excon-0.58.0-5.el7.noarch
* tfm-rubygem-facter-2.4.0-7.el7.x86_64
* tfm-rubygem-faraday-0.15.4-2.el7.noarch
* tfm-rubygem-fast_gettext-1.4.1-4.el7.noarch
* tfm-rubygem-ffi-1.12.2-1.el7.x86_64
* tfm-rubygem-fog-aws-3.6.5-1.el7.noarch
* tfm-rubygem-fog-core-2.1.0-3.el7.noarch
* tfm-rubygem-fog-json-1.2.0-3.el7.noarch
* tfm-rubygem-fog-vsphere-3.3.1-1.el7.noarch
* tfm-rubygem-fog-xml-0.1.2-8.el7.noarch
* tfm-rubygem-foreman-tasks-2.0.2-1.fm2_1.el7.noarch
* tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
* tfm-rubygem-foreman_ansible-5.1.3-1.fm2_1.el7.noarch
* tfm-rubygem-foreman_ansible_core-3.0.3-1.fm2_1.el7.noarch
* tfm-rubygem-foreman_bootdisk-17.0.2-2.fm2_1.el7.noarch
* tfm-rubygem-foreman_discovery-16.1.2-1.fm2_1.el7.noarch
* tfm-rubygem-foreman_openscap-4.0.2-1.fm2_1.el7.noarch
* tfm-rubygem-foreman_remote_execution-3.3.7-1.fm2_1.el7.noarch
* tfm-rubygem-foreman_remote_execution_core-1.3.0-1.el7.noarch
* tfm-rubygem-foreman_setup-7.0.0-2.fm2_1.el7.noarch
* tfm-rubygem-foreman_templates-9.0.1-1.fm2_1.el7.noarch
* tfm-rubygem-formatador-0.2.1-12.el7.noarch
* tfm-rubygem-friendly_id-5.3.0-1.el7.noarch
* tfm-rubygem-fx-0.5.0-1.el7.noarch
* tfm-rubygem-get_process_mem-0.2.1-4.el7.noarch
* tfm-rubygem-gettext_i18n_rails-1.8.0-2.el7.noarch
* tfm-rubygem-git-1.5.0-1.el7.noarch
* tfm-rubygem-gitlab-sidekiq-fetcher-0.5.2-2.el7.noarch
* tfm-rubygem-globalid-0.4.2-1.el7.noarch
* tfm-rubygem-graphql-1.8.14-2.el7.noarch
* tfm-rubygem-graphql-batch-0.3.10-2.el7.noarch
* tfm-rubygem-gssapi-1.2.0-7.el7.noarch
* tfm-rubygem-hammer_cli-2.1.2-1.el7.noarch
* tfm-rubygem-hammer_cli_foreman-2.1.3-1.el7.noarch
* tfm-rubygem-hammer_cli_foreman_ansible-0.3.2-1.fm2_1.el7.noarch
* tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
* tfm-rubygem-hammer_cli_foreman_docker-0.0.5-1.el7.noarch
* tfm-rubygem-hammer_cli_foreman_openscap-0.1.11-1.fm2_1.el7.noarch
* tfm-rubygem-hammer_cli_foreman_remote_execution-0.1.2-1.fm2_1.el7.noarch
* tfm-rubygem-hammer_cli_foreman_tasks-0.0.14-1.fm2_1.el7.noarch
* tfm-rubygem-hammer_cli_katello-0.22.2-1.el7.noarch
* tfm-rubygem-hashie-3.6.0-2.el7.noarch
* tfm-rubygem-highline-1.7.8-5.el7.noarch
* tfm-rubygem-http-cookie-1.0.2-4.el7.noarch
* tfm-rubygem-i18n-1.8.2-1.el7.noarch
* tfm-rubygem-ipaddress-0.8.0-12.el7.noarch
* tfm-rubygem-jwt-2.2.1-2.el7.noarch
* tfm-rubygem-kafo-4.1.0-3.el7.noarch
* tfm-rubygem-kafo_parsers-1.1.0-3.el7.noarch
* tfm-rubygem-kafo_wizards-0.0.1-4.el7.noarch
* tfm-rubygem-katello-3.16.1.2-1.el7.noarch
* tfm-rubygem-ldap_fluff-0.4.7-5.el7.noarch
* tfm-rubygem-little-plugger-1.1.4-2.el7.noarch
* tfm-rubygem-locale-2.0.9-14.el7.noarch
* tfm-rubygem-logging-2.2.2-5.el7.noarch
* tfm-rubygem-loofah-2.4.0-1.el7.noarch
* tfm-rubygem-mail-2.7.1-1.el7.noarch
* tfm-rubygem-marcel-0.3.3-1.el7.noarch
* tfm-rubygem-method_source-0.9.2-1.el7.noarch
* tfm-rubygem-mime-types-3.2.2-4.el7.noarch
* tfm-rubygem-mime-types-data-3.2018.0812-4.el7.noarch
* tfm-rubygem-mimemagic-0.3.5-1.el7.noarch
* tfm-rubygem-mini_mime-1.0.2-1.el7.noarch
* tfm-rubygem-mini_portile2-2.4.0-1.el7.noarch
* tfm-rubygem-multi_json-1.14.1-1.el7.noarch
* tfm-rubygem-multipart-post-2.0.0-2.el7.noarch
* tfm-rubygem-mustermann-1.0.2-4.el7.noarch
* tfm-rubygem-net-ldap-0.16.1-2.el7.noarch
* tfm-rubygem-net-ping-2.0.1-4.el7.noarch
* tfm-rubygem-net-scp-1.2.1-4.el7.noarch
* tfm-rubygem-net-ssh-4.2.0-2.el7.noarch
* tfm-rubygem-netrc-0.11.0-5.el7.noarch
* tfm-rubygem-nio4r-2.5.2-1.el7.x86_64
* tfm-rubygem-nokogiri-1.10.9-1.el7.x86_64
* tfm-rubygem-oauth-0.5.4-4.el7.noarch
* tfm-rubygem-openscap-0.4.9-3.el7.noarch
* tfm-rubygem-optimist-3.0.0-2.el7.noarch
* tfm-rubygem-parse-cron-0.1.4-4.fm2_1.el7.noarch
* tfm-rubygem-pg-1.1.4-3.el7.x86_64
* tfm-rubygem-polyglot-0.3.5-2.el7.noarch
* tfm-rubygem-powerbar-2.0.1-2.el7.noarch
* tfm-rubygem-promise.rb-0.7.4-2.el7.noarch
* tfm-rubygem-public_suffix-3.0.3-2.el7.noarch
* tfm-rubygem-pulp_2to3_migration_client-0.2.0-0.1.b6.el7.noarch
* tfm-rubygem-pulp_ansible_client-0.2.0b13.dev01588546902-1.el7.noarch
* tfm-rubygem-pulp_certguard_client-0.1.0rc5-1.el7.noarch
* tfm-rubygem-pulp_container_client-1.4.1-1.el7.noarch
* tfm-rubygem-pulp_file_client-1.0.1-1.el7.noarch
* tfm-rubygem-pulp_rpm_client-3.5.0-1.el7.noarch
* tfm-rubygem-pulpcore_client-3.4.1-1.el7.noarch
* tfm-rubygem-puma-4.3.3-4.el7.x86_64
* tfm-rubygem-puppetdb_foreman-5.0.0-2.fm2_1.el7.noarch
* tfm-rubygem-rabl-0.14.3-1.el7.noarch
* tfm-rubygem-rack-2.2.2-1.el7.noarch
* tfm-rubygem-rack-cors-1.0.2-2.el7.noarch
* tfm-rubygem-rack-jsonp-1.3.1-9.el7.noarch
* tfm-rubygem-rack-protection-2.0.3-4.el7.noarch
* tfm-rubygem-rack-test-1.1.0-4.el7.noarch
* tfm-rubygem-rails-6.0.3.1-1.el7.noarch
* tfm-rubygem-rails-dom-testing-2.0.3-6.el7.noarch
* tfm-rubygem-rails-html-sanitizer-1.3.0-1.el7.noarch
* tfm-rubygem-rails-i18n-6.0.0-2.el7.noarch
* tfm-rubygem-railties-6.0.3.1-1.el7.noarch
* tfm-rubygem-rainbow-2.2.1-3.el7.noarch
* tfm-rubygem-rb-inotify-0.9.7-5.el7.noarch
* tfm-rubygem-rbvmomi-2.2.0-3.el7.noarch
* tfm-rubygem-record_tag_helper-1.0.1-3.el7.noarch
* tfm-rubygem-redis-4.1.2-2.el7.noarch
* tfm-rubygem-responders-3.0.0-3.el7.noarch
* tfm-rubygem-rest-client-2.0.2-3.el7.noarch
* tfm-rubygem-rkerberos-0.1.5-18.el7.x86_64
* tfm-rubygem-roadie-3.4.0-3.el7.noarch
* tfm-rubygem-roadie-rails-2.1.1-2.el7.noarch
* tfm-rubygem-robotex-1.0.0-21.el7.noarch
* tfm-rubygem-rsec-0.4.3-4.el7.noarch
* tfm-rubygem-ruby-libvirt-0.7.1-1.el7.x86_64
* tfm-rubygem-ruby2ruby-2.4.2-3.el7.noarch
* tfm-rubygem-ruby_parser-3.10.1-3.el7.noarch
* tfm-rubygem-rubyipmi-0.10.0-6.el7.noarch
* tfm-rubygem-runcible-2.13.1-1.el7.noarch
* tfm-rubygem-safemode-1.3.5-3.el7.noarch
* tfm-rubygem-scoped_search-4.1.8-1.el7.noarch
* tfm-rubygem-secure_headers-6.3.0-2.el7.noarch
* tfm-rubygem-sequel-5.7.1-3.el7.noarch
* tfm-rubygem-sexp_processor-4.10.0-6.el7.noarch
* tfm-rubygem-sidekiq-5.2.7-3.el7.noarch
* tfm-rubygem-sinatra-2.0.3-4.el7.noarch
* tfm-rubygem-smart_proxy_ansible-3.0.1-6.fm2_1.el7.noarch
* tfm-rubygem-smart_proxy_discovery-1.0.5-6.fm2_1.el7.noarch
* tfm-rubygem-smart_proxy_dynflow-0.2.4-6.fm2_1.el7.noarch
* tfm-rubygem-smart_proxy_dynflow_core-0.2.5-1.fm2_1.el7.noarch
* tfm-rubygem-smart_proxy_openscap-0.7.3-1.fm2_1.el7.noarch
* tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_1.el7.noarch
* tfm-rubygem-smart_proxy_remote_execution_ssh-0.3.0-4.fm2_1.el7.noarch
* tfm-rubygem-sprockets-3.7.2-6.el7.noarch
* tfm-rubygem-sprockets-rails-3.2.1-6.el7.noarch
* tfm-rubygem-sqlite3-1.3.13-6.el7.x86_64
* tfm-rubygem-sshkey-1.9.0-4.el7.noarch
* tfm-rubygem-statsd-instrument-2.1.4-3.el7.noarch
* tfm-rubygem-stomp-1.4.9-1.el7.noarch
* tfm-rubygem-thor-1.0.1-2.el7.noarch
* tfm-rubygem-thread_safe-0.3.6-5.el7.noarch
* tfm-rubygem-tilt-2.0.8-4.el7.noarch
* tfm-rubygem-tzinfo-1.2.6-1.el7.noarch
* tfm-rubygem-unf-0.1.3-8.el7.noarch
* tfm-rubygem-unf_ext-0.0.7.2-3.el7.x86_64
* tfm-rubygem-unicode-0.4.4.4-3.el7.x86_64
* tfm-rubygem-unicode-display_width-1.0.5-4.el7.noarch
* tfm-rubygem-validates_lengths_from_database-0.5.0-7.el7.noarch
* tfm-rubygem-webpack-rails-0.9.8-5.el7.noarch
* tfm-rubygem-websocket-driver-0.7.1-1.el7.x86_64
* tfm-rubygem-websocket-extensions-0.1.4-1.el7.noarch
* tfm-rubygem-will_paginate-3.1.7-3.el7.noarch
* tfm-rubygem-x-editable-rails-1.5.5-5.el7.noarch
* tfm-rubygem-xmlrpc-0.3.0-2.el7.noarch
* tfm-rubygem-zeitwerk-2.2.2-1.el7.noarch
* tfm-runtime-6.1-1.el7.x86_64

Expected outcome:

Successful Repo Sync

Foreman and Proxy versions:

2.1.3

Foreman and Proxy plugin versions:

|foreman-tasks|The goal of this plugin is to unify the way of showing task statuses across the Foreman instance. It defines Task model for keeping the information about the tasks and Lock for assigning the tasks to resources. The locking allows dealing with preventing multiple colliding tasks to be run on the same resource. It also optionally provides Dynflow infrastructure for using it for managing the tasks.|Ivan Nečas|2.0.2|
|---|---|---|---|
|foreman_ansible|Ansible integration with Foreman|Daniel Lobato Garcia|5.1.3|
|foreman_bootdisk|Plugin for Foreman that creates iPXE-based boot disks to provision hosts without the need for PXE infrastructure.|Dominic Cleal|17.0.2|
|foreman_discovery|MaaS Discovery Plugin engine for Foreman|Aditi Puntambekar, alongoldboim, Alon Goldboim, amirfefer, Amit Karsale, Amit Upadhye, Amos Benari, Avi Sharvit, Bryan Kearney, bshuster, Daniel Lobato, Daniel Lobato Garcia, Daniel Lobato García, Danny Smit, David Davis, Djebran Lezzoum, Dominic Cleal, Eric D. Helms, Ewoud Kohl van Wijngaarden, Frank Wall, Greg Sutcliffe, ChairmanTubeAmp, Ido Kanner, imriz, Imri Zvik, Ivan Nečas, Joseph Mitchell Magen, June Zhang, kgaikwad, Lars Berntzon, ldjebran, Leos Stejskal, Lukas Zapletal, Lukáš Zapletal, Marek Hulan, Marek Hulán, Martin Bačovský, Matt Jarvis, Michael Moll, Nick, odovzhenko, Ohad Levy, Ondrej Prazak, Ondřej Ezr, Ori Rabin, orrabin, Partha Aji, Petr Chalupa, Phirince Philip, Rahul Bajaj, Robert Antoni Buj Gelonch, Scubafloyd, Sean O\'Keeffe, Sebastian Gräßl, Shimon Shtein, Shlomi Zadok, Stephen Benjamin, Swapnil Abnave, Thomas Gelf, Timo Goebel, Tomas Strych, Tom Caspy, Tomer Brisker, and Yann Cézard|16.1.2|
|foreman_openscap|Foreman plug-in for managing security compliance reports|slukasik@redhat.com|4.0.2|
|foreman_remote_execution|A plugin bringing remote execution to the Foreman, completing the config management functionality with remote management functionality.|Foreman Remote Execution team|3.3.7|
|foreman_setup|Plugin for Foreman that helps set up provisioning.|Dominic Cleal|7.0.0|
|foreman_templates|Engine to synchronise provisioning templates from GitHub|Greg Sutcliffe|9.0.1|
|katello|Katello adds Content and Subscription Management to Foreman. For this it relies on Candlepin and Pulp.|N/A|3.16.1.2|
|puppetdb_foreman|Disable hosts on PuppetDB after they are deleted or built in Foreman. Follow https://github.com/theforeman/puppetdb_foreman and raise an issue/submit a pull request if you need extra functionality. You can also find some help in #theforeman IRC channel on Freenode.|Daniel Lobato Garcia|5.0.0|

Distribution and version:

Other relevant data:

Apologies, I pre-formatted the output I thought. Looks like only some of it was preformatted.

Distribution and version: CentOS 7

Multiple repositories have this issue again. Anyone know a fix other than recreating the repos?

Hello! I believe you are hitting this: [Katello] Can't Delete Package "NoMethodError: undefined method `repository_href' for nil:NilClass"

I linked directly to the solution. We’ve been having trouble narrowing down the cause. If you are able to fix the problem with those instructions, would you be able to send /var/log/foreman/* to me? I can pm you my email address.

I’m attempting those instructions now. No luck so far but still trying. About 15 repos flew off the rails overnight. I can send you the logs no problem. I also generated an sos report this morning if you want that as well.

Correction - That solution does appear to have solved the issue. If you want to PM your contact information I’ll be glad to send any logs/cmd results/etc to help troubleshoot.

awesome! I pm’d it to you. If there’s anything you can think happened in the last 24 hours (or between the times you saw a good sync and a bad), that might be helpful. Something like deleting a content, deleting some other product, etc…

Digging through it looks like a Content view version got deleted. So i gave that a try doing the following:

  1. Add a repo to a content view
  2. publish the CV with no filters
  3. remove the CVV from LIbrary
  4. delete the CVV entirely
  5. try to sync

This reproduced the problem!

I opened to track the fix: Bug #31247: “NoMethodError: undefined method `repository_href’ for nil:NilClass” when syncing yum repos - Katello - Foreman

Thanks for providing the logs to help track this down!

1 Like

Awesome! Glad you were able to track it down!