Sounds like the Pulp 2 patch worked, let us know if you see anything else strange applicability-wise.
Yes the patch indeed worked (also adding the forced regeneration task after).
I have since done many system updates, and the data reflecting that patches/errata are now applied on a given client are reflected immediately, which is the way it should be and I’m thrilled. (tonight is patch night, so having this solution today ws great 
The only followup I have is, do you have an idea what katello version will have the updated pulp permanently? If not that’s OK and I will monitor release notes for subsequent versions carefully.
Thank you again everyone for the help.
In my case I was seeing NO packages (no nothing) for anything centos7/redhat7 except, oddly, for ONE host out of 120 hosts. No idea why that one looked ok. My centos8/redhat8 clients in katello were unaffected.
Good to hear your EL8 clients weren’t affected, that lines up with what everyone else is experiencing.
I’ll make sure that Katello 3.15 is matched up with the newer version of Pulp 2 once the issue is fixed.
So, I seem to be left with just 4/789 content hosts which do Combined Profile Updates which end with a ‘warning’ result. All are CentOS 7.x.
Example errors:
Action:
Actions::Katello::Host::UploadProfiles
Input:
{"managed"=>{"id"=>1584, "name"=>"node1.example.com"},
"host_id"=>1584,
"hostname"=>"node1.example.com",
"profile_string"=>"TRIMMED",
"services_checked"=>["pulp", "pulp_auth"],
"current_request_id"=>nil,
"current_timezone"=>"UTC",
"current_user_id"=>1,
"current_organization_id"=>nil,
"current_location_id"=>nil}
Output:
{}
Exception:
ArgumentError: invalid argument: nil.
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/spawn_methods.rb:37:in `merge'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:46:in `upload_modules_to_pulp'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:70:in `import_module_streams'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:114:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:113:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/katello/host/upload_profiles.rb:113:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:563:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:562:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:561:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:561:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:475:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:475:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:475:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:556:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/director.rb:68:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors.rb:12:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'Something is still off. I just noticed one CentOS 7 content host which shows no applicable updates, though yum check-update correctly shows three updates.
I have went through the whole thread again. ::Katello::Pulp::Consumer.new(uuid).applicable_ids(‘rpm’) shows []. I can see katello knows it’s the old versions installed. The new versions are available in the repository (actually I have applied those updates successfully on 14 other CentOS 7 servers before through katello).
I have even reinstalled the pulp-server RPM, applied the pulp PR patch, restarted, ran katello-package-upload -f, and the GenerateApplicability task a couple of times. Nothing helps. Katello still doesn’t pick up those pending changes.
I have noticed the pulp patch is different from the original one. I have now applied the latest version of the patch which was merged. Yet, no difference. It’s just this one host at the moment. Very weird…
Now I have done a unregister and clean with subscription-manager on the host. Delete the host from katello, registered again. katello-package-upload -f, ran GenerateApplicability task again.
No change: the host still doesn’t show any applicable updates even though yum check-update shows them.
yum check-update shows update available:
microcode_ctl.x86_64 2:2.1-61.10.el7_8 DKRZ_centos7_updates_x86_64
python2-tracer.noarch 0.7.4-1.el7 DKRZ_epel7_epel_x86_64
tracer-common.noarch 0.7.4-1.el7 DKRZ_epel7_epel_x86_64
/var/lib/rhsm/cache/profile.json contains:
{
"arch": "x86_64",
"epoch": 2,
"name": "microcode_ctl",
"release": "61.6.el7_8",
"vendor": "CentOS",
"version": "2.1"
},
retrieve_profile contains the installed old version:
::Katello.pulp_server.resources.consumer.retrieve_profile(uuid, "rpm")
...
{"vendor"=>"CentOS", "name"=>"microcode_ctl", "epoch"=>2, "version"=>"2.1", "release"=>"61.6.el7_8", ```"arch"=>"x86_64"},
applicable_ids is empty:
irb(main):024:0> ::Katello::Pulp::Consumer.new(uuid).applicable_ids('rpm')
=> []
I don’t get it. I have added that server just like all the others. Those updates where shown for all the other CentOS 7 servers before I have updated them.
Not much help, sorry, but have the same issue @gvde. This has started before 3.15 for me. Not sure exactly when though.
There is always one host not showing updates. Sometimes unsubscribing/resubscribing makes it work for that host again but then the issue “moves” to a different host.
It gets even weirded now: after I have been away for a few hours, now the host the 2 tracer updates as applicable. The microcode_ctl update is still missing. I don’t get it, what’s going on…
O.K. There must be something going on in the background. I didn’t touch anything, made no updates etc. since my last post. This morning, the server now shows three available updates as it should. There must be a regular cleanup or something like that in pulp, I guess…
You are right: the issue moves. There are some new systemd updates available since this night. Just published those. The host with the missing updates before which looked good this morning also picked those up correctly.
But now there is one other host which shows no updates, even though there should like all the others. yum check-update shows the updates correctly.
Thus it really looks like a transient issue with a single host.
@iballou Any pointers how to deal with this issue?
This single host with the applicability issue, is it always an EL7 client? I’m curious if you see the older packages correctly in the client’s /var/lib/rhsm/cache/profile.json. If not, then I’m still suspecting a Pulp bug.
Do you see any tasks that occurred automatically in your task monitor? I’m curious to figure out what happened there.
For me: so far, the two hosts where I have noticed are CentOS 7.
Well, currently, the server has systemd-219-73.el7_8.6.x86_64 installed and 219-73.el7_8.8 is available. profile.json shows the installed version:
"arch": "x86_64",
"epoch": 0,
"name": "systemd",
"release": "73.el7_8.6",
"vendor": "CentOS",
"version": "219"
},
When I check in the web frontend I also see that the old version is installed.
There are a lot of tasks running automatically. I don’t really know what I should look for. The most curious to me are the “Clean up StoredValues”. Otherwise, there are a lot of syncs, combined profile updates, some generate applicability (one of them ended with an exception/error)
I mistyped before, I’m thinking since the older packages are in the profile correctly then this might be a Pulp 2 bug still.
Can I see the error? Might not be related, but just in case.
Action:
Actions::Katello::Host::GenerateApplicability
Input:
{}
Output:
{}
Exception:
NoMethodError: undefined method `map' for #<Host::Managed:0x000000000abdf8a8>
Did you mean? mac
tap
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-5.2.1/lib/active_model/attribute_methods.rb:430:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:99:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.2/app/lib/actions/katello/host/generate_applicability.rb:10:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:517:in `block (3 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:36:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:36:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.2/app/lib/actions/middleware/keep_locale.rb:7:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_request_id.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_request_id.rb:31:in `with_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_request_id.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_timezone.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_timezone.rb:31:in `with_current_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_timezone.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_user.rb:31:in `with_current_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:36:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:516:in `block (2 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:420:in `concurrence'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:515:in `block in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:475:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:475:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:475:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:514:in `execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/plan_step.rb:55:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan.rb:286:in `block (2 levels) in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan.rb:374:in `with_planning_scope'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan.rb:285:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:44:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/common/transaction.rb:17:in `block in rollback_on_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/transaction_adapters/active_record.rb:6:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/common/transaction.rb:16:in `rollback_on_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/common/transaction.rb:6:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan.rb:284:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/world.rb:211:in `block (2 levels) in plan_with_options'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/coordinator.rb:326:in `acquire'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/world.rb:209:in `block in plan_with_options'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/world.rb:208:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/world.rb:208:in `plan_with_options'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/world.rb:204:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/world.rb:180:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/lib/foreman_tasks.rb:23:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/lib/foreman_tasks.rb:29:in `block in trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/lib/foreman_tasks.rb:49:in `block in rails_safe_trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/lib/foreman_tasks.rb:48:in `rails_safe_trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/lib/foreman_tasks.rb:27:in `trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/lib/foreman_tasks.rb:58:in `sync_task'
(irb):13:in `irb_binding'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb/workspace.rb:85:in `eval'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb/workspace.rb:85:in `evaluate'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb/context.rb:380:in `evaluate'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb.rb:491:in `block (2 levels) in eval_input'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb.rb:623:in `signal_status'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb.rb:488:in `block in eval_input'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb/ruby-lex.rb:246:in `block (2 levels) in each_top_level_statement'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb/ruby-lex.rb:232:in `loop'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb/ruby-lex.rb:232:in `block in each_top_level_statement'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb/ruby-lex.rb:231:in `catch'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb/ruby-lex.rb:231:in `each_top_level_statement'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb.rb:487:in `eval_input'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb.rb:428:in `block in run'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb.rb:427:in `catch'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb.rb:427:in `run'
/opt/rh/rh-ruby25/root/usr/share/ruby/irb.rb:383:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/commands/console/console_command.rb:64:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/commands/console/console_command.rb:19:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/commands/console/console_command.rb:96:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
/opt/theforeman/tfm/root/usr/share/gems/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/command/base.rb:65:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/command.rb:46:in `invoke'
/usr/share/foreman/lib/tasks/console.rake:5:in `block in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
Interesting, it appears that error would only pop up if you try to calculate applicability for a single host. @ppascher do you see this error in your task monitor too?
Not convinced it’s related yet, but it’s a bug for sure.
O.K. I think I have cause that error before. I wanted to generate applicability for one host only and did a
hosts = Host.find_by(name: "test.example.com")
ForemanTasks.sync_task(::Actions::Katello::Host::GenerateApplicability, hosts)
which causes that error.
hosts = Array(Host.find_by(name: "test.example.com"))
would have been better… By bad.
Ah! Thanks for letting me know, I wasn’t seeing where this bug would pop up under normal circumstances.
I will ask the Pulp team about this in case they have any ideas.
It’s really odd. Now for days, it doesn’t show the applicable updates for that one particular host. I can’t tell under which circumstances the problem would “move” to another host…
@iballou Do you have any feedback from the pulp team?