On Foreman/Katello 3.13/4.15 and have a docker repo setup looking like this:
Sync Settings
Registry URL: https://registry-1.docker.io/
Upstream Repository Name: grafana/grafana-oss
Verify SSL: Yes
Download Policy: Immediate
Mirroring Policy: Content Only
Include Tags: latest, main, 9.4.*, 10.4.*, 11.4.*
Exclude Tags: *-source, *-ubuntu, *-beta*
Has worked fine for 1 year+ but now I get this error: “the JSON object must be str, bytes or bytearray, not NoneType”
from production logs:
2025-01-13T12:49:07 [E|bac|6ca52b71] the JSON object must be str, bytes or bytearray, not NoneType (Katello::Errors::Pulp3Error)
6ca52b71 | /usr/share/gems/gems/katello-4.15.0/app/lib/actions/pulp3/abstract_async_task.rb:107:in `block in check_for_errors'
6ca52b71 | /usr/share/gems/gems/katello-4.15.0/app/lib/actions/pulp3/abstract_async_task.rb:105:in `each'
6ca52b71 | /usr/share/gems/gems/katello-4.15.0/app/lib/actions/pulp3/abstract_async_task.rb:105:in `check_for_errors'
6ca52b71 | /usr/share/gems/gems/katello-4.15.0/app/lib/actions/pulp3/abstract_async_task.rb:161:in `poll_external_task'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/polling.rb:22:in `run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/cancellable.rb:15:in `run'
6ca52b71 | /usr/share/gems/gems/katello-4.15.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:33:in `run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
6ca52b71 | /usr/share/gems/gems/katello-4.15.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
6ca52b71 | /usr/share/gems/gems/katello-4.15.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
6ca52b71 | /usr/share/gems/gems/katello-4.15.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
6ca52b71 | /usr/share/gems/gems/katello-4.15.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
6ca52b71 | /usr/share/gems/gems/katello-4.15.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
6ca52b71 | /usr/share/gems/gems/katello-4.15.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
6ca52b71 | /usr/share/gems/gems/activesupport-6.1.7.10/lib/active_support/execution_wrapper.rb:91:in `wrap'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/progress.rb:29:in `with_progress_calculation'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action/progress.rb:15:in `run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:33:in `run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:28:in `pass'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware.rb:20:in `pass'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
6ca52b71 | /usr/share/gems/gems/foreman-tasks-10.0.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/stack.rb:24:in `call'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/middleware/world.rb:31:in `execute'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:588:in `catch'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:588:in `block in execute_run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `block in with_error_handling'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `catch'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:491:in `with_error_handling'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:583:in `execute_run'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/action.rb:304:in `execute'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/director.rb:95:in `execute'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in `block (2 levels) in perform'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors.rb:18:in `run_user_code'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in `block in perform'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in `with_telemetry'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `perform'
6ca52b71 | /usr/share/gems/gems/dynflow-1.9.0/lib/dynflow/executors/sidekiq/serialization.rb:28:in `perform'
6ca52b71 | [ sidekiq ]
6ca52b71 | [ concurrent-ruby ]
Getting the same error from two other repos:
keycloak/keycloak
structurizr/onpremises
Upgraded pretty recent to 3.13/4.15 but a little unsure if the error is arrived with the upgrade or not.