Smart proxy sync broken after upgrading from 3.10 to 3.11.2 (katello 4.13)

Problem:
after upgrading smart proxy from version 3.10.1 to 3.11.2 Foreman sync is obviously broken to smart proxies.
Pulp plugin missing for synchronizable content types: yum.
Repositories containing these content types will not be synced.

Expected outcome:
rpm sync works

Foreman and Proxy versions:
Foreman 3.11.2 Katello 4.13
Smart Proxy 3.11.2 Katello 4.13
Foreman and Proxy plugin versions:

Distribution and version:
RHEL 8.10
Other relevant data:

Actions::Pulp3::ContentGuard::Refresh

Input:

{"smart_proxy_id"=>2, "remote_user"=>"admin", "remote_cp_user"=>"admin", "current_request_id"=>"3b659f17-dd9c-470b-98ba-7da05446cb41", "current_timezone"=>"Europe/Vienna", "current_organization_id"=>1, "current_location_id"=>nil, "current_user_id"=>63}

Output:

{}

Exception:

PulpCertguardClient::ApiError: Error message: the server returns an error HTTP status code: 403 Response headers: {"date"=>"Tue, 24 Sep 2024 09:04:38 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, POST, HEAD, OPTIONS", "x-frame-options"=>"DENY", "content-length"=>"58", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "cross-origin-opener-policy"=>"same-origin", "correlation-id"=>"3b659f17-dd9c-470b-98ba-7da05446cb41", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 capsulegraz.wamas.com"} Response body: {"detail":"Authentication credentials were not provided."}

Backtrace:

/usr/share/gems/gems/pulp_certguard_client-3.49.6/lib/pulp_certguard_client/api_client.rb:83:in `call_api' /usr/share/gems/gems/pulp_certguard_client-3.49.6/lib/pulp_certguard_client/api/contentguards_rhsm_api.rb:250:in `list_with_http_info' /usr/share/gems/gems/pulp_certguard_client-3.49.6/lib/pulp_certguard_client/api/contentguards_rhsm_api.rb:168:in `list' /usr/share/gems/gems/katello-4.13.1/app/services/katello/pulp3/api/content_guard.rb:65:in `list' /usr/share/gems/gems/katello-4.13.1/app/services/katello/pulp3/api/content_guard.rb:32:in `refresh' /usr/share/gems/gems/katello-4.13.1/app/lib/actions/pulp3/content_guard/refresh.rb:10:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:33:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/katello-4.13.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run' /usr/share/gems/gems/katello-4.13.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user' /usr/share/gems/gems/katello-4.13.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config' /usr/share/gems/gems/katello-4.13.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user' /usr/share/gems/gems/katello-4.13.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user' /usr/share/gems/gems/katello-4.13.1/app/lib/actions/middleware/remote_action.rb:16:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run' /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:91:in `wrap' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action/progress.rb:29:in `with_progress_calculation' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action/progress.rb:15:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:33:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/world.rb:31:in `execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:588:in `catch' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:588:in `block in execute_run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `block in with_error_handling' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `catch' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `with_error_handling' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:583:in `execute_run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:304:in `execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/director.rb:70:in `execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in `block (2 levels) in perform' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors.rb:18:in `run_user_code' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in `block in perform' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in `with_telemetry' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `perform' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/serialization.rb:28:in `perform' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in `execute_job' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in `block (2 levels) in process' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:172:in `invoke' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in `block in process' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in `local' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in `block in <module:Sidekiq>' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in `stats' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in `call' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in `global' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in `block in dispatch' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in `prepare' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in `dispatch' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in `process' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in `process_one' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in `run' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in `watchdog' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread' /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

output from foreman rake console:

root@foreman01# (git)-[master] # cat << EOF | foreman-rake console
capsule = SmartProxy.find_by_name('capsulegraz.example.com')
pp capsule.capabilities(::SmartProxy::PULP3_FEATURE)
pp capsule.feature_details
smart_proxy_helper = ::Katello::SmartProxyHelper.new(capsule)
pp smart_proxy_helper.unsyncable_content_types
EOF
Loading production environment (Rails 6.1.7.8)
Switch to inspect mode.
capsule = SmartProxy.find_by_name('capsulegraz.example.com')
#<SmartProxy id: 2, name: "capsulegraz.example.com", url: "https://capsulegraz.example.com:9090", created_at: "2022-02-25 13:10:59.096159000 +0000", updated_at: "2024-08-28 12:31:35.401983000 +0000", pubkey: nil, expired_logs: "167083749...
pp capsule.capabilities(::SmartProxy::PULP3_FEATURE)
[]
[]
pp capsule.feature_details
{"Pulpcore"=>
  {:settings=>
    {"pulp_url"=>"https://capsulegraz.example.com",
     "mirror"=>true,
     "content_app_url"=>"https://capsulegraz.example.com/pulp/content",
     "username"=>nil,
     "password"=>nil,
     "client_authentication"=>["client_certificate"],
     "rhsm_url"=>"https://capsulegraz.example.com:443/rhsm"},
   :capabilities=>[]},
 "Logs"=>{:settings=>{}, :capabilities=>[]}}
{"Pulpcore"=>{:settings=>{"pulp_url"=>"https://capsulegraz.example.com", "mirror"=>true, "content_app_url"=>"https://capsulegraz.example.com/pulp/content", "username"=>nil, "password"=>nil, "client_authentication"=>["client_certificate"], "rhsm_url"=>"https://capsulegraz.example.com:443/rhsm"}, :capabilities=>[]}, "Logs"=>{:settings=>{}, :capabilities=>[]}}
smart_proxy_helper = ::Katello::SmartProxyHelper.new(capsule)
#<Katello::SmartProxyHelper:0x000055b99e3f76b0 @smart_proxy=#<SmartProxy id: 2, name: "capsulegraz.example.com", url: "https://capsulegraz.example.com:9090", created_at: "2022-02-25 13:10:59.096159000 +0000", updated_at: "2024-08-28 12:3...
pp smart_proxy_helper.unsyncable_content_types
["yum"]
["yum"]
root@capsulegraz# (git)-[master] # foreman-installer --help | grep yum
    --foreman-proxy-content-enable-yum                                     Enable the RPM content feature. This allows syncing, managing and serving RPM content to be consumed by package managers like yum and dnf. (current: true)

Can you show us the /etc/httpd/conf.d/10-rhsm-pulpcore-https-443.conf file from the capsulegraz host? Especially the <Location "/pulp/api/v3"> section?

And then the output of rpm -q foreman-installer python3.11-gunicorn

Hi @evgeni

after a restore to 3.10.1 and another upgrade to 3.11.2 the sync task succeeds, however I still get the message that yum is not syncable, also the foreman-rake console command displays the same as in my previous post.

here the output:

root@capsulegraz /etc/httpd/conf.d (git)-[master] # cat 10-rhsm-pulpcore-https-443.conf
# ************************************
# Vhost template in module puppetlabs-apache
# Managed by Puppet
# ************************************
#
<VirtualHost *:443>
  ServerName capsulegraz.example.com

  ## Vhost docroot
  DocumentRoot "/var/www/"

  ## Directories, there should at least be a declaration for /var/www/

  <Directory "/var/www/">
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Require all granted
  </Directory>

  ## Logging
  ErrorLog "/var/log/httpd/rhsm-pulpcore-https-443_error_ssl.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/rhsm-pulpcore-https-443_access_ssl.log" combined
  ErrorDocument 500 '{"displayMessage": "Internal error, contact administrator", "errors": ["Internal error, contact administrator"], "status": "500" }'
  ErrorDocument 503 '{"displayMessage": "Service unavailable or restarting, try later", "errors": ["Service unavailable or restarting, try later"], "status": "503" }'

  ## Request header rules
  ## as per http://httpd.apache.org/docs/2.4/mod/mod_headers.html#requestheader
  RequestHeader set X_RHSM_SSL_CLIENT_CERT "%{SSL_CLIENT_CERT}s"

  # SSL Proxy directives
  SSLProxyEngine On
  SSLProxyCACertificateFile "/etc/pki/katello/certs/katello-default-ca.crt"
  SSLProxyMachineCertificateFile "/etc/pki/katello/private/capsulegraz.example.com-foreman-proxy-client-bundle.pem"
  ProxyPass /katello/api/v2/repositories/ https://foreman01.example.com/katello/api/v2/repositories/
  <Location /katello/api/v2/repositories/>
    ProxyPassReverse https://foreman01.example.com/
  </Location>
  SSLProxyEngine On

  <Location "/pulp/repos">
    RequestHeader unset X-CLIENT-CERT
    RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
    ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600
    ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content
  </Location>
Alias /pub /var/www/html/pub

<Location /pub>
  <IfModule mod_passenger.c>
    PassengerEnabled off
  </IfModule>
  Options +FollowSymLinks +Indexes
  Require all granted
</Location>

  <Location "/pulp/content">
    RequestHeader unset X-CLIENT-CERT
    RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
    ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600
    ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content
  </Location>

  <Location "/pulp/api/v3">
    RequestHeader unset REMOTE-USER
    RequestHeader unset REMOTE_USER
    RequestHeader set REMOTE-USER "%{SSL_CLIENT_S_DN_CN}s" env=SSL_CLIENT_S_DN_CN
    RequestHeader set REMOTE-USER "admin" "expr=%{SSL_CLIENT_S_DN_CN} == 'foreman01.example.com'"
    RequestHeader set REMOTE-USER "admin" "expr=%{SSL_CLIENT_S_DN_CN} == 'capsulegraz.example.com'"
    ProxyPass unix:///run/pulpcore-api.sock|http://pulpcore-api/pulp/api/v3 timeout=600
    ProxyPassReverse unix:///run/pulpcore-api.sock|http://pulpcore-api/pulp/api/v3
  </Location>

  ProxyPass /pulp/assets/ unix:///run/pulpcore-api.sock|http://pulpcore-api/pulp/assets/
  ProxyPassReverse /pulp/assets/ unix:///run/pulpcore-api.sock|http://pulpcore-api/pulp/assets/

  ## Proxy rules
  ProxyRequests Off
  ProxyPreserveHost Off
  ProxyPass / h2://foreman01.example.com/ disablereuse=on retry=0
  ProxyPassReverse / h2://foreman01.example.com/

  ## SSL directives
  SSLEngine on
  SSLCertificateFile      "/etc/pki/katello/certs/katello-apache.crt"
  SSLCertificateKeyFile   "/etc/pki/katello/private/katello-apache.key"
  SSLCertificateChainFile "/etc/pki/katello/certs/katello-server-ca.crt"
  SSLVerifyClient         optional
  SSLVerifyDepth          10
  SSLCACertificateFile    "/etc/pki/katello/certs/katello-default-ca.crt"
  SSLOptions +StdEnvVars +ExportCertData +FakeBasicAuth
</VirtualHost>

rpm output

root@capsulegraz /etc/httpd/conf.d (git)-[master] # rpm -q foreman-installer python3.11-gunicorn
foreman-installer-3.11.2-1.el8.noarch
python3.11-gunicorn-22.0.0-1.el8.noarch

Good. The best guess I can offer is: you forgot to run the installer after the first upgrade and thus the REMOTE-USER section wasn’t correct for gunicorn 22.

Mhh. Can we see curl -k https://capsulegraz.example.com/pulp/api/v3/status/ and rpm -qa |grep pulp_ (on the capsule) please?

curl output:

{
    "versions": [
        {
            "component": "core",
            "version": "3.49.19",
            "package": "pulpcore",
            "module": "pulpcore.app",
            "domain_compatible": true
        },
        {
            "component": "certguard",
            "version": "3.49.19",
            "package": "pulpcore",
            "module": "pulp_certguard.app",
            "domain_compatible": true
        },
        {
            "component": "file",
            "version": "3.49.19",
            "package": "pulpcore",
            "module": "pulp_file.app",
            "domain_compatible": true
        },
        {
            "component": "rpm",
            "version": "3.26.1",
            "package": "pulp-rpm",
            "module": "pulp_rpm.app",
            "domain_compatible": true
        },
        {
            "component": "python",
            "version": "3.11.3",
            "package": "pulp-python",
            "module": "pulp_python.app",
            "domain_compatible": false
        },
        {
            "component": "deb",
            "version": "3.2.1",
            "package": "pulp_deb",
            "module": "pulp_deb.app",
            "domain_compatible": false
        },
        {
            "component": "container",
            "version": "2.20.3",
            "package": "pulp-container",
            "module": "pulp_container.app",
            "domain_compatible": false
        },
        {
            "component": "ansible",
            "version": "0.21.8",
            "package": "pulp-ansible",
            "module": "pulp_ansible.app",
            "domain_compatible": false
        }
    ],
    "online_workers": [
        {
            "pulp_href": "/pulp/api/v3/workers/01922399-f40a-75ab-89e3-9399c9adfe44/",
            "pulp_created": "2024-09-24T10:35:45.036170Z",
            "pulp_last_updated": "2024-09-24T10:35:45.036190Z",
            "name": "10962@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:12.239986Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            },
            "current_task": null
        },
        {
            "pulp_href": "/pulp/api/v3/workers/01922399-f74f-7dc0-b26f-30bbac18566f/",
            "pulp_created": "2024-09-24T10:35:45.872930Z",
            "pulp_last_updated": "2024-09-24T10:35:45.872950Z",
            "name": "10994@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:13.276322Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            },
            "current_task": null
        },
        {
            "pulp_href": "/pulp/api/v3/workers/01922399-f150-7eb7-b470-530cdb5d1a0b/",
            "pulp_created": "2024-09-24T10:35:44.338110Z",
            "pulp_last_updated": "2024-09-24T10:35:44.338133Z",
            "name": "10924@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:11.580473Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            },
            "current_task": null
        },
        {
            "pulp_href": "/pulp/api/v3/workers/01922399-f316-79c9-934d-3ed1eb380d34/",
            "pulp_created": "2024-09-24T10:35:44.791266Z",
            "pulp_last_updated": "2024-09-24T10:35:44.791286Z",
            "name": "10930@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:12.038923Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            },
            "current_task": null
        }
    ],
    "online_api_apps": [
        {
            "name": "11142@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:15.368507Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            }
        },
        {
            "name": "11172@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:15.372534Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            }
        },
        {
            "name": "11146@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:15.374523Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            }
        },
        {
            "name": "11174@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:15.375276Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            }
        },
        {
            "name": "11173@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:15.377342Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            }
        }
    ],
    "online_content_apps": [
        {
            "name": "11269@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:14.681130Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            }
        },
        {
            "name": "11274@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:14.878911Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            }
        },
        {
            "name": "11275@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:15.416017Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            }
        },
        {
            "name": "11307@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:08.653543Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            }
        },
        {
            "name": "11312@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:08.763885Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            }
        },
        {
            "name": "11313@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:08.469324Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            }
        },
        {
            "name": "11318@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:08.470627Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            }
        },
        {
            "name": "11321@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:08.740421Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            }
        },
        {
            "name": "11414@capsulegraz.example.com",
            "last_heartbeat": "2024-09-24T11:43:15.438106Z",
            "versions": {
                "deb": "3.2.1",
                "rpm": "3.26.1",
                "core": "3.49.19",
                "file": "3.49.19",
                "python": "3.11.3",
                "ansible": "0.21.8",
                "certguard": "3.49.19",
                "container": "2.20.3"
            }
        }
    ],
    "database_connection": {
        "connected": true
    },
    "redis_connection": {
        "connected": true
    },
    "storage": {
        "total": 528306708480,
        "used": 14865788928,
        "free": 491778023424
    },
    "content_settings": {
        "content_origin": "https://capsulegraz.example.com",
        "content_path_prefix": "/pulp/content/"
    },
    "domain_enabled": false
}

rpm -qa |grep pulp_

root@capsulegraz /etc/httpd/conf.d (git)-[master] # rpm -qa |grep pulp_
1 
root@capsulegraz /etc/httpd/conf.d (git)-[master] # rpm -qa |grep pulp                                                                                                                                                                   
pulpcore-obsolete-packages-1.2.0-1.el8.noarch
python3.11-pulp-glue-0.27.2-1.el8.noarch
python3.11-pulp-ansible-0.21.8-1.el8.noarch
python3.11-pulp-python-3.11.3-1.el8.noarch
python3.11-pulp-container-2.20.3-1.el8.noarch
rubygem-smart_proxy_pulp-3.3.0-1.fm3_10.el8.noarch
python3.11-pulp-rpm-3.26.1-1.el8.noarch
python3.11-pulpcore-3.49.19-1.el8.noarch
python3.11-pulp-deb-3.2.1-1.el8.noarch

thanks! this looks correct to me.

@katello can one of you have a look here, why the capsule is listed as not being able to sync yum repos?

Wonder if running “Refresh Features” on the smart proxy would resolve it. That message comes from looking at the features and capabilities on the proxy at runtime.

I ran refresh features at foreman, but still unsyncable content yum.

root@foreman01 /etc/wireguard (git)-[master] # hammer capsule refresh-features --name capsulegraz.example.com
Smart proxy features were refreshed.

Anything else I could look into?

Interestingly task Update Content Counts fails with Pulp 3 not supported:

Actions::Katello::CapsuleContent::UpdateContentCounts

Input:

{"smart_proxy_id"=>2, "current_request_id"=>nil, "current_timezone"=>"UTC", "current_organization_id"=>nil, "current_location_id"=>nil, "current_user_id"=>1}

Output:

{}

Exception:

RuntimeError: Pulp 3 not supported

Backtrace:

/usr/share/gems/gems/katello-4.13.1/app/models/katello/repository.rb:215:in `backend_service' /usr/share/gems/gems/katello-4.13.1/app/models/katello/concerns/smart_proxy_extensions.rb:152:in `block in update_content_counts!' /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/delegation.rb:88:in `each' /usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/delegation.rb:88:in `each' /usr/share/gems/gems/katello-4.13.1/app/models/katello/concerns/smart_proxy_extensions.rb:151:in `update_content_counts!' /usr/share/gems/gems/katello-4.13.1/app/lib/actions/katello/capsule_content/update_content_counts.rb:15:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:590:in `block (3 levels) in execute_run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run' /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/execution_wrapper.rb:91:in `wrap' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action/progress.rb:29:in `with_progress_calculation' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action/progress.rb:15:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:33:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:28:in `pass' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware.rb:20:in `pass' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user' /usr/share/gems/gems/foreman-tasks-9.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/stack.rb:24:in `call' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/middleware/world.rb:31:in `execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:589:in `block (2 levels) in execute_run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:588:in `catch' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:588:in `block in execute_run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `block in with_error_handling' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `catch' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:491:in `with_error_handling' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:583:in `execute_run' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/action.rb:304:in `execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract.rb:168:in `with_meta_calculation' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/director.rb:70:in `execute' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:12:in `block (2 levels) in perform' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors.rb:18:in `run_user_code' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:10:in `block in perform' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:26:in `with_telemetry' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `perform' /usr/share/gems/gems/dynflow-1.8.4/lib/dynflow/executors/sidekiq/serialization.rb:28:in `perform' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in `execute_job' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in `block (2 levels) in process' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:172:in `invoke' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in `block in process' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in `local' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in `block in <module:Sidekiq>' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in `stats' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in `call' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in `global' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in `block in dispatch' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in `prepare' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in `dispatch' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in `process' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in `process_one' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in `run' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in `watchdog' /usr/share/gems/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread' /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

After updating the katello-ca-consumer-latest.noarch.rpm on all non working smart proxies,
I reran the foreman-installer. All Smart-Proxy features are there and sync works now.

May a Katello dev could explain why this step was necessary.

1 Like

Had the same problem - fixed with:

  1. dnf install http://foreman/pub/katello-ca-consumer-latest.noarch.rpm
  2. foreman-installer
  3. foreman-maintain service restart
  4. trigger a smart-sync from /smart_proxies/:id