Foreman Upgrade failure - "A smart proxy seems to have been refreshed without pulpcore being running"

Support
, , ,
1

Problem:
We have a Foreman/Puppet server with katello that has been neglected for a while. I now wanted to update it from version 3.13 to 3.14 and then to 3.15. The upgrade fails at some point with the error that “A smart proxy seems to have been refreshed without pulpcore being running”. The whole message is below in the additional section.

The logs suggest that pulpcore hasn’t been running but all pulpcore services are up and running. Executing curl -k https://$(hostname -f)/pulp/api/v3/status/ returns a list of components so I assume pulp is running correctly.

We are not running any other smart proxies than the one that is built in.

I have tried upgrading this host multiple times by now and I can’t figure out where exactly this issue comes from. I have the feeling it has something to do with a problematic package but I can’t find concrete evidence for that.

Expected outcome:
The upgrade works without issue

Foreman and Proxy versions:
3.13.0

Foreman and Proxy plugin versions:

Name Version
foreman-tasks 10.0.1
foreman_openscap 10.0.1
foreman_puppet 8.0.0
foreman_remote_execution 14.0.2
katello 4.15.0
puppetdb_foreman 6.0.2

Distribution and version:
Rocky Linux 9.5
Other relevant data:
Error Message:

2025-09-09 15:45:06 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[puppetserver]: Could not evaluate: Error making GET request to Foreman at https://puppetserver/api/v2/smart_proxies: Response  
: 500 Internal Server Error: Check /var/log/foreman/production.log on puppetserver for detailed information  
2025-09-09 15:45:10 [NOTICE] [configure] System configuration has finished.  
  
Error 1: Puppet Foreman_smartproxy resource 'puppetserver' failed. Logs:  
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[puppetserver]/before  
   before to Cron[puppet]  
   before to Service[puppet]  
   before to Service[puppet-run.timer]  
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[puppetserver]  
   Adding autorequire relationship with Anchor[foreman::service]  
   Adding autorequire relationship with Anchor[foreman::providers::oauth]  
   Starting to evaluate the resource (1674 of 1702)  
   Could not evaluate: Error making GET request to Foreman at https://puppetserver/api/v2/smart_proxies: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on puppetserver for detailed information  
   Evaluated in 0.30 seconds  
 Foreman_smartproxy[puppetserver](provider=rest_v3)  
   Making get request to https://puppetserver/api/v2/smart_proxies?search=name%3D%22puppetserver%22  
   Received response 500 from request to https://puppetserver/api/v2/smart_proxies?search=name%3D%22puppetserver%22  
  
1 error was detected during installation.  
Please address the errors and re-run the installer to ensure the system is properly configured.  
Failing to do so is likely to result in broken functionality.

I can provide the production.log with the stack trace but it is very long so here is an excerpt

2025-09-09T15:45:06 [I|app|e69888e7] Backtrace for 'Action failed' error (ActionView::Template::Error): A smart proxy seems to have been refreshed without pulpcore being running. Please refresh the smart proxy af  
ter ensuring that pulpcore services are running.  
e69888e7 | /usr/share/gems/gems/katello-4.16.3/app/services/katello/repository_type_manager.rb:29:in `fix_pulp3_capabilities'  
e69888e7 | /usr/share/gems/gems/katello-4.16.3/app/services/katello/repository_type_manager.rb:35:in `enabled_repository_types'  
e69888e7 | /usr/share/gems/gems/katello-4.16.3/app/models/katello/concerns/smart_proxy_extensions.rb:422:in `supported_pulp_types'  
(more stacktrace...)

Installed Packages:


    candlepin-4.4.20-1.el9.noarch
    candlepin-selinux-4.4.20-1.el9.noarch
    dynflow-utils-1.6.3-1.el9.x86_64
    foreman-3.13.0-1.el9.noarch
    foreman-cli-3.13.0-1.el9.noarch
    foreman-dynflow-sidekiq-3.13.0-1.el9.noarch
    foreman-installer-3.13.0-1.el9.noarch
    foreman-installer-katello-3.13.0-1.el9.noarch
    foreman-obsolete-packages-1.10-1.el9.noarch
    foreman-postgresql-3.13.0-1.el9.noarch
    foreman-proxy-3.13.0-1.el9.noarch
    foreman-redis-3.13.0-1.el9.noarch
    foreman-release-3.13.0-1.el9.noarch
    foreman-selinux-3.13.0-1.el9.noarch
    foreman-service-3.13.0-1.el9.noarch
    katello-4.15.0-1.el9.noarch
    katello-certs-tools-2.10.0-1.el9.noarch
    katello-client-bootstrap-1.7.9-2.el9.noarch
    katello-common-4.15.0-1.el9.noarch
    katello-repos-4.15.0-1.el9.noarch
    katello-selinux-5.0.2-1.el9.noarch
    pulpcore-selinux-2.0.1-1.el9.x86_64
    python3.11-pulp-ansible-0.22.2-1.el9.noarch
    python3.11-pulp-cli-0.29.2-2.el9.noarch
    python3.11-pulp-container-2.22.0-1.el9.noarch
    python3.11-pulp-deb-3.3.1-1.el9.noarch
    python3.11-pulp-glue-0.29.2-2.el9.noarch
    python3.11-pulp-python-3.12.5-1.el9.noarch
    python3.11-pulp-rpm-3.27.2-1.el9.noarch
    python3.11-pulpcore-3.63.1-1.el9.noarch
    rubygem-dynflow-1.9.0-1.el9.noarch
    rubygem-foreman-tasks-10.0.1-1.fm3_13.el9.noarch
    rubygem-foreman_maintain-1.8.1-2.el9.noarch
    rubygem-foreman_openscap-10.0.1-1.fm3_13.el9.noarch
    rubygem-foreman_puppet-8.0.0-1.fm3_13.el9.noarch
    rubygem-foreman_remote_execution-14.0.2-1.fm3_13.el9.noarch
    rubygem-hammer_cli-3.13.0-1.el9.noarch
    rubygem-hammer_cli_foreman-3.13.0-1.el9.noarch
    rubygem-hammer_cli_foreman_puppet-0.1.0-1.fm3_11.el9.noarch
    rubygem-hammer_cli_foreman_remote_execution-0.3.0-1.el9.noarch
    rubygem-hammer_cli_foreman_tasks-0.0.21-1.fm3_11.el9.noarch
    rubygem-hammer_cli_katello-1.15.0-0.1.pre.main.20241104082525gitd2c7a35.el9.noarch
    rubygem-katello-4.15.0-1.el9.noarch
    rubygem-pulp_ansible_client-0.22.2-1.el9.noarch
    rubygem-pulp_certguard_client-3.63.1-1.el9.noarch
    rubygem-pulp_container_client-2.22.0-1.el9.noarch
    rubygem-pulp_deb_client-3.3.1-1.el9.noarch
    rubygem-pulp_file_client-3.63.1-1.el9.noarch
    rubygem-pulp_ostree_client-2.4.4-1.el9.noarch
    rubygem-pulp_python_client-3.12.5-1.el9.noarch
    rubygem-pulp_rpm_client-3.27.2-1.el9.noarch
    rubygem-pulpcore_client-3.63.1-1.el9.noarch
    rubygem-puppetdb_foreman-6.0.2-1.fm3_10.el9.noarch
    rubygem-puppetdb_foreman-doc-6.0.2-1.fm3_10.el9.noarch
    rubygem-smart_proxy_pulp-3.4.0-1.fm3_13.el9.noarch

Thanks to everybody reading through all of this :slight_smile:

1 Like
2

We’ll need to dig into how capabilities are registered on the smart proxy to answer, I can’t recall how it works off the top of my head. Maybe @ekohl or @aruzicka has has a tip.

When the upgrade fails, you’re able to see that Pulp is still running? I wasn’t sure if you meant Pulp was running fine pre or post the upgrade attempt.

3

I can’t recall how it works off the top of my head

Glancing at the code, before a smart proxy object is saved, foreman reaches out to the actual proxy to load it’s features, capabilities are a part of that. On the proxy side, they’re filled in during plugin initialization in any way the authors deem fit iirc. In smart_proxy_pulp’s case, that means calling to /pulp/api/v3/status/ and picking some bits out of that.

1 Like
4

Thank you for your answers.

Yes Pulp is running post failed upgrade as well.

Since I was tearing my hairs out trying to make this upgrade work we decided it would be easier to just set up a new Foreman and move everything over than risk having a half-working system. Especially since 3.16 came out. I think we can close this topic.

5

Thanks for the update. If other folks hit this in the future, please show us the output of Pulp status. It’s possible too that a certificate issue on upgrade could cause Foreman to be unable to talk to Pulp.

6

Hello!
I have same problem after update from 3.15 to 3.16.1. Also I try update from 3.16.1 to night version for fix, but error not fixed.

When run foreman-installer have some errors with “Puppet Foreman_host” and “Puppet Foreman_smartproxy“ resource, in /var/log/foreman/production.log all write the same:
8139cda5 | Katello::Errors::PulpcoreMissingCapabilities (A smart proxy seems to have been refreshed without pulpcore being running. Please refresh the smart proxy after ensuring that pulpcore services are running.):
8139cda5 |
8139cda5 | katello (4.20.0.pre.master) app/services/katello/repository_type_manager.rb:29:in `fix_pulp3_capabilities'
8139cda5 | katello (4.20.0.pre.master) app/services/katello/repository_type_manager.rb:40:in `enabled_repository_types'
8139cda5 | katello (4.20.0.pre.master) app/services/katello/repository_type_manager.rb:81:in `generic_content_types'
8139cda5 | katello (4.20.0.pre.master) app/controllers/katello/api/v2/generic_content_units_controller.rb:5:in `block (2 levels) in <class:GenericContentUnitsController>'
8139cda5 | config/initializers/apipie.rb:191:in `values'
8139cda5 | config/initializers/apipie.rb:199:in `description'
8139cda5 | lib/foreman/middleware/logging_context_session.rb:22:in `call'
8139cda5 | lib/foreman/middleware/logging_context_request.rb:11:in `call'
8139cda5 | katello (4.20.0.pre.master) lib/katello/prevent_json_parsing.rb:12:in `call'

Pulp status and curl foreman.local/pulp/api/v3/status/ show correct status of pulp, but I can’t attach output file becouse just register.
Component version and final status:
{
"versions": [
{
"component": "core",
"version": "3.85.1",
"package": "pulpcore",
"module": "pulpcore.app",
"domain_compatible": true
},
{
"component": "python",
"version": "3.19.1",
"package": "pulp-python",
"module": "pulp_python.app",
"domain_compatible": true
},
{
"component": "certguard",
"version": "3.85.1",
"package": "pulpcore",
"module": "pulp_certguard.app",
"domain_compatible": true
},
{
"component": "file",
"version": "3.85.1",
"package": "pulpcore",
"module": "pulp_file.app",
"domain_compatible": true
},
{
"component": "ansible",
"version": "0.28.0",
"package": "pulp-ansible",
"module": "pulp_ansible.app",
"domain_compatible": true
},
{
"component": "container",
"version": "2.26.3",
"package": "pulp-container",
"module": "pulp_container.app",
"domain_compatible": true
},
{
"component": "deb",
"version": "3.7.0",
"package": "pulp_deb",
"module": "pulp_deb.app",
"domain_compatible": true
},
{
"component": "ostree",
"version": "2.5.3",
"package": "pulp-ostree",
"module": "pulp_ostree.app",
"domain_compatible": true
},
{
"component": "rpm",
"version": "3.32.5",
"package": "pulp-rpm",
"module": "pulp_rpm.app",
"domain_compatible": true
}
],
"online_workers": [
{… cut … }
],
"database_connection": {
"connected": true
},
"redis_connection": {
"connected": true
},
"storage": {
"total": 749784940544,
"used": 571924254720,
"free": 146255073280
},
"content_settings": {
"content_origin": "``https://foreman.local``",
"content_path_prefix": "/pulp/content/"
},
"domain_enabled": false
}

7

Were you expecting the ostree plugin? We usually do not enable this on installs.

8

I enable all pulp modules when install:

…

–foreman-proxy-content-enable-ansible=true \
–foreman-proxy-content-enable-deb=true \
–foreman-proxy-content-enable-docker=true \
–foreman-proxy-content-enable-file=true \
–foreman-proxy-content-enable-ostree=true \
–foreman-proxy-content-enable-python=true \
–foreman-proxy-content-enable-yum=true \
–foreman-proxy-content-pulpcore-allowed-content-checksums=md5 \
–foreman-proxy-content-pulpcore-allowed-content-checksums=sha1 \
–foreman-proxy-content-pulpcore-allowed-content-checksums=sha224 \
–foreman-proxy-content-pulpcore-allowed-content-checksums=sha256 \
–foreman-proxy-content-pulpcore-allowed-content-checksums=sha384 \
–foreman-proxy-content-pulpcore-allowed-content-checksums=sha512 \

…

Having not solved the problem till 2 days, I performed a clean install today. I use latest stable version foreman 3.16 and katello 4.18 with my opts restored from backuped katello-answers.yaml. It failed with the same errors.

I try to disable some option. And I found that after resetting these options to the default value, everything worked:

–foreman-proxy-dns=true \
–foreman-proxy-http=true

1 Like
9

Interesting, so somehow --foreman-proxy-dns and --foreman-proxy-http affect the connection from Foreman to Pulp. I’ll have to check up on what those settings do.