Foreman smart-proxies sync problem ERF12-0457

Support
1

Problem: When i sync my smart proxies i get: ERF12-0457 [ProxyAPI::ProxyException]: Unable to update hosts ([RestClient::InternalServerError]: 500 Internal Server Error) for proxy https://foreman-smartproxie.com:9090/container_gateway/update_hosts

Expected outcome:

Foreman and Proxy versions:

  • foreman-3.16.2-1.el9.noarch

  • katello-4.18.1-1.el9.noarch

Foreman and Proxy plugin versions:

Distribution and version:

RHEL 9.7

Other relevant data:

2

Hello senetm,
Thank you for sharing error details.

  • If I am correct then this error is associated with a 500 Internal Server Error on the Smart Proxy side, specifically when trying to refresh host-repository mappings or updating host records.
  • The error ERF12-0457 usually indicates that the Foreman server is unable to update host information on the Smart Proxy (often returning a 500 Internal Server Error).
  • Could you check the following and share?
  1. Smart Proxy Logs: Check the logs on the affected Smart Proxy at /var/log/foreman-proxy/proxy.log right after a failed update or sync.
  2. Service Status: Run foreman-maintain service status on both the main server and the Smart Proxy to ensure all pulpcore and katello services are running correctly.
  3. Certificates: Ensure there aren’t any expired certificates by running foreman-maintain health check --all on the main server
1 Like
3

500 could be also out of space (RAM or storage) or anything, logs would help here.

When i sync my smart proxies …

Hmm, more proxies - do all of them fail, or only one particular? Optimized sync or Complete sync? Immediate or on_demand download policy on the failing one(s)?

4

The problem was that the plugin crashes when a host UUID is nil:

var/log/foreman-proxy/proxy.log

2026-02-18T14:29:07 1f244ce4 [I] Started PUT /container_gateway/update_hosts/2026-02-18T14:29:07 1f244ce4 [W] Error processing request '1f244ce4-4560-4c0c-bd66-e1c9c6314866: Sequel::NotNullConstraintViolation: PG::NotNullViolation: ERROR:  null value in column “uuid” of relation “hosts” violates not-null constraintDETAIL:  Failing row contains (204, null).

So i then repaired the UUIDs and sync is working again.

foreman-rake console

require 'securerandom'

Host.where(uuid: [nil, ""]).find_each(batch_size: 200) do |h|
  h.update_column(:uuid, SecureRandom.uuid)
  puts "Fixed #{h.name}"
end
1 Like
5

What is the version of smart_proxy_container_gateway on your smart proxy? We have pushed a fix for this in version: 3.4.2 and Katello 4.20.

A host not having a UUID would mean it is unregistered but not deleted.There is a setting that controls that called “Delete Host upon unregister” which by default is no, which leaves these host records with UUIDs cleaned out in the system which are reused if the system re-registers.

6

It is:

rubygem-smart_proxy_container_gateway.noarch 3.4.1-1.fm3_16.el9 @foreman-plugins

7

We have been fighting this exact same issue for weeks. We also have rubygem-smart_proxy_container_gateway-3.4.1-1.fm3_16.el9.noarch installed on our proxy server.

What exactly needs to be done to resolve the proxy sync issue?

How do we upgrade the container gateway package to 3.4.2 and Katello to 4.20?

Are the upgraded packages required only on the proxy server(s)?

8

@senetm It looks like you’ve identified a data inconsistency where hosts were missing UUIDs, which the container_gateway on the Smart Proxy requires. your foreman-rake console snippet is the correct way to backfill those missing IDs.
Which version of Foreman/Katello are you running?

Assuming you are running Foreman 3.18 / Katello 4.20, I would highly suggest upgrading your environment to the latest stable release (Foreman 3.18 / Katello 4.20).
To ensure a successful transition, please make sure these things:

Sequential Path: Katello requires a sequential upgrade path. Ensure you are moving from 4.18 → 4.19 → 4.20 without skipping versions.

Version Parity: It is critical that your Main Foreman Server and all Smart Proxies stay on the same major/minor version (e.g., all on 3.18.x). Mismatched versions are a frequent cause of API communication failures.

The Upgrade Process:

  1. Sync the new 3.18 repositories.

  2. Run foreman-maintain upgrade run --target-version 3.18 on the main server.

  3. Once the main server is updated, repeat the process for your Smart Proxies.

Pre-check: Before starting, run foreman-rake katello:upgrade_check to identify any paused tasks or stalled syncs that might block the installer.

Keeping everything on 3.18.x should resolve the database constraint issues we’ve seen in earlier versions.

9

I am running now:

  • foreman-3.16.2-1.el9.noarch

  • katello-4.18.1-1.el9.noarch

1 Like
11

Apologies, please correct the upgrade process, follow steps given below

  1. Update repositories
# dnf upgrade https://yum.theforeman.org/releases/3.18/el9/x86_64/foreman-release.rpm \
https://yum.theforeman.org/katello/4.20/katello/el9/x86_64/katello-repos-latest.rpm
  1. Upgrade all packages
# dnf upgrade
  1. Run foreman installer command
# foreman-installer
  1. Optional, below command determines if system needs reboot
# dnf needs-restarting --reboothint
  1. If previous command told you to reboot the system, then use command
# reboot
12

@vijaysawant Will there be fixes for the “proxy failing to sync” (aka the null value being inserted into the uuid field) issue implemented into Foreman versions 3.16 and 3.17? Or is the only path forward to move to 3.18?

13

@erlaprad : The fix is available in https://yum.theforeman.org/plugins/3.17/el9/x86_64/rubygem-smart_proxy_container_gateway-3.4.2-1.fm3_17.el9.noarch.rpm which is available in foreman 3.17 and onwards..

Independently, we also have this: Bug #38862: Smart proxy sync fails to update hosts table - Katello - Foreman which also fixes this. This is targeted to 4.18.z but no z streams have been released since we merged this. You don’t need this for the issue you’re seeing. You can upgrade to 3.4.2 container_gateway with foreman 3.17 and you should be good.

2 Likes
14

We have 4.19.1 in the oven and you should see it out soon: Katello 4.19.1 Release Process

15

@sajha We successfully upgraded our development Foreman server and proxy to 3.17 yesterday (which included container_gateway 3.4.2) and it appears the uuid null problem has been resolved. However, we are now seeing the following errors in the proxy.log when attempting to sync our proxy server:

2026-02-25T11:58:13 bd73d959 [I] Started PUT /container_gateway/host_repository_mapping/
2026-02-25T11:58:13 bd73d959 [W] Error processing request ‘bd73d959-3e11-4dfd-acc3-629b159814ff: : undefined method filter_map' for nil:NilClass /usr/share/gems/gems/smart_proxy_container_gateway-3.4.2/lib/smart_proxy_container_gateway/container_gateway_main.rb:209:in block in build_host_repository_mapping’
/usr/share/gems/gems/smart_proxy_container_gateway-3.4.2/lib/smart_proxy_container_gateway/container_gateway_main.rb:208:in each' /usr/share/gems/gems/smart_proxy_container_gateway-3.4.2/lib/smart_proxy_container_gateway/container_gateway_main.rb:208:in flat_map’
/usr/share/gems/gems/smart_proxy_container_gateway-3.4.2/lib/smart_proxy_container_gateway/container_gateway_main.rb:208:in build_host_repository_mapping' /usr/share/gems/gems/smart_proxy_container_gateway-3.4.2/lib/smart_proxy_container_gateway/container_gateway_main.rb:193:in update_host_repo_mapping’
/usr/share/gems/gems/smart_proxy_container_gateway-3.4.2/lib/smart_proxy_container_gateway/container_gateway_api.rb:273:in `block in class:Api’

1 Like
Foreman proxy sync failing
16

Could someone please point me to the section of the release notes for 4.16 showing where this null field issue relates to? We upgraded not so long ago and saw nothing about it