Foreman remote executions fail due to incorrect smart proxy selection as it load balancing between local and remote smart proxy

**Problem:foreman remote executions fail due to incorrect smart proxy selection as it load balancing between local and remote smart proxy

**Expected outcome:100% remote exection success

*Foreman and Proxy versions:
candlepin-4.1.10-1.el7.noarch
candlepin-selinux-4.1.10-1.el7.noarch
foreman-3.2.0-1.el7.noarch
foreman-cli-3.2.0-1.el7.noarch
foreman-debug-3.2.0-1.el7.noarch
foreman-dynflow-sidekiq-3.2.0-1.el7.noarch
foreman-installer-3.2.0-1.el7.noarch
foreman-installer-katello-3.2.0-1.el7.noarch
foreman-postgresql-3.2.0-1.el7.noarch
foreman-proxy-3.2.0-1.el7.noarch
foreman-release-3.2.0-1.el7.noarch
foreman-service-3.2.0-1.el7.noarch
katello-4.4.0-1.el7.noarch
katello-ca-consumer-sul-d4t-lcsm.d4t4solutions.com-1.0-1.noarch
katello-certs-tools-2.8.2-1.el7.noarch
katello-client-bootstrap-1.7.8-1.el7.noarch
katello-common-4.4.0-1.el7.noarch
katello-debug-4.4.0-1.el7.noarch
katello-repos-4.4.0-1.el7.noarch
python-pulp-bindings-2.21.5-1.el7.noarch
python-pulp-client-lib-2.21.5-1.el7.noarch
python-pulp-common-2.21.5-1.el7.noarch
python-pulp-rpm-common-2.21.5-1.el7.noarch
qpid-proton-c-0.37.0-1.el7.x86_64
rubygem-foreman_maintain-1.0.3-1.el7.noarch
tfm-pulpcore-python3-aiodns-3.0.0-2.el7.noarch
tfm-pulpcore-python3-aiofiles-0.7.0-2.el7.noarch
tfm-pulpcore-python3-aiohttp-3.8.1-2.el7.x86_64
tfm-pulpcore-python3-aiohttp-socks-0.6.0-2.el7.noarch
tfm-pulpcore-python3-aiohttp-xmlrpc-1.5.0-1.el7.noarch
tfm-pulpcore-python3-aioredis-2.0.0-2.el7.noarch
tfm-pulpcore-python3-aiosignal-1.2.0-1.el7.noarch
tfm-pulpcore-python3-ansible-builder-1.0.1-2.el7.noarch
tfm-pulpcore-python3-asgiref-3.4.1-1.el7.noarch
tfm-pulpcore-python3-async-lru-1.0.2-2.el7.noarch
tfm-pulpcore-python3-async-timeout-4.0.2-1.el7.noarch
tfm-pulpcore-python3-asyncio-throttle-1.0.2-2.el7.noarch
tfm-pulpcore-python3-attrs-21.2.0-2.el7.noarch
tfm-pulpcore-python3-backoff-1.11.1-1.el7.noarch
tfm-pulpcore-python3-bandersnatch-4.4.0-4.el7.noarch
tfm-pulpcore-python3-bindep-2.10.1-1.el7.noarch
tfm-pulpcore-python3-bleach-3.3.1-1.el7.noarch
tfm-pulpcore-python3-bleach-allowlist-1.0.3-2.el7.noarch
tfm-pulpcore-python3-brotli-1.0.9-1.el7.x86_64
tfm-pulpcore-python3-cchardet-2.1.7-1.el7.x86_64
tfm-pulpcore-python3-certifi-2020.6.20-2.el7.noarch
tfm-pulpcore-python3-cffi-1.15.0-1.el7.x86_64
tfm-pulpcore-python3-chardet-3.0.4-3.el7.noarch
tfm-pulpcore-python3-charset-normalizer-2.0.7-1.el7.noarch
tfm-pulpcore-python3-click-8.0.3-1.el7.noarch
tfm-pulpcore-python3-colorama-0.4.4-2.el7.noarch
tfm-pulpcore-python3-createrepo_c-0.17.7-5.1.el7.x86_64
tfm-pulpcore-python3-cryptography-3.1.1-1.el7.x86_64
tfm-pulpcore-python3-dateutil-2.8.2-1.el7.noarch
tfm-pulpcore-python3-debian-0.1.42-1.el7.noarch
tfm-pulpcore-python3-defusedxml-0.7.1-2.el7.noarch
tfm-pulpcore-python3-diff-match-patch-20200713-2.el7.noarch
tfm-pulpcore-python3-distro-1.6.0-2.el7.noarch
tfm-pulpcore-python3-django-3.2.9-1.el7.noarch
tfm-pulpcore-python3-django-currentuser-0.5.3-3.el7.noarch
tfm-pulpcore-python3-django-filter-21.1-1.el7.noarch
tfm-pulpcore-python3-django-guardian-2.4.0-3.el7.noarch
tfm-pulpcore-python3-django-guid-3.2.0-2.el7.noarch
tfm-pulpcore-python3-django-import-export-2.6.1-1.el7.noarch
tfm-pulpcore-python3-django-lifecycle-0.9.3-1.el7.noarch
tfm-pulpcore-python3-django-prometheus-2.1.0-2.el7.noarch
tfm-pulpcore-python3-django-readonly-field-1.0.5-3.el7.noarch
tfm-pulpcore-python3-djangorestframework-3.12.4-4.el7.noarch
tfm-pulpcore-python3-djangorestframework-queryfields-1.0.0-4.el7.noarch
tfm-pulpcore-python3-drf-access-policy-1.1.0-1.el7.noarch
tfm-pulpcore-python3-drf-nested-routers-0.93.3-3.el7.noarch
tfm-pulpcore-python3-drf-spectacular-0.20.1-1.el7.noarch
tfm-pulpcore-python3-dynaconf-3.1.7-2.el7.noarch
tfm-pulpcore-python3-ecdsa-0.13.3-3.el7.noarch
tfm-pulpcore-python3-et-xmlfile-1.1.0-1.el7.noarch
tfm-pulpcore-python3-filelock-3.0.12-2.el7.noarch
tfm-pulpcore-python3-flake8-3.9.2-3.el7.noarch
tfm-pulpcore-python3-frozenlist-1.3.0-1.el7.x86_64
tfm-pulpcore-python3-future-0.18.2-4.el7.noarch
tfm-pulpcore-python3-galaxy-importer-0.4.1-2.el7.noarch
tfm-pulpcore-python3-gnupg-0.4.7-2.el7.noarch
tfm-pulpcore-python3-gunicorn-20.1.0-3.el7.noarch
tfm-pulpcore-python3-idna-3.3-1.el7.noarch
tfm-pulpcore-python3-idna-ssl-1.1.0-4.el7.noarch
tfm-pulpcore-python3-importlib-metadata-1.7.0-2.el7.noarch
tfm-pulpcore-python3-importlib-resources-5.0.0-2.el7.noarch
tfm-pulpcore-python3-inflection-0.5.1-2.el7.noarch
tfm-pulpcore-python3-iniparse-0.4-34.el7.noarch
tfm-pulpcore-python3-jinja2-3.0.2-1.el7.noarch
tfm-pulpcore-python3-jsonschema-3.2.0-7.el7.noarch
tfm-pulpcore-python3-libcomps-0.1.18-1.el7.x86_64
tfm-pulpcore-python3-lxml-4.6.4-1.el7.x86_64
tfm-pulpcore-python3-markdown-3.3.4-4.el7.noarch
tfm-pulpcore-python3-markuppy-1.14-2.el7.noarch
tfm-pulpcore-python3-markupsafe-2.0.1-2.el7.x86_64
tfm-pulpcore-python3-mccabe-0.6.1-2.el7.noarch
tfm-pulpcore-python3-multidict-5.2.0-1.el7.x86_64
tfm-pulpcore-python3-odfpy-1.4.1-5.el7.noarch
tfm-pulpcore-python3-openpyxl-3.0.9-1.el7.noarch
tfm-pulpcore-python3-packaging-21.2-1.el7.noarch
tfm-pulpcore-python3-parsley-1.3-1.el7.noarch
tfm-pulpcore-python3-pbr-5.6.0-1.el7.noarch
tfm-pulpcore-python3-pkginfo-1.7.1-3.el7.noarch
tfm-pulpcore-python3-productmd-1.33-2.el7.noarch
tfm-pulpcore-python3-prometheus-client-0.8.0-2.el7.noarch
tfm-pulpcore-python3-psycopg2-2.9.1-1.el7.x86_64
tfm-pulpcore-python3-pulp-ansible-0.10.1-1.el7.noarch
tfm-pulpcore-python3-pulp-certguard-1.5.1-1.el7.noarch
tfm-pulpcore-python3-pulp-container-2.9.2-1.el7.noarch
tfm-pulpcore-python3-pulp-deb-2.16.1-1.el7.noarch
tfm-pulpcore-python3-pulp-file-1.10.1-1.el7.noarch
tfm-pulpcore-python3-pulp-python-3.5.2-1.el7.noarch
tfm-pulpcore-python3-pulp-rpm-3.17.3-2.el7.noarch
tfm-pulpcore-python3-pulpcore-3.16.6-1.el7.noarch
tfm-pulpcore-python3-pyOpenSSL-19.1.0-2.el7.noarch
tfm-pulpcore-python3-pycairo-1.20.1-2.el7.x86_64
tfm-pulpcore-python3-pycares-4.1.2-1.el7.x86_64
tfm-pulpcore-python3-pycodestyle-2.7.0-4.el7.noarch
tfm-pulpcore-python3-pycparser-2.20-2.el7.noarch
tfm-pulpcore-python3-pycryptodomex-3.11.0-1.el7.x86_64
tfm-pulpcore-python3-pyflakes-2.3.1-4.el7.noarch
tfm-pulpcore-python3-pygobject-3.40.1-2.el7.x86_64
tfm-pulpcore-python3-pygtrie-2.4.2-2.el7.noarch
tfm-pulpcore-python3-pyjwkest-1.4.2-5.el7.noarch
tfm-pulpcore-python3-pyjwt-1.7.1-7.el7.noarch
tfm-pulpcore-python3-pyparsing-2.4.7-2.el7.noarch
tfm-pulpcore-python3-pyrsistent-0.18.0-1.el7.x86_64
tfm-pulpcore-python3-pytz-2021.3-1.el7.noarch
tfm-pulpcore-python3-pyyaml-5.4.1-3.el7.x86_64
tfm-pulpcore-python3-redis-3.5.3-2.el7.noarch
tfm-pulpcore-python3-requests-2.26.0-3.el7.noarch
tfm-pulpcore-python3-requirements-parser-0.2.0-2.el7.noarch
tfm-pulpcore-python3-rhsm-1.19.2-2.el7.x86_64
tfm-pulpcore-python3-semantic-version-2.8.5-2.el7.noarch
tfm-pulpcore-python3-setuptools-1.0-4.el7.x86_64
tfm-pulpcore-python3-six-1.16.0-1.el7.noarch
tfm-pulpcore-python3-socks-1.2.4-2.el7.noarch
tfm-pulpcore-python3-solv-0.7.20-4.el7.x86_64
tfm-pulpcore-python3-sqlparse-0.4.2-2.el7.noarch
tfm-pulpcore-python3-tablib-3.1.0-1.el7.noarch
tfm-pulpcore-python3-typing-extensions-3.10.0.2-1.el7.noarch
tfm-pulpcore-python3-uritemplate-4.1.1-1.el7.noarch
tfm-pulpcore-python3-url-normalize-1.4.3-3.el7.noarch
tfm-pulpcore-python3-urllib3-1.26.7-1.el7.noarch
tfm-pulpcore-python3-urlman-1.4.0-2.el7.noarch
tfm-pulpcore-python3-webencodings-0.5.1-2.el7.noarch
tfm-pulpcore-python3-whitenoise-5.3.0-1.el7.noarch
tfm-pulpcore-python3-xlrd-2.0.1-4.el7.noarch
tfm-pulpcore-python3-xlwt-1.3.0-2.el7.noarch
tfm-pulpcore-python3-yarl-1.7.2-1.el7.x86_64
tfm-pulpcore-python3-zipp-3.4.0-3.el7.noarch
tfm-pulpcore-runtime-1.0-4.el7.x86_64
tfm-rubygem-actioncable-6.0.3.7-1.el7.noarch
tfm-rubygem-actionmailbox-6.0.3.7-1.el7.noarch
tfm-rubygem-actionmailer-6.0.3.7-1.el7.noarch
tfm-rubygem-actionpack-6.0.3.7-1.el7.noarch
tfm-rubygem-actiontext-6.0.3.7-1.el7.noarch
tfm-rubygem-actionview-6.0.3.7-1.el7.noarch
tfm-rubygem-activejob-6.0.3.7-1.el7.noarch
tfm-rubygem-activemodel-6.0.3.7-1.el7.noarch
tfm-rubygem-activerecord-6.0.3.7-1.el7.noarch
tfm-rubygem-activerecord-import-1.1.0-1.el7.noarch
tfm-rubygem-activerecord-session_store-2.0.0-1.el7.noarch
tfm-rubygem-activestorage-6.0.3.7-1.el7.noarch
tfm-rubygem-activesupport-6.0.3.7-1.el7.noarch
tfm-rubygem-addressable-2.8.0-1.el7.noarch
tfm-rubygem-algebrick-0.7.3-8.el7.noarch
tfm-rubygem-amazing_print-1.1.0-2.el7.noarch
tfm-rubygem-ancestry-3.0.7-2.el7.noarch
tfm-rubygem-anemone-0.7.2-23.el7.noarch
tfm-rubygem-angular-rails-templates-1.1.0-2.el7.noarch
tfm-rubygem-ansi-1.5.0-3.el7.noarch
tfm-rubygem-apipie-bindings-0.4.0-2.el7.noarch
tfm-rubygem-apipie-dsl-2.4.0-1.el7.noarch
tfm-rubygem-apipie-params-0.0.5-5.el7.noarch
tfm-rubygem-apipie-rails-0.5.17-4.el7.noarch
tfm-rubygem-audited-4.9.0-4.el7.noarch
tfm-rubygem-bcrypt-3.1.12-4.el7.x86_64
tfm-rubygem-builder-3.2.4-2.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-6.el7.noarch
tfm-rubygem-clamp-1.1.2-7.el7.noarch
tfm-rubygem-colorize-0.8.1-2.el7.noarch
tfm-rubygem-concurrent-ruby-1.1.6-3.el7.noarch
tfm-rubygem-concurrent-ruby-edge-0.6.0-3.fm2_5.el7.noarch
tfm-rubygem-connection_pool-2.2.2-3.el7.noarch
tfm-rubygem-crass-1.0.6-2.el7.noarch
tfm-rubygem-css_parser-1.4.7-5.el7.noarch
tfm-rubygem-daemons-1.2.3-7.el7.noarch
tfm-rubygem-deacon-1.0.0-5.el7.noarch
tfm-rubygem-deep_cloneable-3.0.0-4.el7.noarch
tfm-rubygem-deface-1.5.3-3.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-5.el7.noarch
tfm-rubygem-dynflow-1.6.4-1.fm3_2.el7.noarch
tfm-rubygem-erubi-1.9.0-2.el7.noarch
tfm-rubygem-excon-0.76.0-2.el7.noarch
tfm-rubygem-facter-4.0.51-2.el7.x86_64
tfm-rubygem-faraday-0.17.3-2.el7.noarch
tfm-rubygem-fast_gettext-1.4.1-5.el7.noarch
tfm-rubygem-ffi-1.12.2-2.el7.x86_64
tfm-rubygem-fog-core-2.1.0-4.el7.noarch
tfm-rubygem-foreman-tasks-6.0.1-1.fm3_2.el7.noarch
tfm-rubygem-foreman_puppet-3.0.7-1.fm3_2.el7.noarch
tfm-rubygem-foreman_remote_execution-6.0.0-1.fm3_2.el7.noarch
tfm-rubygem-formatador-0.2.1-13.el7.noarch
tfm-rubygem-friendly_id-5.3.0-2.el7.noarch
tfm-rubygem-fx-0.5.0-2.el7.noarch
tfm-rubygem-get_process_mem-0.2.7-2.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.8.0-3.el7.noarch
tfm-rubygem-gitlab-sidekiq-fetcher-0.6.0-2.el7.noarch
tfm-rubygem-globalid-0.4.2-2.el7.noarch
tfm-rubygem-graphql-1.8.14-3.el7.noarch
tfm-rubygem-graphql-batch-0.3.10-3.el7.noarch
tfm-rubygem-gssapi-1.2.0-8.el7.noarch
tfm-rubygem-hammer_cli-3.2.0-1.20220214173651git27087bf.el7.noarch
tfm-rubygem-hammer_cli_foreman-3.2.0-1.20220214175114git323f240.el7.noarch
tfm-rubygem-hammer_cli_foreman_puppet-0.0.5-1.fm3_2.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.17-1.fm3_2.el7.noarch
tfm-rubygem-hammer_cli_katello-1.4.1-1.el7.noarch
tfm-rubygem-hashie-3.6.0-3.el7.noarch
tfm-rubygem-highline-2.0.3-2.el7.noarch
tfm-rubygem-hocon-1.3.1-2.el7.noarch
tfm-rubygem-http-cookie-1.0.2-5.el7.noarch
tfm-rubygem-i18n-1.8.2-2.el7.noarch
tfm-rubygem-jwt-2.2.2-2.el7.noarch
tfm-rubygem-kafo-6.4.0-1.el7.noarch
tfm-rubygem-kafo_parsers-1.2.1-1.el7.noarch
tfm-rubygem-kafo_wizards-0.0.2-2.el7.noarch
tfm-rubygem-katello-4.4.0.2-2.el7.noarch
tfm-rubygem-ldap_fluff-0.6.0-1.el7.noarch
tfm-rubygem-little-plugger-1.1.4-3.el7.noarch
tfm-rubygem-locale-2.0.9-15.el7.noarch
tfm-rubygem-logging-2.3.0-2.el7.noarch
tfm-rubygem-loofah-2.4.0-2.el7.noarch
tfm-rubygem-mail-2.7.1-2.el7.noarch
tfm-rubygem-marcel-1.0.1-1.el7.noarch
tfm-rubygem-method_source-0.9.2-3.el7.noarch
tfm-rubygem-mime-types-3.3.1-2.el7.noarch
tfm-rubygem-mime-types-data-3.2018.0812-5.el7.noarch
tfm-rubygem-mini_mime-1.0.2-2.el7.noarch
tfm-rubygem-mini_portile2-2.5.1-1.el7.noarch
tfm-rubygem-mqtt-0.5.0-1.el7.noarch
tfm-rubygem-msgpack-1.3.3-2.el7.x86_64
tfm-rubygem-multi_json-1.14.1-3.el7.noarch
tfm-rubygem-multipart-post-2.0.0-3.el7.noarch
tfm-rubygem-mustermann-1.1.1-1.el7.noarch
tfm-rubygem-net-ldap-0.17.0-2.el7.noarch
tfm-rubygem-net-ping-2.0.1-5.el7.noarch
tfm-rubygem-net-scp-1.2.1-5.el7.noarch
tfm-rubygem-net-ssh-4.2.0-3.el7.noarch
tfm-rubygem-net_http_unix-0.2.2-2.el7.noarch
tfm-rubygem-netrc-0.11.0-6.el7.noarch
tfm-rubygem-nio4r-2.5.4-2.el7.x86_64
tfm-rubygem-nokogiri-1.11.3-2.el7.x86_64
tfm-rubygem-oauth-0.5.4-5.el7.noarch
tfm-rubygem-parallel-1.19.1-2.el7.noarch
tfm-rubygem-parse-cron-0.1.4-5.fm2_5.el7.noarch
tfm-rubygem-pg-1.1.4-4.el7.x86_64
tfm-rubygem-polyglot-0.3.5-3.el7.noarch
tfm-rubygem-powerbar-2.0.1-3.el7.noarch
tfm-rubygem-promise.rb-0.7.4-3.el7.noarch
tfm-rubygem-public_suffix-3.0.3-3.el7.noarch
tfm-rubygem-pulp_ansible_client-0.10.1-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.5.0-1.el7.noarch
tfm-rubygem-pulp_container_client-2.9.0-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.16.0-1.el7.noarch
tfm-rubygem-pulp_file_client-1.10.0-1.el7.noarch
tfm-rubygem-pulp_ostree_client-2.0.0-0.1.a1.el7.noarch
tfm-rubygem-pulp_python_client-3.5.2-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.17.4-1.el7.noarch
tfm-rubygem-pulpcore_client-3.16.0-1.el7.noarch
tfm-rubygem-puma-5.5.2-1.el7.x86_64
tfm-rubygem-puma-status-1.3-1.el7.noarch
tfm-rubygem-qpid_proton-0.37.0-1.el7.x86_64
tfm-rubygem-rabl-0.14.3-2.el7.noarch
tfm-rubygem-racc-1.5.2-1.el7.x86_64
tfm-rubygem-rack-2.2.3-2.el7.noarch
tfm-rubygem-rack-cors-1.0.2-3.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-10.el7.noarch
tfm-rubygem-rack-protection-2.1.0-2.el7.noarch
tfm-rubygem-rack-test-1.1.0-5.el7.noarch
tfm-rubygem-rails-6.0.3.7-1.el7.noarch
tfm-rubygem-rails-dom-testing-2.0.3-7.el7.noarch
tfm-rubygem-rails-html-sanitizer-1.3.0-2.el7.noarch
tfm-rubygem-rails-i18n-6.0.0-3.el7.noarch
tfm-rubygem-railties-6.0.3.7-1.el7.noarch
tfm-rubygem-rainbow-2.2.2-1.el7.noarch
tfm-rubygem-rb-inotify-0.9.7-6.el7.noarch
tfm-rubygem-redfish_client-0.5.2-2.el7.noarch
tfm-rubygem-redis-4.1.2-3.el7.noarch
tfm-rubygem-responders-3.0.0-4.el7.noarch
tfm-rubygem-rest-client-2.0.2-4.el7.noarch
tfm-rubygem-rkerberos-0.1.5-20.el7.x86_64
tfm-rubygem-roadie-3.4.0-4.el7.noarch
tfm-rubygem-roadie-rails-2.1.1-3.el7.noarch
tfm-rubygem-robotex-1.0.0-22.el7.noarch
tfm-rubygem-rsec-0.4.3-5.el7.noarch
tfm-rubygem-ruby-libvirt-0.7.1-2.el7.x86_64
tfm-rubygem-ruby2_keywords-0.0.4-1.el7.noarch
tfm-rubygem-ruby2ruby-2.4.2-4.el7.noarch
tfm-rubygem-ruby_parser-3.10.1-4.el7.noarch
tfm-rubygem-rubyipmi-0.11.0-1.el7.noarch
tfm-rubygem-runcible-2.13.1-2.el7.noarch
tfm-rubygem-safemode-1.3.6-2.el7.noarch
tfm-rubygem-scoped_search-4.1.10-1.el7.noarch
tfm-rubygem-sd_notify-0.1.0-2.el7.noarch
tfm-rubygem-secure_headers-6.3.0-3.el7.noarch
tfm-rubygem-sequel-5.42.0-2.el7.noarch
tfm-rubygem-server_sent_events-0.1.2-2.el7.noarch
tfm-rubygem-sexp_processor-4.10.0-7.el7.noarch
tfm-rubygem-sidekiq-5.2.7-4.el7.noarch
tfm-rubygem-sinatra-2.1.0-2.el7.noarch
tfm-rubygem-smart_proxy_dynflow-0.7.0-1.fm3_2.el7.noarch
tfm-rubygem-smart_proxy_pulp-3.2.0-2.fm3_2.el7.noarch
tfm-rubygem-smart_proxy_remote_execution_ssh-0.5.3-1.fm3_2.el7.noarch
tfm-rubygem-sprockets-4.0.2-2.el7.noarch
tfm-rubygem-sprockets-rails-3.2.1-7.el7.noarch
tfm-rubygem-sqlite3-1.3.13-7.el7.x86_64
tfm-rubygem-sshkey-1.9.0-5.el7.noarch
tfm-rubygem-statsd-instrument-2.1.4-4.el7.noarch
tfm-rubygem-stomp-1.4.9-2.el7.noarch
tfm-rubygem-thor-1.0.1-3.el7.noarch
tfm-rubygem-thread_safe-0.3.6-6.el7.noarch
tfm-rubygem-tilt-2.0.8-5.el7.noarch
tfm-rubygem-tzinfo-1.2.6-2.el7.noarch
tfm-rubygem-unf-0.1.3-9.el7.noarch
tfm-rubygem-unf_ext-0.0.7.2-4.el7.x86_64
tfm-rubygem-unicode-0.4.4.4-4.el7.x86_64
tfm-rubygem-unicode-display_width-1.7.0-2.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-8.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-6.el7.noarch
tfm-rubygem-websocket-driver-0.7.1-2.el7.x86_64
tfm-rubygem-websocket-extensions-0.1.5-2.el7.noarch
tfm-rubygem-will_paginate-3.1.7-4.el7.noarch
tfm-rubygem-xmlrpc-0.3.0-3.el7.noarch
tfm-rubygem-zeitwerk-2.2.2-2.el7.noarch
tfm-runtime-7.0-5.el7.x86_64
*

**Foreman and Proxy plugin versions:katello 4.4.0.2
foreman-tasks 6.0.1
foreman_puppet 3.0.7
foreman_remote_execution 6.0.0

Distribution and version:

**Other relevant data:*company policy does not allow remote execution from local foreman server and each datacenter locations have their own smart proxies and allow local ssh connetion to foreman client. We have configured all remote execution clients to have content source to nearest smart proxy. When a remote task is running chosing single server it choses correct proxy and remote execution is successful but multiple hosts execution shares remote execution with local proxy on 50:50 basis and since local proxy does not have ssh access remote execution task fails. Local proxy is a puppet smart proxy when removing puppet environment, puppet ca id and proxy id on all clients fixes this odd behavior *

There’s a setting called “Restrict registered smart proxy” or " Prefer registered through Smart Proxy for remote execution" (I think it’s under Authentication.) This should force jobs to use the smart proxy the host is registered through.

Restrict registered smart proxy options is set to yes. Just to clarify issue here is having two proxies 1) puppet server registration using smart proxy and 2) katello subscription registration using smart proxy to do remote execution so both are registered proxies in our setup.

Depending on your setup, I have 10+ proxies all over the world and since I do not use/configure subnets in Foreman, I had to enable “Fallback to Any Proxy” in the REX settings + make sure “Content Source” is set to correct proxy on the host/host group to allow REX to use the right proxy.

Fallback to Any Proxy and Enable Global Proxy are set to Yes and have content source configured to the intended proxy (i.e. proxy1) on each host.

Proxy1 (Katello + remote execution) << multiple proxies based on location (especially remote ssh connections only from a local proxy)
Proxy2 (Puppetserver centralized proxy for all clients in a dc without ssh connection allowed out to clients due to security reasons)

somehow rex is distributed between proxy1 and proxy2 at a 50:50 basis and it fails 50% execution because proxy2 is not allowing ssh connections

Here’s the logic that governs how it chooses a smart proxy:

https://docs.theforeman.org/nightly/Managing_Hosts/index-katello.html#remote-execution-workflow_managing-hosts

Would it help to disable the remote execution plugin on the Foreman server? Is there a reason you need it there?

Foreman server will be set for remote execution which is an exception ssh policy where remote proxy is not installed in a very small environment due to cost and config overhead. Though setting up “Prefer registered through Smart Proxy” for remote execution to true fixes the issue of unnecessary load balancing in our case.