Katello nightly pipeline failed:
https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/984/
foreman-pipeline-katello-nightly-centos8-install (passed) (remote job)
foreman-pipeline-katello-nightly-centos8-stream-install (passed) (remote job)
foreman-pipeline-katello-nightly-centos8-upgrade (failed) (remote job)
foreman-pipeline-katello-nightly-centos7-upgrade (failed) (remote job)
foreman-pipeline-katello-nightly-centos7-install (passed) (remote job)
Looks like the proxy refresh call fails after upgrade to 2.6.0:
[2021-06-12T01:11:35.168Z] e[0;31m 2021-06-12 01:11:13 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[pipe-up-katello-nightly-centos7.n42.example.com]: Failed to call refresh: Error making PUT request to https://pipe-up-katello-nightly-centos7.n42.example.com/api/v2/smart_proxies/1/refresh: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on Foreman server for detailed informatione[0m
[2021-06-12T01:11:35.168Z] e[0;31m 2021-06-12 01:11:13 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[pipe-up-katello-nightly-centos7.n42.example.com]: Error making PUT request to https://pipe-up-katello-nightly-centos7.n42.example.com/api/v2/smart_proxies/1/refresh: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on Foreman server for detailed informatione[0m
and production.log:
2021-06-12T01:11:10 [I|app|c7bd9843] Started PUT "/api/v2/smart_proxies/1/refresh" for 192.168.121.219 at 2021-06-12 01:11:10 +0000
2021-06-12T01:11:10 [I|app|c7bd9843] Processing by Api::V2::SmartProxiesController#refresh as JSON
2021-06-12T01:11:10 [I|app|c7bd9843] Parameters: {"apiv"=>"v2", "id"=>"1", "smart_proxy"=>{}}
2021-06-12T01:11:10 [I|app|c7bd9843] Authorized user foreman_api_admin(API Admin)
2021-06-12T01:11:13 [W|app|c7bd9843] Action failed
2021-06-12T01:11:13 [I|app|c7bd9843] Backtrace for 'Action failed' error (ProxyAPI::ProxyException): ERF12-9411 [ProxyAPI::ProxyException]: Unable to fetch public key ([RestClient::InternalServerError]: 500 Internal Server Error) for proxy https://pipe-up-katello-nightly-centos7.n42.example.com:9090/ssh
c7bd9843 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.6.0/app/lib/proxy_api/remote_execution_ssh.rb:11:in `rescue in pubkey'
c7bd9843 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.6.0/app/lib/proxy_api/remote_execution_ssh.rb:8:in `pubkey'
c7bd9843 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.6.0/app/models/concerns/foreman_remote_execution/smart_proxy_extensions.rb:10:in `update_pubkey'
c7bd9843 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.6.0/app/models/concerns/foreman_remote_execution/smart_proxy_extensions.rb:21:in `refresh'
c7bd9843 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.0.pre.master/app/models/katello/concerns/smart_proxy_extensions.rb:12:in `refresh'
c7bd9843 | /usr/share/foreman/app/controllers/api/v2/smart_proxies_controller.rb:60:in `refresh'and proxy.log has:
2021-06-14T01:09:57 d469d222 [W] Error processing request 'd469d222-d85f-4e17-a3c6-93fdec0eeecf: <Errno::ENOENT>: No such file or directory @ rb_sysopen - /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy.pub
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.4.0/lib/smart_proxy_remote_execution_ssh/api.rb:11:in `read'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.4.0/lib/smart_proxy_remote_execution_ssh/api.rb:11:in `block in <class:Api>'
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.
I think this change Fixes #32542 - Make REX core only a compatibility layer proxy plugin … · theforeman/foreman_remote_execution@f321808 · GitHub started pulling in smart_proxy_remote_execution_ssh which lays down a configuration that is unmanaged and turns the feature on. Given for katello scenario this is not on by default (foreman-installer/katello-answers.yaml at develop · theforeman/foreman-installer · GitHub) the installer logic such as generating the key, is not running and leading the smart-proxy to be in an inconsistent state (puppet-foreman_proxy/ssh.pp at master · theforeman/puppet-foreman_proxy · GitHub). There could be a flaw in my logic as this seems to only happen on upgrades and not fresh installs.
I know you are doing a lot of re-factoring work to drop the need for _core gems across the ecosystem @aruzicka, could you suggest how we should proceed to get nightly flowing again?
Any insights why rex core gets pulled in in the first place if it is not needed?
Was too hasty with the previous post
Remote execution < 4.6.0 used to depend on rex core so it got pulled in before the upgrade. Then after the upgrade it is not really needed, but remains installed and brings in additional deps which cause this mess.
Not sure what we can do here, other packages can still depend on REX core in its current form, so we cannot obsolete it in packaging just yet. If I’m looking right then the only thing depending on REX core is ansible core, which should go away once Update rubygem-smart_proxy_ansible to 3.2.1 by adamruzicka · Pull Request #6816 · theforeman/foreman-packaging · GitHub gets in.
IMHO it should not fail with an internal server error at all. Can’t we make it deal with a proxy without a public key? Or if it’s always needed, make the Smart Proxy plugin fail to start up.
I think this would give more guarantees.
That also gave me a thought: should it generate a private key on startup if it’s missing? Perhaps that would even allow us to remove the code from the installer at some point.
The release of this fix is blocked by a repoclosure failure with plugins. The fix for that repoclosure issue is an update to foreman_ansible, foreman_ansible fails to build right now due to Refs #32399: Reference theforeman/vendor variables and not patternfly… by ehelms · Pull Request #423 · theforeman/foreman_ansible · GitHub
After cleaning up foreman_ansible_core, the problem remains, perhaps due to foreman_remote_execution_core ?
rubygem-foreman-tasks-core noarch 0.4.0-1.fm2_6.el8 @foreman-plugins-koji 37 k
Removing dependent packages:
rubygem-foreman_remote_execution_core noarch 1.5.0-1.el8 @foreman-plugins-koji 34 k
Removing unused dependencies:
rubygem-bcrypt_pbkdf x86_64 1.1.0-1.el8 @foreman-plugins 36 k
rubygem-ed25519 x86_64 1.2.4-1.el8 @foreman-plugins 128 k
rubygem-smart_proxy_dynflow noarch 0.5.2-1.fm2_6.el8 @foreman-plugins-koji 80 k
rubygem-smart_proxy_dynflow_core noarch 0.4.1-1.fm2_6.el8 @foreman-plugins-koji 38 k
rubygem-smart_proxy_remote_execution_ssh noarch 0.4.0-1.fm2_6.el8 @foreman-plugins-koji 81 k
rubygem-sqlite3