Katello-nightly-rpm-pipeline 1022 failed

jenkins · July 9, 2021, 12:02pm

Katello nightly pipeline failed:

https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/1022/

foreman-pipeline-katello-nightly-centos8-stream-install (passed) (remote job)
foreman-pipeline-katello-nightly-centos8-install (passed) (remote job)
foreman-pipeline-katello-nightly-centos7-install (passed) (remote job)
foreman-pipeline-katello-nightly-centos8-upgrade (failed) (remote job)
foreman-pipeline-katello-nightly-centos7-upgrade (failed) (remote job)

evgeni · July 9, 2021, 12:12pm

okay, we’re back to the state “pre-pulpcore-3.14”, upgrades are failing because rex can’t find its key:

[2021-07-09T11:44:58.569Z] e[0;31m    2021-07-09 11:44:39 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[pipe-up-katello-nightly-centos8.n22.example.com]: Failed to call refresh: Error making PUT request to https://pipe-up-katello-nightly-centos8.n22.example.com/api/v2/smart_proxies/1/refresh: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on Foreman server for detailed informatione[0m
[2021-07-09T11:44:58.569Z] e[0;31m    2021-07-09 11:44:39 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[pipe-up-katello-nightly-centos8.n22.example.com]: Error making PUT request to https://pipe-up-katello-nightly-centos8.n22.example.com/api/v2/smart_proxies/1/refresh: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on Foreman server for detailed informatione[0m

2021-07-09T11:57:33 6830a2ae [I] Started GET /ssh/pubkey 
2021-07-09T11:57:33 6830a2ae [W] Error processing request '6830a2ae-6886-40dd-8b4a-c97afa4efca2: <Errno::ENOENT>: No such file or directory @ rb_sysopen - /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy.pub
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.4.1/lib/smart_proxy_remote_execution_ssh/api.rb:11:in `read'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.4.1/lib/smart_proxy_remote_execution_ssh/api.rb:11:in `block in <class:Api>'

ekohl · July 9, 2021, 3:24pm

This PR would make the Smart Proxy plugin fail to start up if the public key is not readable:

github.com/theforeman/smart_proxy_remote_execution_ssh

Use more native methods of SmartProxy for validation

theforeman:master ← ekohl:more-native-methods

opened 08:45AM - 24 Jun 21 UTC

ekohl

+53 -45

This uses more built in validators. It also creates a setting for the public key… file and derives it if not specified. The only downside is that the validate_readable doesn't print a helpful command to generate it if needed. Programmable settings can be used to generate it on the fly if the parent directory is readable. Another alternative is to introduce another custom validator. It also still uses the Smart Proxy's LegacyModuleLoader due to the presence of after_activation. Currently a draft since I can't test this locally (smart_proxy_dynflow is not Ruby 3 compatible). Consider it more of an RFC.

While it’s not a guarantee (a user could remove the key after it was started), it’s a much better start. The user experience then becomes that the feature is not recognized on Foreman, but at least there is no internal server error due to the assumption that /ssh/pubkey returns a result.

evgeni · July 9, 2021, 3:52pm

Does that mean the proxy would start, but the plugin won’t be loaded? That sounds good.

ehelms · July 9, 2021, 4:07pm

I think the fix proposed above is a good code change for robustness. It does not address to me the core problem that the upgrade recognizes. This is just now the same problem but presented in a new way. Take a look at my proposal for my thoughts on the technical cause and the user facing problem and how we could fix it

github.com/theforeman/foreman-packaging

Do not Obsolete foreman_remote_execution_core from smart_proxy_remote_execution_ssh

theforeman:rpm/develop ← ehelms:fix-smart-proxy-rex

opened 03:49PM - 09 Jul 21 UTC

ehelms

+4 -2

@adamruzicka @evgeni @ekohl Please take a look The interesting part here is, …and I don't know the code enough, is that by smart_proxy_remote_execution_ssh obsoleting foreman_remote_execution_core we are *introducing* functionality that was not present before. That is, previously only the core code parts existed via the foreman_remote_execution_core gem. Now we are introducing the actual smart_proxy_remote_execution plugin even though a user may not have had it configured. On upgrade, the yum output is: ``` tfm-rubygem-smart_proxy_remote_execution_ssh noarch 0.4.1-1.fm2_6.el7 foreman-plugins-koji 35 k replacing tfm-rubygem-foreman_remote_execution_core.noarch 1.4.3-1.el7 ``` This is, what in turn results in: ``` # cat /etc/foreman-proxy/settings.d/remote_execution_ssh.yml --- :enabled: true :ssh_identity_key_file: '~/.ssh/id_rsa_foreman_proxy' :local_working_dir: '/var/tmp' :remote_working_dir: '/var/tmp' # :kerberos_auth: false # :async_ssh: false # Defines how often (in seconds) should the runner check # for new data leave empty to use the runner's default # (1 second for regular, 60 seconds with async_ssh enabled) # :runner_refresh_interval: # Defines the verbosity of logging coming from Net::SSH # one of :debug, :info, :warn, :error, :fatal # must be lower than general log level # :ssh_log_level: fatal # Remove working directories on job completion # :cleanup_working_dirs: true ``` Even though, a user has: ``` # grep -r remote_execution::ssh /etc/foreman-installer/scenarios.d/katello-answers.yaml foreman_proxy::plugin::remote_execution::ssh: false ``` So I think that this is what was breaking the upgrade before: https://github.com/theforeman/foreman-packaging/pull/6855/files#diff-5639bb35f3b9b6c5d8eb5121c28b627340d9fa15bbee42e94682fc58cb51a4d6L24 And now, the Obsoletes is breaking us. Something maybe ought to clean up the system, but not the smart proxy plugin.

ekohl · July 9, 2021, 4:24pm

Yes it does.

ekohl · July 9, 2021, 4:30pm

Thanks for that part. It now looks like the problem is that the module is enabled after yum install while all plugins are shipped disabled by default. That means the user must opt in to configure. I’ve submitted this PR:

This is in addition to others.

evgeni · July 12, 2021, 12:53pm

We merged Erics PR and things are passing now.

Whether we want also “do not enable by default”, is up to y’all

system · July 19, 2021, 12:53pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.