Did Dynflow/foreman-tasks become slower to start up?

evgeni · January 26, 2023, 9:44am

Ohai,

recently we have been seeing an increased amount of failed pipelines, that all fail during the “enable puppet feature” step:

github.com

theforeman/forklift/blob/7b52fced3ba0a59d296cea40593a8533a255a7f9/bats/fb-test-puppet.bats#L22-L24

    
      
          foreman-installer --enable-foreman-plugin-puppet --enable-foreman-cli-puppet --foreman-proxy-puppet true --foreman-proxy-puppetca true --foreman-proxy-content-puppet true --enable-puppet --puppet-server true --puppet-server-foreman-ssl-ca /etc/pki/katello/puppet/puppet_client_ca.crt --puppet-server-foreman-ssl-cert /etc/pki/katello/puppet/puppet_client.crt --puppet-server-foreman-ssl-key /etc/pki/katello/puppet/puppet_client.key
          # Force hammer to reload the apidoc cache - https://projects.theforeman.org/issues/28283
          hammer --reload-cache ping

The installer would finish fine, but hammer ping fails with:

 foreman_tasks:
     Status:          FAIL
     Server Response: Message: some executors are not responding, check /foreman_tasks/dynflow/status

It seems to happen especially on the luna pipes, which have a bigger number of plugins enabled, but also there not always, which leads me to believe that this is a timing/performance issue.

evgeni · January 26, 2023, 10:11am

Mind you, the hammer issue is fixed since 3.3, so we could probably get away with dropping that hammer ping call, but it’s just hiding the issue?