1.19 -> 1.20 upgrade fail: "add queue katello_event_queue --durable' returned 1 instead of one of [0]"

Lachlan_Musicman · November 27, 2018, 11:56pm

Problem:

Installed updates successfully. Running foreman-installer --scenario katello --upgrade
gives the error:

foreman-rake db:migrate finished successfully!
Upgrade Step: Running installer...
 'qpid-config --ssl-certificate /etc/pki/katello/certs/vmpr-res-utils.company.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/vmpr-res-utils.company.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable' returned 1 instead of one of [0]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:164:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:236:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_needed'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `block in perform_changes'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `perform_changes'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:233:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:249:in `eval_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:163:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:163:in `block (2 levels) in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:507:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:506:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:163:in `block in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:154:in `evaluate'
/usr/share/gems/gems/kafo-2.1.0/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:222:in `block in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:155:in `with_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:221:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:171:in `block in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:224:in `block in benchmark'
/opt/puppetlabs/puppet/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:223:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:170:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:343:in `run_internal'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:221:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:306:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:195:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:348:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:274:in `block in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:306:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:225:in `main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:170:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:662:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:132:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
 /Stage[main]/Katello::Qpid/Qpid::Config::Queue[katello_event_queue]/Qpid::Config_cmd[ensure queue katello_event_queue]/Exec[qpid-config ensure queue katello_
event_queue]/returns: change from notrun to 0 failed: 'qpid-config --ssl-certificate /etc/pki/katello/certs/vmpr-res-utils.company.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/vmpr-res-utils.company.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable' re
turned 1 instead of one of [0]
Installing             Done                                               [100%] [...........................................................................]
  Something went wrong! Check the log for ERROR-level output

Foreman and Proxy plugin versions:

1.20, Katello 3.9.0.1

Zhunting · November 28, 2018, 2:33pm

Thanks for reporting this issue!

Would you mind running:

qpid-config --ssl-certificate /etc/pki/katello/certs/vmpr-res-utils.company.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/vmpr-res-utils.company.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable

Then posting the traceback or error message from that?

dcfrmn · November 28, 2018, 8:03pm

I have the same problem, and what I get back is:

Failed:Exception: Exception from Agent: {u’error_code’: 7, u’error_text’: ‘object already exists: katello_event_queue’}

I also tried this:

qpid-config --ssl-certificate /etc/pki/katello/certs/vmpr-res-utils.company.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/vmpr-res-utils.company.com-qpid-broker.key -b amqps://localhost:5671 del queue katello_event_queue --durable

This returned an error stating that the queue was in use.

dcfrmn · November 28, 2018, 8:25pm

Update:

I was able to navigate past this by shutting everything down (foreman-maintain service stop), starting qpidd (systemctl start qpidd.service) and removing the katello_event_queue:

qpid-config --ssl-certificate /etc/pki/katello/certs/vmpr-res-utils.company.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/vmpr-res-utils.company.com-qpid-broker.key -b amqps://localhost:5671 del queue katello_event_queue --durable

I then stopped qpidd and re-tried the installer, which completed largely without complaint. There was one message about the cron provider, but I’ll omit it since it does not seem to be related.

Lachlan_Musicman · November 28, 2018, 10:05pm

Can confirm that this solution also worked for me - thank you @dcfrmn

Bernhard_Suttner · February 8, 2019, 9:15am

Does someone already know if this issue is fixed in latest foreman-installer?
Why does it happen? Is it required to remove the katello event queue first?

Bernhard_Suttner · February 8, 2019, 9:44am

Looks like the issue should is fixed in:

github.com/theforeman/puppet-qpid

Fixes #25909 - make qpidd.service wait until the port is open

theforeman:4.3-stable ← ehelms:cherry-pick-wait-for-port

opened 06:26PM - 31 Jan 19 UTC

ehelms

+43 -0

qpidd does not open the ssl-port immediately after starting, but after some ini…tialization happened. Let's make qpidd.service not return until the port is really open, as otherwise actions depending on the service being available might fail (cherry picked from commit 7534cd902129921d5f86722ea9e41df7ff188f52)

(issue: Bug #25909: satellite-installer --upgrade qpid-config returned 1 instead of one of [0] - Installer - Foreman)

roheve · April 19, 2019, 11:08am

same issue here updating 1.20.1 to 1.20.2 but I kept getting the ‘queue in use’ message. adding --force to the ‘del’ command solved that. Re-running the installer then went through without this issue.