Problem:
virt-who reporting/update hypervisors continues to fail.
This has been reported previously, but I’ve not stumbled upon a fix for this issue.
Currenting running Foreman 1.19 / Katello 3.8 – on multiple machines, same condition persists.
Dynflow console:
3: Actions::Candlepin::AsyncHypervisors (skipped) [ 6.15s / 0.26s ]
Started at: 2018-10-16 18:59:53 UTC
Ended at: 2018-10-16 18:59:59 UTC
Real time: 6.15s
Execution time (excluding suspended state): 0.26s
Input:
task_id: hypervisor_update_50178ebc-bdd1-4d9b-a315-560261ff4606
session_id: 86b46964-2ee5-4797-9383-4851308852c9
remote_user: admin
remote_cp_user: admin
Output:
---
task:
created: '2018-10-16T18:59:52+0000'
updated: '2018-10-16T18:59:52+0000'
id: hypervisor_update_50178ebc-bdd1-4d9b-a315-560261ff4606
state: RUNNING
startTime: '2018-10-16T18:59:52+0000'
finishTime:
result:
principalName: foreman_admin
targetType: owner
targetId: ......
ownerId: .......
correlationId: 86b46964
resultData:
done: false
statusPath: "/jobs/hypervisor_update_50178ebc-bdd1-4d9b-a315-560261ff4606"
group: async group
poll_attempts:
total: 6
failed: 3
Error:
**NoMethodError**
**undefined method `[]' for nil:NilClass**
Expected outcome:
Hypervisors update and are displayed within Katello as normal.
Foreman and Proxy versions:
Installed Packages
* candlepin-2.4.0-1.el7.noarch
* candlepin-selinux-2.4.0-1.el7.noarch
* foreman-1.19.0-1.el7.noarch
* foreman-cli-1.19.0-1.el7.noarch
* foreman-compute-1.19.0-1.el7.noarch
* foreman-debug-1.19.0-1.el7.noarch
* foreman-ec2-1.19.0-1.el7.noarch
* foreman-installer-1.19.0-1.el7.noarch
* foreman-installer-katello-3.8.0-1.el7.noarch
* foreman-postgresql-1.19.0-1.el7.noarch
* foreman-proxy-1.19.0-1.el7.noarch
* foreman-release-1.19.0-1.el7.noarch
* foreman-release-scl-7-1.el7.noarch
* foreman-selinux-1.19.0-1.el7.noarch
* foreman-vmware-1.19.0-1.el7.noarch
* katello-3.8.0-5.el7.noarch
* katello-agent-3.3.5-1.el7.noarch
* katello-ca-consumer-lprnedifkap002v.int.asurion.com-1.0-1.noarch
* katello-certs-tools-2.4.0-1.el7.noarch
* katello-client-bootstrap-1.4.2-1.el7.noarch
* katello-common-3.8.0-5.el7.noarch
* katello-debug-3.8.0-5.el7.noarch
* katello-default-ca-1.0-1.noarch
* katello-host-tools-3.3.5-1.el7.noarch
* katello-host-tools-fact-plugin-3.3.5-1.el7.noarch
* katello-installer-base-3.8.0-1.el7.noarch
* katello-repos-3.8.0-4.el7.noarch
* katello-selinux-3.0.3-1.el7.noarch
* katello-server-ca-1.0-1.noarch
* katello-service-3.8.0-5.el7.noarch
* katello.us.prod.asurion.net-apache-1.0-1.noarch
* katello.us.prod.asurion.net-foreman-client-1.0-1.noarch
* katello.us.prod.asurion.net-foreman-proxy-1.0-1.noarch
* katello.us.prod.asurion.net-foreman-proxy-client-1.0-1.noarch
* katello.us.prod.asurion.net-puppet-client-1.0-1.noarch
* katello.us.prod.asurion.net-qpid-broker-1.0-2.noarch
* katello.us.prod.asurion.net-qpid-client-cert-1.0-1.noarch
* katello.us.prod.asurion.net-qpid-router-client-1.0-1.noarch
* katello.us.prod.asurion.net-qpid-router-server-1.0-1.noarch
* katello.us.prod.asurion.net-tomcat-1.0-1.noarch
* pulp-admin-client-2.16.4-1.el7.noarch
* pulp-client-1.0-1.noarch
* pulp-deb-plugins-1.7.1-1.el7.noarch
* pulp-docker-plugins-3.1.4-2.el7.noarch
* pulp-katello-1.0.2-1.el7.noarch
* pulp-puppet-plugins-2.16.4-1.el7.noarch
* pulp-puppet-tools-2.16.4-1.el7.noarch
* pulp-rpm-admin-extensions-2.16.4-1.el7.noarch
* pulp-rpm-plugins-2.16.4-1.el7.noarch
* pulp-selinux-2.16.4-1.el7.noarch
* pulp-server-2.16.4-1.el7.noarch
* python-gofer-qpid-2.12.1-1.el7.noarch
* python-isodate-0.5.0-4.pulp.el7.noarch
* python-pulp-bindings-2.16.4-1.el7.noarch
* python-pulp-client-lib-2.16.4-1.el7.noarch
* python-pulp-common-2.16.4-1.el7.noarch
* python-pulp-deb-common-1.7.1-1.el7.noarch
* python-pulp-docker-common-3.1.4-2.el7.noarch
* python-pulp-oid_validation-2.16.4-1.el7.noarch
* python-pulp-puppet-common-2.16.4-1.el7.noarch
* python-pulp-repoauth-2.16.4-1.el7.noarch
* python-pulp-rpm-common-2.16.4-1.el7.noarch
* python-pulp-streamer-2.16.4-1.el7.noarch
* python2-qpid-1.37.0-4.el7.noarch
* python2-qpid-proton-0.24.0-1.el7.x86_64
* python2-qpid-qmf-1.38.0-2.el7.x86_64
* qpid-cpp-client-1.38.0-2.el7.x86_64
* qpid-cpp-client-devel-1.38.0-2.el7.x86_64
* qpid-cpp-server-1.38.0-2.el7.x86_64
* qpid-cpp-server-linearstore-1.38.0-2.el7.x86_64
* qpid-dispatch-router-1.2.0-1.el7.x86_64
* qpid-proton-c-0.24.0-1.el7.x86_64
* qpid-qmf-1.38.0-2.el7.x86_64
* qpid-tools-1.38.0-2.el7.noarch
* rubygem-foreman_maintain-0.2.4-1.el7.noarch
* rubygem-smart_proxy_pulp-1.3.0-1.el7.noarch
* tfm-ror51-rubygem-actioncable-5.1.6-1.el7.centos.noarch
* tfm-ror51-rubygem-actionmailer-5.1.6-1.el7.centos.noarch
* tfm-ror51-rubygem-actionpack-5.1.6-1.el7.centos.noarch
* tfm-ror51-rubygem-actionview-5.1.6-1.el7.centos.noarch
* tfm-ror51-rubygem-activejob-5.1.6-1.el7.centos.noarch
* tfm-ror51-rubygem-activemodel-5.1.6-1.el7.centos.noarch
* tfm-ror51-rubygem-activerecord-5.1.6-1.el7.centos.noarch
* tfm-ror51-rubygem-activesupport-5.1.6-1.el7.centos.noarch
* tfm-ror51-rubygem-arel-8.0.0-2.el7.centos.noarch
* tfm-ror51-rubygem-builder-3.2.3-2.el7.centos.noarch
* tfm-ror51-rubygem-coffee-rails-4.2.2-2.el7.centos.noarch
* tfm-ror51-rubygem-coffee-script-2.4.1-2.el7.centos.noarch
* tfm-ror51-rubygem-coffee-script-source-1.12.2-2.el7.centos.noarch
* tfm-ror51-rubygem-concurrent-ruby-1.0.5-4.el7.centos.noarch
* tfm-ror51-rubygem-crass-1.0.2-2.el7.centos.noarch
* tfm-ror51-rubygem-erubi-1.7.0-2.el7.centos.noarch
* tfm-ror51-rubygem-execjs-2.7.0-2.el7.centos.noarch
* tfm-ror51-rubygem-globalid-0.4.1-2.el7.centos.noarch
* tfm-ror51-rubygem-i18n-0.9.1-3.el7.centos.noarch
* tfm-ror51-rubygem-loofah-2.1.1-3.el7.centos.noarch
* tfm-ror51-rubygem-mail-2.7.0-3.el7.centos.noarch
* tfm-ror51-rubygem-method_source-0.9.0-2.el7.centos.noarch
* tfm-ror51-rubygem-mime-types-3.1-2.el7.centos.noarch
* tfm-ror51-rubygem-mime-types-data-3.2016.0521-2.el7.centos.noarch
* tfm-ror51-rubygem-mini_mime-1.0.0-2.el7.centos.noarch
* tfm-ror51-rubygem-multi_json-1.12.2-2.el7.centos.noarch
* tfm-ror51-rubygem-mustermann-1.0.1-2.el7.centos.noarch
* tfm-ror51-rubygem-nio4r-2.1.0-2.el7.centos.x86_64
* tfm-ror51-rubygem-nokogiri-1.8.1-2.el7.centos.x86_64
* tfm-ror51-rubygem-rack-2.0.3-2.el7.centos.noarch
* tfm-ror51-rubygem-rack-protection-2.0.0-2.el7.centos.noarch
* tfm-ror51-rubygem-rack-test-0.7.0-2.el7.centos.noarch
* tfm-ror51-rubygem-rails-5.1.6-1.el7.centos.noarch
* tfm-ror51-rubygem-rails-dom-testing-2.0.3-2.el7.centos.noarch
* tfm-ror51-rubygem-rails-html-sanitizer-1.0.3-3.el7.centos.noarch
* tfm-ror51-rubygem-railties-5.1.6-1.el7.centos.noarch
* tfm-ror51-rubygem-sinatra-2.0.0-2.el7.centos.noarch
* tfm-ror51-rubygem-sprockets-3.7.1-2.el7.noarch
* tfm-ror51-rubygem-sprockets-rails-3.2.1-2.el7.centos.noarch
* tfm-ror51-rubygem-sqlite3-1.3.13-2.el7.centos.x86_64
* tfm-ror51-rubygem-thor-0.20.0-2.el7.centos.noarch
* tfm-ror51-rubygem-thread_safe-0.3.6-2.el7.centos.noarch
* tfm-ror51-rubygem-tilt-2.0.8-2.el7.centos.noarch
* tfm-ror51-rubygem-turbolinks-2.5.4-2.el7.centos.noarch
* tfm-ror51-rubygem-tzinfo-1.2.4-3.el7.centos.noarch
* tfm-ror51-rubygem-websocket-driver-0.6.5-3.el7.centos.x86_64
* tfm-ror51-rubygem-websocket-extensions-0.1.2-2.el7.centos.noarch
* tfm-ror51-runtime-1.1-2.el7.centos.x86_64
* tfm-rubygem-activerecord-session_store-1.1.0-4.el7.noarch
* tfm-rubygem-addressable-2.3.6-5.el7.noarch
* tfm-rubygem-algebrick-0.7.3-5.el7.noarch
* tfm-rubygem-ancestry-3.0.0-2.el7.noarch
* tfm-rubygem-anemone-0.7.2-15.el7.noarch
* tfm-rubygem-angular-rails-templates-1.0.2-2.el7.noarch
* tfm-rubygem-apipie-bindings-0.2.2-1.el7.noarch
* tfm-rubygem-apipie-params-0.0.5-2.el7.noarch
* tfm-rubygem-apipie-rails-0.5.9-1.el7.noarch
* tfm-rubygem-audited-4.7.1-1.el7.noarch
* tfm-rubygem-awesome_print-1.8.0-2.el7.noarch
* tfm-rubygem-bastion-6.1.11-1.fm1_19.el7.noarch
* tfm-rubygem-bundler_ext-0.4.1-2.el7.noarch
* tfm-rubygem-clamp-1.1.2-2.el7.noarch
* tfm-rubygem-concurrent-ruby-edge-0.2.4-1.el7.noarch
* tfm-rubygem-css_parser-1.4.7-2.el7.noarch
* tfm-rubygem-daemons-1.2.3-4.el7.noarch
* tfm-rubygem-deacon-1.0.0-2.el7.noarch
* tfm-rubygem-deep_cloneable-2.3.2-1.el7.noarch
* tfm-rubygem-deface-1.2.0-8.el7.noarch
* tfm-rubygem-docker-api-1.28.0-3.el7.noarch
* tfm-rubygem-domain_name-0.5.20160310-2.el7.noarch
* tfm-rubygem-dynflow-1.1.0-1.el7.noarch
* tfm-rubygem-excon-0.58.0-2.el7.noarch
* tfm-rubygem-facter-2.4.0-5.el7.x86_64
* tfm-rubygem-fast_gettext-1.4.1-2.el7.noarch
* tfm-rubygem-fog-1.42.1-1.el7.noarch
* tfm-rubygem-fog-aws-1.3.0-2.el7.noarch
* tfm-rubygem-fog-core-1.45.0-2.el7.noarch
* tfm-rubygem-fog-digitalocean-0.3.0-2.el7.noarch
* tfm-rubygem-fog-google-0.1.0-3.el7.noarch
* tfm-rubygem-fog-json-1.0.2-5.el7.noarch
* tfm-rubygem-fog-openstack-0.1.25-1.el7.noarch
* tfm-rubygem-fog-ovirt-1.1.1-1.el7.noarch
* tfm-rubygem-fog-rackspace-0.1.4-2.el7.noarch
* tfm-rubygem-fog-vsphere-2.3.0-1.el7.noarch
* tfm-rubygem-fog-xenserver-0.2.3-2.el7.noarch
* tfm-rubygem-fog-xml-0.1.2-5.el7.noarch
* tfm-rubygem-foreman-tasks-0.13.4-1.fm1_19.el7.noarch
* tfm-rubygem-foreman-tasks-core-0.2.5-1.fm1_18.el7.noarch
* tfm-rubygem-foreman_ansible-2.2.9-1.fm1_19.el7.noarch
* tfm-rubygem-foreman_ansible_core-2.1.1-1.fm1_19.el7.noarch
* tfm-rubygem-foreman_bootdisk-12.0.0-2.fm1_18.el7.noarch
* tfm-rubygem-foreman_cockpit-2.0.3-3.fm1_18.el7.noarch
* tfm-rubygem-foreman_default_hostgroup-5.0.0-2.fm1_18.el7.noarch
* tfm-rubygem-foreman_discovery-13.0.1-1.fm1_19.el7.noarch
* tfm-rubygem-foreman_docker-4.1.0-1.fm1_18.el7.noarch
* tfm-rubygem-foreman_remote_execution-1.5.6-1.fm1_19.el7.noarch
* tfm-rubygem-foreman_remote_execution_core-1.1.3-1.el7.noarch
* tfm-rubygem-foreman_setup-6.0.0-2.fm1_18.el7.noarch
* tfm-rubygem-formatador-0.2.1-10.el7.noarch
* tfm-rubygem-friendly_id-5.2.4-1.el7.noarch
* tfm-rubygem-get_process_mem-0.2.1-2.el7.noarch
* tfm-rubygem-gettext_i18n_rails-1.2.1-4.el7.noarch
* tfm-rubygem-hammer_cli-0.14.0-1.el7.noarch
* tfm-rubygem-hammer_cli_foreman-0.14.0-1.el7.noarch
* tfm-rubygem-hammer_cli_foreman_bootdisk-0.1.3-6.el7.noarch
* tfm-rubygem-hammer_cli_foreman_docker-0.0.4-3.el7.noarch
* tfm-rubygem-hammer_cli_foreman_tasks-0.0.12-2.fm1_17.el7.noarch
* tfm-rubygem-hammer_cli_katello-0.14.1-1.el7.noarch
* tfm-rubygem-hashie-2.0.5-5.el7.noarch
* tfm-rubygem-highline-1.7.8-3.el7.noarch
* tfm-rubygem-http-cookie-1.0.2-2.el7.noarch
* tfm-rubygem-ipaddress-0.8.0-10.el7.noarch
* tfm-rubygem-katello-3.8.0-1.el7.noarch
* tfm-rubygem-ldap_fluff-0.4.7-2.el7.noarch
* tfm-rubygem-little-plugger-1.1.3-22.el7.noarch
* tfm-rubygem-locale-2.0.9-12.el7.noarch
* tfm-rubygem-logging-2.2.2-2.el7.noarch
* tfm-rubygem-net-ldap-0.15.0-2.el7.noarch
* tfm-rubygem-net-ping-2.0.1-2.el7.noarch
* tfm-rubygem-net-scp-1.2.1-2.el7.noarch
* tfm-rubygem-net-ssh-4.0.1-4.el7.noarch
* tfm-rubygem-netrc-0.11.0-2.el7.noarch
* tfm-rubygem-oauth-0.5.4-2.el7.noarch
* tfm-rubygem-ovirt-engine-sdk-4.2.3-1.el7.x86_64
* tfm-rubygem-parse-cron-0.1.4-3.fm1_17.el7.noarch
* tfm-rubygem-passenger-4.0.18-9.12.el7.x86_64
* tfm-rubygem-passenger-native-4.0.18-9.12.el7.x86_64
* tfm-rubygem-passenger-native-libs-4.0.18-9.12.el7.x86_64
* tfm-rubygem-pg-0.21.0-2.el7.x86_64
* tfm-rubygem-polyglot-0.3.5-1.el7.noarch
* tfm-rubygem-powerbar-1.0.17-2.el7.noarch
* tfm-rubygem-qpid_messaging-1.36.0-2.el7.x86_64
* tfm-rubygem-rabl-0.13.1-1.el7.noarch
* tfm-rubygem-rack-jsonp-1.3.1-6.el7.noarch
* tfm-rubygem-rails-i18n-5.0.4-1.el7.noarch
* tfm-rubygem-rainbow-2.2.1-2.el7.noarch
* tfm-rubygem-rbovirt-0.1.6-1.el7.noarch
* tfm-rubygem-rbvmomi-1.10.0-2.el7.noarch
* tfm-rubygem-record_tag_helper-1.0.0-1.el7.noarch
* tfm-rubygem-responders-2.4.0-1.el7.noarch
* tfm-rubygem-rest-client-2.0.1-2.el7.noarch
* tfm-rubygem-roadie-3.2.2-1.el7.noarch
* tfm-rubygem-roadie-rails-1.2.1-1.el7.noarch
* tfm-rubygem-robotex-1.0.0-20.el7.noarch
* tfm-rubygem-ruby2ruby-2.4.0-1.el7.noarch
* tfm-rubygem-ruby_parser-3.10.1-1.el7.noarch
* tfm-rubygem-runcible-2.8.1-1.el7.noarch
* tfm-rubygem-safemode-1.3.5-1.el7.noarch
* tfm-rubygem-scoped_search-4.1.3-1.el7.noarch
* tfm-rubygem-secure_headers-5.0.5-1.el7.noarch
* tfm-rubygem-sequel-5.7.1-1.el7.noarch
* tfm-rubygem-sexp_processor-4.10.0-4.el7.noarch
* tfm-rubygem-smart_proxy_dynflow_core-0.2.0-1.fm1_18.el7.noarch
* tfm-rubygem-sprockets-rails-3.2.0-2.el7.noarch
* tfm-rubygem-sshkey-1.9.0-2.el7.noarch
* tfm-rubygem-trollop-2.1.2-2.el7.noarch
* tfm-rubygem-unf-0.1.3-6.el7.noarch
* tfm-rubygem-unf_ext-0.0.6-7.el7.x86_64
* tfm-rubygem-unicode-0.4.4.1-4.el7.x86_64
* tfm-rubygem-unicode-display_width-1.0.5-2.el7.noarch
* tfm-rubygem-useragent-0.16.8-2.el7.noarch
* tfm-rubygem-validates_lengths_from_database-0.5.0-4.el7.noarch
* tfm-rubygem-webpack-rails-0.9.8-2.el7.noarch
* tfm-rubygem-wicked-1.3.2-1.el7.noarch
* tfm-rubygem-will_paginate-3.1.5-2.el7.noarch
* tfm-rubygem-x-editable-rails-1.5.5-2.el7.noarch
* tfm-runtime-4.0-3.el7.x86_64
Foreman and Proxy plugin versions:
Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]
(for logs, surround with three back-ticks to get proper formatting, e.g.)
logs
Digging into the database, found hypervisors in state 5;
candlepin=# select id, state from cp_job where id like 'hypervisor_update%';
id | state
--------------------------------------------------------+-------
hypervisor_update_deae54a8-52f7-41df-a963-769baf64a3b4 | 5
hypervisor_update_fb653882-9d91-4be9-bdeb-acb3fe587c49 | 5
hypervisor_update_50178ebc-bdd1-4d9b-a315-560261ff4606 | 5
hypervisor_update_1fcbf443-e14f-4aa5-bc9b-df5d7f954e25 | 5
(4 rows)
hypervisor_update_fb653882-9d91-4be9-bdeb-acb3fe587c49 | 2018-10-15 15:15:49.387-05 | 2018-10-15 15:16:33.725-05 | 2018-10-15 15:15:53.241-05 | async group | foreman_admin | javax.persistence.RollbackException: Error while committing the transaction |
2018-10-15 15:15:49.42-05 | 5 | Company| 0 | org.candlepin.pinsetter.tasks.HypervisorUpdateJob