Virt-who service won't start, times out

Support
23

Sorry for the out of order replies:

From Dynflow, here is the text on the “Execution History” :


event	time	world
start execution	2019-09-16 18:20:21 UTC	5c62e6bf-8733-4508-a8fb-9ff6e39dd505 {"hostname"=>"MY_FOREMAN_FQDN", "pid"=>3079, "queues"=>{"default"=>{"pool_size"=>5}, "remote_execution"=>{"pool_size"=>5}}, "last_seen"=>"2019-09-17 11:56:31.367", "delayed_executor"=>true}
24

Are other things in foreman-tasks working fine? Things like repository syncing or content view publish?

25

This seems to be the only stuck task. Repo sync’s succeeded after this task was started. Other tasks are finishing with state: stopped, Result: success. No Hypervisor task ever moves.

26

Can you elaborate on this please? After installing the RPM, this file exists (I don’t recall creating it when I was installing from source) with two hashes in it. I figured out the second hash is the rhsm password, but what is the first hash? Your reply seems to infer that the first hash is the “password”, but password for what? The only other password in the /etc/virt-who.d/virt-who-config-1.conf is the password for the user for vSphere and that first hash does not work for that. What is that first hash in /var/lib/virt-who/key for?

27

Please detail the command used to clean these Hypervisor tasks. Oddly enough, I already have this plugin installed in my Foreman.

I attempted to do:

foreman-rake foreman_tasks:cleanup TASK_SEARCH="Actions::Candlepin::AsyncHypervisors" STATE="running" AFTER=1s NOOP=true
foreman-rake foreman_tasks:cleanup TASK_SEARCH="Actions::Katello::Host::HypervisorsUpdate" STATE="running" AFTER=1s NOOP=true

and neither command reported any tasks that would be cleaned, even though there are running tasks with state=running and older than 1s. I’d like to use the correct process for this task cleanup, but following the documentation has so far yielded no results.

28

nvm on the foreman_tasks:cleanup command, I had the syntax all wrong. Some trial/error yielded:

foreman-rake foreman_tasks:cleanup TASK_SEARCH="action=Hypervisors" STATES=running AFTER=2h NOOP=true

to at least acknowledge there was a task that needed cleaning.

29

This is a master password encrypted with user provided pasphrase I believe. It’s auto generated with first virt-who-password run I think. I think that’s then used for encrypting/decrypting any other password.

30

Ok, that makes some sense. I’m disappointed it won’t help me solve my issue though.

I’d be more than happy to publish my complete installation steps. I took meticulous notes cuz I’m that kinda guy. Maybe it helps someone reproduce? I really, really, really, need this virt-who business to work and I feel like I’m right on the verge (had it working on old version actually, but now, doesn’t work again).

31

So, a complete service restart seems to have let new Hypervisor tasks actually run.

Solution:

foreman-maintain service stop && foreman-maintain service start

I’m seeing ESXi hosts populating in Hosts->All Hosts now as expected.

/var/lib/virt-who/key hashes explained.

Correct foreman-rake commands for killing a stuck Hypervisor task seems to be close to :

foreman-rake foreman_tasks:cleanup TASK_SEARCH="action=Hypervisors" STATES=running AFTER=1s

(tweak time as needed, this will kill all Hypervisors tasks in state running older than 1s, which is probably all of them, so don’t just c/p)

32

Also side note: they moved the virt-who-0.24.7 to main from cr repo very very recently.