Remote hypervisor using qemu+ssh hangs

Support
#1

This is from a recent message I sent to the libvirt-users mailing list
with no responses…maybe someone who uses Foreman has come across
this. Any help troubleshooting is greatly appreciated. Still at the
phase of testing the URI to add to Foreman.

··· -----------------------

I’m attempting to remote connect to my KVM instance using virsh, but
all the commands hang.

When issuing the below command, nothing on the remote system happens,
and no errors are displayed, (hostname changed)
$ virsh --debug 5 --log /var/lib/foreman/virsh.log -c qemu+ssh://
foreman@kvmhost.tld:16509/system?no_tty=1

This is the uncommented lines in /etc/libvirt/libvirtd.conf

listen_tls = 0
listen_tcp = 1
listen_addr = "<omitted, set to management NIC>"
log_level = 1
log_filters="1:remote 1:event 1:qemu"
log_outputs=“1:syslog:libvirtd 1:file:/var/log/libvirt/libvirtd.log”

This is the only debug output I get in /var/log/libvirt/libvirtd.log
during the remote connection attempt

17:56:04.579: debug : virEventRunOnce:595 : Poll got 1 event
17:56:04.580: debug : virEventDispatchTimeouts:405 : Dispatch 3
17:56:04.580: debug : virEventDispatchHandles:450 : Dispatch 10
17:56:04.580: debug : virEventDispatchHandles:464 : i=0 w=1
17:56:04.580: debug : virEventDispatchHandles:464 : i=1 w=2
17:56:04.580: debug : virEventDispatchHandles:464 : i=2 w=3
17:56:04.580: debug : virEventDispatchHandles:464 : i=3 w=4
17:56:04.580: debug : virEventDispatchHandles:464 : i=4 w=5
17:56:04.580: debug : virEventDispatchHandles:464 : i=5 w=6
17:56:04.580: debug : virEventDispatchHandles:464 : i=6 w=7
17:56:04.580: debug : virEventDispatchHandles:464 : i=7 w=8
17:56:04.580: debug : virEventDispatchHandles:477 : Dispatch n=7 f=13
w=8 e=1 0x1629640
17:56:04.580: debug : virEventAddHandleImpl:113 : Add handle fd=20
events=1 cb=0x4196e0 opaque=0x1629640
17:56:04.580: debug : virEventInterruptLocked:664 : Skip interrupt, 1
-1447459072
17:56:04.580: debug : virEventDispatchHandles:464 : i=8 w=9
17:56:04.580: debug : virEventDispatchHandles:464 : i=9 w=10
17:56:04.580: debug : virEventCleanupTimeouts:495 : Cleanup 3
17:56:04.580: debug : virEventCleanupHandles:536 : Cleanupo 11
17:56:04.580: debug : virEventCleanupTimeouts:495 : Cleanup 3
17:56:04.580: debug : virEventCleanupHandles:536 : Cleanupo 11
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=0 w=1, f=5
e=1
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=1 w=2, f=7
e=1
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=2 w=3, f=14
e=1
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=3 w=4, f=15
e=1
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=4 w=5, f=17
e=25
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=5 w=6, f=18
e=25
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=6 w=7, f=19
e=25
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=7 w=8, f=13
e=25
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=8 w=9, f=12
e=25
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=9 w=10, f=11
e=25
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=10 w=15,
f=20 e=1
17:56:04.580: debug : virEventCalculateTimeout:314 : Calculate expiry
of 3 timers
17:56:04.580: debug : virEventCalculateTimeout:344 : Timeout at 0 due
in -1 ms
17:56:04.580: debug : virEventRunOnce:593 : Poll on 11 handles
0x7f35a4001240 timeout -1

I’ve already opened up the firewall for port 16509, and allowed the
user foreman (member of libvirt_admin) to manage libvirt via PolicyKit
Relevant line in iptables,
5 ACCEPT tcp – 0.0.0.0/0 0.0.0.0/0 tcp
dpt:16509

/etc/polkit-1/localauthority/50-local.d/50-libvirt-remote-access.pkla

[libvirt Remote Access]
Identity=unix-group:libvirt_admin
Action=org.libvirt.unix.manage
ResultAny=yes
ResultInactive=yes
ResultActive=yes

Originally I had created the file /etc/polkit-1/localauthority/50-
local.d/51-libvirt-foreman-remote-access.pkla with contents below, and
had the file 50-libvirt-remote-access.pkla only allowing a single
user.
/etc/polkit-1/localauthority/50-local.d/51-libvirt-foreman-remote-
access.pkla

[libvirt Foreman Remote Access]
Identity=unix-user:foreman
Action=org.libvirt.unix.manage
ResultAny=yes
ResultInactive=yes
ResultActive=yes

However I wasn’t able to connect to libvirt on the host itself, and
the logs indicated it was a PolicyKit block, so my second problem/
question…Is it possible to have multiple local PolicyKit *.pkla
files or can only one exist? From the documentation here,
http://wiki.libvirt.org/page/SSHPolicyKitSetup, it seems like so long
as the names are unique then multiple would be allowed. Reason that’s
key is I’m using Puppet and will have multiple servers/applications
needing access and being restricted to a single file to manage will be
a problem.

Connecting locally with a specific pkla for “foreman”…

$ virsh -c qemu:///system
error: authentication failed
error: failed to connect to the hypervisor

/var/log/libvirt/libvirtd.log

17:50:06.102: debug : virRunWithHook:914 : Command stderr: Not
authorized.

17:50:06.103: error : remoteDispatchAuthPolkit:3810 : Policy kit
denied action org.libvirt.unix.manage from pid 29640, uid 503, result:
256

Thanks

  • Trey
#2

> This is from a recent message I sent to the libvirt-users mailing list
> with no responses…maybe someone who uses Foreman has come across
> this. Any help troubleshooting is greatly appreciated. Still at the
> phase of testing the URI to add to Foreman.
>
>
> -----------------------
>
> I'm attempting to remote connect to my KVM instance using virsh, but
> all the commands hang.
>
> When issuing the below command, nothing on the remote system happens,
> and no errors are displayed, (hostname changed)
> $ virsh --debug 5 --log /var/lib/foreman/virsh.log -c qemu+ssh://
> foreman@kvmhost.tld:16509/system?no_tty=1
Are you using SSH or tcp?

I'm using libvirth with pure SSL certs, reusing the same set of certs
that puppet generated.
but for a quick test, just get it work with libvirt in listening mode
and turn of sasl, this should be as trivial to use with uri such as
qemu+tcp://hostname/system (if i remember correctly).

Ohad

··· On Tue, Sep 20, 2011 at 8:21 PM, treydock wrote:

This is the uncommented lines in /etc/libvirt/libvirtd.conf

listen_tls = 0
listen_tcp = 1
listen_addr = "<omitted, set to management NIC>"
log_level = 1
log_filters="1:remote 1:event 1:qemu"
log_outputs=“1:syslog:libvirtd 1:file:/var/log/libvirt/libvirtd.log”

This is the only debug output I get in /var/log/libvirt/libvirtd.log
during the remote connection attempt

17:56:04.579: debug : virEventRunOnce:595 : Poll got 1 event
17:56:04.580: debug : virEventDispatchTimeouts:405 : Dispatch 3
17:56:04.580: debug : virEventDispatchHandles:450 : Dispatch 10
17:56:04.580: debug : virEventDispatchHandles:464 : i=0 w=1
17:56:04.580: debug : virEventDispatchHandles:464 : i=1 w=2
17:56:04.580: debug : virEventDispatchHandles:464 : i=2 w=3
17:56:04.580: debug : virEventDispatchHandles:464 : i=3 w=4
17:56:04.580: debug : virEventDispatchHandles:464 : i=4 w=5
17:56:04.580: debug : virEventDispatchHandles:464 : i=5 w=6
17:56:04.580: debug : virEventDispatchHandles:464 : i=6 w=7
17:56:04.580: debug : virEventDispatchHandles:464 : i=7 w=8
17:56:04.580: debug : virEventDispatchHandles:477 : Dispatch n=7 f=13
w=8 e=1 0x1629640
17:56:04.580: debug : virEventAddHandleImpl:113 : Add handle fd=20
events=1 cb=0x4196e0 opaque=0x1629640
17:56:04.580: debug : virEventInterruptLocked:664 : Skip interrupt, 1
-1447459072
17:56:04.580: debug : virEventDispatchHandles:464 : i=8 w=9
17:56:04.580: debug : virEventDispatchHandles:464 : i=9 w=10
17:56:04.580: debug : virEventCleanupTimeouts:495 : Cleanup 3
17:56:04.580: debug : virEventCleanupHandles:536 : Cleanupo 11
17:56:04.580: debug : virEventCleanupTimeouts:495 : Cleanup 3
17:56:04.580: debug : virEventCleanupHandles:536 : Cleanupo 11
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=0 w=1, f=5
e=1
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=1 w=2, f=7
e=1
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=2 w=3, f=14
e=1
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=3 w=4, f=15
e=1
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=4 w=5, f=17
e=25
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=5 w=6, f=18
e=25
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=6 w=7, f=19
e=25
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=7 w=8, f=13
e=25
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=8 w=9, f=12
e=25
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=9 w=10, f=11
e=25
17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=10 w=15,
f=20 e=1
17:56:04.580: debug : virEventCalculateTimeout:314 : Calculate expiry
of 3 timers
17:56:04.580: debug : virEventCalculateTimeout:344 : Timeout at 0 due
in -1 ms
17:56:04.580: debug : virEventRunOnce:593 : Poll on 11 handles
0x7f35a4001240 timeout -1

I’ve already opened up the firewall for port 16509, and allowed the
user foreman (member of libvirt_admin) to manage libvirt via PolicyKit
Relevant line in iptables,
5 ACCEPT tcp – 0.0.0.0/0 0.0.0.0/0 tcp
dpt:16509

/etc/polkit-1/localauthority/50-local.d/50-libvirt-remote-access.pkla

[libvirt Remote Access]
Identity=unix-group:libvirt_admin
Action=org.libvirt.unix.manage
ResultAny=yes
ResultInactive=yes
ResultActive=yes

Originally I had created the file /etc/polkit-1/localauthority/50-
local.d/51-libvirt-foreman-remote-access.pkla with contents below, and
had the file 50-libvirt-remote-access.pkla only allowing a single
user.
/etc/polkit-1/localauthority/50-local.d/51-libvirt-foreman-remote-
access.pkla

[libvirt Foreman Remote Access]
Identity=unix-user:foreman
Action=org.libvirt.unix.manage
ResultAny=yes
ResultInactive=yes
ResultActive=yes

However I wasn’t able to connect to libvirt on the host itself, and
the logs indicated it was a PolicyKit block, so my second problem/
question…Is it possible to have multiple local PolicyKit *.pkla
files or can only one exist? From the documentation here,
http://wiki.libvirt.org/page/SSHPolicyKitSetup, it seems like so long
as the names are unique then multiple would be allowed. Reason that’s
key is I’m using Puppet and will have multiple servers/applications
needing access and being restricted to a single file to manage will be
a problem.

Connecting locally with a specific pkla for “foreman”…

$ virsh -c qemu:///system
error: authentication failed
error: failed to connect to the hypervisor

/var/log/libvirt/libvirtd.log

17:50:06.102: debug : virRunWithHook:914 : Command stderr: Not
authorized.

17:50:06.103: error : remoteDispatchAuthPolkit:3810 : Policy kit
denied action org.libvirt.unix.manage from pid 29640, uid 503, result:
256

Thanks

  • Trey

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en.

#3

I was able to connect using "sudo -u foreman virsh -c qemu+tcp://
kvmhost.tld/system?no_verify=1" once I turned off sasl. However now
when I try my original connection I get this…

$ virsh -c qemu+ssh://kvmhost.tld/system?no_verify=1
error: packet received from server too large
error: failed to connect to the hypervisor

Nothing comes to the KVM server's logs except,

Sep 28 17:44:01 kvmhost sshd[32664]: Accepted publickey for foreman
from … port 57421 ssh2
Sep 28 17:44:01 kvmhost sshd[32664]: pam_unix(sshd:session): session
opened for user foreman by (uid=0)
Sep 28 17:44:01 kvmhost sshd[32664]: pam_unix(sshd:session): session
closed for user foreman

Any ideas?

The configuration is the same as my original post.

  • Trey
··· On Sep 20, 12:35 pm, Ohad Levy wrote: > On Tue, Sep 20, 2011 at 8:21 PM, treydock wrote: > > This is from a recent message I sent to the libvirt-users mailing list > > with no responses...maybe someone who uses Foreman has come across > > this. Any help troubleshooting is greatly appreciated. Still at the > > phase of testing the URI to add to Foreman. > > > ----------------------- > > > I'm attempting to remote connect to my KVM instance using virsh, but > > all the commands hang. > > > When issuing the below command, nothing on the remote system happens, > > and no errors are displayed, (hostname changed) > > $ virsh --debug 5 --log /var/lib/foreman/virsh.log -c qemu+ssh:// > > fore...@kvmhost.tld:16509/system?no_tty=1 > > Are you using SSH or tcp? > > I'm using libvirth with pure SSL certs, reusing the same set of certs > that puppet generated. > but for a quick test, just get it work with libvirt in listening mode > and turn of sasl, this should be as trivial to use with uri such as > qemu+tcp://hostname/system (if i remember correctly). > > Ohad > > > > > > > > > > > This is the uncommented lines in /etc/libvirt/libvirtd.conf > > ---------- > > > listen_tls = 0 > > listen_tcp = 1 > > listen_addr = "" > > log_level = 1 > > log_filters="1:remote 1:event 1:qemu" > > log_outputs="1:syslog:libvirtd 1:file:/var/log/libvirt/libvirtd.log" > > > This is the only debug output I get in /var/log/libvirt/libvirtd.log > > during the remote connection attempt > > ----------- > > 17:56:04.579: debug : virEventRunOnce:595 : Poll got 1 event > > 17:56:04.580: debug : virEventDispatchTimeouts:405 : Dispatch 3 > > 17:56:04.580: debug : virEventDispatchHandles:450 : Dispatch 10 > > 17:56:04.580: debug : virEventDispatchHandles:464 : i=0 w=1 > > 17:56:04.580: debug : virEventDispatchHandles:464 : i=1 w=2 > > 17:56:04.580: debug : virEventDispatchHandles:464 : i=2 w=3 > > 17:56:04.580: debug : virEventDispatchHandles:464 : i=3 w=4 > > 17:56:04.580: debug : virEventDispatchHandles:464 : i=4 w=5 > > 17:56:04.580: debug : virEventDispatchHandles:464 : i=5 w=6 > > 17:56:04.580: debug : virEventDispatchHandles:464 : i=6 w=7 > > 17:56:04.580: debug : virEventDispatchHandles:464 : i=7 w=8 > > 17:56:04.580: debug : virEventDispatchHandles:477 : Dispatch n=7 f=13 > > w=8 e=1 0x1629640 > > 17:56:04.580: debug : virEventAddHandleImpl:113 : Add handle fd=20 > > events=1 cb=0x4196e0 opaque=0x1629640 > > 17:56:04.580: debug : virEventInterruptLocked:664 : Skip interrupt, 1 > > -1447459072 > > 17:56:04.580: debug : virEventDispatchHandles:464 : i=8 w=9 > > 17:56:04.580: debug : virEventDispatchHandles:464 : i=9 w=10 > > 17:56:04.580: debug : virEventCleanupTimeouts:495 : Cleanup 3 > > 17:56:04.580: debug : virEventCleanupHandles:536 : Cleanupo 11 > > 17:56:04.580: debug : virEventCleanupTimeouts:495 : Cleanup 3 > > 17:56:04.580: debug : virEventCleanupHandles:536 : Cleanupo 11 > > 17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=0 w=1, f=5 > > e=1 > > 17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=1 w=2, f=7 > > e=1 > > 17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=2 w=3, f=14 > > e=1 > > 17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=3 w=4, f=15 > > e=1 > > 17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=4 w=5, f=17 > > e=25 > > 17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=5 w=6, f=18 > > e=25 > > 17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=6 w=7, f=19 > > e=25 > > 17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=7 w=8, f=13 > > e=25 > > 17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=8 w=9, f=12 > > e=25 > > 17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=9 w=10, f=11 > > e=25 > > 17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=10 w=15, > > f=20 e=1 > > 17:56:04.580: debug : virEventCalculateTimeout:314 : Calculate expiry > > of 3 timers > > 17:56:04.580: debug : virEventCalculateTimeout:344 : Timeout at 0 due > > in -1 ms > > 17:56:04.580: debug : virEventRunOnce:593 : Poll on 11 handles > > 0x7f35a4001240 timeout -1 > > > I've already opened up the firewall for port 16509, and allowed the > > user foreman (member of libvirt_admin) to manage libvirt via PolicyKit > > Relevant line in iptables, > > 5 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp > > dpt:16509 > > > /etc/polkit-1/localauthority/50-local.d/50-libvirt-remote-access.pkla > > ----------- > > [libvirt Remote Access] > > Identity=unix-group:libvirt_admin > > Action=org.libvirt.unix.manage > > ResultAny=yes > > ResultInactive=yes > > ResultActive=yes > > > Originally I had created the file /etc/polkit-1/localauthority/50- > > local.d/51-libvirt-foreman-remote-access.pkla with contents below, and > > had the file 50-libvirt-remote-access.pkla only allowing a single > > user. > > /etc/polkit-1/localauthority/50-local.d/51-libvirt-foreman-remote- > > access.pkla > > ---------- > > [libvirt Foreman Remote Access] > > Identity=unix-user:foreman > > Action=org.libvirt.unix.manage > > ResultAny=yes > > ResultInactive=yes > > ResultActive=yes > > > However I wasn't able to connect to libvirt on the host itself, and > > the logs indicated it was a PolicyKit block, so my second problem/ > > question...Is it possible to have multiple local PolicyKit *.pkla > > files or can only one exist? From the documentation here, > >http://wiki.libvirt.org/page/SSHPolicyKitSetup, it seems like so long > > as the names are unique then multiple would be allowed. Reason that's > > key is I'm using Puppet and will have multiple servers/applications > > needing access and being restricted to a single file to manage will be > > a problem. > > > Connecting locally with a specific pkla for "foreman"... > > ----------- > > $ virsh -c qemu:///system > > error: authentication failed > > error: failed to connect to the hypervisor > > > /var/log/libvirt/libvirtd.log > > --------- > > 17:50:06.102: debug : virRunWithHook:914 : Command stderr: Not > > authorized. > > > 17:50:06.103: error : remoteDispatchAuthPolkit:3810 : Policy kit > > denied action org.libvirt.unix.manage from pid 29640, uid 503, result: > > 256 > > > Thanks > > - Trey > > > -- > > You received this message because you are subscribed to the Google Groups "Foreman users" group. > > To post to this group, send email to foreman-users@googlegroups.com. > > To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com. > > For more options, visit this group athttp://groups.google.com/group/foreman-users?hl=en.