Virtio device /dev/vda I/O issues with anaconda, can't be provisioned

Support
,
1

Hi all,

I’m having an issue with Foreman 1.15

We are trying to deploy CentOS6 and CentOS7 VM guests on CentOS 7 KVM hosts running libvirt x86_64 3.2.0 14.el7_4.7

The deployment process goes smooth initially, foreman shows the guest is pending installation.

When viewing the console for the guests, anaconda dies on both distributions with a similar error:

I/O error, dev vda

The host dmesg also shows the following:

[112225.395018] Buffer I/O error on dev dm-7, logical block 3145712, async page read
[112344.569672] Buffer I/O error on dev dm-7, logical block 3145712, async page read
[113245.075280] Buffer I/O error on dev dm-7, logical block 3145712, async page read
[113430.610101] Buffer I/O error on dev dm-7, logical block 3145712, async page read
[114144.567496] Buffer I/O error on dev dm-7, logical block 3145712, async page read
[115045.259332] Buffer I/O error on dev dm-7, logical block 3145712, async page read
[115230.527487] Buffer I/O error on dev dm-7, logical block 3145712, async page read

What I was expecting to happen is to be able to deploy guests to kvm hosts that are listed in foreman as compute resources.

I have been able to deploy both bus=scsi and bus=virtio on the kvm hosts manually perfectly fine without this error.

I have also been able to use Foreman to deploy to baremetal fine as well, without issue.

There is some notable difference in the options passed in the kvm commands.

This is an example if Foreman deploys:

qemu     165812      1 68 23:34 ?        00:00:06 /usr/libexec/qemu-kvm -name dev-testvm024.mynetwork.net -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off,dump-guest-core=off -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 4ac758ab-7e9f-4b9b-be3e-e0ce04369d5a -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-3-dev-testvm024.myne.s/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/vms/dev-testvm024.mynetwork.net-disk1,format=raw,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -netdev tap,fd=34,id=hostnet0,vhost=on,vhostfd=36 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:ef:a0:fb,bus=pci.0,addr=0x3,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:0,password -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
root     165823 162331  0 23:35 pts/2    00:00:00 grep --color=auto libvirt

This is an example if I manually deploy using virt-install on the host:

qemu      56450      1  2 14:08 ?        00:14:47 /usr/libexec/qemu-kvm -name dev-foreman02.mynetwork.net -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off,dump-guest-core=off -cpu Broadwell,+rtm,+hle -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 2f61f7e3-490c-43b2-996d-6c1f1edb3062 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-dev-foreman02.myne.s/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/dev/vms/dev-foreman02.mynetwork.net-disk1,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=31 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=54:52:00:0e:4d:ac,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-2-dev-foreman02.myne.s/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:1,password -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on

See here for the full anaconda debug output: https://pastebin.com/VAgxa88w

Any help is greatly appreciated in advance.

  • Shawn
2

Hello, I haven’t seen this behavior and it is unlikely this has anything to do with Foreman or libvirt API we create VM with. We do not pass any special options.

I suspect this is error on your backend device. Are you using physical, LVM, raw or qcow2? You can get similar errors when using qcow2 and something is going on with your underlaying filesystem. I saw issues when using NFS, when FS is broken or non-supported (BRFS). I recommend to avoid file-based backend and stick with LVM. Sometimes thin provisioning in LVM can cause issues as well. In all cases, make sure you have enough space!

Good luck.

3

After a great deal of troubleshooting, it turned out to be the boot image provided by the yum repo!

We were using LVM for it, both on baremetal and virtual machines.

Interestingly, this issue didn’t happen with baremetal, only virtual machines, which was really throwing us off the trail.

Problem is solved now, with a working boot image we’re up and running. Thanks so much for the insight!

1 Like