Bug report: wrong memory shown for libvirt-VM in new UI

Support
1

Problem:
libvirt VMs were created without password on the VNC console and console is not working if the compute_resource have “Console Passwords” enabled.

Expected outcome:
Set a randomized passwords on creation so foreman and continue to set a randomized password on connect.

Foreman and Proxy versions:
Foreman version: 3.9.1
Plugins:

  • foreman-tasks 9.0.1
  • foreman_ansible 13.0.3
  • foreman_discovery 23.0.0
  • foreman_remote_execution 12.0.4
  • foreman_statistics 2.1.0
  • foreman_webhooks 3.2.2
  • katello 4.11.0

Distribution and version:

  • CentOS Stream 8 for Foreman
  • CentOS Stream 9 running on the compute resource (libvirtd)

Other relevant data:

[root@tfm ~]# foreman-rake errors:fetch_log request_id=b942d271
Foreman version: 3.9.1
Plugins:
 - foreman-tasks 9.0.1
 - foreman_ansible 13.0.3
 - foreman_discovery 23.0.0
 - foreman_remote_execution 12.0.4
 - foreman_statistics 2.1.0
 - foreman_webhooks 3.2.2
 - katello 4.11.0

2024-01-24T05:51:44 [I|app|b942d271] Started GET "/hosts/tonia-croson.dev.exmaple.org/console" for 10.225.4.24 at 2024-01-24 05:51:44 +0000
2024-01-24T05:51:44 [I|app|b942d271] Processing by HostsController#console as HTML
2024-01-24T05:51:44 [I|app|b942d271]   Parameters: {"id"=>"tonia-croson.dev.exmaple.org"}
2024-01-24T05:51:44 [W|app|b942d271] Failed to set console
2024-01-24T05:51:44 [I|app|b942d271] Backtrace for 'Failed to set console' error (Libvirt::Error): Call to virDomainUpdateDeviceFlags failed: internal error: unable to execute QEMU command 'set_password': Could not set password
 b942d271 | /usr/share/gems/gems/fog-libvirt-0.12.0/lib/fog/libvirt/requests/compute/update_display.rb:22:in `update_device'
 b942d271 | /usr/share/gems/gems/fog-libvirt-0.12.0/lib/fog/libvirt/requests/compute/update_display.rb:22:in `update_display'
 b942d271 | /usr/share/gems/gems/fog-libvirt-0.12.0/lib/fog/libvirt/models/compute/server.rb:230:in `update_display'
 b942d271 | /usr/share/foreman/app/models/compute_resources/foreman/model/libvirt.rb:184:in `console'
 b942d271 | /usr/share/foreman/app/controllers/hosts_controller.rb:357:in `console'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:228:in `process_action'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:30:in `process_action'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 b942d271 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 b942d271 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 b942d271 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 b942d271 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 b942d271 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 b942d271 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 b942d271 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 b942d271 | /usr/share/gems/gems/audited-5.4.2/lib/audited/sweeper.rb:16:in `around'
 b942d271 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 b942d271 | /usr/share/gems/gems/audited-5.4.2/lib/audited/sweeper.rb:16:in `around'
 b942d271 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 b942d271 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:137:in `run_callbacks'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:41:in `process_action'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/rescue.rb:22:in `process_action'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 b942d271 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `block in instrument'
 b942d271 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 b942d271 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
 b942d271 | /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:165:in `process'
 b942d271 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:39:in `process'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:190:in `dispatch'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:254:in `dispatch'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:50:in `block in serve'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `each'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `serve'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:842:in `call'
 b942d271 | /usr/share/gems/gems/katello-4.11.0/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
 b942d271 | /usr/share/gems/gems/katello-4.11.0/lib/katello/middleware/event_daemon.rb:10:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 b942d271 | /usr/share/gems/gems/apipie-dsl-2.6.1/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 b942d271 | /usr/share/gems/gems/apipie-rails-1.2.3/lib/apipie/static_dispatcher.rb:68:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 b942d271 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 b942d271 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 b942d271 | /usr/share/gems/gems/apipie-rails-1.2.3/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 b942d271 | /usr/share/gems/gems/rack-2.2.8/lib/rack/tempfile_reaper.rb:15:in `call'
 b942d271 | /usr/share/gems/gems/rack-2.2.8/lib/rack/etag.rb:27:in `call'
 b942d271 | /usr/share/gems/gems/rack-2.2.8/lib/rack/conditional_get.rb:27:in `call'
 b942d271 | /usr/share/gems/gems/rack-2.2.8/lib/rack/head.rb:12:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
 b942d271 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 b942d271 | /usr/share/gems/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:266:in `context'
 b942d271 | /usr/share/gems/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:260:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/cookies.rb:697:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 b942d271 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:98:in `run_callbacks'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 b942d271 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:37:in `call_app'
 b942d271 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:28:in `call'
 b942d271 | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 b942d271 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 b942d271 | /usr/share/gems/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/request_id.rb:26:in `call'
 b942d271 | /usr/share/gems/gems/katello-4.11.0/lib/katello/prevent_json_parsing.rb:12:in `call'
 b942d271 | /usr/share/gems/gems/rack-2.2.8/lib/rack/method_override.rb:24:in `call'
 b942d271 | /usr/share/gems/gems/rack-2.2.8/lib/rack/runtime.rb:22:in `call'
 b942d271 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/executor.rb:14:in `call'
 b942d271 | /usr/share/gems/gems/rack-2.2.8/lib/rack/sendfile.rb:110:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/ssl.rb:77:in `call'
 b942d271 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
 b942d271 | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
 b942d271 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/engine.rb:539:in `call'
 b942d271 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/railtie.rb:207:in `public_send'
 b942d271 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/railtie.rb:207:in `method_missing'
 b942d271 | /usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:74:in `block in call'
 b942d271 | /usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `each'
 b942d271 | /usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `call'
 b942d271 | /usr/share/gems/gems/puma-6.4.0/lib/puma/configuration.rb:272:in `call'
 b942d271 | /usr/share/gems/gems/puma-6.4.0/lib/puma/request.rb:100:in `block in handle_request'
 b942d271 | /usr/share/gems/gems/puma-6.4.0/lib/puma/thread_pool.rb:378:in `with_force_shutdown'
 b942d271 | /usr/share/gems/gems/puma-6.4.0/lib/puma/request.rb:99:in `handle_request'
 b942d271 | /usr/share/gems/gems/puma-6.4.0/lib/puma/server.rb:443:in `process_client'
 b942d271 | /usr/share/gems/gems/puma-6.4.0/lib/puma/server.rb:241:in `block in run'
 b942d271 | /usr/share/gems/gems/puma-6.4.0/lib/puma/thread_pool.rb:155:in `block in spawn_thread'
 b942d271 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-01-24T05:51:44 [E|app|b942d271] Failed to save:
2024-01-24T05:51:44 [I|app|b942d271] Redirected to https://tfm.dev.exmaple.org/new/hosts/tonia-croson.dev.exmaple.org
2024-01-24T05:51:44 [I|app|b942d271] Completed 302 Found in 259ms (ActiveRecord: 4.1ms | Allocations: 6748)
2

Eh - I didn’t change the title - how can I fixed that? :sweat_smile:

3

This seems like libvirt can’t perform some operation, can you check libvirt logs too? Some thread suggested the password could not be set, because there’s no GPU. Was this VM created from Foreman? Did you modify it’s HW afterwards?

I guess we may need to add some “rescue” code in case the Console can’t be accessed. @lfu I think you worked on this area, any thoughts?

4

Thanks for responding even when I have butchered the title Marek :joy:

I didn’t check libvirtd log for this one but I can reproduce this easily. My understanding is that the VMs were created without a VNC password and the fog plugin fails to set one while the VM is running.

I’ll repo this and post some logs here when I have a chance. :pray:

5

Apologies for the radio silence. Here are more details to the problem.

libvirt log didn’t show much. The VM started fine without a VNC password set and I have included the startup log below.

Some thread suggested the password could not be set, because there’s no GPU. Was this VM created from Foreman? Did you modify it’s HW afterwards?

Yes the password could not be set and it’s show in the stacktrace in the first post.

There is definitely a GPU defined. The VM has the following in the libvirt XML:

...
<graphics type="vnc" port="-1" autoport="yes" listen="0.0.0.0">
  <listen type="address" address="0.0.0.0"/>
</graphics>
...
<video>
  <model type="cirrus" vram="16384" heads="1" primary="yes"/>
  <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
</video>
...

Yes the VM was created from Foreman and no I did not modify the HW afterwards.

After setting a new attribute “passwd” to a random 8 characters string and the console will start working.

Here is the startup log for a new VM created today (name changed):

LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-27-repo101.dev.in \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-27-repo101.dev.in/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-27-repo101.dev.in/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-27-repo101.dev.in/.config \
/usr/libexec/qemu-kvm \
-name guest=repo101.dev,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-27-repo101.dev.in/master-key.aes"}' \
-machine pc-q35-rhel9.2.0,usb=off,dump-guest-core=off,memory-backend=pc.ram,hpet=off,acpi=on \
-accel kvm \
-cpu host,migratable=on \
-m size=4194304k \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":4294967296}' \
-overcommit mem-lock=off \
-smp 4,sockets=4,cores=1,threads=1 \
-uuid e14a566e-aa60-4f35-ac04-176c939e83b2 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=69,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-shutdown \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"}' \
-device '{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}' \
-device '{"driver":"qemu-xhci","id":"usb","bus":"pci.2","addr":"0x0"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.3","addr":"0x0"}' \
-blockdev '{"driver":"host_device","filename":"/dev/VolGroup01/repo101.dev-disk1","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.4","addr":"0x0","drive":"libvirt-1-format","id":"virtio-disk0","bootindex":2}' \
-netdev '{"type":"tap","fd":"72","vhost":true,"vhostfd":"76","id":"hostnet0"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:99:e7:12","bootindex":1,"bus":"pci.1","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=63,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:8,audiodev=audio1 \
-device '{"driver":"cirrus-vga","id":"video0","bus":"pcie.0","addr":"0x1"}' \
-global ICH9-LPC.noreboot=off \
-watchdog-action reset \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.5","addr":"0x0"}' \
-object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/random"}' \
-device '{"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.6","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/12 (label charserial0)
2024-02-05T21:30:02.558761Z qemu-kvm: -device {"driver":"cirrus-vga","id":"video0","bus":"pcie.0","addr":"0x1"}: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
6

At work we are moving VMs from CentOS 7 hosts to CentOS Stream 9 hosts and I ended up writing a Ansible filter function to massage the XML before importing - it goes something like this:


#!/usr/bin/env python3
'''Filters to help dealing with libvirt
'''

import random, string
import xml.etree.ElementTree as ET

def mkpasswd(length=8):
    chars = string.ascii_letters + string.digits + '!@#$%^&*()-+,.'
    return ''.join(random.sample(chars, k=length))

def libvirt_c7_to_cs9(xml):
    root = ET.fromstring(xml)

    # change the machine attribute in domain.os.type to pc
    dom_os_type = root.find('./os/type')
    if dom_os_type is not None: # bool(dom_os_type) == False when found!?
        dom_os_type.attrib['machine'] = 'pc'

    # disk type: file => block
    for disk in root.findall('./devices/disk'):
        disk.attrib['type'] = 'block'

    # disk source attribute changed from file to dev
    for source in root.findall('./devices/disk/source'):
        if 'dev' in source.attrib:
            continue            # nothing to do!
        source.attrib['dev'] = source.attrib['file']
        source.attrib.pop('file')

    # set a random VNC password
    devices_graphic = root.find('./devices/graphics')
    if devices_graphic is not None and 'passwd' not in devices_graphic.attrib:
        devices_graphic.attrib['passwd'] = mkpasswd()

    return ET.tostring(root).decode('utf-8')

class FilterModule(object):
    def filters(self):
        return {
            'libvirt_c7_to_cs9': libvirt_c7_to_cs9,
        }


if __name__ == '__main__':
    import sys
    with open(sys.argv[1]) as infile:
        print(libvirt_c7_to_cs9(infile.read()))

Hope it would be useful.

1 Like