Problem:
I have setup Foreman to provision Virtual Machines via libvirt. My setup works fine for deploying to differnt x86_64 based KVM/libvirt hosts running Debian 9.
Now I have added a Raspberry 4 system running CentOS 7.8 (aarch64) to my setup. Libvirt/KVM works fine and I can set up and run virtual machines using virt-manager manually.
Naturally I tried to set up provisioning to this host witth Foreman. It fails at an early stage due to a libvirt error. It seems libvirt is asking or checking for x86_64 capabilities. Of course there are none, because this is a aarch64 host.
This is what I see in production.log:
2020-06-05T17:45:06 [I|app|be83bf84] Started POST "/hosts" for 10.94.76.105 at 2020-06-05 17:45:06 +0200
2020-06-05T17:45:06 [I|app|be83bf84] Processing by HostsController#create as */*
2020-06-05T17:45:06 [I|app|be83bf84] Parameters: {"utf8"=>"✓", "authenticity_token"=>"ZYBkKcMCZXvQGtfASElpnyH7RMwKDmkUfHhobs3rAlgzntewyVjL006kl2Xxxrok/3wio++11A6rcP1RRDYTjw==", "host"=>{"name"=>"karl-hailey", "organization_id"=>"1", "location_id"=>"2", "hostgroup_id"=>"", "compute_resource_id"=
>"3", "compute_profile_id"=>"", "content_facet_attributes"=>{"lifecycle_environment_id"=>"", "content_view_id"=>"", "content_source_id"=>""}, "environment_id"=>"", "puppet_proxy_id"=>"", "puppet_ca_proxy_id"=>"", "ansible_role_ids"=>[""], "managed"=>"true", "progress_report_id"=>"[FILTERED]", "ty
pe"=>"Host::Managed", "interfaces_attributes"=>{"0"=>{"_destroy"=>"0", "type"=>"Nic::Managed", "mac"=>"", "identifier"=>"", "name"=>"karl-hailey", "domain_id"=>"1", "subnet_id"=>"1", "ip"=>"10.94.76.158", "ip6"=>"", "managed"=>"1", "primary"=>"1", "provision"=>"1", "execution"=>"1", "virtual"=>"0
", "tag"=>"", "attached_to"=>"", "compute_attributes"=>{"type"=>"bridge", "bridge"=>"virbr0", "model"=>"virtio"}}}, "compute_attributes"=>{"cpus"=>"2", "cpu_mode"=>"default", "memory"=>"2147483648", "start"=>"1", "volumes_attributes"=>{"0"=>{"_delete"=>"", "pool_name"=>"default", "capacity"=>"10G
", "allocation"=>"10G", "format_type"=>"raw"}}}, "architecture_id"=>"3", "operatingsystem_id"=>"1", "provision_method"=>"build", "build"=>"1", "medium_id"=>"10", "ptable_id"=>"106", "pxe_loader"=>"PXELinux BIOS", "disk"=>"", "root_pass"=>"[FILTERED]", "is_owned_by"=>"4-Users", "enabled"=>"1", "co
mment"=>"", "overwrite"=>"true"}, "media_selector"=>"install_media"}
2020-06-05T17:45:07 [I|app|f2974fdb] Started GET "/notification_recipients" for 10.94.76.105 at 2020-06-05 17:45:07 +0200
2020-06-05T17:45:07 [I|app|f2974fdb] Processing by NotificationRecipientsController#index as JSON
2020-06-05T17:45:08 [I|app|f2974fdb] Completed 200 OK in 60ms (Views: 0.6ms | ActiveRecord: 21.7ms)
2020-06-05T17:45:08 [I|app|be83bf84] Adding Compute instance for karl-hailey.lan.net
2020-06-05T17:45:08 [I|app|15c3a636] Started GET "/tasks/080911ed-f6d2-4228-9fe0-02387f1485d8" for 10.94.76.105 at 2020-06-05 17:45:08 +0200
2020-06-05T17:45:08 [I|app|15c3a636] Processing by TasksController#show as */*
2020-06-05T17:45:08 [I|app|15c3a636] Parameters: {"id"=>"080911ed-f6d2-4228-9fe0-02387f1485d8"}
2020-06-05T17:45:08 [I|app|15c3a636] Rendered tasks/_list.html.erb (1.4ms)
2020-06-05T17:45:08 [I|app|15c3a636] Completed 200 OK in 48ms (Views: 3.8ms | ActiveRecord: 23.2ms)
2020-06-05T17:45:08 [W|app|be83bf84] Unhandled Libvirt error
2020-06-05T17:45:08 [W|app|be83bf84] Failed to create a compute libvirt@raspb4-01 (Libvirt) instance karl-hailey.lan.net: Error saving the server: Call to virDomainDefineXML failed: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm
2020-06-05T17:45:08 [W|app|be83bf84] Rolling back due to a problem: [#<Orchestration::Task:0x00007feac838bb38 @name="Set up compute instance karl-hailey.lan.net", @id="Set up compute instance karl-hailey.lan.net", @status="failed", @priority=3, @action=[#<Host::Managed id: nil, name: "karl-hailey.lan.net", last_compile: nil, last_report: nil, updated_at: nil, created_at: nil, root_pass: "$5$h2YEAXRXJykZgvQ9$GrYSq9P7ssP01WsV3fRh1Xgv1sw2sq...", architecture_id: 3, operatingsystem_id: 1, environment_id: nil, ptable_id: 106, medium_id: 10, build: true, comment: "", disk: "", installed_at: nil, model_id: nil, hostgroup_id: nil, owner_id: 4, owner_type: "User", enabled: true, puppet_ca_proxy_id: nil, managed: true, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: 3, puppet_proxy_id: nil, certname: nil, image_id: nil, organization_id: 1, location_id: 2, type: "Host::Managed", otp: nil, realm_id: nil, compute_profile_id: nil, provision_method: "build", grub_pass: "$6$ef1F3boXaVTxN5l3$7mipX3Huup..F/JEnEnab2V2bP1FCo...", global_status: 0, lookup_value_matcher: "fqdn=karl-hailey.lan.net", pxe_loader: "PXELinux BIOS", initiated_at: nil, build_errors: nil>, :setCompute], @created=1591371907.3606744, @timestamp=2020-06-05 15:45:08 UTC>]
2020-06-05T17:45:08 [I|app|be83bf84] Processed 1 tasks from queue 'Host::Managed Main', completed 0/12
2020-06-05T17:45:08 [E|app|be83bf84] Task 'Set up compute instance karl-hailey.lan.net' *failed*
2020-06-05T17:45:08 [E|app|be83bf84] Task 'DHCP conflicts removal for karl-hailey.lan.net' *canceled*
2020-06-05T17:45:08 [E|app|be83bf84] Task 'Query instance details for karl-hailey.lan.net' *canceled*
2020-06-05T17:45:08 [E|app|be83bf84] Task 'Create DHCP Settings for karl-hailey.lan.net' *canceled*
2020-06-05T17:45:08 [E|app|be83bf84] Task 'Create IPv4 DNS record for karl-hailey.lan.net' *canceled*
2020-06-05T17:45:08 [E|app|be83bf84] Task 'Create Reverse IPv4 DNS record for karl-hailey.lan.net' *canceled*
2020-06-05T17:45:08 [E|app|be83bf84] Task 'Deploy TFTP PXELinux config for karl-hailey.lan.net' *canceled*
2020-06-05T17:45:08 [E|app|be83bf84] Task 'Deploy TFTP PXEGrub2 config for karl-hailey.lan.net' *canceled*
2020-06-05T17:45:08 [E|app|be83bf84] Task 'Deploy TFTP PXEGrub config for karl-hailey.lan.net' *canceled*
2020-06-05T17:45:08 [E|app|be83bf84] Task 'Deploy TFTP iPXE config for karl-hailey.lan.net' *canceled*
2020-06-05T17:45:08 [E|app|be83bf84] Task 'Fetch TFTP boot files for karl-hailey.lan.net' *canceled*
2020-06-05T17:45:08 [E|app|be83bf84] Task 'Power up compute instance karl-hailey.lan.net' *canceled*
2020-06-05T17:45:08 [E|app|be83bf84] Failed to save: Failed to create a compute libvirt@raspb4-01 (Libvirt) instance karl-hailey.lan.net: Error saving the server: Call to virDomainDefineXML failed: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm
and this on the aarch64 KVM host:
Jun 5 15:45:08 raspb4-01 libvirtd: 2020-06-05 15:45:08.887+0000: 750: error : virCapabilitiesDomainDataLookupInternal:750 : invalid argument: could not find capabilities for arch=x86_64 domaintype=kv
This is the output of virsh --connect qemu:///system capabilities on the aarch64 KVM host:
<capabilities>
<host>
<uuid>4980490f-d37e-448d-977f-c7214259c2c8</uuid>
<cpu>
<arch>aarch64</arch>
<topology sockets='1' cores='4' threads='1'/>
<pages unit='KiB' size='4'/>
</cpu>
<power_management/>
<iommu support='no'/>
<migration_features>
<live/>
<uri_transports>
<uri_transport>tcp</uri_transport>
<uri_transport>rdma</uri_transport>
</uri_transports>
</migration_features>
<topology>
<cells num='1'>
<cell id='0'>
<memory unit='KiB'>8004156</memory>
<cpus num='4'>
<cpu id='0' socket_id='0' core_id='0' siblings='0'/>
<cpu id='1' socket_id='0' core_id='1' siblings='1'/>
<cpu id='2' socket_id='0' core_id='2' siblings='2'/>
<cpu id='3' socket_id='0' core_id='3' siblings='3'/>
</cpus>
</cell>
</cells>
</topology>
<secmodel>
<model>none</model>
<doi>0</doi>
</secmodel>
<secmodel>
<model>dac</model>
<doi>0</doi>
<baselabel type='kvm'>+107:+107</baselabel>
<baselabel type='qemu'>+107:+107</baselabel>
</secmodel>
</host>
<guest>
<os_type>hvm</os_type>
<arch name='armv7l'>
<wordsize>32</wordsize>
<emulator>/usr/libexec/qemu-kvm</emulator>
<machine maxCpus='240'>virt-rhel7.6.0</machine>
<machine canonical='virt-rhel7.6.0' maxCpus='240'>virt</machine>
<domain type='qemu'/>
<domain type='kvm'>
<emulator>/usr/libexec/qemu-kvm</emulator>
</domain>
</arch>
<features>
<cpuselection/>
<deviceboot/>
<disksnapshot default='off' toggle='no'/>
</features>
</guest>
<guest>
<os_type>hvm</os_type>
<arch name='aarch64'>
<wordsize>64</wordsize>
<emulator>/usr/libexec/qemu-kvm</emulator>
<machine maxCpus='240'>virt-rhel7.6.0</machine>
<machine canonical='virt-rhel7.6.0' maxCpus='240'>virt</machine>
<domain type='qemu'/>
<domain type='kvm'>
<emulator>/usr/libexec/qemu-kvm</emulator>
</domain>
</arch>
<features>
<cpuselection/>
<deviceboot/>
<disksnapshot default='off' toggle='no'/>
<acpi default='on' toggle='yes'/>
</features>
</guest>
</capabilities>
Expected outcome:
Create a virtual machine on the aarch64 KVM host.
Foreman and Proxy versions:
foreman-1.24.3-1.el7.noarch
foreman-proxy-1.24.3-1.el7.noarch
foreman-installer-katello-1.24.3-1.el7.noarch
foreman-release-1.24.3-1.el7.noarch
Foreman and Proxy plugin versions:
foreman-libvirt-1.24.3-1.el7.noarch
libvirt-libs-4.5.0-33.el7_8.1.x86_64
on the aarch64 host:
libvirt-daemon-4.5.0-33.el7_8.1.aarch64
libvirt-4.5.0-33.el7_8.1.aarch64
libvirt-libs-4.5.0-33.el7_8.1.aarch64
Distribution and version:
CentOS 7.8 x86_64 on the Forman host
CentOS 7.8. aarch64 on the Raspberry Pi 4 KVM host
Other relevant data: