Hello!
I created some patches to enable Foreman to provision libvirt virtual machines running on S390x (IBM Z). The following are my notes to get everything working. My environment was:
- Fedora 33 S390x LPAR running on an IBM z14 mainframe
- Red Hat Satellite 6.10 Server
- Capsule 6.10 with TFTP feature with the TFTP mounted over NFS to the hypervisor (every hypervisor requires its own Capsule)
In my case, I had a Fedora 33 installation on LPAR, but according to IBM documentation KVM on RHEL 8.3 is supported and should work fine as long as it has new enough libvirt with ccw PXELinux parsing feature. I used Satellite 6.10 however this tutorial will work on Foreman 2.5 or newer.
The client was Red Hat Enterprise Linux 8.3. I also tested RHEL 7.9 but due to bug in Anaconda (or Pulp depending on how you see it) the installer will be unable to find the second stage bootloader. The workaround is to create symlinks named .treeinfo pointing to treeinfo files for all the repositories. Filed a Pulp BZ for this: 1938233 – Unable to kickstart IBM Z due to missing .treeinfo but the customer I was creating this for is only interested in EL8 so I closed the BZ in the end.
There are several ways to do networking. Typically, hypervisors could be setup with bridge libvirt interfaces and a DHCP Capsule (running on Intel x64) would be deployed in these networks for full DHCP management. Another option, which I tested, was to keep libvirt DHCP server in use (dnsmasq) and do static IP provisioning. In this scenario, DHCP is only used to boot over a network and then the installer downloads kickstart and configures networking with static IP addresses. I recommend creating a small pool for dynamic addresses (e.g. 192.168.122.1 to 50) and allocate static IP addresses in the remaining space.
Also double check MD5 sums of initramdisk and kernel if you run into issues, TFTP over NFS can cause issues on the initial download. Also pay attention to SELinux, I tested running Satellite in enforcing mode and no policy changes were required.
Red Hat Enterprise Linux LPAR hypervisor
I did put the main network connection into the “trusted” zone, make sure to enable NFS ports if you want to keep it in the insecure zone.
Install and enable libvirt daemon, make sure the “default” network connection (192.168.122.0) is up and running.
$ cat /etc/exports.d/tftp.exports
/var/lib/tftpboot *(rw,async,insecure,all_squash,anonuid=0,anongid=0)
Satellite Server 6.10 (Foreman 2.5)
Unless you are on Satellite 7.0 or Foreman 3.1 with fog-libvirt library 0.9+, upgrade the package manually from Foreman nightly repositories:
yum upgrade https://yum.theforeman.org/nightly/el7/x86_64/tfm-rubygem-fog-libvirt-0.9.0-1.el7.noarch.rpm
Synchronize IBM Z RHEL 7 or 8 or when not using Katello just define a new architecture named s390x and create an operating system with this architecture. Make sure that PXELinux and Kickstart templates are associated with the OS.
Install NFS client (nfs-utils package). Mount TFTP directory for read-write via NFS.
# grep nfs /etc/fstab
xxxxx.z14.xxxxx.redhat.com:/var/lib/tftpboot /var/lib/tftpboot nfs defaults 0 0
Follow our project or product documentation on generating ssh keys for the foreman user, copy public key over to the hypervisor server, connect from the foreman user and accept key fingerprint. Test that a foreman user from Satellite can connect to the hypervisor without password.
# su foreman -s /bin/bash
# ssh-keygen
# ssh-copy-id root@hyperfisor
# ssh root@hypervisor
Create a compute resource, when not using Katello then Operating System, architecture must be set to “s390x” so templates will detect that and render some custom bits. Build and deploy the reboot dracut image to the TFTP file, check permissions and SELinux.
Make sure the TFTP Capsule is associated with a Subnet. In case of DHCP orchestrated scenario, make sure that DHCP Capsule is associated as well and it is deployed in the (bridge) libvirt network and it is the only DHCP server on that network (disable dnsmasq). In case of unmanaged DHCP do not select and DHCP Capsule for the subnet.
Create a new host via the compute resource, select synchronized content (or operating system when not using Katello), select domain, subnet and submit the host for provisioning.
The console built-in Foreman (Satellite) will not work as there is no VNC or Spice available so virsh or cockpit must be used to access it. I have filed a feature request to implement this.
Older versions of Satellite or Foreman will work too, the following patches are required:
Here is a quick and dirty demonstration of the setup.
