Fedora/Red Hat CoreOS provisioning with Foreman

Tutorials
1

Here are the steps I took to be able to provision/bootstrap an OKD or Openshift 4.11 cluster using Foreman, following the upi baremetal process.

Notes

Same steps are needed for both OKD and Openshift, the only differences are the coreos flavour (rhcos or fcos) and pull-secret for Openshift.

There are probably ways to use native Foreman functions for these steps:

  • Web server to publish the rootfs and rendered ignition files
  • Upload custom initramfs/kernel to tftpboot custom directory

Also note that the Foreman Ansible module doesn’t play well with Fcos/Rhcos operating systems, but all the rest can be Ansible-ised if you are into that kind of thing.

Callback URL is generated by foreman and passed as a kernel argument. It is ignored by installer but can be extracted by dmesg.

External documentation

https://docs.okd.io/latest/installing/installing_bare_metal/installing-bare-metal.html

Prerequisites

Download:
foreman_cos.tar (10 KB)

In the simplest of cases, you need a VM (ocpwks.example.com) with these services:

  • nginx publishing a directory. i.e. /var/www would be http://ocpwks.example.com/
  • Cluster dedicated IP mounted as a secondary IP. DNS record for (api|api-int|*.apps).clustername.example.com points to this secondary IP
  • Haproxy configured to LB cluster dedicated IP to bootstrap/master/worker as per OKD documentation.
  • Docker installed

On foreman:

  • Proxy with a custom tftp directory.
    –foreman-proxy-tftp-dirs /var/lib/tftpboot/custom_boot
  • FedoraCoreOS and RedHatCoreOS operating systems defined, with:
    • Installation media is irrelevant, as kernel/customized initramfs are plced in the custom tftp dir on the proxies
    • Both OSes use the PXELinux template “CoreOS PXELinux Custom Boot”

CoreOS and OKD/Openshift files preparation

On ocpwks:

  • Download the proper openshift-install binary and extract the 3 files for your desired CoreOS version and arch.
    openshift-install coreos print-stream-json | grep -Eo '"https.*(kernel-|initramfs.|rootfs.)\w+(\.img)?"'
  • Download the foreman_callback.sh script in this tutorial.
  • Modify the initramfs to add the foreman_callback.sh as a post-install step. $SOURCE_COS_INITRAMFS_OUTPUT_FILENAME can be the same name as the original initramfs with “foreman” somewhere in the filename to make it clear.
    $ docker run -v .:/workdir quay.io/coreos/coreos-installer:release pxe customize --post-install /workdir/foreman_callback.sh -o /workdir/$SOURCE_COS_INITRAMFS_OUTPUT_FILENAME /workdir/$SOURCE_COS_INITRAMFS_FILENAME
    $ scp $SOURCE_COS_KERNEL $SOURCE_COS_INITRAMFS_OUTPUT_FILENAME foreman-proxy.example.com:/var/lib/tftpboot/custom_boot/
    Note: Make sure the kernel and modified initramfs are world readable on the foreman-proxy.
  • Copy the CoreOS rootfs file to the /var/www folder.
  • Generate the ignition files as per documentation and copy the (bootsrap|master|worker).ign to /var/www.

Host configuration in foreman

Define proper Host/Hostgroup parameters.

Examples in Ansible for Openshift:

Common to all bootstrap/master/worker:

      - name: coreos_ignition_urlprefix
        parameter_type: string
        value: "http://ocpwks.example.com/{{ ocp_cluster_name }}/"
      - name: coreos_rootfs_url
        parameter_type: string
        value: "http://ocpwks.example.com/rhcos-{{ coreos_version }}-live-rootfs.x86_64.img"
      - name: coreos_kernel_args
        parameter_type: string
        value: "ignition.firstboot ignition.platform.id=metal coreos.inst.install_dev=/dev/sda"
      - name: initrd_path
        parameter_type: string
        value: "custom_boot/rhcos-{{ coreos_version }}-live-initramfs-foreman.x86_64.img"
      - name: kernel_path
        parameter_type: string
        value: "custom_boot/rhcos-{{ coreos_version }}-live-kernel-x86_64"

Specific to each bootstrap/master/worker:

            - name: coreos_ignition_filename
              parameter_type: string
              value: bootstrap.ign

Install process

  • Set hosts to build
  • PXE boot
  • Sign all certs (check OKD/Openshift doc) in a while loop
3 Likes