RHCOS // FHCOS provisioned for openshift

elytscha · October 1, 2020, 12:37pm

Hi,

we provision our infrastructure via foreman, so there we get an overview over our servers (barematel, vms from vmware)

we are integrating openshift,(okd and licensed ocp) therefore we need rhel coreos/fedora coreos
we want to provision those vms the same way we provision the rest of our servers, via foreman

so i tested and googled,

i was able to get rhcos installing via foreman:

with this pxetemplate:

DEFAULT pxeboot
TIMEOUT 20
PROMPT 0
LABEL pxeboot
    KERNEL boot/rhcos-4.5.6-x86_64-installer-kernel-x86_64	
    APPEND ip=dhcp rd.neednet=1 initrd=boot/rhcos-4.5.6-x86_64-installer-initramfs.x86_64.img coreos.autologin=yes coreos.first_boot=1 coreos.inst=yes coreos.inst.callhome_url=<%= foreman_url('provision')%> coreos.inst.install_dev=sda coreos.inst.image_url=http://foremantest71.domain.com:8089/pxeboot/rhcos/rhcos-4.5.6-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://foremantest71.oeamtc.at:8089/pxeboot/rhcos/<%= @host.params['role'] %>.ign

its working with that but hangs in an installloop because ‘coreos.inst=yes coreos.inst.callhome_url=<%= foreman_url(‘provision’)%>’ seems not to be the correct thing, so foreman gets never the install success callback after the vm is provisioned, so it provisions the vm every reboot from scratch,

for this i found a workaround, just cancel the vm build after its started, it will provision and dont start a new provisioning after the vm reboots, this is not nice, but can be workarounded so it should not be a problem, this was problem 1.

problem 2 is the following:

<%#	
kind: PXELinux	
name: CoreOS PXELinux	
model: ProvisioningTemplate	
oses:	
- CoreOS	
- Red Hat	
- Rhcos	
%>	
DEFAULT pxeboot
TIMEOUT 20
PROMPT 0
LABEL pxeboot
    KERNEL <%= @kernel %>	
    APPEND ip=dhcp rd.neednet=1 initrd=<%= @initrd %> coreos.autologin=yes coreos.first_boot=1 coreos.inst=yes coreos.inst.callhome_url=<%= foreman_url('provision')%> coreos.inst.install_dev=sda coreos.inst.image_url=http://foremantest71.oeamtc.at:8089/pxeboot/rhcos/rhcos-4.5.6-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://foremantest71.oeamtc.at:8089/pxeboot/rhcos/<%= @host.params['role'] %>.ign

i want to do this: with an installation media and operatings system like this and an applied foreman patch from this ticket Feature #27604: Support RedHat CoreOS and Fedora CoreOS as operating systems - Foreman

class Rhcos < Operatingsystem
    PXEFILES = {:kernel => 'rhcos-installer-kernel-x86_64', :initrd => 'rhcos-installer-initramfs.x86_64.img'}
  
    def pxe_type
      'kickstart'
    end
  
    def mediumpath(medium_provider)
      ''
    end
  
    def pxedir
      ''
    end
  
    def display_family
      'RHCOS'
    end

    def available_loaders                                                                                                                                                                                                                       
      self.class.all_loaders                                                                                                                                                                                                                    
    end
  
  end

so i could dynamically set the installed openshift version via foremans rhcos os, because its possible do determine the openshift download urls from the version, this are the two urls which are needed for the pxe boot to be successful:
https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.5/4.5.6/rhcos-installer-kernel-x86_64
https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.5/4.5.6/rhcos-installer-initramfs.x86_64.img

but with this i only get this error:
There was an error rendering the RHCOS PXE template: wrong number of arguments (given 1, expected 0)

so i seem to have done something wrong but i can’t figure out what

just to explain our way:

our provisioning happens with ansible,

so we let ansible create the vm’s // baremetal hosts on our foreman

we want to do this for openshift the same way

the idea is to provision the rhcos vm’s via ansible on our foreman (powered off), with an extra host_var called ‘role’ which value can be ‘boostrap’,‘master’ or ‘worker’
then ansible triggers the openshift install and creates the ignition files for boostrap,master,worker, uploads them to the foreman http server
then ansible powers up the rhcos vm’s, they find their iginiton file, previously created by the openshift installer and which is now uploaded to the foreman http server, so they are now able to find their igniton file based on their host var value (boostrap, master or worker) and will start with that an openshift 4 install
while this is happening, ansible cancels the build on foreman, so problem 1 is not kicking in.

but if i cant figure out to solve problem 2

that would mean we have to do this in ansible:

create/update via ansible the pxetemplate according to the version with hardcoded values on foreman,
fetch the rhcos kernel/initrd via ansible
put them to the foreman pxe server via ansible
then the openshift node should install with the correct version…

any thoughts or tipps on this?

lzap · October 1, 2020, 1:16pm

The work on CoreOS haven’t finished yet. The blocker was that there was no way of calling home form CoreOS initram phase. There was a ticket, I created a patch for CoreOS but it was not merged therefore your template (kernel parameter) will not work.

https://github.com/coreos/coreos-installer/issues/21

This needs to be done in a different way (I haven’t tried this), it’s a bit clunky but you need to embed a “call home” shell script into ignition itself:

Unfortunately, as you know, Foreman call back URL contains a token. Therefore this means that the ignition config needs to be different for every single host provisioned. Foreman has a powerful templating system and this should not be a problem, except that ignition configuration needs to be “compiled” from YAML to JSON. And Foreman does not support that.

So here we are, CoreOS provisioning is currently not supported. Only if this was easier on the CoreOS side. We probably need to implement some kind of ignition compiler service (smart-proxy API?) that would do this for the templating engine. Or Foreman could call the ignition compiler directly, that’s also an option. Then this could be used in a template like (this will NOT work its just a mockup):

<%= compile_ignition("""
variant: fcos
version: 1.0.0
systemd:
  units:
  - name: run-coreos-installer.service
    enabled: true
    contents: |
      [Unit]
      After=network-online.target
      Wants=network-online.target
      Before=systemd-user-sessions.service
      OnFailure=emergency.target
      OnFailureJobMode=replace-irreversibly
      [Service]
      RemainAfterExit=yes
      Type=oneshot
      ExecStart=/usr/local/bin/run-coreos-installer
      ExecStartPost=/usr/bin/systemctl --no-block reboot
      StandardOutput=kmsg+console
      StandardError=kmsg+console
      [Install]
      WantedBy=multi-user.target
storage:
  files:
    - path: /usr/local/bin/run-coreos-installer
      mode: 0755
      contents:
        inline: |
          #!/usr/bin/bash
          set -x
          poststatus() {
              status=$1
              curl -X POST #{foreman_url}
          }
          main() {
              # Hardcoded values the config.ign file is written out
              # by the Ignition run when the Live environment is booted
              ignition_file='/home/core/config.ign'
              # Image url should be wherever our FCOS image is stored
              # Note you'll want to use https and also copy the image .sig
              # to the appropriate place. Otherwise you'll need to `--insecure`
              image_url='https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/31.20200310.3.0/x86_64/fedora-coreos-31.20200310.3.0-metal.x86_64.raw.xz'
              # Some custom arguments for firstboot
              firstboot_args='console=tty0'
              # Dynamically detect which device to install to.
              # This represents something an admin may want to do to share the
              # same installer automation across various hardware.
              if [ -b /dev/sda ]; then
                  install_device='/dev/sda'
              elif [ -b /dev/nvme0 ]; then
                  install_device='/dev/nvme0'
              else
                  echo "Can't find appropriate device to install to" 1>&2
                  poststatus 'failure'
                  return 1
              fi
              # Call out to the installer and use curl to ping a URL
              # In some provisioning environments it can be useful to
              # post some status information to the environment to let
              # it know the install completed successfully.
              cmd="coreos-installer install --firstboot-args=${firstboot_args}"
              cmd+=" --image-url ${image_url} --ignition=${ignition_file}"
              cmd+=" ${install_device}"
              if $cmd; then
                  echo "Install Succeeded!"
                  poststatus 'success'
                  return 0
              else
                  echo "Install Failed!"
                  poststatus 'failure'
                  return 1
              fi
          }
          main
    - path: /home/core/config.ign
      # A basic Ignition config that will enable autologin on tty1
      contents:
        inline: |
          {
            "ignition": {
              "config": {
                "replace": {
                  "source": null,
                  "verification": {}
                }
              },
              "security": {
                "tls": {}
              },
              "timeouts": {},
              "version": "3.0.0"
            },
            "passwd": {},
            "storage": {},
            "systemd": {
              "units": [
                {
                  "dropins": [
                    {
                      "contents": "[Service]\n# Override Execstart in main unit\nExecStart=\n# Add new Execstart with `-` prefix to ignore failure\nExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM\nTTYVTDisallocate=no\n",
                      "name": "autologin-core.conf"
                    }
                  ],
                  "name": "getty@tty1.service"
                }
              ]
            }
          }
""") %>

Probably a ruby heredoc with proper escaping would need to be used.

If you want to implement this, I will happily do all reviews and merging for you.

TimoGoebel · October 1, 2020, 1:22pm

Just a quick note: We have added this for the original CoreOS as part of Foreman Omaha: https://github.com/theforeman/foreman_omaha/blob/3ea64913dfffa827dcef0acdf15bae70825553ad/app/lib/foreman_omaha/renderer/scope/macros/omaha.rb#L8-L10

lzap · October 1, 2020, 1:28pm

Nice, I forgot. Is there any difference between fcct (Fedora CoreOS) and ct? I think it’s just a rebranded ct.

Have you solved the call home issue in the plugin?

lzap · October 1, 2020, 1:32pm

Also I can’t find any example templates in that plugin, are there any?

Is there any kind of documentation how this works describing the workflow? This isn’t the first time someone is asking for this and this plugin is well hidden to be honest.

TimoGoebel · October 1, 2020, 1:35pm

Well, the workflow with the original CoreOS is:

pxe boot CoreOS, pass a kernel parameter to download a ignition template (the provision template)
the provision template writes the desired host config to disk based on a snippet
the template runs coreos-install, then runs curl to do the call home.

See: https://github.com/theforeman/community-templates/blob/develop/provisioning_templates/provision/coreos_provision.erb

elytscha · October 1, 2020, 1:42pm

just to note there are differences between coreos and rhcos/fcos,

apperently this layout: https://github.com/theforeman/foreman/blob/develop/app/models/operatingsystems/coreos.rb
does not fit for rhcos/fcos

also in case of openshift it will get more complicated in my opinion

the ignition can’t be created by foreman, the igintion file has to be created be the openshift installer
thats why we actually trigger the openshift install in ansible and upload the finished ignition files directly to the foreman http server, so the pxetemplate is able to find them
so i think foreman would have to be able to run the openshift installer in the right version to get working ignition files for an ocp 4 cluster if he wants doing this by itself

in my opinion its far more relevant to enable foreman to able to resolve the rhcos kernel/initrd right

an installation media with this value: https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/$major/$version
and major version 4.5 and minor version 6 values in the os should result somehow in a link like this:
https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.5/4.5.6/rhcos-installer-kernel-x86_64
https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.5/4.5.6/rhcos-installer-initramfs.x86_64.img
which are then downloaded via foreman to the pxe server

the problem with the call home can be work arounded actually by just cancel the build after it started

sure i would be willing to contribute, just my dev knowledge in ruby is very limited, im more the ‘ops’ guy, which can write bash and small golang applications …

lzap · October 1, 2020, 1:57pm

Yes, that’s what we are tracking in Bug #27604: Support RedHat Coreos as operating system in Foreman - Foreman. Currently OS named CoreOS is actually the original CoreOS (Flatcar these days I guess) while Red Hat CoreOS (aka Fedora CoreOS) we don’t have. This should not be difficult, except finding the correct name (Fedora or Red Hat).

But doesn’t OS installer also provide full installation support including discovery? I mean what should be Foreman’s role in this?

That’s ugly, since ignition must be generated externally, is there any way of integrating this better than hitting the cancel button?

I can create the “Fedora CoreOS” operating system with the paths to PXE files for you, but someone needs to patch an instance and test it.

elytscha · October 1, 2020, 2:03pm

yep the os installer does, thats totally true, but if we would do it that way it would mean we have all our servers (baremetal, vms from vmware, maybe some day vms from openstack) in our foreman, where we can keep a nice overview about ALL our servers, regardingless if they are baremetal or vm’s and we also can reprovision them btw build new server the same way the old are built.

if we use the openshift installer, the openshift vm’s will never be in our ‘general server overview’ foreman

maybe a foreman plugin which utilizes the openshift installer would be an idea?

that would be awesome, i’m able to patch our foreman test instance where i could test this out

(just a note we should write okd or ocp [open kubernetes distribution, openshift container plattform] and not os, it can be confusing with os = operating system)

ohadlevy · October 1, 2020, 2:26pm

not that I want to hijack the thread (apologize if I do) OCP has an alternative installation flow using a Sass on cloud.redhat.com that can help simplify the installation, it generates a discovery image which you can boot on your servers, or integrate it with ipxe.

if you want to see a demo of how it works, you can checkout this demo from openshift.tv

lastly, OpenShift manages the nodes/hosts by it self, so in general it can by slightly conflicting with how Foreman does it today, further, the fact that CoreOS was renamed to Fedora Core OS or RHCOS actually makes it a bit confusing to what is working in foreman, I’ll try to find some spare time to implement that integration if I can

I apologize if I offered a non foreman solution to your issues…

elytscha · October 1, 2020, 2:44pm

thank you, i already now this, we did a poc where we deployed the cluster like this, manually

but maybe i should say we are planning 3 openshift clusters and operating at a scale where our linux server count already has 4 digits

that means we rely on heavy automation to get our work done, so we want to be able to hit a button with a few values and get a fully working openshift cluster in our on premise setup, we want the deployment of openshift as automated as possible and as compliant with our already existing infrastructure as possible,

that means
linux server -> foreman
deployments -> ansible

and we would like to keep that, also for openshift

lzap · October 1, 2020, 3:26pm

Can you specify what changes do you need? So far I was able to identify implementing “Fedora CoreOS” operating system with corrected PXE file paths: Bug #27604: Support RedHat Coreos as operating system in Foreman - Foreman Anything else?

elytscha · October 1, 2020, 5:49pm

for beeing able to use foreman os // foreman installation media for rhcos/fcos this is needed:
Bug #27604: Support RedHat Coreos as operating system in Foreman - Foreman x2 (1x for redhat coreos / 1x for fedora coreos)

for fcos it needs to be able to fetch this format, which could be a little more difficult because there the version number is included in the file name, so you can’t hardcode it… :
https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200907.3.0/x86_64/fedora-coreos-32.20200907.3.0-live-initramfs.x86_64.img
https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200907.3.0/x86_64/fedora-coreos-32.20200907.3.0-live-kernel-x86_64

for rhcos it needs to be able to fetch this format:
https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.5/4.5.6/rhcos-installer-kernel-x86_64
https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.5/4.5.6/rhcos-installer-initramfs.x86_64.img

then foreman would need:
a) a plugin which utilizes the openshift installer to create the ignition files for an cluster install? if the goal is to install openshift 4 via foreman

b) as you said some kind of ignition compiler which generates a fcos/rhcos iginition file, pushes it to the foreman http server which the pxe template is then able to pick up? if the goal is to just install rhcos/fcos with foreman

and then yep there is still the problem with the missing functionality for ‘coreos.inst.callhome_url=’, there is not much to do in foreman i think, would it be an idea to talk there directly to redhat? so far i know foreman is the upstream for satellite and fcos is the upstream for rhcos, so redhat is involved in both projects and it should be in their own interest to integrate their provided software as seamless as possible i think

just thinking, wouldn’t it be a good selling point for satellite btw. an awesome feature in foreman? if its possible to just “one klick” an openshift cluster from there?

(for us only the problem with the not supported rhcos/fcos mirrors is an actual pain point, because we can’t use the already existing pxe infrastructure which foreman provides, so we have to upload the kernel/initrd manually to the foreman tftp server and hardcode them in the pxe template)

elytscha · October 1, 2020, 6:07pm

and this i forgot i think this needs to be dynamic too, so a method for getting this into the pxetemplate will be also needed for fhcos/rchos

https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.5/4.5.6/rhcos-metal.x86_64.raw.gz
https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200907.3.0/x86_64/fedora-coreos-32.20200907.3.0-metal.x86_64.raw.xz

lzap · October 2, 2020, 12:06pm

Here is the patch that introduces Fedora/RedHat CoreOS operating systems and installation media. It’s larger than I expected, I tried to improve user experience a bit:

https://github.com/theforeman/foreman/pull/8042

Don’t understand.

Plugin foreman_omaha already provides that, if the workflow provided by this plugin is not what you need, we can simply move this feature from the plugin to core so other plugins or features can take advantage of this.

This is just a simple PXE template change to get the URL to kernel command line. But the design of Foreman does not allow downloading arbitrary files to PXE/HTTPBoot smartproxies other than kernel/initramdisk.

Images are subject of content management (Katello) therefore they should be published somewhere else. Katello/Pulp currently does not provide RHCOS images, I think Pulp2 has some support for arbitrary files (including images) but there is no support in Katello. So for now, you need to make sure to publish this file manually.

I wonder if it makes sense to extend PXE files downloading with additional (3rd) file, the same file can be then exposed using our new HTTPBoot module. This looks like it is against content management (Katello), however after my second thought it is actually hand-in-hand with Katello management - the source content can be from Katello/Pulp just like for initramdisk and kernel. It’s just smart-proxy would make copies of the image closer to clients - it makes sense.

Some intergarion is indeed on our radar, however OpenShift was busy working on the new installer, which @ohadlevy mentioned above. Now when the work is done, we can maybe start talking about integration? @mcorr I think it would make sense to bring stakeholders together in a meeting:

@lzap - Foreman provisioning
@TimoGoebel - DM / Foreman Omaha / CoreOS + Flatcar
@ohadlevy - OpenShift / Red Hat CoreOS
@Marek_Hulan
and others

elytscha · October 5, 2020, 10:46am

just to leave a note here in the thread also:

i started testing the pullrequest you submitted and wrote a comment with my results

lzap · October 5, 2020, 10:49am

Great, let’s continue there. I have a power outage today so working from my Grandma on my backup laptop. I will get back to the PR tomorrow when I get back to my PC Let’s continue the technical details in regard to the new OS there.

elytscha · October 5, 2020, 3:26pm

lzap:

This is just a simple PXE template change to get the URL to kernel command line. But the design of Foreman does not allow downloading arbitrary files to PXE/HTTPBoot smartproxies other than kernel/initramdisk.

Images are subject of content management (Katello) therefore they should be published somewhere else. Katello/Pulp currently does not provide RHCOS images, I think Pulp2 has some support for arbitrary files (including images) but there is no support in Katello. So for now, you need to make sure to publish this file manually.

I wonder if it makes sense to extend PXE files downloading with additional (3rd) file, the same file can be then exposed using our new HTTPBoot module. This looks like it is against content management (Katello), however after my second thought it is actually hand-in-hand with Katello management - the source content can be from Katello/Pulp just like for initramdisk and kernel. It’s just smart-proxy would make copies of the image closer to clients - it makes sense.

we use foreman without katello, so basically i have no katello experience,

but i think it would make sense to add a additional (optional) 3rd file to extend PXE files downloading

as said could be optional so you dont need it if yous os does not use it, but you can use it for rhcos/fcos

as i have seen in the actual fedora docs there is support for the 3rd image in the initrd param like this:

DEFAULT pxeboot
TIMEOUT 20
PROMPT 0
LABEL pxeboot
    KERNEL fedora-coreos-32.20200726.3.1-live-kernel-x86_64
    APPEND initrd=fedora-coreos-32.20200726.3.1-live-initramfs.x86_64.img,fedora-coreos-32.20200726.3.1-live-rootfs.x86_64.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://192.168.1.101:8000/config.ign
IPAPPEND 2

so maybe it could be just appended to here https://github.com/theforeman/foreman/blob/9adb9a9eb16466349b177498772ddad5cf93a37e/app/models/operatingsystems/rhcos.rb#L9

then a basic ignition file for the install should be created somehow by foreman (if its not already possible) and should be put to the foreman httpd server so the pxe parameter ‘coreos.inst.ignition_url=http://192.168.1.101:8000/config.ign’ is able to find there his ignition file and then the install should be able run successfully.
Producing an Ignition Config :: Fedora Docs

(in an install loop, if its that far, we could wait until the call_home function is available in rhcos/fcos and we are good to go or its still necessary to find a solution for this problem )

lzap · October 6, 2020, 11:34am

I agree and this shouldn’t be huge amount of work. Once core OS change is merged I can proceed with patches (both fore core + proxy).

Yeah, I need opinion from @TimoGoebel in here as foreman_omaha already provides some ignition support. But this is all doable.

TimoGoebel · October 6, 2020, 1:14pm

… just shells out to ct. So there is a template macro, that calls ct and provides the command output.

@lzap: Do you need anything more?