Autoinstalling ubuntu-server 20.04.3 from the live-iso

Tutorials
,
9

@lzap ,

Update:

I have created a finish script based on a clone of ‘preseed default finish’ that will be executed at first boot after the host has been installed with ubuntu 20.04.
Therefor the cloud-init autoinstall file “user-data” will need some additional configurations to make it possible to execute it at first boot:

  user-data:
    runcmd:
      - wget -Y off <%= @static ? "'#{foreman_url('finish', static: 'true')}'" : foreman_url('finish') %> -O /tmp/finish.sh
      - chmod +x /tmp/finish.sh
      - /tmp/finish.sh

I just need to make one more small change in it in order to apply a custom snippet for those who want to apply some additional actions which are not part of the default finish-actions, like for example creating specific additional users, installing additional packages, …

The template will execute following actions at first-boot:

  • configure /etc/hosts with the ip, fqdn and short hostname (necessary for actions that will follow)
  • execute the actions in the custom snippet if it exist
  • add ssh-key for foreman-proxy to roots authorized_keys file
  • install alien in order to install the katello-ca-consumer.rpm (probably not necessary for those who have installed foreman on a system with Ubuntu/Debian, so might need some modifications)
  • install/configure and execute the puppet-agent in order to sign the host in foreman
  • install the ‘subscription manager’ from Atix
  • subscribe the host in foreman with the subscription manager
  • move the initial sources.list file to foreman_backup_sources.list as it will not be necassary anymore (can be removed if desired)
  • send the end of installation to foreman
  • and finaly, when the host has the new boolean parameter ‘upgrade_OS’ = ‘true’, it will upgrade the OS and perform a reboot afterwards.

As of tomorrow I’m going to review and adapt the templates were needed and as soon they are ready I will make them available in github.

Of course, they probably might need some changes, but they will be a good start for the community to implement them in the product.

Note:
Despite the subscription works, there is still one thing not going well.
The host reports for ‘Errata’: “No installed packages and/or enabled repositories have been reported by subscription-manager.” which results the host gets the status “Warning”

I hope that someone from Atix would be so kind to provide us the solution for this.

Question:
Have you or someone else already worked out a solution in order to create the cloud-init autoinstall “user-data” file in /var/www/html/pub/preseed/autoinstall/< hostname-short >?
If so, is it possible for me to apply the changes in our foreman 3.0.1 / katello 4.2.1 environment, and how?

10

@lzap Thanks for your reply, I left a review but no action has been taken so far.

11

@lzap ,

Here the modified and new templates:

Please let me know if you need more info.

2 Likes
12

@atix will probably know more but I think the RPM package only contains a certificate that should already be available as a plain file on /pub or at least it used to be. Maybe we should consider publishing it again if its no longer the case so alien is not necessary.

Yeah Debian Errata is @atix’s domain.

Remind me why you cannot use the build-in Foreman’s userdata endpoint?

And why you did not use the default finish preseed template? What was wrong with it?

Edit: Initially I did not see the wget command now I see it.

13

Ok I have incorporated your latest changes into my big PR:

I do not like introducing a new Default Finish template tho, there is already a one, can you simply extend it with parameters or snippet_if_exists so you would be able to extend it as you wish?

@ezr-ondrej let’s merge the PR so @langesmalle can file smaller improvements based on that, I really don’t have much time for this at the moment.

14

As described in the initial post, the PXElinux template generates a file in /var/lib/tftpboot/pxelinux.cfg/ with following content:

DEFAULT linux autoinstall
LABEL linux autoinstall
    KERNEL ubuntu/focal/vmlinuz
    INITRD ubuntu/focal/initrd
    APPEND ip=dhcp url=http://< foreman server >/pub/installation_media/ISO/ubuntu-20.04.3-live-server-amd64.iso autoinstall ds=nocloud-net;s=http://< foreman server >/pub/preseed/autoinstall/< hostname.short >/ root=/dev/ram0 ramdisk_size=1500000 fsck.mode=skip

For each host you provision the part autoinstall ds=nocloud-net;s=http://< foreman server >/pub/preseed/autoinstall/< hostname.short >/ needs to have 2 files, meta-data and the user-data, where the last one contains the cloud-config how the host should be build.
Based on the docu I referred to.

As multiple host provisions might be launched simultaneously I have decided to create them in a separate path for each host. I have not tested it by using other paths/locations for these files.

As I’m not a developer I was trying to find out a solution to make it possible to provision a host with the new method Ubuntu implemented.
So I was hoping that which what I created as a solution, Foreman developers would be able to implement it in a correct way in the product.

But if necessary I will have a look at the build-in userdata endpoint if it is possible to migrate my solution in it.

I started from a clean template in order to create a finish script that would work as expected.
It might be possible to merge it in the existing one, but at the moment I do not have the time as well to have a look at it.
If no one else has looked at it in the meantime I will pick it up as soon as I have time for.

15

Yeah sorry I missed it above, the thing is we actually do have both endpoints in Foreman now:

config/routes.rb
469:  get 'userdata/meta-data', controller: 'userdata', action: 'metadata', format: 'text'
470:  get 'userdata/user-data', controller: 'userdata', action: 'userdata', format: 'text'

Userdata can be fully customized, however, metadata is hardcoded in the codebase:

  def metadata
    data = {
      :'instance-id' => "i-#{Digest::SHA1.hexdigest(@host.id.to_s)[0..17]}",
      :hostname => @host.name,
      :mac => @host.mac,
      :'local-ipv4' => @host.ip,
      :'local-hostname' => @host.name,
    }
    render plain: data.map { |key, value| "#{key}: #{value}" }.join("\n")
  end

Is there anything we can change in this design what would enable you to do what you want to do?

Understood, the problem is that we do not want to ship several fine-tuned templates because every single environment is different. What we are doing is shipping a single but extendable template for each individual type which serves as a good starting point for others. Not a problem, we can just use our standard finish template for now, unless there is something really missing.

Apologies I was on a short vacation, I will check PRs soon, I hope we can change the big one (my PR) into develop so you guys can start improving it.

16

Hey! I’m currently working on integrating the new autoinstall method into Foreman. Thanks for providing the great tutorial @langesmalle!
I’m already deploying 20.04.3 hosts with custom userdata templates as suggested by @lzap. Now, I want to automate the whole integration & I see the following steps:

  • Fix installation media such that it provides the correct kernel files (see here)
  • In order to configure the network with cloud-init, a network identifier must be set. Previous templates didn’t need an explicit identifier and it would be generated during installation. So, the name wasn’t set in the provisioning template, but now it seems like it must be generated here
  • AFAIK cloud-init and the corresponding userdata/user-data API endpoint is usually used for image-based deplyoment. When using image-based deployment instead of the regular host creation, we must use a different template. Therefore, we must differentiate between these two use cases.

What do you think about it @lzap? Idk whether my approach for the PR is feasible or whether it would be better to create the operating system type Ubuntu, since the different kernel file names to Debian may be seen as a major difference which justifies a new operating system class.

2 Likes
17

Hey, thanks for the patch.

How this is supposed to work? Foreman currently cannot download PXE from Ubuntu mirrors, because they simply aren’t there. Have you seen: RFC: Rearchitecting how PXE files are deployed ?

Can you describe the full end to end workflow?

18

You’re right, it still needs some location with the extracted iso files. With the patch, the user can define an installation media path where the extracted image files are placed (for example, the pub folder of the Foreman itself). Then, the Foreman is going to look for the re-named/non-zipped files (vmlinuz instead of linux and initrd instead of initrd.gz) if the Ubuntu version is higher than 20.04.2.

19

Isn’t renaming the solution? I mean, currently it’s broken and this patch does not solve this, I am not sure if this is worth doing.

20

Do you mean renaming the boot files manually so they’re detected by Foreman again or renaming the expected file names in Foreman as it’s proposed by the patch?
I tested the former by renaming vmlinuz to linux and compressing the init file to initrd.gz, so they look the same like before. This enables Foremen to detect the files again, but the boot process of the VM won’t work since it’s explicitly looking for an uncompressed initrd.
So, I think we should add this exception for newer Ubuntu versions from 20.04.3 in Foreman itself as it’s done in the patch.

What do you think about the second point, I described earlier?

In order to configure the network with cloud-init, a network identifier must be set. Previous templates didn’t need an explicit identifier and it would be generated during installation. So, the name wasn’t set in the provisioning template, but now it seems like it must be generated here

For me, the cloud-init templates crash if I don’t set the network identifier name manually. Can we prevent this by generating a default name that is used if none is given?

1 Like
21

Ok thanks for explaining, I see. Sure, let’s merge this then and let’s wait until the next version when they confirmed to publish PXE files again to solve this. If not, then we need to talk about some bigger redesign.

2 Likes
22

I’d experience some issues with the netplan setup of the preseed_netplan_generic_interface snippet (see here). If the network identifier is not set to a specifc name (in my case ens160), the installation would fail since the DNS setup won’t work properly.

@langesmalle can you verify the network identifier issue which I described in this ticket?

Also, I wanted to know if there is a specific reason why the new templates are still called “preseed”? I thought “preseed” relates only to the debian-installer.

Besides that, I’m about to adapt the templates slightly due to some issue when using root as a default user. You can have a look here if you want. :slight_smile:

1 Like
23

@bastian-src ,

Sorry for the late reply, but I’m currently very busy with other urgent things for the company.
As already said, this was also very difficult for me to build something temporary that works for me as I’m not a developer and had not that much experience regarding defaults, coding, … in foreman.
So, after Canonical decided to use another way of autoinstall as of version 20.04.2 (I guess), I was somehow happy to work out a solution in order to install Ubuntu 20.04.3 with foreman on a systems as we decided to move from CentOS to Ubuntu (OS requirements for some products were Alma and Rocky are not supported (yet)).
That is why I provided my temporary solution to the community in hope that the Foreman’s die-hard developers-team could work out a final version.

Regarding root user for Ubuntu:
By default Ubuntu is always installed with root disabled for login during installation, that’s why we are asked for a password and user during manual installation.

This way is configured in my template “preseed_cloud-init-autoinstall-user-data” I created, more specific:
identity: {hostname: <%= @host.shortname %>, password: <%= userpassword_to_create %>,
realname: <%= realname_to_create %>, username: <%= username_to_create %>}
These fields (username, realname and password) are configured on the host or hostgroup.

Beside this, I have seen following mistake in that file:
<%= indent 4 do snippet ‘preseed_cloud-init-autoinstall-user-data custom sources_list’ end -%>
should be replaced by:
<%= indent 4 do snippet ‘preseed_cloud-init-autoinstall-user-data custom sources_list’ end %>

Remove “-” at the end.

Regarding the netplan setup:
In our case, as we also use bond-interfaces with vlans for which we define static addresses, I always configure/modify the network interfaces manually before starting the provisioning of the system.
However, we also have a separate pxe-vlan linked to a specific interface.
For this interface the ip is provided by foreman by dhcp and the network identifier is always filled in with the name discovered during the discovery-process.
So for this interface I never encountered any problems during provisioning.

Hope this will help you further and otherwise do not hesitate to ask further details.

3 Likes
24

I would like to use these endpoints in my templates, but I can’t seem to find how to do it in the docs. I know we have the new autoinstall templates in 3.2, but I can’t seem to find how to enable the endpoints that the cloud-init templates should be available at. Is there a configuration file for these?

25

Hi, I’ve tried your tutorial but since I’m using bootdisk as boot method, it does not work as expected
Any ideas on where I can dig to look into this issue ?
The error message I get:

ERF42-8554 [Foreman::Exception]: Unable to download boot file http://foreman-server.local/pub/installation_media/ubuntu/amd64/20-04//dists/focal/main/installer-amd64/current/legacy-images/netboot/ubuntu-installer/amd64/linux, HTTP return code 404"

It makes sense since in the ISO image this path does not exists, but I don’t know why the bootdisk method search for this path exactly

26

That path is for the now deprecated “old” Ubuntu installer, based on d-i. That is the location that the old installer is in the usual Ubuntu mirrors. For my environment, not using a bootdisk, using the latest templates in 3.2, the “Preseed default PXELinux Autoinstall” PXELinux template has the following line:

APPEND ip=dhcp url=http://<%= @preseed_server %><%= @preseed_path %> autoinstall ds=nocloud-net;s=http://<%= @preseed_server %>/pub/preseed/autoinstall/<%= @host.shortname %>/ root=/dev/ram0 ramdisk_size=1500000 fsck.mode=skip

The url value is the location of the installer iso (configured in installation media as just the ISO, no folders).

This is how I understand it, but I don’t even have a working configuration so it is probably incorrect. I hope it is a good starting point though.

27

To get around the network identifier being unknown for newly created hosts I changed the preseed_netplan_generic_interface template to match on mac address like this

3 Likes
28

@Zedeau ,

Sorry for the late reply, but was to bussy with other things.

I worked out a this solution in order to pxe-install ubunto 20.04, but this error does not affect the installation.
As I’m not a developer, I have no idea from where this comes, but I will make some time to find out why it refers to this location.
Initially I extracted the ISO in this path “installation_media/ubuntu”, but it should not be necessary as the installation should be done using the ISO-file.

As @upeter metioned, you need to apply the method in the “Preseed default PXELinux Autoinstall” PXELinux template.

Today I was able to autoinstall ubuntu 22.04 as well with my workaround, and even without having an extract of the ISO for this version the installation, I get the same message in the apache log, but the installation went also well.
Only problem with ubuntu 22.04 is that the subscription-manager packages from atix ends up in an error (dependency for python3 version which is higher in 22.04).
I already informed @maximilian about this issue, so it is just a matter of waiting till they provide a new repo for ubuntu 22.04.

1 Like