Provisioning using (image based) VMWare Templates

Hi,

I have a "vmware template" available I've been trying to follow this guide:

http://theforeman.org/manuals/1.9/index.html#5.2.9VMwareNotes

Originally I tried taking the ssh approach which involves making using of a
finish template. I couldn't get this to work because foreman (v1.9) appears
to just does a "clone" of the vmware template rather than "Deploy virtual
machine from this Template…". Therefore when the new VM is created on
vsphere (v5.5), it doesn't have any networking, and therefore the ssh part
times out.

I then tried, "Image provisioning without SSH". However after a few trial
and errors I've discovered that I can't do this without a YAML user data
template for the VMware customisation. At the moment the closest thing we
have appears to only work for cloudinit:

https://github.com/theforeman/community-templates/tree/develop/cloudinit

I've also had a look at this:

http://www.herbert.org.nz/?p=245

But this appears to deal with provisioning windows machine, whereas I'm
interested in provisioning a RHEL machine.

If anyone can create a community version then I'm happy to test it out.

Has anyone else had any success in provisioning a vm from a vmware template?

Thanks,
Sher Chowdhury

> Hi,
>
> I have a "vmware template" available I've been trying to follow this guide:
>
> Foreman :: Manual
>
> Originally I tried taking the ssh approach which involves making using of a
> finish template. I couldn't get this to work because foreman (v1.9) appears
> to just does a "clone" of the vmware template rather than "Deploy virtual
> machine from this Template…". Therefore when the new VM is created on
> vsphere (v5.5), it doesn't have any networking, and therefore the ssh part
> times out.

> I then tried, "Image provisioning without SSH". However after a few trial
> and errors I've discovered that I can't do this without a YAML user data
> template for the VMware customisation. At the moment the closest thing we
> have appears to only work for cloudinit:
>
> https://github.com/theforeman/community-templates/tree/develop/cloudinit
>
> I've also had a look at this:
>
> http://www.herbert.org.nz/?p=245
>
> But this appears to deal with provisioning windows machine, whereas I'm
> interested in provisioning a RHEL machine.
It works, I tested it with RHEL 6 and RHEL 7 machines.

Did you follow vmware documentations? You need mainly 3 things :

  • open-vm-tools (RHEL7) or VMtools (RHEL6) installed
  • perl installed
  • for RHEL7, open-vm-tools-deploypkg from packages.vmware.com repository

The network interface number must match the number installed on the template.
Here is my sample yaml file :
-----------8<-----------8<-----------8<------------

YAML file

<%#
kind: user_data
name: UserData default
oses:

  • CentOS 4
  • CentOS 5
  • CentOS 6
  • CentOS 7
  • SLES 9
  • SLES 10
  • SLES 11
  • SLES 12
  • Ubuntu 10.04
  • Ubuntu 11.04
  • Ubuntu 11.10
  • Ubuntu 12.04
  • Ubuntu 12.10
  • Ubuntu 13.04
  • Ubuntu 13.10
  • Ubuntu 14.04
  • Ubuntu 14.10
  • Ubuntu 15.04
    -%>

https://pubs.vmware.com/vsphere-55/index.jsp#com.vmware.wssdk.apiref.doc/vim.vm.customization.Specification.html

encryptionKey expects an array

globalIPSettings expects a hash, REQUIRED

identity expects an hash, REQUIRED

nicSettingMap expects an array

options expects an hash

* LinuxPrep: <~Hash> - Optional, contains machine-wide settings (note the uppercase P)

* domain: <~String> - REQUIRED, The fully qualified domain name.

* hostName: <~String> - REQUIRED, the network host name

* hwClockUTC: <~Boolean> - Optional, Specifies whether the hardware clock is in UTC or local time

* timeZone: <~String> - Optional, Case sensistive timezone, valid values can be found at https://pubs.vmware.com/vsphere-51/topic/com.vmware.wssdk.apiref.doc/timezone.html

identity:
LinuxPrep:
domain: <%= @host.domain %>
hostName: <%= @host.shortname%>
hwClockUTC: true
timeZone: <%= @host.params['time-zone'] || 'UTC' %>

* globalIPSettings <~Hash> - REQUIRED

* dnsServerList <~Array> - Optional, list of dns servers - Example: ["10.0.0.2", "10.0.0.3"]

* dnsSuffixList <~Array> - Optional, List of name resolution suffixes - Example: ["dev.example.com", "example.com"]

globalIPSettings:
dnsSuffixList: [<%= @host.domain %>]

* nicSettingMap: <~Array> - Optional, IP settings that are specific to a particular virtual network adapter

* Each item in array:

* adapter: <~Hash> - REQUIRED, IP settings for the associated virtual network adapter

* dnsDomain: <~String> - Optional, DNS domain suffix for adapter

* dnsServerList: <~Array> - Optional, list of dns server ip addresses - Example: ["10.0.0.2", "10.0.0.3"]

* gateway: <~Array> - Optional, list of gateways - Example: ["10.0.0.2", "10.0.0.3"]

* ip: <~String> - Optional, but required if static IP

* ipV6Spec: <~Hash> - Optional, IPv^ settings

* ipAddress: <~String> - Optional, but required if setting static IP

* gateway: <~Array> - Optional, list of ipv6 gateways

* netBIOS: <~String> - Optional, NetBIOS settings, if supplied must be one of: disableNetBIOS','enableNetBIOS','enableNetBIOSViaDhcp'

* primaryWINS: <~String> - Optional, IP address of primary WINS server

* secondaryWINS: <~String> - Optional, IP address of secondary WINS server

* subnetMask: <~String> - Optional, subnet mask for adapter

* macAddress: <~String> - Optional, MAC address of adapter being customized. This cannot be set by the client

nicSettingMap:
<% @host.interfaces.each do |interface| %>

  • adapter:
    dnsDomain: <%= interface.domain %>
    dnsServerList: [<%= interface.subnet.dns_primary %>, <%= interface.subnet.dns_secondary %>]
    gateway: [<%= interface.subnet.gateway %>]
    ip: <%= interface.ip %>
    subnetMask: <%= interface.subnet.mask %>
    <% end %>

* options: <~Hash> Optional operations, currently only win options have any value

* changeSID: <~Boolean> - REQUIRED, The customization process should modify the machine's security identifier

* deleteAccounts: <~Boolean> - REQUIRED, If deleteAccounts is true, then all user accounts are removed from the system

* reboot: <~String> - Optional, (defaults to reboot), Action to be taken after running sysprep, must be one of: 'noreboot', 'reboot', 'shutdown'

#options:

changeSID: true

deleteAccounts: false

-----------8<-----------8<-----------8<------------

> If anyone can create a community version then I'm happy to test it out.
>
> Has anyone else had any success in provisioning a vm from a vmware template?

Yes I did succeed. What I didnt workout yet, is puppet registration. I don't
understand why I can't combine userscript with ssh finish script. maybe a
ticket to open in redmine :slight_smile:

Regards,

Claer

··· On Mon, Oct 26 2015 at 39:08, Sher Chowdhury wrote:

Hi,
Spent quite a while getting non-SSH provisioning to work, these were the
issues I ran into. Now that it works for me, it works beautifully (with
RHEL 7.1). The way it all works is a bit convoluted - the host parameters
in foreman host creation populate a user-data template, which the fog gem
converts to a customization spec that is passed to vCenter when the
template is cloned.

  1. Check that you're running a supported combination of guest OS and
    vCenter (only works with vCenter managed hypervisor
    hosts): http://partnerweb.vmware.com/programs/guestOS/guest-os-customization-matrix.pdf

  2. Check that your virtual machine template is the latest VM hardware
    version, I think version 9 is the minimum for customization specs.

  3. open-vm-tools didn't work for me. Despite Red Hat recommending
    open-vm-tools for RHEL 7, which is what I was using. When I installed
    vmware tools on the template, downloaded from the vmware website, it worked
    (but see 4))

  4. If you create a vm template elsewhere (I use Packer too), and then
    deploy it to vCenter, vCenter does not detect that the template has VMware
    tools installed. I had to convert the template to a VM (as far as I can
    tell, the difference between a template and a VM is just the file
    extension), turn it on with networking disabled so vCenter detects VMware
    tools, turn it off, enable networking on boot, and convert back to a
    template.

  5. I ran into some issues with the userdata_cloudinit.erb template and
    had to modify it (can dig up my changes to the template tomorrow). But the
    best way is to troubleshoot errors in fog - if the conversion from
    cloud-init to customization spec fails, the stack trace will show you
    errors in one of the two files below (remember to check the version of fog
    installed)

https://github.com/fog/fog/blob/master/lib/fog/vsphere/requests/compute/vm_clone.rb
https://github.com/fog/fog/blob/master/lib/fog/vsphere/requests/compute/cloudinit_to_customspec.rb

The whole process isn't very elegant, but I don't know of an alternative
when using VM templates. Installing the OS from scratch each time you
provision a VM is painfully slow. Once I got things working it was useful -
I can change disk size and thin / thick provisioning, network interfaces
and vlans, CPU, memory etc.And deploying VMs is extremely fast.

··· On Tuesday, October 27, 2015 at 2:39:23 AM UTC+11, Sher Chowdhury wrote: > > Hi, > > > > I have a "vmware template" available I've been trying to follow this guide: > > http://theforeman.org/manuals/1.9/index.html#5.2.9VMwareNotes > > Originally I tried taking the ssh approach which involves making using of > a finish template. I couldn't get this to work because foreman (v1.9) > appears to just does a "clone" of the vmware template rather than "Deploy > virtual machine from this Template...". Therefore when the new VM is > created on vsphere (v5.5), it doesn't have any networking, and therefore > the ssh part times out. > > I then tried, "Image provisioning without SSH". However after a few trial > and errors I've discovered that I can't do this without a YAML user data > template for the VMware customisation. At the moment the closest thing we > have appears to only work for cloudinit: > > https://github.com/theforeman/community-templates/tree/develop/cloudinit > > I've also had a look at this: > > http://www.herbert.org.nz/?p=245 > > But this appears to deal with provisioning windows machine, whereas I'm > interested in provisioning a RHEL machine. > > If anyone can create a community version then I'm happy to test it out. > > Has anyone else had any success in provisioning a vm from a vmware > template? > > > Thanks, > Sher Chowdhury > > >

Hello all! Really Glad, to find this topic :smiley:

I have katello-2.2.1-0.el7.noarch foreman-release-1.8.4-1.el7.noarch
and vSphere v5.1

Cloud Images deployment with SSH works really well for CentOS 6.7 and 7.1.

Now I'm interested in Image deploy without SSH (Cloud-Init based), Foreman
can't reach some of the networks were I need to deploy VMs.
I've created user_data template but when creating the new host it fails.
I can see the following messages:

New in Progress
> rollbacked - Render user data template for cloudinit.ntw.impresa.pt
> failed - Set up compute instance cloudinit.ntw.impresa.pt
> canceled - Query instance details for cloudinit.ntw.impresa.pt
> canceled - Power up compute instance cloudinit.ntw.impresa.pt

Then it rolls back and fails

Unable to save
> Failed to create a compute VMware-PASR1 (VMware) instance
> cloudinit.ntw.impresa.pt: domain is required

I don't have a clue which logs I should check.
Can anyone help me?

··· On Monday, October 26, 2015 at 3:39:23 PM UTC, Sher Chowdhury wrote: > > Hi, > > > > I have a "vmware template" available I've been trying to follow this guide: > > http://theforeman.org/manuals/1.9/index.html#5.2.9VMwareNotes > > Originally I tried taking the ssh approach which involves making using of > a finish template. I couldn't get this to work because foreman (v1.9) > appears to just does a "clone" of the vmware template rather than "Deploy > virtual machine from this Template...". Therefore when the new VM is > created on vsphere (v5.5), it doesn't have any networking, and therefore > the ssh part times out. > > I then tried, "Image provisioning without SSH". However after a few trial > and errors I've discovered that I can't do this without a YAML user data > template for the VMware customisation. At the moment the closest thing we > have appears to only work for cloudinit: > > https://github.com/theforeman/community-templates/tree/develop/cloudinit > > I've also had a look at this: > > http://www.herbert.org.nz/?p=245 > > But this appears to deal with provisioning windows machine, whereas I'm > interested in provisioning a RHEL machine. > > If anyone can create a community version then I'm happy to test it out. > > Has anyone else had any success in provisioning a vm from a vmware > template? > > > Thanks, > Sher Chowdhury > > >

I am still struggling with this.

I am trying to build a proof-of-concept setup as part of a larger
deployment. We have a lot of objectives and we think Freman can accomplish,
but there is a lot we haven't been able to make work. If I build a machine
from scratch, which takes a huge amount of time, I can get things working.
However if I try using the pre-gen VMware images I can various errors.

Does anyone know of a company that specializes in Foreman integration and
deployment?

··· On Monday, October 26, 2015 at 11:39:23 PM UTC+8, Sher Chowdhury wrote: > > Hi, > > > > I have a "vmware template" available I've been trying to follow this guide: > > http://theforeman.org/manuals/1.9/index.html#5.2.9VMwareNotes > > Originally I tried taking the ssh approach which involves making using of > a finish template. I couldn't get this to work because foreman (v1.9) > appears to just does a "clone" of the vmware template rather than "Deploy > virtual machine from this Template...". Therefore when the new VM is > created on vsphere (v5.5), it doesn't have any networking, and therefore > the ssh part times out. > > I then tried, "Image provisioning without SSH". However after a few trial > and errors I've discovered that I can't do this without a YAML user data > template for the VMware customisation. At the moment the closest thing we > have appears to only work for cloudinit: > > https://github.com/theforeman/community-templates/tree/develop/cloudinit > > I've also had a look at this: > > http://www.herbert.org.nz/?p=245 > > But this appears to deal with provisioning windows machine, whereas I'm > interested in provisioning a RHEL machine. > > If anyone can create a community version then I'm happy to test it out. > > Has anyone else had any success in provisioning a vm from a vmware > template? > > > Thanks, > Sher Chowdhury > > >

Hey,
There is another workaround when using a dhcp server
There are two options:

  1. Use dhcp for all your environment (do not introduce any static
    configuration on the guest)
  2. Use dhcp only for the provisioning part (the ssh finish script will
    configure the static network configuration)

I have a deployment with multiple VLANs so even here there are a few ways
to tackle this:
For the dhcpd configuration:

  1. Run multiple dhcp + smart proxy instances (per VLAN)
  2. Configure a dhcp relay on each VLAN (on the switch) and relay to one
    single dhcp + smart proxy instance
  3. Run one dhcp + smart proxy and configure the NIC as a trunk (on all
    those VLANs), you will have to create a VLAN interface per subnet (you can
    automate all of this with puppet + theforeman)
    If you are running this dhcp server in a VMware (or any other Cloud
    environment) you can create a distributed port group that is actually a
    trunk VLAN and then attach it to the VM running the dhcp.

Because some VMs in this deployment i am working on requires a very basic
image (really really clean image) i have to use this approach, I make sure
before i pack the image (I also automate the creation of those images
through foreman) that the first interface (primary, managed, provisioning
interface) on this VM (usually eth0) is configured to boot from dhcp and i
make sure to remove the mac address part in the guest configuration.
For RHEL based OS you can do something like this in the post installation
script when creating the image, before packing it:
$ rm -f /etc/udev/rules.d/70* || true
$ sed -i'' '/^(HWADDR)/d' /etc/sysconfig/network-scripts/ifcfg-eth* ||
true
And make sure the configuration in the ifcfg-eth<N> is set to boot from
DHCP. (if your kickstart file networking is dhcp then it will be
automatically configured for you)

Create appropriate Subnets (per VLAN as example) in foreman and make them
static or dhcp (your choice, it will work both way starting from foreman
1.11.3 Bug #14905: DHCP reservation for PXE boot not added when using static boot mode - Foreman)

When you create a new host and attach the interface (make sure it's managed

  • provisioning interface) to a Subnet, doesn't matter if the subnet is
    configured to dhcp or static, Foreman will create an entry on the DHCP
    server.
    When the new VM (from the Image/Template created earlier) will boot, it
    will receive an IP address from the DHCP server, Foreman will be aware of
    that address and run the SSH finish script on the new VM.
    The default SSH Finish script is aware of different networking
    configuration (fixed also in 1.11.3 but you can backport it pretty easily)
    and as part of this script it will configure the ifcfg-eth<N> accordingly
    (removing it from DHCP to STATIC as example, it can also set the HW address
    back).

It's seems complicated, but it's pretty simple :slight_smile:

··· On Monday, October 26, 2015 at 5:39:23 PM UTC+2, Sher Chowdhury wrote: > > Hi, > > > > I have a "vmware template" available I've been trying to follow this guide: > > http://theforeman.org/manuals/1.9/index.html#5.2.9VMwareNotes > > Originally I tried taking the ssh approach which involves making using of > a finish template. I couldn't get this to work because foreman (v1.9) > appears to just does a "clone" of the vmware template rather than "Deploy > virtual machine from this Template...". Therefore when the new VM is > created on vsphere (v5.5), it doesn't have any networking, and therefore > the ssh part times out. > > I then tried, "Image provisioning without SSH". However after a few trial > and errors I've discovered that I can't do this without a YAML user data > template for the VMware customisation. At the moment the closest thing we > have appears to only work for cloudinit: > > https://github.com/theforeman/community-templates/tree/develop/cloudinit > > I've also had a look at this: > > http://www.herbert.org.nz/?p=245 > > But this appears to deal with provisioning windows machine, whereas I'm > interested in provisioning a RHEL machine. > > If anyone can create a community version then I'm happy to test it out. > > Has anyone else had any success in provisioning a vm from a vmware > template? > > > Thanks, > Sher Chowdhury > > >

Hi Claer,

I tried your user_data template and it worked brilliantly!!! :o)

I did make a few tweaks, first off I added the following to my list of Oses:

  • OracleLinux 6
  • OracleLinux 7

Since we only work Oracle Linux 6 and 7.

Next the following line didn't work for me:

dnsServerList: [<%= interface.subnet.dns_primary %>, <%=
interface.subnet.dns_secondary %>]

The ENC yaml dump did include this, but they got ignored because of:

https://pubs.vmware.com/vsphere-55/index.jsp#com.vmware.wssdk.apiref.doc/vim.vm.customization.AdapterMapping.html

To get round this, I inserted the following lines into your template:

globalIPSettings:
dnsSuffixList: [<%= @host.domain %>]

<% @host.interfaces.each do |interface| %>
# new line
dnsServerList: [<%= interface.subnet.dns_primary %>, <%=
interface.subnet.dns_secondary %>] # new line
<% end %>
# new line

We always build single network interface machine so the above worked for
us. But it might cause problems if you have more than one network
interface. But there might be a better of doing this.

Many thanks for responding to my question.

With respect to the "finish" script, I have found a workaround for this.
Let me do a quick write-up and I'll post it tomorrow.

Thanks again Claer, I'm now in your debt.

Kind regards,
Sher

By the way something I forgot to clarify in my initial post, for the
benefit of other readers. In Foreman there are 2 ways to do image
provisioning with vmware templates and foreman, image provision with ssh,
and image provision without ssh:

http://theforeman.org/manuals/1.9/index.html#5.2.9VMwareNotes

From what I can see, the ssh approach is the better option. However
unfortunately in my company we don't use dhcp, which means that when I
tried the ssh approach, foreman did successfully instantiate a new vm out
of the vmware template, but the machine can't configure it's network
settings because of the unavailability of dhcp, so it ends up just hanging
and not doing anything else. That's why I took the 'without ssh' approach,
which involves using a user_data provisioning template.

> > Yes I did succeed. What I didnt workout yet, is puppet registration. I
> don't
> > understand why I can't combine userscript with ssh finish script. maybe
> a
> > ticket to open in redmine :slight_smile:
>

Thanks Claer, I will raise a ticket for this as suggested.

However as a workaround, here's what I did, first I had some prerequisites:

  1. I enabled autosigning of certificates on my foreman server.
  2. I have pre-installed puppet agent into my template, with the puppet.conf
    already configured to point to my foreman server and puppetmaster. But I
    have disabled the puppet agent service.

I then created an "at" job. "at" is like cron but runs a one-off job at
some date into the future, e.g. in my case i set it it to "now+5min".
Before the 5 minutes run out I quickly converted my vm into a template.
This resulted in the "at" job frozen into the template, and will actually
end up running the next time create I a vm from the template and start up
that vm. In my case I will be using my vm the following day, so well past
the 5min deadline, in this situation the 'at' command simply defaults to
running the next time the machine starts up.

In my "at" job I scheduled the task to re-enable the puppet service. When
foreman generates a new vm from this template, the vm starts up, but only
for a couple of seconds for the vmware customisation process to take place
(as well as trigger the 'at' job). The machine then reboots and this time
the puppet service started up the puppet run happens.

As a side note, I used to packer, https://packer.io/, to create my base
image vm (as well as schedule in the 'at' job). Then used a powercli script
to convert this vm into a vmware template. We then automated the whole
process into a jenkins pipeline consisting of several jobs. Some of the
latter jobs in the pipeline creates a test vm from the template, wait until
the puppet run has ended, and then and tests it with serverspec to validate
our template as well as the puppet code.

Kind regards,
Sher

Hi Abir

Interesting stuff. I forgot to mention that I only tried this with linux
6.7 so far. Haven't tried it with Linux 7 yet.

Also with respects to packer, from what I can see, packer doesn't create
vmware templates, it just creates a normal vm. By the way, one of the last
steps I do in my packer build is delete the file
/etc/sysconfig/network-scripts/ifcfg-{interface-name}. Maybe a future
release of packer might add in this template creation capability.

Since I have converted my packer stuff into a jenkins job, I then created a
new downstream jenkins job to convert the newly generated vm into a
template using a powershell/powercli.

I'll report back my findings on how I get on with Linux 7.1, when I
eventually get round to doing it.

I've noticed some important corrections for this were made in Foreman 1.9,
I'm updating.

··· On Wednesday, November 18, 2015 at 12:09:26 PM UTC, Helio Clemente wrote: > > Hello all! Really Glad, to find this topic :D > > I have *katello*-2.2.1-0.el7.noarch *foreman*-release-1.8.4-1.el7.noarch > and vSphere v5.1 > > Cloud Images deployment with SSH works really well for CentOS 6.7 and 7.1. > > Now I'm interested in Image deploy without SSH (Cloud-Init based), Foreman > can't reach some of the networks were I need to deploy VMs. > I've created user_data template but when creating the new host it fails. > I can see the following messages: > > New in Progress >> rollbacked - Render user data template for cloudinit.ntw.impresa.pt >> failed - Set up compute instance cloudinit.ntw.impresa.pt >> canceled - Query instance details for cloudinit.ntw.impresa.pt >> canceled - Power up compute instance cloudinit.ntw.impresa.pt > > Then it rolls back and fails > > Unable to save >> Failed to create a compute VMware-PASR1 (VMware) instance >> cloudinit.ntw.impresa.pt: domain is required > > > I don't have a clue which logs I should check. > Can anyone help me? > > > On Monday, October 26, 2015 at 3:39:23 PM UTC, Sher Chowdhury wrote: >> >> Hi, >> >> >> >> I have a "vmware template" available I've been trying to follow this >> guide: >> >> http://theforeman.org/manuals/1.9/index.html#5.2.9VMwareNotes >> >> Originally I tried taking the ssh approach which involves making using of >> a finish template. I couldn't get this to work because foreman (v1.9) >> appears to just does a "clone" of the vmware template rather than "Deploy >> virtual machine from this Template...". Therefore when the new VM is >> created on vsphere (v5.5), it doesn't have any networking, and therefore >> the ssh part times out. >> >> I then tried, "Image provisioning without SSH". However after a few trial >> and errors I've discovered that I can't do this without a YAML user data >> template for the VMware customisation. At the moment the closest thing we >> have appears to only work for cloudinit: >> >> https://github.com/theforeman/community-templates/tree/develop/cloudinit >> >> I've also had a look at this: >> >> http://www.herbert.org.nz/?p=245 >> >> But this appears to deal with provisioning windows machine, whereas I'm >> interested in provisioning a RHEL machine. >> >> If anyone can create a community version then I'm happy to test it out. >> >> Has anyone else had any success in provisioning a vm from a vmware >> template? >> >> >> Thanks, >> Sher Chowdhury >> >> >>

Hi Claer & Sher Chowdhury & All,

The template worked good for me but couple of issues like subnetMask: <%= "255.255.255.0" %>
I could not get the subnet and mask from the host or ip. So I hard coded.

When i created a host it created the VM from template and powered on and i can able to login also.
but one issue is from foreman still i can see status of new VM is Pending installation only. I did not understand why its still showing "Pending installation"

I dont have any puppet classes . is that the reason? Did i miss anything in the steps.

Please give me some clues or references pelase

Also seems my ip got chnaged I have 136.89.113.15 but after the VM creation it got changed to 136.89.113.1.
Not sure why this happened. any ideas please…