VMware bootdisk provisioning will fail with Failed to upload ISO image for instance

Support
, ,
#1

Problem:

Hey guys,

for some time past our foreman instance is unable to upload the bootdisk iso to our VMware Datastores. We assume that this is not working since we updated our vCenter Server to Version 6.7.0 Build 15129973

Is there any known bug?

The upload process will be aborted with the following error message:

Failed to save: Failed to upload ISO image for instance HOSTNAME: FileAlreadyExists: Cannot complete the operation because the file or fold
er [DATASTORENAME] foreman_isos already exists

Expected outcome:
Bootdisk iso will be uploaded to the datastore and attached to the virtual machine.

Foreman and Proxy versions:
foreman-1.20.1.36-1.el7sat.noarch
foreman-proxy-1.20.0-1.el7sat.noarch

Foreman and Proxy plugin versions:
tfm-rubygem-foreman_bootdisk-14.0.0.2-1.el7sat.noarch

Distribution and version:
Red Hat Enterprise Linux Server release 7.7 (Maipo)

Other relevant data:

2019-12-20T14:19:11 [W|app|1f103512] Failed to upload ISO image for instance HOSTNAME: FileAlreadyExists: Cannot complete the operation because the file or folder [DATASTORENAME] foreman_isos already exists

#2

Yeah itā€™s broken a bit. @ezr-ondrej if you have a working setup I can assist you setting up bootdisk plugin, itā€™s as easy as enabling it and selecting bootdisk provisioning method - no other things are really required. If you could spend some time debugging this Iā€™d appreciate I hear about this every week.

@TimoGoebel are you no longer interested in this feature? Itā€™s pain in ass, I suggest either fixing it or removing it completely. We have many reports downstream about it, also people complain that bootdisk does not remove bootdisks which if I understand the workflow correctly renders this pretty much unusable (I know ejecting it is some kind of a workaround - Iā€™ve never got it working).

#3

We have never used it at work so I donā€™t have a particular interest in the feature. I know that there are users that rely on the feature.

@ezr-ondrej: Is this something you could take a look at? Itā€™s actually a pretty trivial implementation.

i.e. deleting the iso from the datastore? or as in after the installation, the disk is not properly ejected?

#4

I think he means that the ISO remains undeleted on VMware datastore after successful completion of provisioning. (https://bugzilla.redhat.com/show_bug.cgi?id=1694082)

Regarding the ā€œforeman_isos already existsā€ error, this seems to be the troublesome part:

/opt/theforeman/tfm/root/usr/share/gems/gems/fog-vsphere-2.4.0/lib/fog/vsphere/requests/compute/upload_iso.rb (lines 24-28)

      unless datastore.exists? options['upload_directory'] + '/'
        connection.serviceContent.fileManager.MakeDirectory name: "[#{options['datastore']}] #{options['upload_directory']}",
                                                            datacenter: datacenter,
                                                            createParentDirectories: false
      end
1 Like
#5

Unfortunately most of the time we get a ā€œlocked cdromā€ error in vmware when re-provisioning an existing vm via bootdisk, the existing bootdisk does not get ejected properly.

VMWare vCenter Version: 6.7.0
VMWare vCenter Build: 15129973

#6

Iā€™ll take a look at whole process during the next week, I think we are just missing update of the booting (ejecting the ISO) and deleting the bootdisk from datastore if I understand it correctly, so it should be so hard to implement.

#7

Honestly, I donā€™t want to support it as it has never worked on my trivial setup and I know nothing about VMWare. Unless @ezr-ondrej wants to dive in and fix this, I suggest to do something with it (disable, remove, add warning or have someone fix this).

What do people suggest in this case? Can someone with this setup add the missing code? Does not sound to be too much work, just put it down and restart and test it out.

Iā€™d try to remove ā€œ+ ā€˜/ā€™ā€ if it helps. If it does not, the whole section could be removed and we would ask the user to explicitly create the directory in advance. Worst case: encapsulate in catch block and only put warning into logs - this one I can implement without testing.

Oh I should start reading from the bottom, great! Let me know if you need assistance.

#8

Thank you very much! The upload of the iso works as expected when I remove the + '/'

Unfortunately I have no knowledge in ruby programming, is this a safe way to fix the upload issue?

Best regards!

#9

If I understand this code block correctly, the + '/' appends a slash sign at the end of the upload_directory variable, so the upload directory (named ā€œforeman_isosā€ in our case) will be correctly detected, when the slash at the end is removed!?

#10

I can write Ruby code, however that does not guarantee me I understand this :slight_smile: This depends on the VMWare (Ruby) API, it is likely a bug - paths should be probably normalized before doing any actions with them like comparing.

Anyway, if you tested this please file a PR with the (tested) change and I am willing to merge this. I am unable to find a relevant issue for this problem tho, probably create one and assiciate with the commit: Search - Boot disk - Foreman

#11

@brotaxt: Do you mind testing if removing the + '/' helps? I found similar code that does not do this magic.

  unless datastore.exists? options['upload_directory']
    connection...

Just change the code in the file you already found, restart foreman (systemctl restart httpd.service) and see if the issue is gone. Please report back or file a PR that removes the problematic code.

#12

Hi Timo, the bootdisk upload works as expected if I remove the + '/' at line 24

I ran two tests, to verify this change. The first one with a vmware datastore where the foreman_isos directory already exists. And a second one where the foreman_isos directory didnā€™t exist and has to be created first. Everything worked without any issues.

Would be nice if foreman would delete the old bootdisks after removing the vm, though :smiley:

#13

This is actually a little bit more challenging than it sounds.

  • In the provisioning template we tell the installer to eject the disk after the installation is complete. I believe this does not work reliably. @lzap: Any ideas?
  • The iso needs to be ejected (and the file unlocked) before we can remove it from the datastore. If the installer does not eject the disk, Foreman could do it - in theory. Currently, we donā€™t have a callback that tells us when a host booted after provisioning that we could leverage to tell VMware to eject the disk.

I donā€™t remember if itā€™s possible that during re-provisioning of a host the iso file on the datastore is overwritten if the iso is ejected. If that would be the case, we would not need to delete the file.
In any case, ejecting the disk from the vm would be the first priority in my opinion.

#14

Thanks a bunch for testing, Iā€™ve filed a PR with the fix.

#15

This would actually be easy. I believe this would be the method weā€™d need to call:

      def iso_detach(vm_uuid)
        client.vm_reconfig_cdrom(
          'instance_uuid' => vm_uuid
         )
      end

Question is: When do we call it? @lzap? @ezr-ondrej?

#16

@brotaxt: Sorry, I misread your post earlier.
This would actually be fairly easy to do. We would just need to add a task to the delete queue of the host that deletes the iso file if itā€™s present.

To fix the rebuild issue, we could just call it here and detach any existing iso before we upload and attach the new one:

#17

Wild guess - during ā€œbuiltā€ call?

#18

Thatā€˜s actually too early. The system needs to have bootet from the harddisk.

#19

Then we need to create what we call ā€œfirstbootā€ script that would call ā€œscriptā€ template which will have the only purpose - calling a function that will do it through Foreman? I mean, this is crazy and out of control at this point.

Bootdisk and VMWare integration is not really needed since VMWare does support iPXE through custom net firmware and then it boots anything straight up from HTTP through Foreman. No bootdisk dance needed.

#20

But you still need dhcp.Thats one reason why we prefer to install our hosts via bootdisk.