Can't deploy to certain esx hosts / clusters

widhalmt · August 8, 2018, 9:45am

Problem:

I can’t deploy to certain ESX hosts. It works on some and on some it doesn’t.

I get quite long errormessages but it seems to come down to:

2018-08-03T14:21:22 [W|app|437e0] Failed to upload ISO image for instance nag-test-07.ad.dsegroup.eu: InvalidArgument: 
RbVmomi::Fault: InvalidArgument:

Expected outcome:

Getting a new VM deployed.

Foreman and Proxy versions:

Debian 1.18.0

(My colleague @dirk applied some patches about storage handling from 1.18.1 )

Foreman and Proxy plugin versions:

Other relevant data:

This seems not to be ESX version specific. It work’s on a 6.5 in the main datacenter but it doesn’t work on 5.5, 6.0 and 6.5 in remote datacenters.

I found an article at RedHat about missing permissions but the user Foreman uses is managed centrally and this doesn’t seem to apply.

https://access.redhat.com/solutions/1339483

Could you give me a hint how to debug this issue any further? Or do you know a possible solution?

Thanks in advance,
Cheers,
Thomas

Marek_Hulan · August 8, 2018, 10:25am

Hello,

I think it might help to see the whole trace, if there’s some. Do users in your both working and nonworking ESX instances have same permissions?

widhalmt · August 8, 2018, 3:27pm

Hi,

Thanks for your reply. I spefically asked if the permissions are the same. I can’t check it myself but they told me: yes.

Here’s the rest:

2018-08-03T14:21:22 [W|app|437e0] Failed to upload ISO image for instance ***: InvalidArgument: 
RbVmomi::Fault: InvalidArgument: 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rbvmomi-1.13.0/lib/rbvmomi/connection.rb:63:in `parse_response' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rbvmomi-1.13.0/lib/rbvmomi/connection.rb:92:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rbvmomi-1.13.0/lib/rbvmomi/basic_types.rb:213:in `_call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rbvmomi-1.13.0/lib/rbvmomi/basic_types.rb:76:in `block (2 levels) in init' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/fog-vsphere-2.1.1/lib/fog/vsphere/requests/compute/upload_iso.rb:25:in `upload_iso' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_bootdisk-12.0.0/app/models/concerns/foreman_bootdisk/compute_resources/vmware.rb:29:in `iso_upload' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_bootdisk-12.0.0/app/models/concerns/foreman_bootdisk/orchestration/compute.rb:39:in `bootdisk_upload_iso' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_bootdisk-12.0.0/app/models/concerns/foreman_bootdisk/orchestration/compute.rb:57:in `setIsoImage' 
/usr/share/foreman/app/models/concerns/orchestration.rb:222:in `execute' 
/usr/share/foreman/app/models/concerns/orchestration.rb:145:in `block in process' 
/usr/share/foreman/app/models/concerns/orchestration.rb:137:in `each' 
/usr/share/foreman/app/models/concerns/orchestration.rb:137:in `process' 
/usr/share/foreman/app/models/concerns/orchestration.rb:44:in `around_save_orchestration' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:135:in `run_callbacks' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:827:in `_run_save_callbacks' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/callbacks.rb:336:in `create_or_update' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/persistence.rb:129:in `save' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/validations.rb:44:in `save' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/attribute_methods/dirty.rb:35:in `save' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/transactions.rb:308:in `block (2 levels) in save' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/transactions.rb:384:in `block in with_transaction_returning_status' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `block in transaction' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/transaction.rb:194:in `block in within_new_transaction' 
/usr/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/transaction.rb:191:in `within_new_transaction' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `transaction' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/transactions.rb:210:in `transaction' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/transactions.rb:381:in `with_transaction_returning_status' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/transactions.rb:308:in `block in save' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/transactions.rb:323:in `rollback_active_record_state!' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/transactions.rb:307:in `save' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/suppressor.rb:42:in `save' 
/usr/share/foreman/app/models/concerns/foreman/sti.rb:26:in `save' 
/usr/share/foreman/app/controllers/hosts_controller.rb:104:in `create' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/abstract_controller/base.rb:186:in `process_action' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_controller/metal/rendering.rb:30:in `process_action' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/abstract_controller/callbacks.rb:20:in `block in process_action' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:108:in `block in run_callbacks' 
/usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks' 
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks' 
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:135:in `run_callbacks' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/abstract_controller/callbacks.rb:19:in `process_action' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_controller/metal/rescue.rb:20:in `process_action' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/notifications.rb:166:in `block in instrument' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/notifications/instrumenter.rb:21:in `instrument' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/notifications.rb:166:in `instrument' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_controller/metal/instrumentation.rb:30:in `process_action' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_controller/metal/params_wrapper.rb:252:in `process_action' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.1.6/lib/active_record/railties/controller_runtime.rb:22:in `process_action' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/abstract_controller/base.rb:124:in `process' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionview-5.1.6/lib/action_view/rendering.rb:30:in `process' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_controller/metal.rb:189:in `dispatch' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_controller/metal.rb:253:in `dispatch' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/routing/route_set.rb:49:in `dispatch' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/routing/route_set.rb:31:in `serve' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/journey/router.rb:50:in `block in serve' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/journey/router.rb:33:in `each' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/journey/router.rb:33:in `serve' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/routing/route_set.rb:844:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/apipie-rails-0.5.9/lib/apipie/static_dispatcher.rb:65:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/apipie-rails-0.5.9/lib/apipie/extractor/recorder.rb:136:in `call' 
/usr/share/foreman/lib/middleware/telemetry.rb:9:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/apipie-rails-0.5.9/lib/apipie/middleware/checksum_in_headers.rb:27:in `call' 
/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.5/lib/rack/conditional_get.rb:38:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.5/lib/rack/head.rb:12:in `call' 
/usr/share/foreman/lib/middleware/logging_context.rb:23:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/middleware/cookies.rb:613:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/middleware/callbacks.rb:26:in `block in call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/callbacks.rb:97:in `run_callbacks' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/middleware/callbacks.rb:24:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/middleware/debug_exceptions.rb:59:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/rack/logger.rb:36:in `call_app' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/rack/logger.rb:26:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/middleware/remote_ip.rb:79:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/middleware/request_id.rb:25:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.1.6/lib/active_support/cache/strategy/local_cache_middleware.rb:27:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/middleware/executor.rb:12:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/actionpack-5.1.6/lib/action_dispatch/middleware/static.rb:125:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/secure_headers-5.0.5/lib/secure_headers/middleware.rb:13:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/engine.rb:522:in `call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/railtie.rb:185:in `public_send' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-5.1.6/lib/rails/railtie.rb:185:in `method_missing' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call' 
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' 
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:160:in `accept_and_process_next_request' 
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop' 
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads' 
/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' 
/usr/share/foreman/vendor/ruby/2.3.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' 
2018-08-03T14:21:22 [W|app|437e0] Rolling back due to a problem: [#<Orchestration::Task:0x007fcb0b20cd48 @name="Upload ISO image to datastore for ***", @id="Upload ISO image to datastore for ***", @status="failed", @priority=6, @action=[#<Host::Managed id: nil, name: "***", last_compile: nil, last_report: nil, updated_at: nil, created_at: nil, root_pass: "***", architecture_id: 1, operatingsystem_id: 92, environment_id: 1, ptable_id: 56, medium_id: 2, build: true, comment: "", disk: "", installed_at: nil, model_id: nil, hostgroup_id: 1, owner_id: 4, owner_type: "User", enabled: true, puppet_ca_proxy_id: 1, managed: true, use_image: nil, image_file: nil, uuid: "***", compute_resource_id: 4, puppet_proxy_id: 1, certname: nil, image_id: nil, organization_id: nil, location_id: nil, type: "Host::Managed", otp: nil, realm_id: nil, compute_profile_id: 4, provision_method: "bootdisk", grub_pass: "***", global_status: 0, lookup_value_matcher: "fqdn=***", pxe_loader: "PXELinux BIOS">, :setIsoImage], @created=1533298880.4729137, @timestamp=2018-08-03 12:21:22 UTC>] 
2018-08-03T14:21:22 [I|app|437e0] Removing Compute instance for *** 
2018-08-03T14:21:22 [I|app|437e0] Processed 4 tasks from queue 'Host::Managed Main', completed 0/6 
2018-08-03T14:21:22 [E|app|437e0] Task 'Set up compute instance ****' *rollbacked* 
2018-08-03T14:21:22 [E|app|437e0] Task 'Query instance details for ***' *rollbacked* 
2018-08-03T14:21:22 [E|app|437e0] Task 'Generating ISO image for ***' *rollbacked* 
2018-08-03T14:21:22 [E|app|437e0] Task 'Upload ISO image to datastore for ***' *failed* 
2018-08-03T14:21:22 [E|app|437e0] Task 'Power up compute instance ***' *canceled* 
2018-08-03T14:21:22 [E|app|437e0] Task 'Attach ISO image to CDROM drive for ***' *canceled* 
2018-08-03T14:21:22 [E|app|437e0] Failed to save: Failed to upload ISO image for instance ***: InvalidArgument:

Sorry for the anonymizing but since this is from a customer of ours I have to double check not to reveal anything sensible.

ekohl · August 8, 2018, 3:42pm

It’s good to note that we’re working on including fog-vsphere 2.3.0 in 1.18.1. That contains various important fixes but yesterday we found a regression that prevents from provisioning with distributed virtual switches. That is considered a release blocker for 1.18.1. That bug is also present in 1.19 and nightly branches.

widhalmt · August 8, 2018, 7:14pm

Ok, so you think we hit this bug? So searching further is futile until we upgraded to an upcoming 1.19.1?

tbrisker · August 9, 2018, 8:18am

This doesn’t looks like the regression in 2.3.0, since you are using 2.1.1 version of fog-vsphere. @vmware team - have you run across this issue before? This may be a separate issue that may or may not have been fixed in version 2.3 of fog-vsphere

widhalmt · August 9, 2018, 8:40am

Ok, thank you. I’ll wait if your @vmware has anything to add.

Thanks so far!

Marek_Hulan · August 9, 2018, 11:14am

Alright, so the trace revels that this is bootdisk based deployment and the operation of uploading the ISO image fails. I suppose PXE and traditional image based provisioning works fine. I wonder if there can be some limitation such as bootdisk iso is too big or something.

Since we don’t get more info from our logs, could you try to obtain error log from ESX admins? All we can tell from this is that the uploadnig operation failed and the fault was InvalidArgument. We need details of that fault type.

widhalmt · August 9, 2018, 11:27am

Thank you. I’ll try and get the info and come to you.

widhalmt · August 22, 2018, 12:17pm

Sorry for the delay. I’m still waiting for the logs.

cintrix84 · August 22, 2018, 3:01pm

@widhalmt

Hi,

This was fixed in fog-vsphere 2.3.0

widhalmt · August 22, 2018, 3:24pm

Thank you!

I’ll check if this fixes this issue.

Cheers,
Thomas

Dokon · August 28, 2018, 12:06pm

Hi!

In addition to my colleague widhalmt i have to report that it doesn’t seem to fix the issue.

When the actual last foreman update is applied there is the following error shown.

“Unable to save
Query instance details for host.xx.com failed with the following error: undefined method `raw_networks’ for #FOG::Compute::Vsphere::Real:0x007f4e6f096fb8 Did you mean? raw_network_vlan”

If more Information and Vsphere logs are required i can provide them as well.

cintrix84 · August 28, 2018, 1:23pm

Hi @Dokon

Sorry about that, there is some additional steps that are needed.

Stop Foreman services

wget https://raw.githubusercontent.com/theforeman/foreman/develop/app/models/concerns/fog_extensions/vsphere/server.rb
mv server.rb /usr/share/foreman/app/models/concerns/fog_extensions/vsphere/

Overwrite and startup Foreman services and let me know if that works.

Dokon · August 28, 2018, 3:58pm

Thanks ! For the help. We try it out and i will give as fast as possible feedback if it is working.

Dokon · August 29, 2018, 8:37am

Hi !

We tried it but the error is still present we get the following “new” error shown in the WebGUI.

“Failed to upload ISO image for instance host.xx: InvalidArgument:”

Dirk · August 29, 2018, 8:45am

My colleague asked me to upload the screenshot from the error as he has not yet reached the status to do so.

cintrix84 · August 29, 2018, 1:27pm

Hi,

I assume you are you using boot-disk plugin since it is erroring out about uploading the iso? Can you try with just network provisioning and see if it works correctly?

Dokon · August 29, 2018, 3:17pm

Hi ! We sorry to report, but thats no option in this environment.
Because there is no dhcp available, also the “discovery” option is not available in this environment here.

Anything else we can provide to help to close in on the error.

TimoGoebel · August 29, 2018, 3:56pm

Can you guys please add a debug command to Foreman. Just extend https://github.com/theforeman/foreman_bootdisk/blob/2a50cc35ab327ad9fcd5b41c213ffc6ad348e6b2/app/models/concerns/foreman_bootdisk/compute_resources/vmware.rb#L22-L30 with something like this:

      def iso_upload(iso, vm_uuid)
        options = {
          'local_path' => iso,
          'datacenter' => dc.name,
          'datastore' => bootdisk_datastore(vm_uuid),
          'upload_directory' => 'foreman_isos' # fog creates the directory if it does not exist
        }
        Rails.logger.error "DEBUG: upload_iso options: #{options.inspect}"
        client.upload_iso options
      end

The stack trace indicates, that this fails when creating the directory on the remote datastore.
With the patch above we should see (after restarting foreman ofc) in the logs what options are passed from foreman to fog-vsphere. I hope this helps to isolate the issue and maybe find some common pattern (datastore name or similar).