Error - illegal option -o when updating packets on a server/vm

Support
1

Problem:
I want to update a server but the job fails, other jobs, like Run Ansible Roles are ok.
Expected outcome:
Foreman/Katello will update the system and not fail before even starting.
Foreman and Proxy versions:
Foreman 3.11.2 - Katello 4.12.x
Foreman and Proxy plugin versions:

ansible-collection-theforeman-foreman-4.0.0-2.el8.noarch
candlepin-4.4.14-1.el8.noarch
candlepin-selinux-4.4.14-1.el8.noarch
dynflow-utils-1.6.3-1.el8.x86_64
foreman-3.11.2-1.el8.noarch
foreman-cli-3.11.2-1.el8.noarch
foreman-debug-3.11.2-1.el8.noarch
foreman-dynflow-sidekiq-3.11.2-1.el8.noarch
foreman-installer-3.11.2-1.el8.noarch
foreman-installer-katello-3.11.2-1.el8.noarch
foreman-obsolete-packages-1.9-1.el8.noarch
foreman-postgresql-3.11.2-1.el8.noarch
foreman-proxy-3.11.2-1.el8.noarch
foreman-redis-3.11.2-1.el8.noarch
foreman-release-3.11.2-1.el8.noarch
foreman-selinux-3.11.2-1.el8.noarch
foreman-service-3.11.2-1.el8.noarch
foreman-vmware-3.11.2-1.el8.noarch
katello-4.13.1-1.el8.noarch
katello-certs-tools-2.10.0-1.el8.noarch
katello-client-bootstrap-1.7.9-2.el8.noarch
katello-common-4.13.1-1.el8.noarch
katello-debug-4.13.1-1.el8.noarch
katello-repos-4.13.1-1.el8.noarch
katello-selinux-5.0.2-1.el8.noarch
pulpcore-obsolete-packages-1.2.0-1.el8.noarch
pulpcore-selinux-2.0.1-1.el8.x86_64
python3.11-pulp-ansible-0.21.8-1.el8.noarch
python3.11-pulp-cli-0.27.2-1.el8.noarch
python3.11-pulp-container-2.20.3-1.el8.noarch
python3.11-pulp-deb-3.2.1-1.el8.noarch
python3.11-pulp-glue-0.27.2-1.el8.noarch
python3.11-pulp-python-3.11.3-1.el8.noarch
python3.11-pulp-rpm-3.26.1-1.el8.noarch
python3.11-pulpcore-3.49.19-1.el8.noarch
rubygem-dynflow-1.8.4-1.el8.noarch
rubygem-foreman-tasks-9.1.1-1.fm3_11.el8.noarch
rubygem-foreman_ansible-14.0.0-2.fm3_11.el8.noarch
rubygem-foreman_discovery-24.0.1-2.fm3_11.el8.noarch
rubygem-foreman_maintain-1.6.9-1.el8.noarch
rubygem-foreman_remote_execution-13.1.0-1.fm3_11.el8.noarch
rubygem-foreman_remote_execution-cockpit-13.1.0-1.fm3_11.el8.noarch
rubygem-foreman_statistics-2.1.0-3.fm3_11.el8.noarch
rubygem-foreman_templates-9.4.0-3.fm3_11.el8.noarch
rubygem-hammer_cli-3.11.0-1.el8.noarch
rubygem-hammer_cli_foreman-3.11.0-1.el8.noarch
rubygem-hammer_cli_foreman_ansible-0.7.0-1.fm3_11.el8.noarch
rubygem-hammer_cli_foreman_discovery-1.3.0-1.fm3_11.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.3.0-1.fm3_10.el8.noarch
rubygem-hammer_cli_foreman_ssh-0.0.3-1.fm3_9.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.21-1.fm3_11.el8.noarch
rubygem-hammer_cli_katello-1.13.0-0.2.pre.master.el8.noarch
rubygem-katello-4.13.1-1.el8.noarch
rubygem-pulp_ansible_client-0.21.3-1.el8.noarch
rubygem-pulp_certguard_client-3.49.6-1.el8.noarch
rubygem-pulp_container_client-2.20.0-1.el8.noarch
rubygem-pulp_deb_client-3.2.0-1.el8.noarch
rubygem-pulp_file_client-3.49.6-1.el8.noarch
rubygem-pulp_ostree_client-2.3.0-1.el8.noarch
rubygem-pulp_python_client-3.11.1-1.el8.noarch
rubygem-pulp_rpm_client-3.26.1-1.el8.noarch
rubygem-pulpcore_client-3.49.6-1.el8.noarch
rubygem-smart_proxy_dynflow-0.9.2-1.fm3_11.el8.noarch
rubygem-smart_proxy_pulp-3.3.0-1.fm3_10.el8.noarch

Distribution and version:
CentOS 8 stream - latest
Other relevant data:

Most of the Foreman/Katello installation is default. No custom templates or other stuff.
The system was working ok before 2 days. I have updated from F3.9/K4.10 first to F3.10.x/K4.11.x and after to F3.11.2/K4.12.x

Now i am kida stuck as I have no snapshot to turn back to and if I migrate to a new foreman (did one install on centos 9 stream and restored) and restore the data i still get the same error.

Error below:

image
image2308×1271 167 KB

LOG -

2024-09-19T16:34:19 [I|app|2954e264]   Rendered api/v2/hosts/show.json.rabl (Duration: 19381.9ms | Allocations: 52327)
2024-09-19T16:34:19 [I|app|a44c19f5] Started GET "/notification_recipients" for 000.000.35.20 at 2024-09-19 16:34:19 +0300
2024-09-19T16:34:19 [I|app|a44c19f5] Processing by NotificationRecipientsController#index as JSON
2024-09-19T16:34:19 [I|app|a44c19f5] Completed 200 OK in 5ms (Views: 0.1ms | ActiveRecord: 0.6ms | Allocations: 1876)
2024-09-19T16:34:19 [I|app|2954e264] Completed 200 OK in 33224ms (Views: 24338.9ms | ActiveRecord: 51.3ms | Allocations: 60463)
2024-09-19T16:34:20 [I|app|ad910f62] Started POST "/api/v2/job_invocations" for 000.000.35.20 at 2024-09-19 16:34:20 +0300
2024-09-19T16:34:20 [I|app|ad910f62] Processing by Api::V2::JobInvocationsController#create as JSON
2024-09-19T16:34:20 [I|app|ad910f62]   Parameters: {"job_invocation"=>{"feature"=>"katello_packages_update_by_search", "inputs"=>{"Packages search query"=>"", "Selected update versions"=>"[]"}, "search_query"=>"name ^ (bitbucket)"}, "apiv"=>"v2"}
2024-09-19T16:34:20 [I|aud|ad910f62] JobInvocation (338) create event on job_category Katello
2024-09-19T16:34:20 [I|aud|ad910f62] JobInvocation (338) create event on description Update package(s)
2024-09-19T16:34:20 [I|aud|ad910f62] JobInvocation (338) create event on concurrency_level
2024-09-19T16:34:20 [I|aud|ad910f62] JobInvocation (338) create event on execution_timeout_interval
2024-09-19T16:34:20 [I|aud|ad910f62] JobInvocation (338) create event on password [redacted]
2024-09-19T16:34:20 [I|aud|ad910f62] JobInvocation (338) create event on key_passphrase
2024-09-19T16:34:20 [I|aud|ad910f62] JobInvocation (338) create event on remote_execution_feature_id 4
2024-09-19T16:34:20 [I|aud|ad910f62] JobInvocation (338) create event on effective_user_password
2024-09-19T16:34:20 [I|aud|ad910f62] JobInvocation (338) create event on ssh_user
2024-09-19T16:34:20 [I|aud|ad910f62] JobInvocation (338) create event on time_to_pickup
2024-09-19T16:34:20 [I|bac|ad910f62] Task {label: , execution_plan_id: c7d3aa96-ab3c-49c0-8774-26fc47bb2e99} state changed: pending
2024-09-19T16:34:20 [I|bac|ad910f62] Task {label: Actions::RemoteExecution::RunHostsJob, id: 44c8ad1b-7874-4a6a-b3bc-c94a817566f7, execution_plan_id: c7d3aa96-ab3c-49c0-8774-26fc47bb2e99} state changed: planning
2024-09-19T16:34:20 [I|bac|ad910f62] Task {label: Actions::RemoteExecution::RunHostsJob, id: 44c8ad1b-7874-4a6a-b3bc-c94a817566f7, execution_plan_id: c7d3aa96-ab3c-49c0-8774-26fc47bb2e99} state changed: planned
2024-09-19T16:34:20 [I|app|ad910f62]   Rendered /usr/share/gems/gems/foreman_remote_execution-13.1.0/app/views/api/v2/job_invocations/create.json.rabl (Duration: 34.1ms | Allocations: 15078)
2024-09-19T16:34:20 [I|app|ad910f62] Completed 201 Created in 324ms (Views: 21.9ms | ActiveRecord: 138.7ms | Allocations: 73949)
2024-09-19T16:34:20 [I|bac|ad910f62] Task {label: Actions::RemoteExecution::RunHostsJob, id: 44c8ad1b-7874-4a6a-b3bc-c94a817566f7, execution_plan_id: c7d3aa96-ab3c-49c0-8774-26fc47bb2e99} state changed: running
2024-09-19T16:34:20 [I|app|] Performing ForemanStatistics::TrendCounterJob (Job ID: 17c7a341-51c0-4867-be1f-d89d367d9aea) from Dynflow(default) enqueued at 2024-09-19T13:04:20Z with arguments: {}
2024-09-19T16:34:20 [I|bac|ad910f62] Task {label: , execution_plan_id: da8d213b-3b21-455d-ace5-fca65b7c86f8} state changed: pending
2024-09-19T16:34:20 [I|bac|ad910f62] Task {label: Actions::RemoteExecution::RunHostJob, id: 09f01e40-ad88-494e-ba2b-9706f37199fb, execution_plan_id: da8d213b-3b21-455d-ace5-fca65b7c86f8} state changed: planning
2024-09-19T16:34:20 [I|app|] Enqueued ForemanStatistics::TrendCounterJob (Job ID: fba21f40-2a58-4f13-8bcb-639037d22df8) to Dynflow(default) at 2024-09-19 14:04:20 UTC with arguments: {}
2024-09-19T16:34:20 [I|app|] Performed ForemanStatistics::TrendCounterJob (Job ID: 17c7a341-51c0-4867-be1f-d89d367d9aea) from Dynflow(default) in 231.89ms
2024-09-19T16:34:20 [I|bac|ad910f62] Task {label: Actions::RemoteExecution::RunHostJob, id: 09f01e40-ad88-494e-ba2b-9706f37199fb, execution_plan_id: da8d213b-3b21-455d-ace5-fca65b7c86f8} state changed: planned
2024-09-19T16:34:21 [I|bac|ad910f62] Task {label: Actions::RemoteExecution::RunHostJob, id: 09f01e40-ad88-494e-ba2b-9706f37199fb, execution_plan_id: da8d213b-3b21-455d-ace5-fca65b7c86f8} state changed: running
2024-09-19T16:34:24 [I|app|6e2e117a] Started POST "/foreman_tasks/api/tasks/callback" for 000.000.5.41 at 2024-09-19 16:34:24 +0300
2024-09-19T16:34:24 [I|app|6e2e117a] Processing by ForemanTasks::Api::TasksController#callback as HTML
2024-09-19T16:34:24 [I|app|6e2e117a]   Parameters: {"callback"=>{"task_id"=>"09f01e40-ad88-494e-ba2b-9706f37199fb", "step_id"=>3}, "data"=>{"result"=>[{"output_type"=>"stdout", "output"=>"rex login: ", "timestamp"=>1726752862.3901713}, {"output_type"=>"stdout", "output"=>"\n/var/tmp/foreman-ssh-cmd-6211671a-9abe-4030-adbc-9b3c62955880/script: 20: set: Illegal option -o pipefail\n", "timestamp"=>1726752864.393257}], "runner_id"=>"6211671a-9abe-4030-adbc-9b3c62955880", "exit_status"=>2}, "task"=>{}}
2024-09-19T16:34:24 [I|app|6e2e117a] Completed 200 OK in 12ms (Views: 0.1ms | ActiveRecord: 6.3ms | Allocations: 2343)
2024-09-19T16:34:24 [I|bac|ad910f62] Event delivered by request 6e2e117a-80bb-407a-ab68-b7a7f66f2c73
2024-09-19T16:34:24 [E|bac|ad910f62] Job execution failed
2024-09-19T16:34:24 [I|bac|ad910f62] Task {label: Actions::RemoteExecution::RunHostJob, id: 09f01e40-ad88-494e-ba2b-9706f37199fb, execution_plan_id: da8d213b-3b21-455d-ace5-fca65b7c86f8} state changed: stopped  result: error
2024-09-19T16:34:25 [I|app|d4c2c9f9] Started GET "/notification_recipients" for 000.000.23.13 at 2024-09-19 16:34:25 +0300
2024-09-19T16:34:25 [I|app|d4c2c9f9] Processing by NotificationRecipientsController#index as JSON
2024-09-19T16:34:25 [I|app|d4c2c9f9] Completed 200 OK in 5ms (Views: 0.1ms | ActiveRecord: 0.6ms | Allocations: 1900)
2024-09-19T16:34:29 [I|app|2914aa8e] Started GET "/notification_recipients" for 000.000.35.20 at 2024-09-19 16:34:29 +0300
2024-09-19T16:34:29 [I|app|2914aa8e] Processing by NotificationRecipientsController#index as JSON
2024-09-19T16:34:29 [I|app|2914aa8e] Completed 200 OK in 5ms (Views: 0.1ms | ActiveRecord: 0.7ms | Allocations: 1876)

Actual error:

image
image848×131 4.33 KB

2

After upgrading to latest - f3.12/k4.14 RC2 it seems that everything is ok :confused:

1 Like
3

@aruzicka for the record books, do you recall if there were any remote execution issues fixed that might’ve been related to the issue here?
Specifically the

Illegal option -o pipefail

error.
I’m not 100% sure if that would be an issue with REX itself or potentially the job template.

4

It was definitely an issue with a job template, issue with rex itself would have looked a bit different.

The failure was most likely introduced by this change Fix diff behavior of package removal on debian/rhel based system by sbernhard · Pull Request #881 · theforeman/foreman_remote_execution · GitHub and later resolved by Fixes #37545 - Fix package_action for dash by m-bucher · Pull Request #903 · theforeman/foreman_remote_execution · GitHub

1 Like
5

thank you,

is good to know. maybe someone will actually benefit from this in the future. i know i did fix some past issues in my installation from threads like this one.

cheers for an awesome job to everyone here !!!

also i would have said is not rex issue because rex was working in other jobs without any problems… anything that was not working with apt.

if you guys want to do some more testing i can restore the snapshot and we can debug a bit (1-2 days) but i would like to know in the next 1-2 hours because it’s prod and if too much time passes and jobs get ran then i will have problems in the future.

6

Nice, thanks for the analysis there @aruzicka.