Not able to install any package in client using foreman

**Problem: Not able to install any package in any client from Foreman, getting below error.
Errors during downloading metadata for repository ‘rhel-8-for-x86_64-baseos-rpms’:

When I go to respective directory all symbolic links are dead.

root@z1foreman01 Packages]# pwd
/var/lib/pulp/published/yum/https/repos/Otsuka/Otsuka-Extended_Test/Extended_content_view/content/eus/rhel/server/7/7Server/x86_64/os/Packages
[root@z1foreman01 Packages]# ll
total 1424
drwxr-xr-x 2 apache apache 4096 Oct 27 21:06 3
drwxr-xr-x 2 apache apache 28672 Oct 27 21:06 a
drwxr-xr-x 2 apache apache 32768 Oct 27 21:05 b
drwxr-xr-x 2 apache apache 32768 Oct 27 21:06 c
drwxr-xr-x 2 apache apache 28672 Oct 27 21:06 d
drwxr-xr-x 2 apache apache 12288 Oct 27 21:06 e
drwxr-xr-x 2 apache apache 61440 Oct 27 21:05 f
drwxr-xr-x 2 apache apache 73728 Oct 27 21:06 g
drwxr-xr-x 2 apache apache 24576 Oct 27 21:06 h
drwxr-xr-x 2 apache apache 36864 Oct 27 21:06 i
drwxr-xr-x 2 apache apache 20480 Oct 27 21:06 j
drwxr-xr-x 2 apache apache 69632 Oct 27 21:05 k
drwxr-xr-x 2 apache apache 212992 Oct 27 21:06 l
drwxr-xr-x 2 apache apache 32768 Oct 27 21:06 m
drwxr-xr-x 2 apache apache 49152 Oct 27 21:06 n
drwxr-xr-x 2 apache apache 32768 Oct 27 21:05 o
drwxr-xr-x 2 apache apache 110592 Oct 27 21:05 p
drwxr-xr-x 2 apache apache 24576 Oct 27 21:06 q
drwxr-xr-x 2 apache apache 32768 Oct 27 21:06 r
drwxr-xr-x 2 apache apache 94208 Oct 27 21:06 s
drwxr-xr-x 2 apache apache 53248 Oct 27 21:06 t
drwxr-xr-x 2 apache apache 8192 Oct 27 21:06 u
drwxr-xr-x 2 apache apache 8192 Oct 27 21:06 v
drwxr-xr-x 2 apache apache 4096 Oct 27 21:06 w
drwxr-xr-x 2 apache apache 16384 Oct 27 21:05 x
drwxr-xr-x 2 apache apache 4096 Oct 27 21:06 y
drwxr-xr-x 2 apache apache 4096 Oct 27 21:06 z
[root@z1foreman01 Packages]#

[root@z1foreman01 a]# pwd
/var/lib/pulp/published/yum/https/repos/Otsuka/Otsuka-Extended_Test/Extended_content_view/content/eus/rhel/server/7/7Server/x86_64/os/Packages/a
[root@z1foreman01 a]#

[root@z1foreman01 a]# ll
total 0
lrwxrwxrwx 1 apache apache 145 Nov 10 10:14 abattis-cantarell-fonts-0.0.12-3.el7.noarch.rpm -> /var/lib/pulp/content/units/rpm/a5/df900693a5589079b420d2eaa8c64ee5e81aad1246e00bf25818134c76f2da/abattis-cantarell-fonts-0.0.12-3.el7.noarch.rpm
lrwxrwxrwx 1 apache apache 145 Nov 10 10:14 abattis-cantarell-fonts-0.0.16-3.el7.noarch.rpm -> /var/lib/pulp/content/units/rpm/8d/180254afcf563190ae749be71d8b526322524eeaeb873bdc0140d1a53d1ff9/abattis-cantarell-fonts-0.0.16-3.el7.noarch.rpm
lrwxrwxrwx 1 apache apache 145 Nov 10 10:14 abattis-cantarell-fonts-0.0.25-1.el7.noarch.rpm -> /var/lib/pulp/content/units/rpm/94/5ec8227415acdff59d3e87412a0a18c2b9bddeb13693243de7b14ed7b1e389/abattis-cantarell-fonts-0.0.25-1.el7.noarch.rpm
lrwxrwxrwx 1 apache apache 127 Nov 10 10:14 abrt-2.1.11-12.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/c8/18e0f60f5cf624ce9aa2bf010525f5be72678969501e27f2f7030cc9302096/abrt-2.1.11-12.el7.x86_64.rpm
lrwxrwxrwx 1 apache apache 127 Nov 10 10:14 abrt-2.1.11-19.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/6c/895b51b5f402cd1119ab2263a57c12ce2922fd2fec5f86b0d4e7ba5e47da5e/abrt-2.1.11-19.el7.x86_64.rpm
lrwxrwxrwx 1 apache apache 129 Nov 10 10:14 abrt-2.1.11-22.el7_1.x86_64.rpm -> /var/lib/pulp/content/units/rpm/54/b6bd029a0ff1d9f7e6be3c2d7de0a30b06608c0c580386dbfa8e6ef6f1db6e/abrt-2.1.11-22.el7_1.x86_64.rpm
lrwxrwxrwx 1 apache apache 127 Nov 10 10:14 abrt-2.1.11-34.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/17/00a7a724f370274211978da186da5f031cbb6dbfd1bf9d337f0dbe92e1c279/abrt-2.1.11-34.el7.x86_64.rpm
lrwxrwxrwx 1 apache apache 127 Nov 10 10:14 abrt-2.1.11-35.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/b0/8903d2e6c75dcdc994a8ec710d7bf0dccdfca2bfca0ed6778571efabb169d7/abrt-2.1.11-35.el7.x86_64.rpm
lrwxrwxrwx 1 apache apache 127 Nov 10 10:14 abrt-2.1.11-45.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/15/0950b32c37a594fd5d2144c884d68b6087e8af7c58f054976b783d5e201146/abrt-2.1.11-45.el7.x86_64.rpm
**

Expected outcome: directory in symbolic link should exist at /var/lib/pulp/content/units/rpm/

Foreman and Proxy versions: foreman-release-1.18.1-1.el7.noarch

Foreman and Proxy plugin versions:

Distribution and version: RHEL 7.6

Other relevant data:

Hi @priyank_gupta

The broken symlinks are normal.

Do you mind sharing what repos are in this content view? Also, the error seems to be on a RHEL8 repo. Is the client on RHEL 7 or 8?

Hello jeremylenz, Thanks for reply,

In Content View we are using RHEL 6, 7 & 8 Repos including Ansible Repos.

Output was from RHEL 8 but issue is in RHEL 6,7 & 8 in all repos.

Tried on RHEL 6 7 & 8 and getting same error after running yum install

Let me know if I have to share any log

Created new content view and published it and promoted patches, still getting same error in both RHEL 7 & 8

Can you give the exact names of all repositories in the content view?

It kinda looks like from your request that you have marked that clients as using an ‘8.3’ release version. That means that many of the RHEL 8 repos won’t work as is. You have to explicitly enable and sync the 8.3 repos, which we don’t typically recommend. So if you’ve set the release version on the content host to ‘8.3’ ,try setting it to just ‘8’.

Hello Justin, Tried that also, also created new content view for RHEL 8 and RHEL 7 and still getting same error.

Yesterday I ran below command which ended being fail, could you please check if it can be fix.

nohup hammer repository synchronize --id 8117 --validate-contents yes &
Getting below error after running this command.

Error: RPM1004: Error retrieving metadata: Forbidden
No new packages.

Below is error getting in RHEL 7 servers.

js-1.8.5-20.el7.x86_64.rpm FAILED
https://z1foreman01.mocr-nt1.otsuka.com/pulp/repos/Otsuka/Otsuka-Extended_Test/Extended_content_view/content/dist/rhel/server/7/7Server/x86_64/os/Packages/j/js-1.8.5-20.el7.x86_64.rpm: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article

If above article doesn’t help to resolve this issue please open a ticket with Red Hat Support.

elinks-0.12-0.37.pre6.el7.x86_ FAILED
https://z1foreman01.mocr-nt1.otsuka.com/pulp/repos/Otsuka/Otsuka-Extended_Test/Extended_content_view/content/dist/rhel/server/7/7Server/x86_64/os/Packages/e/elinks-0.12-0.37.pre6.el7.x86_64.rpm: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.
nss_compat_ossl-0.9.6-8.el7.x8 FAILED
https://z1foreman01.mocr-nt1.otsuka.com/pulp/repos/Otsuka/Otsuka-Extended_Test/Extended_content_view/content/eus/rhel/server/7/7Server/x86_64/os/Packages/n/nss_compat_ossl-0.9.6-8.el7.x86_64.rpm: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.

For the ‘forbidden’ error on sync, you may need to refresh your manifest (under Content > Subscriptions > manage manifest > Refresh button)

For the 404, can you try doing a ‘yum clean all’ on the client?

Thanks Justin, will try to refresh manifest, regarding 404 trying yum clean all and removing content of /var/cache/yum

Tried refreshing manifest and getting below error.

RestClient::BadRequest: 400 Bad Request

Action:

Actions::Candlepin::Owner::UpstreamUpdate

Input:

{“organization_id”=>1,
“upstream”=>
{“created”=>“2020-01-10T17:41:45+0000”,
“updated”=>“2020-01-10T17:41:45+0000”,
“id”=>“8a92c2af6f670286016f908c347207bb”,
“uuid”=>“11486355-6f14-4639-b74f-51e2edd165a0”,
“name”=>“z1foreman01”,
“apiUrl”=>“https://subscription.rhsm.redhat.com/subscription/consumers/”,
“webUrl”=>“access.redhat.com/management/subscription_allocations/”,
“ownerId”=>“8a92c2af656ce10901656ce172e70001”,
“contentAccessMode”=>nil,
“type”=>{“id”=>“1006”, “label”=>“sam”, “manifest”=>true},
“idCert”=>
{“created”=>“2019-06-28T15:37:51+0000”,
“updated”=>“2020-01-10T17:41:45+0000”,
“id”=>“8a92c2af6f670286016f908c33c007ba”,
“key”=>
“-----BEGIN RSA PRIVATE KEY-----\n” +
“MIIEpAIBAAKCAQEAiky6U4wyy7AkGWjzS9z+HUJf4vAdRerlhU2RLV41IfRVAs15\n” +
“apj4Apty74zqx23er8csZz3psCJib3JkPq+laGOGJ8LqORHGMgLDeWyAv1isvmc4\n” +
“H2wpH+sVFXMnvEdCDr3aR7uouH/E6BkoWCos1j1aWvBkSavqNwjVQKmZQ8O/x3rV\n” +
“fYpxB60ZHHi1fWYpJDIOyeVpwgj+QvMfOhljCuMkHErtEalKD0M7sPqpLGTFvRZ0\n” +
“NIBas966xod/O0BZC0W3SzUvVAhN8u1XRaenA6P7TeoeestQtTCa2WD5lerZE0J7\n” +
“9+Q2Lkb5UWk0uzjpznal35OYx8BXJnmSRimhGwIDAQABAoIBAEnO6SHtkK5kMt/c\n” +
“wvuZ1YSIB+R1gSnvUs7xtqkfvfuBCl2Ui1f4i2PY0l5GNoZ4fIATK/9ta2o075PK\n” +
“2PUbiItNkitmOtp6QewuzWPgJFX6XTbAn5Mlhk0iiP4JlwtEfFyhtfzSihjtF4lG\n” +
“4+EpoQQ1eiNOaDdpKHZ2oFkCKmmfI1+XSrBXro8Zxt9LyobBbjcUJDWcPFIoFzq2\n” +
“DpcKkBLzg7erxSnt77JSNUR76uAxR2ae9oXrZUUZnZPiqUuIMen6+i1yAXMTJSzU\n” +
“3mTNcnCNASL+4m4ml3Zt5S4XSCGe/ipjbr4AY6JcJ1ZOJ/rUYDNzSHwBADtDzF0i\n” +
“AF0fawECgYEA24Tmt5iU/vsxhBXrFkyxzG02Yuq9BDs8/JlXHlMr4OSTe+nILRTr\n” +
“o+NuaICW3pzbWmbVps2GQR+tE9YkEcgDLZESJepSvvyAwKj342hUhV9V61n2N2cR\n” +
“jv8vZws6Eyxlp3vzAsRBdeWtFRYb/aB6xP0ePxgK+B6yWUjgVdkGgoUCgYEAoUh5\n” +
“G0MY0gdE2TvReRx5a4Z7lriCNHg+PG5BSw2fsi63e9kpEVYSe4qs89A8GJVrnjix\n” +
“809OkhSNrhnX2rnYPZ0BajPTVwJbk2D0qvO5FIpPkoeCgGTu4g4IBt1m6b5BRapF\n” +
“N9guGUQnDcmMd3LV1hA1EHm0ODoBhbSBXTkkdx8CgYEAt36qVQHIWjOjOmQP9L37\n” +
“SYRpF9pcUbMq0YNkHUcswHqRBd1vpK9+Iaq02cnTUxYPDcwz/5CBDuLpjFFOKUbl\n” +
“0SsY3nS24kqzXVhgjcjiDfKZIXg7pigTt1J8T0hFLU3JU2z6v29JwMFSIvQuRZ3l\n” +
“wdBLfjYcxTbzokqWq2wUUvkCgYAIIsnJMEdOQQOeGcBlZaRSYkRVxBT9ZJtvHPi6\n” +
“OoO1z1l+K+RAdaEVDlfPLM/7RX5aSu3fIj8Ka3MCN1eGNx3LbeImY1rj5sCL3LY6\n” +
“8XSCrr942tEgfPb3/FE3V2/blrdgNkirmoIo0CFQNryZQfe7hrHGlP0cYQC2p/Zm\n” +
“p31JbQKBgQCtmoHkYLC0oEBxmw/QbDrDYNcHNp+1Iny0TgE7TWNi+cOHjUr/5Rf7\n” +
“SGaGZazcDH7djWPEi29bpzPChVwJBG0VU/VP9AHZQK3ulRD3ZMQQvfWhTOO/lgej\n” +
“B8IAEIk/GRgE4pjgujA5jY3uY6l4Kc1D15h5bFmjP3C9ujoIwDI7BA==\n” +
“-----END RSA PRIVATE KEY-----\n”,
“cert”=>
“-----BEGIN CERTIFICATE-----\n” +
“MIIF6TCCA9GgAwIBAgIIZF+j4fJetO4wDQYJKoZIhvcNAQELBQAwgaQxCzAJBgNV\n” +
“BAYTAlVTMRcwFQYDVQQIDA5Ob3J0aCBDYXJvbGluYTEWMBQGA1UECgwNUmVkIEhh\n” +
“dCwgSW5jLjEYMBYGA1UECwwPUmVkIEhhdCBOZXR3b3JrMSQwIgYDVQQDDBtSZWQg\n” +
“SGF0IENhbmRsZXBpbiBBdXRob3JpdHkxJDAiBgkqhkiG9w0BCQEWFWNhLXN1cHBv\n” +
“cnRAcmVkaGF0LmNvbTAeFw0xOTA2MjgxNDM3NTFaFw0yMDA2MjgxNTM3NTFaMC8x\n” +
“LTArBgNVBAMMJDExNDg2MzU1LTZmMTQtNDYzOS1iNzRmLTUxZTJlZGQxNjVhMDCC\n” +
“ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAIpMulOMMsuwJBlo80vc/h1C\n” +
“X+LwHUXq5YVNkS1eNSH0VQLNeWqY+AKbcu+M6sdt3q/HLGc96bAiYm9yZD6vpWhj\n” +
“hifC6jkRxjICw3lsgL9YrL5nOB9sKR/rFRVzJ7xHQg692ke7qLh/xOgZKFgqLNY9\n” +
“WlrwZEmr6jcI1UCpmUPDv8d61X2KcQetGRx4tX1mKSQyDsnlacII/kLzHzoZYwrj\n” +
“JBxK7RGpSg9DO7D6qSxkxb0WdDSAWrPeusaHfztAWQtFt0s1L1QITfLtV0WnpwOj\n” +
“+03qHnrLULUwmtlg+ZXq2RNCe/fkNi5G+VFpNLs46c52pd+TmMfAVyZ5kkYpoRsC\n” +
“AwEAAaOCAZEwggGNMBEGCWCGSAGG+EIBAQQEAwIFoDALBgNVHQ8EBAMCBLAwgd4G\n” +
“A1UdIwSB1jCB04AUdy6lzTcNqOHT6mHFVmdD6JHHbRyhgbekgbQwgbExCzAJBgNV\n” +
“BAYTAlVTMRcwFQYDVQQIDA5Ob3J0aCBDYXJvbGluYTEWMBQGA1UECgwNUmVkIEhh\n” +
“dCwgSW5jLjEYMBYGA1UECwwPUmVkIEhhdCBOZXR3b3JrMTEwLwYDVQQDDChSZWQg\n” +
“SGF0IEVudGl0bGVtZW50IE9wZXJhdGlvbnMgQXV0aG9yaXR5MSQwIgYJKoZIhvcN\n” +
“AQkBFhVjYS1zdXBwb3J0QHJlZGhhdC5jb22CAT8wHQYDVR0OBBYEFK/An1kJPaWm\n” +
“Ozr2yqsFsW0WHM99MBMGA1UdJQQMMAoGCCsGAQUFBwMCMFYGA1UdEQRPME2kMTAv\n” +
“MS0wKwYDVQQDDCQxMTQ4NjM1NS02ZjE0LTQ2MzktYjc0Zi01MWUyZWRkMTY1YTCk\n” +
“GDAWMRQwEgYDVQQDDAt6MWZvcmVtYW4wMTANBgkqhkiG9w0BAQsFAAOCAgEAiDmR\n” +
“5Cz2rrHJmC3zwuHgV9GEAjM864oxUKXEJuwN+iz3jWshe367BzwSLW0Up4tSoR0v\n” +
“xgWUEt4VX7PBA4AgrYaqgExk5iB6MAZLCwcAYmm6o55NHok5PjG2fcYaKYcFUmsL\n” +
“5ZSC7TEjqqntWfsADehTvyY3uXeEH9dyOVTOf9HqfJh1U/5PoTE+HphBpdqCaUIl\n” +
“kuq6cktb/yrJhRM2Xb0ineNi3Kdib+zKw2xSUweRJhCxNUcXJomn3/8gBLCeVJLH\n” +
“ic/S9Vkq3tN0R0xRrvouOtrvkrFj/eL+lVgYIeq67jb6w+YpjjhA8FBDqeOU3w8P\n” +
“StfuHTPAxo6Vr8KOuGuo59uo+ZeKvK9CKhzNuSagwF+jhxeBCpyPhDfBY6hmaKP0\n” +
“2z4TBs9mfDsK36zKRhuWoKqLFfTiK9VePR9JEJigwE+ZirRHD3D8KWrtl3Z4KOz1\n” +
“LQ+jlBFSZT8O1OeKwGk+ufbCphg/ryXKYMjXZHJM/TKEKwIUVmsy0ywmcm6s0iOY\n” +
“mZPICbL6iSQEJNm/6+N5JOGg5LSXvaGaoiW7HcByPfBDkc8VTE0VLDxh5tETgZ57\n” +
“F0btyZRFM2PothzLLiDZRBSd0Qk13QZJMdc1KvzJPshgEe5fxj1oJvGW4lAa7Y9b\n” +
“W8xRDNxQN3iglmamYvNlIC01dsp1Ny7lnCgtmhM=\n” +
“-----END CERTIFICATE-----\n”,
“serial”=>
{“created”=>“2019-06-28T15:37:51+0000”,
“updated”=>“2020-01-10T17:41:45+0000”,
“id”=>3569093140393345175,
“serial”=>3569093140393345175,
“expiration”=>“2020-06-28T15:37:51+0000”,
“collected”=>false,
“revoked”=>false}}},
“session_id”=>“c3fb4033-e646-4471-b19f-a528e95ddc64”,
“remote_user”=>“admin”,
“remote_cp_user”=>“admin”}

Output:

{}

Exception:

RestClient::BadRequest: 400 Bad Request

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:223:in exception_with_response' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:103:in return!’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:809:in process_result' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:725:in block in transmit’
/opt/rh/rh-ruby24/root/usr/share/ruby/net/http.rb:877:in start' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in transmit’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:76:in put' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/katello/resources/candlepin/upstream_consumer.rb:28:in update’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/models/katello/glue/provider.rb:75:in owner_upstream_update' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/candlepin/owner/upstream_update.rb:12:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:538:in block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/middleware/keep_session_id.rb:11:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/middleware/keep_session_id.rb:23:in with_session_id' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/middleware/keep_session_id.rb:11:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:31:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:9:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:19:in propagate_candlepin_errors' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:9:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/middleware/remote_action.rb:16:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/models/katello/concerns/user_extensions.rb:21:in cp_config’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/middleware/remote_action.rb:16:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/execution_wrapper.rb:85:in wrap’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/progress.rb:30:in with_progress_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/progress.rb:16:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/middleware/keep_locale.rb:11:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/middleware/keep_locale.rb:22:in with_locale' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/lib/actions/middleware/keep_locale.rb:11:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/keep_current_user.rb:15:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/keep_current_user.rb:43:in restore_curent_user' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/keep_current_user.rb:15:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:31:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/world.rb:30:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:537:in block (2 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:536:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:536:in block in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:451:in block in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:451:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:451:in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:531:in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:278:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block (2 levels) in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract.rb:162:in with_meta_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:30:in open_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:15:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/director.rb:43:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/executors/parallel/worker.rb:12:in block in on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/executors.rb:12:in run_user_code’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/executors/parallel/worker.rb:11:in on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/context.rb:46:in on_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/executes_context.rb:7:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/actor.rb:26:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/awaits.rb:15:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:38:in process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:31:in process_envelopes?’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:20:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/termination.rb:55:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/removes_child.rb:10:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:161:in process_envelope’
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:95:in block in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:118:in block (2 levels) in schedule_execution’
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in block in synchronize' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in synchronize’
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:115:in block in schedule_execution’
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:18:in call' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:96:in work’
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:77:in block in call_job' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in run_task’
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in block (3 levels) in create_worker' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in loop’
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in block (2 levels) in create_worker' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in catch’
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in block in create_worker' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’