[Katello] Continue to have Actions::Katello::Host::UploadPackageProfile backfilled

NeilHanlon · June 13, 2017, 7:34pm

So mostly we use foreman/katello just for syncing operating systems out to
capsules/smart proxies, and for managing system lifecycle (basically, just
kickstarts). I went last week to add a XenServer 7.2 OS into my Katello
Products, and started having tons of issues, specifically in that I had
20k+ 'Actions::Katello::Host::UploadPackageProfile' tasks sitting in my
list as state = running, result = pending–but not doing anything at all.

To be honest, I don't really care why they're not working–I just want them
to stop being scheduled at all. Today I've shut off foreman-tasks and
pulp_workers and tried to set all these tasks which were running/pending to
stopped/error, and that worked. Seemed to help out with the issues of
Foreman. However I still had some tasks that weren't being scheduled, so I
checked and sure enough, foreman tasks was running, as was pulp, but it was
complaining about not having a dynflow environment. Restarting foreman
tasks and waiting a few minutes helped, at which point I started seeing
stuff like this spam my production.log, and tons of Actions::Katello::Host::UploadPackageProfile
tasks being scheduled again–with dates from months ago.

2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
df8fcf4c-0de7-4cca-8aaf-f38adcd2b6ec: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
f96f15cb-5933-4b18-956d-a2e4c7052004: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
dc8976be-99e7-4dc4-8a3f-893d3687a1cc: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get
"https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/",
"Accept"=>"/; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
deflate", "accept"=>"application/json", "content_type"=>"application/json"
> \nRestClient.get
"https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/",
"Accept"=>"/; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
deflate", "accept"=>"application/json", "content_type"=>"application/json"
> \n# => 200 OK | application/json 326 bytes
>
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get
"https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/",
"Accept"=>"/; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
deflate", "accept"=>"application/json", "content_type"=>"application/json"
> \nRestClient.get
"https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/",
"Accept"=>"/; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
deflate", "accept"=>"application/json", "content_type"=>"application/json"
> \n# => 200 OK | application/json 326 bytes
> \n# => 200 OK | application/json 328 bytes
>
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get
"https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/",
"Accept"=>"/; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
deflate", "accept"=>"application/json", "content_type"=>"application/json"
> \nRestClient.get
"https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/",
"Accept"=>"/; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
deflate", "accept"=>"application/json", "content_type"=>"application/json"
> \n# => 200 OK | application/json 326 bytes
> \n# => 200 OK | application/json 328 bytes
> \n# => 200 OK | application/json 328 bytes
>
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
f96f15cb-5933-4b18-956d-a2e4c7052004: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get
"https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/",
"Accept"=>"/; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
deflate", "accept"=>"application/json", "content_type"=>"application/json"
> \n# => 200 OK | application/json 329 bytes
>
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get
"https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/",
"Accept"=>"/; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
deflate", "accept"=>"application/json", "content_type"=>"application/json"
> \n# => 200 OK | application/json 329 bytes
> \n# => 200 OK | application/json 329 bytes
>
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get
"https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/",
"Accept"=>"/; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
deflate", "accept"=>"application/json", "content_type"=>"application/json"
> \n# => 200 OK | application/json 329 bytes
> \n# => 200 OK | application/json 329 bytes
> \n# => 200 OK | application/json 330 bytes
>
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get
"https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/",
"Accept"=>"/; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
deflate", "accept"=>"application/json", "content_type"=>"application/json"
> \n# => 200 OK | application/json 329 bytes
> \n# => 200 OK | application/json 329 bytes
> \n# => 200 OK | application/json 330 bytes
> \n# => 200 OK | application/json 327 bytes
>
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get
"https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/",
"Accept"=>"/; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
deflate", "accept"=>"application/json", "content_type"=>"application/json"
> \n# => 200 OK | application/json 329 bytes
> \n# => 200 OK | application/json 329 bytes

<https://lh3.googleusercontent.com/-sEewHxc4UuA/WUA9z6RVaWI/AAAAAAAABJM/mWeOvhRvpgknjIOZs8Unwt1l2791tugbACLcBGAs/s1600/Selection_133.png>

I'm happy to upload any logs anyone might want to look at to help me with
this… but I'm at my wits end with this. I'm about ready to tear it all
down and just rebuild it from scratch…

John_Mitsch · June 13, 2017, 8:59pm

That is very strange, this may be a good candidate for a foreman-debug so
we can dig through logs. Can you create one and send it to me privately?

-John

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

···

On Tue, Jun 13, 2017 at 3:34 PM, wrote:

So mostly we use foreman/katello just for syncing operating systems out to
capsules/smart proxies, and for managing system lifecycle (basically, just
kickstarts). I went last week to add a XenServer 7.2 OS into my Katello
Products, and started having tons of issues, specifically in that I had
20k+ ‘Actions::Katello::Host::UploadPackageProfile’ tasks sitting in my
list as state = running, result = pending–but not doing anything at all.

To be honest, I don’t really care why they’re not working–I just want
them to stop being scheduled at all. Today I’ve shut off foreman-tasks and
pulp_workers and tried to set all these tasks which were running/pending to
stopped/error, and that worked. Seemed to help out with the issues of
Foreman. However I still had some tasks that weren’t being scheduled, so I
checked and sure enough, foreman tasks was running, as was pulp, but it was
complaining about not having a dynflow environment. Restarting foreman
tasks and waiting a few minutes helped, at which point I started seeing
stuff like this spam my production.log, and tons of
Actions::Katello::Host::UploadPackageProfile tasks being scheduled
again–with dates from months ago.

2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
df8fcf4c-0de7-4cca-8aaf-f38adcd2b6ec: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
f96f15cb-5933-4b18-956d-a2e4c7052004: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
dc8976be-99e7-4dc4-8a3f-893d3687a1cc: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
41b06756-a517-415b-bcb9-9002dc0e5fec/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4./pulp/api/v2/tasks/
ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”
\n# => 200 OK | application/json 326 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
41b06756-a517-415b-bcb9-9002dc0e5fec/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4./pulp/api/v2/tasks/
ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”
\n# => 200 OK | application/json 326 bytes
\n# => 200 OK | application/json 328 bytes

2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
41b06756-a517-415b-bcb9-9002dc0e5fec/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4./pulp/api/v2/tasks/
ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”
\n# => 200 OK | application/json 326 bytes
\n# => 200 OK | application/json 328 bytes
\n# => 200 OK | application/json 328 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
f96f15cb-5933-4b18-956d-a2e4c7052004: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes

2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 330 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 330 bytes
\n# => 200 OK | application/json 327 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes

https://lh3.googleusercontent.com/-sEewHxc4UuA/WUA9z6RVaWI/AAAAAAAABJM/mWeOvhRvpgknjIOZs8Unwt1l2791tugbACLcBGAs/s1600/Selection_133.png

I’m happy to upload any logs anyone might want to look at to help me with
this… but I’m at my wits end with this. I’m about ready to tear it all
down and just rebuild it from scratch…

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

NeilHanlon · June 13, 2017, 9:06pm

Certainly. Going to let that run overnight

···

On Tue, Jun 13, 2017 at 5:00 PM John Mitsch wrote:

That is very strange, this may be a good candidate for a foreman-debug so
we can dig through logs. Can you create one and send it to me privately?

-John

John Mitsch
Red Hat Engineering
(860)-967-7285 <(860)%20967-7285>
irc: jomitsch

On Tue, Jun 13, 2017 at 3:34 PM, nhanlon@kayak.com wrote:

So mostly we use foreman/katello just for syncing operating systems out
to capsules/smart proxies, and for managing system lifecycle (basically,
just kickstarts). I went last week to add a XenServer 7.2 OS into my
Katello Products, and started having tons of issues, specifically in that I
had 20k+ ‘Actions::Katello::Host::UploadPackageProfile’ tasks sitting in
my list as state = running, result = pending–but not doing anything at all.

To be honest, I don’t really care why they’re not working–I just want
them to stop being scheduled at all. Today I’ve shut off foreman-tasks and
pulp_workers and tried to set all these tasks which were running/pending to
stopped/error, and that worked. Seemed to help out with the issues of
Foreman. However I still had some tasks that weren’t being scheduled, so I
checked and sure enough, foreman tasks was running, as was pulp, but it was
complaining about not having a dynflow environment. Restarting foreman
tasks and waiting a few minutes helped, at which point I started seeing
stuff like this spam my production.log, and tons of Actions::Katello::Host::UploadPackageProfile
tasks being scheduled again–with dates from months ago.

2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
df8fcf4c-0de7-4cca-8aaf-f38adcd2b6ec: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
f96f15cb-5933-4b18-956d-a2e4c7052004: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
dc8976be-99e7-4dc4-8a3f-893d3687a1cc: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4./pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”
\n# => 200 OK | application/json 326 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4./pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”
\n# => 200 OK | application/json 326 bytes
\n# => 200 OK | application/json 328 bytes

2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4./pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”
\n# => 200 OK | application/json 326 bytes
\n# => 200 OK | application/json 328 bytes
\n# => 200 OK | application/json 328 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
f96f15cb-5933-4b18-956d-a2e4c7052004: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes

2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 330 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 330 bytes
\n# => 200 OK | application/json 327 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes

https://lh3.googleusercontent.com/-sEewHxc4UuA/WUA9z6RVaWI/AAAAAAAABJM/mWeOvhRvpgknjIOZs8Unwt1l2791tugbACLcBGAs/s1600/Selection_133.png

I’m happy to upload any logs anyone might want to look at to help me with
this… but I’m at my wits end with this. I’m about ready to tear it all
down and just rebuild it from scratch…

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

John_Mitsch · June 15, 2017, 2:06pm

Seems like something funny is going on with the pulp tasks, can you install
pulp-admin and take a look at the tasks? We may need to kill something at
that level

to install:

gist.github.com

https://gist.github.com/johnpmitsch/866fb430780ba0e95a5554566b989416

gistfile1.txt

pulpversion=$(rpm -qa pulp-server --queryformat "%{VERSION}")
yum install -y pulp-admin-client-${pulpversion} pulp-rpm-admin-extensions.noarch pulp-rpm-handlers.noarch
pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)
pulp-admin -u admin -p $pulpAdminPassword repo list

-John

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

···

On Tue, Jun 13, 2017 at 5:06 PM, Neil Hanlon wrote:

Certainly. Going to let that run overnight

On Tue, Jun 13, 2017 at 5:00 PM John Mitsch jomitsch@redhat.com wrote:

That is very strange, this may be a good candidate for a foreman-debug so
we can dig through logs. Can you create one and send it to me privately?

-John

John Mitsch
Red Hat Engineering
(860)-967-7285 <(860)%20967-7285>
irc: jomitsch

On Tue, Jun 13, 2017 at 3:34 PM, nhanlon@kayak.com wrote:

So mostly we use foreman/katello just for syncing operating systems out
to capsules/smart proxies, and for managing system lifecycle (basically,
just kickstarts). I went last week to add a XenServer 7.2 OS into my
Katello Products, and started having tons of issues, specifically in that I
had 20k+ ‘Actions::Katello::Host::UploadPackageProfile’ tasks sitting
in my list as state = running, result = pending–but not doing anything at
all.

To be honest, I don’t really care why they’re not working–I just want
them to stop being scheduled at all. Today I’ve shut off foreman-tasks and
pulp_workers and tried to set all these tasks which were running/pending to
stopped/error, and that worked. Seemed to help out with the issues of
Foreman. However I still had some tasks that weren’t being scheduled, so I
checked and sure enough, foreman tasks was running, as was pulp, but it was
complaining about not having a dynflow environment. Restarting foreman
tasks and waiting a few minutes helped, at which point I started seeing
stuff like this spam my production.log, and tons of
Actions::Katello::Host::UploadPackageProfile tasks being scheduled
again–with dates from months ago.

2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
df8fcf4c-0de7-4cca-8aaf-f38adcd2b6ec: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
f96f15cb-5933-4b18-956d-a2e4c7052004: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
dc8976be-99e7-4dc4-8a3f-893d3687a1cc: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
41b06756-a517-415b-bcb9-9002dc0e5fec/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4.<
/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/
json”
\n# => 200 OK | application/json 326 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
41b06756-a517-415b-bcb9-9002dc0e5fec/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4.<
/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/
json”
\n# => 200 OK | application/json 326 bytes
\n# => 200 OK | application/json 328 bytes

2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
41b06756-a517-415b-bcb9-9002dc0e5fec/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4.<
/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/
json”
\n# => 200 OK | application/json 326 bytes
\n# => 200 OK | application/json 328 bytes
\n# => 200 OK | application/json 328 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
f96f15cb-5933-4b18-956d-a2e4c7052004: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes

2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 330 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 330 bytes
\n# => 200 OK | application/json 327 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes

https://lh3.googleusercontent.com/-sEewHxc4UuA/WUA9z6RVaWI/AAAAAAAABJM/mWeOvhRvpgknjIOZs8Unwt1l2791tugbACLcBGAs/s1600/Selection_133.png

I’m happy to upload any logs anyone might want to look at to help me
with this… but I’m at my wits end with this. I’m about ready to tear it
all down and just rebuild it from scratch…

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

NeilHanlon · June 15, 2017, 2:13pm

Thanks for taking a look for me. Here's the output of the tasks list.

···

On Thu, Jun 15, 2017 at 10:07 AM John Mitsch wrote:

Seems like something funny is going on with the pulp tasks, can you
install pulp-admin and take a look at the tasks? We may need to kill
something at that level

to install:
https://gist.github.com/johnpmitsch/866fb430780ba0e95a5554566b989416

-John

John Mitsch
Red Hat Engineering
(860)-967-7285 <(860)%20967-7285>
irc: jomitsch

On Tue, Jun 13, 2017 at 5:06 PM, Neil Hanlon nhanlon@kayak.com wrote:

Certainly. Going to let that run overnight

On Tue, Jun 13, 2017 at 5:00 PM John Mitsch jomitsch@redhat.com wrote:

That is very strange, this may be a good candidate for a foreman-debug
so we can dig through logs. Can you create one and send it to me privately?

-John

John Mitsch
Red Hat Engineering
(860)-967-7285 <(860)%20967-7285>
irc: jomitsch

On Tue, Jun 13, 2017 at 3:34 PM, nhanlon@kayak.com wrote:

So mostly we use foreman/katello just for syncing operating systems out
to capsules/smart proxies, and for managing system lifecycle (basically,
just kickstarts). I went last week to add a XenServer 7.2 OS into my
Katello Products, and started having tons of issues, specifically in that I
had 20k+ ‘Actions::Katello::Host::UploadPackageProfile’ tasks sitting
in my list as state = running, result = pending–but not doing anything at
all.

To be honest, I don’t really care why they’re not working–I just want
them to stop being scheduled at all. Today I’ve shut off foreman-tasks and
pulp_workers and tried to set all these tasks which were running/pending to
stopped/error, and that worked. Seemed to help out with the issues of
Foreman. However I still had some tasks that weren’t being scheduled, so I
checked and sure enough, foreman tasks was running, as was pulp, but it was
complaining about not having a dynflow environment. Restarting foreman
tasks and waiting a few minutes helped, at which point I started seeing
stuff like this spam my production.log, and tons of Actions::Katello::Host::UploadPackageProfile
tasks being scheduled again–with dates from months ago.

2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
df8fcf4c-0de7-4cca-8aaf-f38adcd2b6ec: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
f96f15cb-5933-4b18-956d-a2e4c7052004: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
dc8976be-99e7-4dc4-8a3f-893d3687a1cc: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4./pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”
\n# => 200 OK | application/json 326 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4./pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”
\n# => 200 OK | application/json 326 bytes
\n# => 200 OK | application/json 328 bytes

2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4./pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”
\n# => 200 OK | application/json 326 bytes
\n# => 200 OK | application/json 328 bytes
\n# => 200 OK | application/json 328 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
f96f15cb-5933-4b18-956d-a2e4c7052004: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes

2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 330 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 330 bytes
\n# => 200 OK | application/json 327 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes

https://lh3.googleusercontent.com/-sEewHxc4UuA/WUA9z6RVaWI/AAAAAAAABJM/mWeOvhRvpgknjIOZs8Unwt1l2791tugbACLcBGAs/s1600/Selection_133.png

I’m happy to upload any logs anyone might want to look at to help me
with this… but I’m at my wits end with this. I’m about ready to tear it
all down and just rebuild it from scratch…

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

John_Mitsch · June 15, 2017, 2:25pm

Neil,

Pulp is really backed up with tasks, my guess would be there was some sort
of deadlock in qpid and it "unlocked", releasing all of these tasks. My
advice would be first try stopping then starting these services
pulp_celerybeat.service pulp_workers.service pulp_resource_manager.service,
and then restarting qpidd and httpd. Wait a bit and see if tasks start
clearing out.

If that doesn't work and you want to go a more forceful route, you can try
deleting the tasks in pulp. This can cause some serious issues if you have
open tasks that you actually care about (like a repo sync), so if you do
wind up doing this, make sure any tasks you care about are finished up.
You've been warned

pulp-admin -u admin -p $pulpAdminPassword tasks list | grep "Task Id"| awk
'{print $3}' | xargs -I % pulp-admin -u admin -p $pulpAdminPassword tasks
cancel --task-id %

If you get unstuck, I would also advise upgrading to 3.4. Let me know how
it goes!

-John

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

···

On Thu, Jun 15, 2017 at 10:13 AM, Neil Hanlon wrote:

Thanks for taking a look for me. Here’s the output of the tasks list.

On Thu, Jun 15, 2017 at 10:07 AM John Mitsch jomitsch@redhat.com wrote:

Seems like something funny is going on with the pulp tasks, can you
install pulp-admin and take a look at the tasks? We may need to kill
something at that level

to install: https://gist.github.com/johnpmitsch/
866fb430780ba0e95a5554566b989416

-John

John Mitsch
Red Hat Engineering
(860)-967-7285 <(860)%20967-7285>
irc: jomitsch

On Tue, Jun 13, 2017 at 5:06 PM, Neil Hanlon nhanlon@kayak.com wrote:

Certainly. Going to let that run overnight

On Tue, Jun 13, 2017 at 5:00 PM John Mitsch jomitsch@redhat.com wrote:

That is very strange, this may be a good candidate for a foreman-debug
so we can dig through logs. Can you create one and send it to me privately?

-John

John Mitsch
Red Hat Engineering
(860)-967-7285 <(860)%20967-7285>
irc: jomitsch

On Tue, Jun 13, 2017 at 3:34 PM, nhanlon@kayak.com wrote:

So mostly we use foreman/katello just for syncing operating systems
out to capsules/smart proxies, and for managing system lifecycle
(basically, just kickstarts). I went last week to add a XenServer 7.2 OS
into my Katello Products, and started having tons of issues, specifically
in that I had 20k+ 'Actions::Katello::Host::UploadPackageProfile’
tasks sitting in my list as state = running, result = pending–but not
doing anything at all.

To be honest, I don’t really care why they’re not working–I just want
them to stop being scheduled at all. Today I’ve shut off foreman-tasks and
pulp_workers and tried to set all these tasks which were running/pending to
stopped/error, and that worked. Seemed to help out with the issues of
Foreman. However I still had some tasks that weren’t being scheduled, so I
checked and sure enough, foreman tasks was running, as was pulp, but it was
complaining about not having a dynflow environment. Restarting foreman
tasks and waiting a few minutes helped, at which point I started seeing
stuff like this spam my production.log, and tons of
Actions::Katello::Host::UploadPackageProfile tasks being scheduled
again–with dates from months ago.

2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 suspended >> running in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
df8fcf4c-0de7-4cca-8aaf-f38adcd2b6ec: 4 running >> suspended in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 suspended >> running in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
f96f15cb-5933-4b18-956d-a2e4c7052004: 4 suspended >> running in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
dc8976be-99e7-4dc4-8a3f-893d3687a1cc: 4 running >> suspended in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
41b06756-a517-415b-bcb9-9002dc0e5fec/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4.<
/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/
json”
\n# => 200 OK | application/json 326 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 running >> suspended in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
41b06756-a517-415b-bcb9-9002dc0e5fec/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4.<
/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/
json”
\n# => 200 OK | application/json 326 bytes
\n# => 200 OK | application/json 328 bytes

2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
41b06756-a517-415b-bcb9-9002dc0e5fec/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4.<
/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/
json”
\n# => 200 OK | application/json 326 bytes
\n# => 200 OK | application/json 328 bytes
\n# => 200 OK | application/json 328 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
f96f15cb-5933-4b18-956d-a2e4c7052004: 4 running >> suspended in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 running >> suspended in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 suspended >> running in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 suspended >> running in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 suspended >> running in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 suspended >> running in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 suspended >> running in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes

2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 running >> suspended in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 running >> suspended in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 330 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 running >> suspended in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 suspended >> running in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 330 bytes
\n# => 200 OK | application/json 327 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 running >> suspended in
phase Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/
f2f90682-f0c9-404d-98d7-8227fceda2b7/”, “Accept”=>"/; q=0.5,
application/xml", “Accept-Encoding”=>“gzip, deflate”,
“accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes

https://lh3.googleusercontent.com/-sEewHxc4UuA/WUA9z6RVaWI/AAAAAAAABJM/mWeOvhRvpgknjIOZs8Unwt1l2791tugbACLcBGAs/s1600/Selection_133.png

I’m happy to upload any logs anyone might want to look at to help me
with this… but I’m at my wits end with this. I’m about ready to tear it
all down and just rebuild it from scratch…

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

NeilHanlon · June 19, 2017, 5:25pm

Updating here for posterity in the list:

I ultimately had to go nuclear and destroy all the tasks using pulp-admin.
This got me unstuck enough to be able to go in and take care of the rest of
the 'stuck' tasks (orphaned stuff from hosts now deleted, etc).

···

On Thu, Jun 15, 2017 at 10:26 AM John Mitsch wrote:

Neil,

Pulp is really backed up with tasks, my guess would be there was some
sort of deadlock in qpid and it “unlocked”, releasing all of these tasks.
My advice would be first try stopping then starting these services
pulp_celerybeat.service pulp_workers.service pulp_resource_manager.service,
and then restarting qpidd and httpd. Wait a bit and see if tasks start
clearing out.

If that doesn’t work and you want to go a more forceful route, you can try
deleting the tasks in pulp. This can cause some serious issues if you have
open tasks that you actually care about (like a repo sync), so if you do
wind up doing this, make sure any tasks you care about are finished up.
You’ve been warned

pulp-admin -u admin -p $pulpAdminPassword tasks list | grep “Task Id”| awk
’{print $3}’ | xargs -I % pulp-admin -u admin -p $pulpAdminPassword tasks
cancel --task-id %

If you get unstuck, I would also advise upgrading to 3.4. Let me know how
it goes!

-John

John Mitsch
Red Hat Engineering
(860)-967-7285 <(860)%20967-7285>
irc: jomitsch

On Thu, Jun 15, 2017 at 10:13 AM, Neil Hanlon nhanlon@kayak.com wrote:

Thanks for taking a look for me. Here’s the output of the tasks list.

On Thu, Jun 15, 2017 at 10:07 AM John Mitsch jomitsch@redhat.com wrote:

Seems like something funny is going on with the pulp tasks, can you
install pulp-admin and take a look at the tasks? We may need to kill
something at that level

to install:
https://gist.github.com/johnpmitsch/866fb430780ba0e95a5554566b989416

-John

John Mitsch
Red Hat Engineering
(860)-967-7285 <(860)%20967-7285>
irc: jomitsch

On Tue, Jun 13, 2017 at 5:06 PM, Neil Hanlon nhanlon@kayak.com wrote:

Certainly. Going to let that run overnight

On Tue, Jun 13, 2017 at 5:00 PM John Mitsch jomitsch@redhat.com >>>> wrote:

That is very strange, this may be a good candidate for a foreman-debug
so we can dig through logs. Can you create one and send it to me privately?

-John

John Mitsch
Red Hat Engineering
(860)-967-7285 <(860)%20967-7285>
irc: jomitsch

On Tue, Jun 13, 2017 at 3:34 PM, nhanlon@kayak.com wrote:

So mostly we use foreman/katello just for syncing operating systems
out to capsules/smart proxies, and for managing system lifecycle
(basically, just kickstarts). I went last week to add a XenServer 7.2 OS
into my Katello Products, and started having tons of issues, specifically
in that I had 20k+ 'Actions::Katello::Host::UploadPackageProfile’
tasks sitting in my list as state = running, result = pending–but not
doing anything at all.

To be honest, I don’t really care why they’re not working–I just
want them to stop being scheduled at all. Today I’ve shut off foreman-tasks
and pulp_workers and tried to set all these tasks which were
running/pending to stopped/error, and that worked. Seemed to help out with
the issues of Foreman. However I still had some tasks that weren’t being
scheduled, so I checked and sure enough, foreman tasks was running, as was
pulp, but it was complaining about not having a dynflow environment.
Restarting foreman tasks and waiting a few minutes helped, at which point I
started seeing stuff like this spam my production.log, and tons of Actions::Katello::Host::UploadPackageProfile
tasks being scheduled again–with dates from months ago.

2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
df8fcf4c-0de7-4cca-8aaf-f38adcd2b6ec: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
f96f15cb-5933-4b18-956d-a2e4c7052004: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
dc8976be-99e7-4dc4-8a3f-893d3687a1cc: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4./pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”
\n# => 200 OK | application/json 326 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4./pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”
\n# => 200 OK | application/json 326 bytes
\n# => 200 OK | application/json 328 bytes

2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\nRestClient.get “https://c4katello02.c4./pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”
\n# => 200 OK | application/json 326 bytes
\n# => 200 OK | application/json 328 bytes
\n# => 200 OK | application/json 328 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
f96f15cb-5933-4b18-956d-a2e4c7052004: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes

2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
2392d32f-0886-4473-ae5a-7595504acb39: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 330 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
a03805a6-1725-4a92-b3f5-05514cc52c81: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 got event
Dynflow::Action::Polling::Poll
2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 suspended >> running in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 330 bytes
\n# => 200 OK | application/json 327 bytes

2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
903f497f-a539-45af-a220-4cf3d7dd70e2: 4 running >> suspended in phase
Run Actions::Pulp::Consumer::GenerateApplicability
2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get “
https://c4katello02.c4./pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/”,
“Accept”=>"/; q=0.5, application/xml", “Accept-Encoding”=>“gzip,
deflate”, “accept”=>“application/json”, “content_type”=>“application/json”

\n# => 200 OK | application/json 329 bytes
\n# => 200 OK | application/json 329 bytes

https://lh3.googleusercontent.com/-sEewHxc4UuA/WUA9z6RVaWI/AAAAAAAABJM/mWeOvhRvpgknjIOZs8Unwt1l2791tugbACLcBGAs/s1600/Selection_133.png

I’m happy to upload any logs anyone might want to look at to help me
with this… but I’m at my wits end with this. I’m about ready to tear it
all down and just rebuild it from scratch…

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it,
send an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
Y