Deployed a new proxy last week and it was syncing fine. It struggled today but that’s because selinux got enabled. Have set to permissive mode and seems better (as in progress bar moves) however it’s getting stuck at 83% then if i refresh it shows 75%
When I check that task I see a success, so this seems to be cosmetic. Anyone else had this before? Any ideas on how to clear it?
Thanks
Foreman and Proxy versions:
3.6.1 / 4.8
Progress still stuck at 75% in the GUI, even though no tasks are running and content got synced to it and other proxies yesterday. I see a couple of sync tasks for it in the task list, both are showing state as stopped but result pending.
How can I clear these to see if this resets the proxy progress bar?
Tried foreman-rake foreman_tasks:cleanup RESULT=pending but no results came back for that
I usually run 2 cleanup scripts i have
First:
#!/bin/bash
echo “cleaning up all tasks”
foreman-rake foreman_tasks:cleanup TASK_SEARCH=‘result == pending’ STATES=‘running’ VERBOSE=true
foreman-rake foreman_tasks:cleanup TASK_SEARCH=‘result == warning’ STATES=‘running’ VERBOSE=true
foreman-rake foreman_tasks:cleanup TASK_SEARCH=‘result == error’ STATES=‘running’ VERBOSE=trueforeman-rake foreman_tasks:cleanup TASK_SEARCH=‘result == pending’ STATES=‘paused’ VERBOSE=true
foreman-rake foreman_tasks:cleanup TASK_SEARCH=‘result == warning’ STATES=‘paused’ VERBOSE=true
foreman-rake foreman_tasks:cleanup TASK_SEARCH=‘result == error’ STATES=‘paused’ VERBOSE=trueforeman-rake foreman_tasks:cleanup TASK_SEARCH=‘result == pending’ STATES=‘stopped’ VERBOSE=true
foreman-rake foreman_tasks:cleanup TASK_SEARCH=‘result == warning’ STATES=‘stopped’ VERBOSE=true
foreman-rake foreman_tasks:cleanup TASK_SEARCH=‘result == error’ STATES=‘stopped’ VERBOSE=true
second:
systemctl stop pulpcore* --all
sudo -u pulp PULP_SETTINGS=‘/etc/pulp/settings.py’ DJANGO_SETTINGS_MODULE=‘pulpcore.app.settings’ pulpcore-manager shell -c “import pulpcore; pulpcore.app.models.ReservedResource.objects.all().delete()”
systemctl start pulpcore* --all
then foreman-maintain service restart
hope this helps
what is that pulp script doing exactly?
Thanks for the task clear ups - will try that and see if it removes the stale sync which is still showing from 4 days ago
Hmm no joy with any of those cleanup commands. Said no tasks matching filter found for each one
Thought STATES it might be a typo but tried STATE as per the tasks in the GUI and no different.
I have a lot of old tasks that are in stopped state with result as pending but running below does not find and clear them?
foreman-rake foreman_tasks:cleanup TASK_SEARCH=‘result == pending’ STATES=‘stopped’ VERBOSE=true
anyone else have an idea on why the smart proxy sync progress bar is stuck?
I’ve just noticed that the timezone was out on this machine, so reset to UTC and hoped it would fix the issue.
Just run a complete sync and watched it climb up to 83% (whilst watching the pulpcore-api and worker log entries for all my content views data in /var/log/messages
It got stuck at 83% and logs stopped - I’ve pasted the last 20 below. Refreshed the smart proxy page and it shows 75% again.
It says synchronized and the last sync is correct but graphic shows 75% !!
Last sync: 2023-11-15 13:36:19 UTC
Nov 15 08:36:17 dy-prod-foreman-proxy01 pulpcore-worker-1[49232]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 8b5a0e87-af4e-4232-9cda-760dff51f369
Nov 15 08:36:17 dy-prod-foreman-proxy01 pulpcore-api[1236]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: - - [15/Nov/2023:13:36:17 +0000] “PATCH /pulp/api/v3/distributions/rpm/rpm/ccbf757a-c1ec-4e33-a497-bdaf8f41d514/ HTTP/1.1” 202 67 “-” “OpenAPI-Generator/3.19.0/ruby”
Nov 15 08:36:17 dy-prod-foreman-proxy01 pulpcore-worker-1[49232]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 8b5a0e87-af4e-4232-9cda-760dff51f369
Nov 15 08:36:17 dy-prod-foreman-proxy01 pulpcore-worker-2[49235]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 540ce93b-6a63-4409-81ab-61181ca7767c
Nov 15 08:36:17 dy-prod-foreman-proxy01 pulpcore-api[1238]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: - - [15/Nov/2023:13:36:17 +0000] “GET /pulp/api/v3/tasks/89f0a54a-ef69-4144-8b3e-85f76d9f0fb1/ HTTP/1.1” 200 562 “-” “OpenAPI-Generator/3.22.2/ruby”
Nov 15 08:36:17 dy-prod-foreman-proxy01 pulpcore-api[1236]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: - - [15/Nov/2023:13:36:17 +0000] “GET /pulp/api/v3/tasks/145a288e-0b47-4184-8603-098454d8f75b/ HTTP/1.1” 200 562 “-” “OpenAPI-Generator/3.22.2/ruby”
Nov 15 08:36:17 dy-prod-foreman-proxy01 pulpcore-api[1119]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: - - [15/Nov/2023:13:36:17 +0000] “GET /pulp/api/v3/publications/rpm/rpm/?repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Frpm%2Frpm%2F05e978bd-67bf-426e-915c-15e511ec553f%2Fversions%2F4%2F HTTP/1.1” 200 500 “-” “OpenAPI-Generator/3.19.0/ruby”
Nov 15 08:36:17 dy-prod-foreman-proxy01 pulpcore-worker-2[49235]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 540ce93b-6a63-4409-81ab-61181ca7767c
Nov 15 08:36:17 dy-prod-foreman-proxy01 pulpcore-api[1236]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: - - [15/Nov/2023:13:36:17 +0000] “GET /pulp/api/v3/tasks/8b5a0e87-af4e-4232-9cda-760dff51f369/ HTTP/1.1” 200 562 “-” “OpenAPI-Generator/3.22.2/ruby”
Nov 15 08:36:17 dy-prod-foreman-proxy01 pulpcore-api[1236]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: - - [15/Nov/2023:13:36:17 +0000] “GET /pulp/api/v3/tasks/540ce93b-6a63-4409-81ab-61181ca7767c/ HTTP/1.1” 200 562 “-” “OpenAPI-Generator/3.22.2/ruby”
Nov 15 08:36:17 dy-prod-foreman-proxy01 pulpcore-api[1238]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: - - [15/Nov/2023:13:36:17 +0000] “GET /pulp/api/v3/distributions/rpm/rpm/?base_path=BT_SPLAT%2FLibrary%2Fcustom%2FOEL9_Upstream%2FOEL9-EPEL_Upstream HTTP/1.1” 200 575 “-” “OpenAPI-Generator/3.19.0/ruby”
Nov 15 08:36:18 dy-prod-foreman-proxy01 pulpcore-api[1236]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: - - [15/Nov/2023:13:36:18 +0000] “PATCH /pulp/api/v3/distributions/rpm/rpm/96701ffd-5798-478f-bac5-e3259803eb65/ HTTP/1.1” 202 67 “-” “OpenAPI-Generator/3.19.0/ruby”
Nov 15 08:36:18 dy-prod-foreman-proxy01 pulpcore-worker-2[49244]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 081d3aeb-1afe-49a7-bd3d-4d3babef8a45
Nov 15 08:36:18 dy-prod-foreman-proxy01 pulpcore-worker-2[49244]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 081d3aeb-1afe-49a7-bd3d-4d3babef8a45
Nov 15 08:36:18 dy-prod-foreman-proxy01 pulpcore-api[1236]: pulp [ce8e2b47-7f53-4f98-ae3f-5e15099b4ec6]: - - [15/Nov/2023:13:36:18 +0000] “GET /pulp/api/v3/tasks/081d3aeb-1afe-49a7-bd3d-4d3babef8a45/ HTTP/1.1” 200 562 “-” “OpenAPI-Generator/3.22.2/ruby”
Nov 15 08:37:14 dy-prod-foreman-proxy01 smart-proxy[1027]: 172.18.171.7 - - [15/Nov/2023:08:37:14 EST] “GET /version HTTP/1.1” 200 166
Nov 15 08:37:14 dy-prod-foreman-proxy01 smart-proxy[1027]: - → /version
Nov 15 08:37:14 dy-prod-foreman-proxy01 smart-proxy[1027]: 172.18.171.7 - - [15/Nov/2023:08:37:14 EST] “GET /tftp/serverName HTTP/1.1” 200 29
Nov 15 08:37:14 dy-prod-foreman-proxy01 smart-proxy[1027]: - → /tftp/serverName
Nov 15 08:37:14 dy-prod-foreman-proxy01 pulpcore-api[1236]: pulp [3507d6f3-13c6-4779-89bf-8c8340d73364]: - - [15/Nov/2023:13:37:14 +0000] “GET /pulp/api/v3/status HTTP/1.1” 301 0 “-” “rest-client/2.1.0 (linux x86_64) ruby/2.7.8p225”
Nov 15 08:37:14 dy-prod-foreman-proxy01 smart-proxy[1027]: 172.18.171.7 - - [15/Nov/2023:08:37:14 EST] “GET /logs/?from_timestamp=0 HTTP/1.1” 200 48805
Nov 15 08:37:14 dy-prod-foreman-proxy01 smart-proxy[1027]: - → /logs/?from_timestamp=0
Nov 15 08:37:15 dy-prod-foreman-proxy01 smart-proxy[1027]: 172.18.171.7 - - [15/Nov/2023:08:37:14 EST] “GET /logs/?from_timestamp=0 HTTP/1.1” 200 49294
Nov 15 08:37:15 dy-prod-foreman-proxy01 smart-proxy[1027]: - → /logs/?from_timestamp=0
Nov 15 08:37:15 dy-prod-foreman-proxy01 smart-proxy[1027]: 172.18.171.7 - - [15/Nov/2023:08:37:14 EST] “GET /logs/?from_timestamp=0 HTTP/1.1” 200 49536
Nov 15 08:37:15 dy-prod-foreman-proxy01 smart-proxy[1027]: - → /logs/?from_timestamp=0
Nov 15 08:37:15 dy-prod-foreman-proxy01 pulpcore-api[1238]: pulp [3507d6f3-13c6-4779-89bf-8c8340d73364]: - - [15/Nov/2023:13:37:15 +0000] “GET /pulp/api/v3/status/ HTTP/1.1” 200 1827 “-” “rest-client/2.1.0 (linux x86_64) ruby/2.7.8p225”
Nov 15 08:37:15 dy-prod-foreman-proxy01 smart-proxy[1027]: 172.18.171.7 - - [15/Nov/2023:08:37:15 EST] “GET /logs/?from_timestamp=0 HTTP/1.1” 200 49779
Nov 15 08:37:15 dy-prod-foreman-proxy01 smart-proxy[1027]: - → /logs/?from_timestamp=0
Does anyone have an idea on what might be causing this “problem”? Seems to be a GUI glitch as we are still getting content from the proxy, but we have 4 other proxies and it’s never happened to any others.