Problem:
I am trying to sync a repository. The sync is working or has completed. However the progress of the task is showing at 0% for three days. It doesn’t show any movement, whatsoever.
Expected outcome:
Task progress should give a reasonable percentage of how much is complete and how much is pending.
Foreman and Proxy versions:
[root@katellohost ~]# rpm -qa | grep -i foreman
foreman-selinux-1.16.1-1.el7.noarch
foreman-release-scl-3-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-0.11.0-1.el7.noarch
foreman-postgresql-1.16.1-1.el7.noarch
foreman-release-1.16.1-1.el7.noarch
foreman-debug-1.16.1-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.1.3-5.el7.noarch
foreman-1.16.1-1.el7.noarch
foreman-compute-1.16.1-1.el7.noarch
foreman-cli-1.16.1-1.el7.noarch
foreman-installer-katello-3.5.1.1-1.el7.noarch
foreman-proxy-1.16.1-1.el7.noarch
tfm-rubygem-foreman-tasks-core-0.2.4-1.fm1_16.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.4-2.el7.noarch
tfm-rubygem-foreman-tasks-0.10.9-1.fm1_16.el7.noarch
tfm-rubygem-foreman_docker-3.2.1-1.fm1_16.el7.noarch
katellohost.example.com-foreman-proxy-1.0-1.noarch
katellohost.example.com-foreman-client-1.0-1.noarch
katellohost.example.com-foreman-proxy-client-1.0-1.noarch
foreman-installer-1.16.1-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.12-1.fm1_16.el7.noarch
[root@katellohost ~]#
Foreman and Proxy plugin versions:
Other relevant data:
| [root@katellohost ~]# hammer task list |
head -10 |
| ID |
NAME |
| ------------------------------------- |
------ |
| 6ef47da9-5690-4768-8f7b-f5b6d2ac0d14 |
|
| a29cc774-1547-491a-8f35-fd52ee009cce |
|
| 1265a5a1-e634-44e8-9ab7-169f49572f06 |
|
| f5350c83-46f4-4f89-bad2-c88346b4b653 |
|
| 9bbf5b83-2e56-4f8f-bea0-93a77b7c402b |
|
| d3b24bd0-660b-4d36-ad37-9c691f362a53 |
|
| 4f630158-48a3-45c8-be2e-501556a69f77 |
|
[root@katellohost ~]#
[root@katellohost ~]# hammer task progress --id f5350c83-46f4-4f89-bad2-c88346b4b653
[ ] [0%]
[ ] [0%]
[root@katellohost ~]# hammer task progress --id 1265a5a1-e634-44e8-9ab7-169f49572f06
[ ] [0%]
[ ] [0%]
As can be seen in the screenshot, the packages are showing up, but I cannot confirm whether the sync job is complete or not.
Hey,
if you go to $FOREMAN_URL/foreman_tasks/tasks/$TASK_UUID, where FOREMAN_URL is url of your foreman server and TASK_UUID is one of the listed task IDs and there click the “Dynflow console” button, you should get to a more detailed view about the task. Try looking around there.
Repository sync tasks usually just kick off the synchronization in pulp and the periodically poll for its status. Because the package numbers are going up, we can safely assume the tasks in pulp are running so the error is somewhere on our side.
Are other tasks being processed normally? Is the foreman-tasks service running?
Hey Adam,
I looked into the dynflow console and captured few screenshots. Honestly I am not that efficient in ruby to understand what is happening at the bare bone level, but hopefully this sheds some light.
Also, the number of packages is still the same. It is still what I showed you earlier in the screenshot. So number of packages is not increasing, thereby rendering the hypothesis that the tasks are running properly, wrong (or so I assume).
foreman-tasks service is running.
/usr/sbin/service-wait foreman-tasks status
● foreman-tasks.service - Foreman jobs daemon
Loaded: loaded (/usr/lib/systemd/system/foreman-tasks.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2018-04-20 03:27:31 EDT; 4 days ago
Docs: https://github.com/theforeman/foreman-tasks
Process: 14395 ExecStop=/usr/bin/foreman-tasks stop (code=exited, status=0/SUCCESS)
Process: 15684 ExecStart=/usr/bin/foreman-tasks start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/foreman-tasks.service
├─15724 dynflow_executor_monitor
└─15727 dynflow_executor
Apr 20 03:27:27 hostname systemd[1]: Starting Foreman jobs daemon...
Apr 20 03:27:30 hostname foreman-tasks[15684]: /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
Apr 20 03:27:30 hostname foreman-tasks[15684]: /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
Apr 20 03:27:30 hostname foreman-tasks[15684]: Dynflow Executor: start in progress
Apr 20 03:27:30 hostname foreman-tasks[15684]: dynflow_executor: process with pid 15727 started.
Apr 20 03:27:31 hostname systemd[1]: Started Foreman jobs daemon.
Success!
# ps aux | grep 15727
root 2949 0.0 0.0 112660 976 pts/0 S+ 04:40 0:00 grep --color=auto 15727
foreman 15727 0.4 2.9 1796200 481260 ? Sl Apr20 28:37 dynflow_executor
Screenshots here:
EDIT: I am tempted to cancel and see what happens. Do you think that is a viable proposition?
If you click into any of those “boxes” they will expand, showing additional details about it. If you click on 3: Actions::Pulp::Repository::Sync it should show a id of the task in pulp, if there’s any. You could then use pulp-admin tool to inspect the task’s state in pulp. Sadly I have only general idea about how pulp works so I can’t provide specific instructions.
Or you could just cancel the task, repository sync tasks should be safe to cancel.
