Help needed: List of all processes in a Katello deployment

Hey,

one thing I really like about Puma app server is how consistently it renames process titles:

foreman   9282  0.0  2.9 947964 293884 ?       Ssl  Dec02   0:48 puma 5.3.2 (unix:///run/foreman.sock) [foreman]
foreman   9308  0.0  4.1 992672 416576 ?       Sl   Dec02   0:39 puma: cluster worker 0: 9282 [foreman]
foreman   9314  0.1  5.3 1143056 536972 ?      Sl   Dec02   1:15 puma: cluster worker 1: 9282 [foreman]

It is very much clear that these processes are instances of foreman app, which is the master process and which are workers. It also renames them consistently which is actually something that is very much needed for the Prometheus Ruby client library if my patch gets accepted:

Now, in a different thread there is a user asking for help with increased memory problem and we struggle to understand which process does what. In this case, the suspicious process is named just rails and it is likely the Katello Event Daemon.

So what I propose is to find all of these processes and then make patches so process title is changed during startup so they appear nicely in utilities like top or ps. But since I do not understand all the details of Katello or other plugins, help me to identify with me all the Foreman and Katello related processes which are running on a Katello deployment (the snippet is from Satellite 6.10):

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0 194108  4592 ?        Ss   Dec02   0:18 /usr/lib/systemd/systemd --system --deserialize 26
root         2  0.0  0.0      0     0 ?        S    Dec02   0:00 [kthreadd]
root         4  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kworker/0:0H]
root         6  0.0  0.0      0     0 ?        S    Dec02   0:02 [ksoftirqd/0]
root         7  0.0  0.0      0     0 ?        S    Dec02   0:00 [migration/0]
root         8  0.0  0.0      0     0 ?        S    Dec02   0:00 [rcu_bh]
root         9  0.0  0.0      0     0 ?        S    Dec02   1:05 [rcu_sched]
root        10  0.0  0.0      0     0 ?        S<   Dec02   0:00 [lru-add-drain]
root        11  0.0  0.0      0     0 ?        S    Dec02   0:00 [watchdog/0]
root        12  0.0  0.0      0     0 ?        S    Dec02   0:00 [watchdog/1]
root        13  0.0  0.0      0     0 ?        S    Dec02   0:00 [migration/1]
root        14  0.0  0.0      0     0 ?        S    Dec02   0:00 [ksoftirqd/1]
root        16  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kworker/1:0H]
root        17  0.0  0.0      0     0 ?        S    Dec02   0:00 [watchdog/2]
root        18  0.0  0.0      0     0 ?        S    Dec02   0:00 [migration/2]
root        19  0.0  0.0      0     0 ?        S    Dec02   0:01 [ksoftirqd/2]
root        21  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kworker/2:0H]
root        22  0.0  0.0      0     0 ?        S    Dec02   0:00 [watchdog/3]
root        23  0.0  0.0      0     0 ?        S    Dec02   0:00 [migration/3]
root        24  0.0  0.0      0     0 ?        S    Dec02   0:01 [ksoftirqd/3]
root        26  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kworker/3:0H]
root        28  0.0  0.0      0     0 ?        S    Dec02   0:00 [kdevtmpfs]
root        29  0.0  0.0      0     0 ?        S<   Dec02   0:00 [netns]
root        30  0.0  0.0      0     0 ?        S    Dec02   0:00 [khungtaskd]
root        31  0.0  0.0      0     0 ?        S<   Dec02   0:00 [writeback]
root        32  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kintegrityd]
root        33  0.0  0.0      0     0 ?        S<   Dec02   0:00 [bioset]
root        34  0.0  0.0      0     0 ?        S<   Dec02   0:00 [bioset]
root        35  0.0  0.0      0     0 ?        S<   Dec02   0:00 [bioset]
root        36  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kblockd]
root        37  0.0  0.0      0     0 ?        S<   Dec02   0:00 [md]
root        38  0.0  0.0      0     0 ?        S<   Dec02   0:00 [edac-poller]
root        39  0.0  0.0      0     0 ?        S<   Dec02   0:00 [watchdogd]
root        48  0.0  0.0      0     0 ?        S    Dec02   0:46 [kswapd0]
root        49  0.0  0.0      0     0 ?        SN   Dec02   0:00 [ksmd]
root        50  0.0  0.0      0     0 ?        SN   Dec02   0:00 [khugepaged]
root        51  0.0  0.0      0     0 ?        S<   Dec02   0:00 [crypto]
root        59  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kthrotld]
root        62  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kmpath_rdacd]
root        63  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kaluad]
root        64  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kpsmoused]
root        66  0.0  0.0      0     0 ?        S<   Dec02   0:00 [ipv6_addrconf]
root        79  0.0  0.0      0     0 ?        S<   Dec02   0:00 [deferwq]
root       116  0.0  0.0      0     0 ?        S    Dec02   0:00 [kauditd]
root       298  0.0  0.0      0     0 ?        S<   Dec02   0:00 [firewire]
root       300  0.0  0.0      0     0 ?        S<   Dec02   0:00 [ata_sff]
root       306  0.0  0.0      0     0 ?        S    Dec02   0:00 [scsi_eh_0]
root       307  0.0  0.0      0     0 ?        S<   Dec02   0:00 [scsi_tmf_0]
root       308  0.0  0.0      0     0 ?        S    Dec02   0:00 [scsi_eh_1]
root       312  0.0  0.0      0     0 ?        S<   Dec02   0:00 [scsi_tmf_1]
root       313  0.0  0.0      0     0 ?        S    Dec02   0:00 [scsi_eh_2]
root       314  0.0  0.0      0     0 ?        S<   Dec02   0:00 [scsi_tmf_2]
root       315  0.0  0.0      0     0 ?        S    Dec02   0:00 [scsi_eh_3]
root       316  0.0  0.0      0     0 ?        S<   Dec02   0:00 [scsi_tmf_3]
root       317  0.0  0.0      0     0 ?        S    Dec02   0:00 [scsi_eh_4]
root       318  0.0  0.0      0     0 ?        S<   Dec02   0:00 [scsi_tmf_4]
root       319  0.0  0.0      0     0 ?        S    Dec02   0:00 [scsi_eh_5]
root       320  0.0  0.0      0     0 ?        S<   Dec02   0:00 [scsi_tmf_5]
root       327  0.0  0.0      0     0 ?        S    Dec02   0:00 [scsi_eh_6]
root       328  0.0  0.0      0     0 ?        S<   Dec02   0:00 [scsi_tmf_6]
root       329  0.0  0.0      0     0 ?        S    Dec02   0:20 [usb-storage]
root       330  0.0  0.0      0     0 ?        S<   Dec02   0:00 [nvkm-disp]
root       360  0.0  0.0      0     0 ?        S<   Dec02   0:00 [ttm_swap]
root       366  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kworker/1:1H]
root       367  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kworker/3:1H]
root       369  0.0  0.0      0     0 ?        S<   Dec02   0:08 [kworker/2:1H]
root       373  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kworker/0:1H]
root       415  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kdmflush]
root       427  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kdmflush]
root       428  0.0  0.0      0     0 ?        S<   Dec02   0:00 [bioset]
root       441  0.0  0.0      0     0 ?        S<   Dec02   0:00 [bioset]
root       442  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfsalloc]
root       443  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs_mru_cache]
root       444  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-buf/dm-0]
root       445  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-data/dm-0]
root       446  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-conv/dm-0]
root       447  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-cil/dm-0]
root       448  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-reclaim/dm-]
root       449  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-log/dm-0]
root       450  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-eofblocks/d]
root       451  0.0  0.0      0     0 ?        S    Dec02   0:21 [xfsaild/dm-0]
root       531  0.0  0.0  40216  6348 ?        Ss   Dec02   0:02 /usr/lib/systemd/systemd-journald
root       554  0.0  0.0 201104     0 ?        Ss   Dec02   0:00 /usr/sbin/lvmetad -f
root       564  0.0  0.0  48892    36 ?        Ss   Dec02   0:00 /usr/lib/systemd/systemd-udevd
root       609  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kvm-irqfd-clean]
root       647  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-buf/sda1]
root       648  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-data/sda1]
root       649  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-conv/sda1]
root       650  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-cil/sda1]
root       651  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-reclaim/sda]
root       652  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-log/sda1]
root       653  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-eofblocks/s]
root       654  0.0  0.0      0     0 ?        S    Dec02   0:00 [xfsaild/sda1]
root       704  0.0  0.0  55532   236 ?        S<sl Dec02   0:00 /sbin/auditd
polkitd    730  0.0  0.0 612332  2880 ?        Ssl  Dec02   0:00 /usr/lib/polkit-1/polkitd --no-debug
root       731  0.0  0.0  27992  1028 ?        Ss   Dec02   0:02 /usr/lib/systemd/systemd-logind
root       732  0.0  0.0  21536   368 ?        Ss   Dec02   0:03 /usr/sbin/irqbalance --foreground
dbus       739  0.0  0.0  66572  1204 ?        Ssl  Dec02   0:05 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
chrony     747  0.0  0.0  22536   592 ?        S    Dec02   0:00 /usr/sbin/chronyd
root       784  0.0  0.0 474312  3184 ?        Ssl  Dec02   0:02 /usr/sbin/NetworkManager --no-daemon
root       789  0.0  0.0 110204     8 tty1     Ss+  Dec02   0:00 /sbin/agetty --noclear tty1 linux
root       851  0.0  0.0 102988   868 ?        S    Dec02   0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-enp1s0.pid -lf /var
root      1071  0.0  0.0 586428  2632 ?        Ssl  Dec02   0:06 /usr/bin/python2 -Es /usr/sbin/tuned -l -P
root      1074  0.0  0.0 112984   248 ?        Ss   Dec02   0:00 /usr/sbin/sshd -D
root      1076  0.0  0.0 216400  3060 ?        Ssl  Dec02   0:03 /usr/sbin/rsyslogd -n
root      1105  0.0  0.0 115808   376 ?        Ss   Dec02   0:00 /usr/bin/rhsmcertd
root      1636  0.0  0.0  89784   248 ?        Ss   Dec02   0:00 /usr/libexec/postfix/master -w
postfix   1639  0.0  0.0  90064   528 ?        S    Dec02   0:00 qmgr -l -t unix -u
root      2699  0.0  0.0 126420   452 ?        Ss   Dec02   0:01 /usr/sbin/crond -n
root      2727  0.0  0.0 108068   472 ?        Ss   Dec02   0:00 rhnsd
redis     8570  0.1  0.0 156616  5660 ?        Ssl  Dec02   1:31 /opt/rh/rh-redis5/root/usr/bin/redis-server 127.0.0.1:6379
postgres  8744  0.0  0.4 834368 46504 ?        Ss   Dec02   0:03 postmaster -D /var/opt/rh/rh-postgresql12/lib/pgsql/data
postgres  8749  0.0  0.0 252084   980 ?        Ss   Dec02   0:00 postgres: logger
postgres  8751  0.0  1.2 834640 130420 ?       Ss   Dec02   0:02 postgres: checkpointer
postgres  8752  0.0  0.1 834504 16312 ?        Ss   Dec02   0:01 postgres: background writer
postgres  8753  0.0  0.1 834368 17008 ?        Ss   Dec02   0:07 postgres: walwriter
postgres  8754  0.0  0.0 835324  1644 ?        Ss   Dec02   0:01 postgres: autovacuum launcher
postgres  8755  0.0  0.0 252936  1384 ?        Ss   Dec02   0:27 postgres: stats collector
postgres  8756  0.0  0.0 835212  1100 ?        Ss   Dec02   0:00 postgres: logical replication launcher
tomcat    8907  0.4  9.5 8279496 956224 ?      Ssl  Dec02   4:01 /usr/lib/jvm/jre-11/bin/java -Xms1024m -Xmx4096m -Djava.security.auth.login.config=/usr/share/
postgres  9120  0.0  0.0 836796  9068 ?        Ss   Dec02   0:00 postgres: candlepin candlepin 127.0.0.1(35578) idle
postgres  9121  0.0  0.0 835848  2528 ?        Ss   Dec02   0:00 postgres: candlepin candlepin 127.0.0.1(35580) idle
postgres  9122  0.0  0.0 836916  9180 ?        Ss   Dec02   0:00 postgres: candlepin candlepin 127.0.0.1(35582) idle
root      9202  0.0  0.0 184228  1052 ?        Ss   Dec02   0:02 /usr/sbin/httpd -DFOREGROUND
apache    9203  0.0  0.0 184928  3552 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9204  0.0  0.0 185064  3560 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9205  0.0  0.0 184896  3664 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9206  0.0  0.0 184936  3452 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9207  0.0  0.0 184992  3748 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9208  0.0  0.0 185216  3960 ?        S    Dec02   0:03 /usr/sbin/httpd -DFOREGROUND
apache    9209  0.0  0.0 184944  3816 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9210  0.0  0.0 184892  3732 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
foreman   9282  0.0  2.9 947964 293884 ?       Ssl  Dec02   0:48 puma 5.3.2 (unix:///run/foreman.sock) [foreman]
postgres  9304  0.0  0.0 836856  5652 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
foreman   9308  0.0  4.1 992672 416448 ?       Sl   Dec02   0:39 puma: cluster worker 0: 9282 [foreman]
foreman   9314  0.1  5.3 1143056 536972 ?      Sl   Dec02   1:15 puma: cluster worker 1: 9282 [foreman]
postgres  9320  0.0  0.0 835716  2448 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
postgres  9321  0.0  0.0 835716  2552 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
postgres  9325  0.0  0.0 836620  9884 ?        Ss   Dec02   0:02 postgres: foreman foreman [local] idle
postgres  9326  0.0  0.1 836600 10060 ?        Ss   Dec02   0:03 postgres: foreman foreman [local] idle
postgres  9330  0.0  0.0 836348  6308 ?        Ss   Dec02   0:02 postgres: foreman foreman [local] idle
postgres  9331  0.0  0.0 836488  6532 ?        Ss   Dec02   0:02 postgres: foreman foreman [local] idle
postgres  9332  0.0  0.0 837024  6704 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
postgres  9333  0.0  0.0 837024  6064 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
foreman   9389  0.2  4.0 988712 411220 ?       Ssl  Dec02   1:58 sidekiq 5.2.7  [0 of 1 busy]
postgres  9415  0.0  0.0 835720  3436 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
postgres  9419  0.0  0.1 836672 17556 ?        Ss   Dec02   0:02 postgres: foreman foreman [local] idle
postgres  9420  0.0  0.1 836624 17676 ?        Ss   Dec02   0:02 postgres: foreman foreman [local] idle
foreman   9459  0.6  6.7 1216080 676400 ?      Ssl  Dec02   6:25 sidekiq 5.2.7  [0 of 2 busy]
postgres  9484  0.0  0.0 835720  3452 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
postgres  9488  0.0  0.1 836672 11648 ?        Ss   Dec02   0:01 postgres: foreman foreman [local] idle
postgres  9489  0.0  0.1 836624 11468 ?        Ss   Dec02   0:01 postgres: foreman foreman [local] idle
foreman   9536  0.1  3.8 984064 388352 ?       Ssl  Dec02   1:19 sidekiq 5.2.7  [0 of 2 busy]
postgres  9563  0.0  0.0 835720  2408 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
postgres  9567  0.0  0.0 836004  5276 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
postgres  9568  0.0  0.0 835836  5184 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
pulp      9721  0.0  0.4 498476 46828 ?        Ss   Dec02   0:17 /usr/bin/python3 /usr/bin/pulpcore-worker
pulp      9757  0.0  0.6 498416 68936 ?        Ss   Dec02   0:17 /usr/bin/python3 /usr/bin/pulpcore-worker
pulp      9834  0.0  0.0 218772  8580 ?        Ss   Dec02   0:08 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-lo
pulp      9838  0.0 10.4 1468268 1047116 ?     S    Dec02   0:32 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-lo
pulp      9848  0.0  4.1 836188 417376 ?       S    Dec02   0:08 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-lo
pulp      9857  0.0  2.3 637972 231672 ?       S    Dec02   0:13 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-lo
pulp      9874  0.0  2.0 626308 207076 ?       S    Dec02   0:07 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-lo
pulp      9875  0.0  3.7 792996 372400 ?       S    Dec02   0:25 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-lo
pulp      9920  0.0  0.0 267680  8280 ?        Ss   Dec02   0:09 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp
pulp      9924  0.0  0.9 668120 91432 ?        Sl   Dec02   0:38 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp
pulp      9928  0.0  0.7 665208 74972 ?        Sl   Dec02   0:36 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp
pulp      9931  0.0  0.7 665204 79428 ?        Sl   Dec02   0:36 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp
pulp      9938  0.0  0.7 665204 75816 ?        Sl   Dec02   0:36 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp
pulp      9939  0.0  0.8 668160 88020 ?        Sl   Dec02   0:40 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp
pulp      9941  0.0  0.7 665204 78296 ?        Sl   Dec02   0:36 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp
pulp      9945  0.0  0.7 665208 73064 ?        Sl   Dec02   0:36 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp
pulp      9946  0.0  0.7 665204 72844 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp
pulp      9947  0.0  0.7 665204 75728 ?        Sl   Dec02   0:36 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp
foreman+  9968  0.0  0.6 1374256 64416 ?       Ssl  Dec02   0:24 ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize
postgres  9989  0.0  0.2 836544 21308 ?        Ss   Dec02   0:04 postgres: pulp pulpcore ::1(52986) idle
postgres  9990  0.0  0.2 836544 21288 ?        Ss   Dec02   0:04 postgres: pulp pulpcore ::1(52988) idle
postgres 10136  0.0  0.1 835692 19056 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(53018) idle
postgres 10137  0.0  0.0 835696  5352 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53020) idle
postgres 10141  0.0  0.0 835696  8544 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53022) idle
postgres 10142  0.0  0.1 835836 10244 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53024) idle
postgres 10143  0.0  0.1 835696 16464 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(53026) idle
postgres 10147  0.0  0.1 835692 18776 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(53028) idle
postgres 10148  0.0  0.1 835692 19064 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(53030) idle
postgres 10149  0.0  0.0 835692  8052 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53032) idle
postgres 10293  0.0  0.1 835696 13076 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53070) idle
postgres 10294  0.0  0.1 835696 18932 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(53072) idle
postgres 10493  0.0  0.0 835696  5292 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53078) idle
postgres 10494  0.0  0.0 835696  5140 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53080) idle
postgres 10495  0.0  0.1 835696 18412 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(53082) idle
postgres 10496  0.0  0.0 835836  8052 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53084) idle
postgres 10497  0.0  0.1 835696 18136 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(53086) idle
postgres 10498  0.0  0.0 835696  5132 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53088) idle
apache   10955  0.0  0.0 184764  3492 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache   10956  0.0  0.0 184900  3836 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache   10957  0.0  0.0 184940  3816 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
postgres 11265  0.0  0.1 836612 17672 ?        Ss   Dec02   0:02 postgres: foreman foreman [local] idle
postgres 11430  0.0  0.1 838604 14712 ?        Ss   Dec02   0:08 postgres: foreman foreman [local] idle
postgres 11635  0.0  0.2 836984 24128 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(54086) idle
postgres 11637  0.0  0.2 836984 23580 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(54092) idle
root     15604  0.0  0.0      0     0 ?        R    Dec02   0:02 [kworker/3:3]
root     16896  0.0  0.0      0     0 ?        S    01:00   0:01 [kworker/3:0]
root     20190  0.0  0.0      0     0 ?        S    05:47   0:00 [kworker/u32:2]
postfix  21312  0.0  0.0  89888  4096 ?        S    07:18   0:00 pickup -l -t unix -u
root     21363  0.0  0.0      0     0 ?        S    07:27   0:00 [kworker/u32:0]
root     21686  0.0  0.0      0     0 ?        S    07:51   0:00 [kworker/2:0]
root     21694  0.0  0.0      0     0 ?        S    07:52   0:00 [kworker/0:0]
root     21727  0.0  0.0      0     0 ?        S    07:58   0:00 [kworker/1:1]
root     21929  0.0  0.0      0     0 ?        S    08:03   0:00 [kworker/0:1]
root     21942  0.0  0.0      0     0 ?        S    08:05   0:00 [kworker/2:1]
postgres 21959  0.0  0.0 835720  4340 ?        Ss   08:08   0:00 postgres: candlepin candlepin 127.0.0.1(38828) idle
root     21961  0.0  0.0      0     0 ?        S    08:09   0:00 [kworker/0:2]
postgres 21966  0.0  0.0 835720  4336 ?        Ss   08:09   0:00 postgres: candlepin candlepin 127.0.0.1(38830) idle
root     21970  0.0  0.0      0     0 ?        S    08:10   0:00 [kworker/1:2]
postgres 21973  0.0  0.0 835720  4340 ?        Ss   08:11   0:00 postgres: candlepin candlepin 127.0.0.1(38832) idle
postgres 21979  0.0  0.0 835720  4340 ?        Ss   08:12   0:00 postgres: candlepin candlepin 127.0.0.1(38834) idle
root     21985  0.0  0.0      0     0 ?        S    08:13   0:00 [kworker/3:1]
postgres 21987  0.0  0.0 835720  4340 ?        Ss   08:13   0:00 postgres: candlepin candlepin 127.0.0.1(38836) idle
root     21992  9.5  0.0 153120  5948 ?        Ds   08:14   0:00 sshd: root@pts/0
root     21995  0.0  0.0 115544  1976 pts/0    Ss   08:14   0:00 -bash
root     22012  0.0  0.0 155448  1876 pts/0    R+   08:14   0:00 ps axu
root     29790  0.0  0.4 384844 49468 ?        Ssl  Dec02   0:06 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent --no-daemonize
root     30884  0.0  0.0      0     0 ?        S<   Dec02   0:00 [bioset]
root     31357  0.0  0.0      0     0 ?        S    Dec02   0:04 [kworker/1:0]
[root@zzzap ~]# ps axu | grep puma
foreman   9282  0.0  2.9 947964 293884 ?       Ssl  Dec02   0:48 puma 5.3.2 (unix:///run/foreman.sock) [foreman]
foreman   9308  0.0  4.1 992672 416576 ?       Sl   Dec02   0:39 puma: cluster worker 0: 9282 [foreman]
foreman   9314  0.1  5.3 1143056 536972 ?      Sl   Dec02   1:15 puma: cluster worker 1: 9282 [foreman]
root     22017  0.0  0.0 112812   980 pts/0    S+   08:15   0:00 grep --color=auto puma
[root@zzzap ~]# ^C
[root@zzzap ~]#
[root@zzzap ~]# ps axu | grep puma
foreman   9282  0.0  2.9 947964 293884 ?       Ssl  Dec02   0:48 puma 5.3.2 (unix:///run/foreman.sock) [foreman]
foreman   9308  0.0  4.1 992672 417164 ?       Sl   Dec02   0:39 puma: cluster worker 0: 9282 [foreman]
foreman   9314  0.1  5.3 1143056 536972 ?      Sl   Dec02   1:15 puma: cluster worker 1: 9282 [foreman]
root     22041  0.0  0.0 112812   980 pts/0    S+   08:19   0:00 grep --color=auto puma
[root@zzzap ~]# ps axuwww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0 194108  4592 ?        Ss   Dec02   0:18 /usr/lib/systemd/systemd --system --deserialize 26
root         2  0.0  0.0      0     0 ?        S    Dec02   0:00 [kthreadd]
root         4  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kworker/0:0H]
root         6  0.0  0.0      0     0 ?        S    Dec02   0:02 [ksoftirqd/0]
root         7  0.0  0.0      0     0 ?        S    Dec02   0:00 [migration/0]
root         8  0.0  0.0      0     0 ?        S    Dec02   0:00 [rcu_bh]
root         9  0.0  0.0      0     0 ?        S    Dec02   1:05 [rcu_sched]
root        10  0.0  0.0      0     0 ?        S<   Dec02   0:00 [lru-add-drain]
root        11  0.0  0.0      0     0 ?        S    Dec02   0:00 [watchdog/0]
root        12  0.0  0.0      0     0 ?        S    Dec02   0:00 [watchdog/1]
root        13  0.0  0.0      0     0 ?        S    Dec02   0:00 [migration/1]
root        14  0.0  0.0      0     0 ?        S    Dec02   0:00 [ksoftirqd/1]
root        16  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kworker/1:0H]
root        17  0.0  0.0      0     0 ?        S    Dec02   0:00 [watchdog/2]
root        18  0.0  0.0      0     0 ?        S    Dec02   0:00 [migration/2]
root        19  0.0  0.0      0     0 ?        S    Dec02   0:01 [ksoftirqd/2]
root        21  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kworker/2:0H]
root        22  0.0  0.0      0     0 ?        S    Dec02   0:00 [watchdog/3]
root        23  0.0  0.0      0     0 ?        S    Dec02   0:00 [migration/3]
root        24  0.0  0.0      0     0 ?        S    Dec02   0:01 [ksoftirqd/3]
root        26  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kworker/3:0H]
root        28  0.0  0.0      0     0 ?        S    Dec02   0:00 [kdevtmpfs]
root        29  0.0  0.0      0     0 ?        S<   Dec02   0:00 [netns]
root        30  0.0  0.0      0     0 ?        S    Dec02   0:00 [khungtaskd]
root        31  0.0  0.0      0     0 ?        S<   Dec02   0:00 [writeback]
root        32  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kintegrityd]
root        33  0.0  0.0      0     0 ?        S<   Dec02   0:00 [bioset]
root        34  0.0  0.0      0     0 ?        S<   Dec02   0:00 [bioset]
root        35  0.0  0.0      0     0 ?        S<   Dec02   0:00 [bioset]
root        36  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kblockd]
root        37  0.0  0.0      0     0 ?        S<   Dec02   0:00 [md]
root        38  0.0  0.0      0     0 ?        S<   Dec02   0:00 [edac-poller]
root        39  0.0  0.0      0     0 ?        S<   Dec02   0:00 [watchdogd]
root        48  0.0  0.0      0     0 ?        S    Dec02   0:46 [kswapd0]
root        49  0.0  0.0      0     0 ?        SN   Dec02   0:00 [ksmd]
root        50  0.0  0.0      0     0 ?        SN   Dec02   0:00 [khugepaged]
root        51  0.0  0.0      0     0 ?        S<   Dec02   0:00 [crypto]
root        59  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kthrotld]
root        62  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kmpath_rdacd]
root        63  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kaluad]
root        64  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kpsmoused]
root        66  0.0  0.0      0     0 ?        S<   Dec02   0:00 [ipv6_addrconf]
root        79  0.0  0.0      0     0 ?        S<   Dec02   0:00 [deferwq]
root       116  0.0  0.0      0     0 ?        S    Dec02   0:00 [kauditd]
root       298  0.0  0.0      0     0 ?        S<   Dec02   0:00 [firewire]
root       300  0.0  0.0      0     0 ?        S<   Dec02   0:00 [ata_sff]
root       306  0.0  0.0      0     0 ?        S    Dec02   0:00 [scsi_eh_0]
root       307  0.0  0.0      0     0 ?        S<   Dec02   0:00 [scsi_tmf_0]
root       308  0.0  0.0      0     0 ?        S    Dec02   0:00 [scsi_eh_1]
root       312  0.0  0.0      0     0 ?        S<   Dec02   0:00 [scsi_tmf_1]
root       313  0.0  0.0      0     0 ?        S    Dec02   0:00 [scsi_eh_2]
root       314  0.0  0.0      0     0 ?        S<   Dec02   0:00 [scsi_tmf_2]
root       315  0.0  0.0      0     0 ?        S    Dec02   0:00 [scsi_eh_3]
root       316  0.0  0.0      0     0 ?        S<   Dec02   0:00 [scsi_tmf_3]
root       317  0.0  0.0      0     0 ?        S    Dec02   0:00 [scsi_eh_4]
root       318  0.0  0.0      0     0 ?        S<   Dec02   0:00 [scsi_tmf_4]
root       319  0.0  0.0      0     0 ?        S    Dec02   0:00 [scsi_eh_5]
root       320  0.0  0.0      0     0 ?        S<   Dec02   0:00 [scsi_tmf_5]
root       327  0.0  0.0      0     0 ?        S    Dec02   0:00 [scsi_eh_6]
root       328  0.0  0.0      0     0 ?        S<   Dec02   0:00 [scsi_tmf_6]
root       329  0.0  0.0      0     0 ?        S    Dec02   0:20 [usb-storage]
root       330  0.0  0.0      0     0 ?        S<   Dec02   0:00 [nvkm-disp]
root       360  0.0  0.0      0     0 ?        S<   Dec02   0:00 [ttm_swap]
root       366  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kworker/1:1H]
root       367  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kworker/3:1H]
root       369  0.0  0.0      0     0 ?        S<   Dec02   0:08 [kworker/2:1H]
root       373  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kworker/0:1H]
root       415  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kdmflush]
root       427  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kdmflush]
root       428  0.0  0.0      0     0 ?        S<   Dec02   0:00 [bioset]
root       441  0.0  0.0      0     0 ?        S<   Dec02   0:00 [bioset]
root       442  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfsalloc]
root       443  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs_mru_cache]
root       444  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-buf/dm-0]
root       445  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-data/dm-0]
root       446  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-conv/dm-0]
root       447  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-cil/dm-0]
root       448  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-reclaim/dm-]
root       449  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-log/dm-0]
root       450  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-eofblocks/d]
root       451  0.0  0.0      0     0 ?        S    Dec02   0:21 [xfsaild/dm-0]
root       531  0.0  0.0  40216  6348 ?        Ss   Dec02   0:02 /usr/lib/systemd/systemd-journald
root       554  0.0  0.0 201104     0 ?        Ss   Dec02   0:00 /usr/sbin/lvmetad -f
root       564  0.0  0.0  48892    36 ?        Ss   Dec02   0:00 /usr/lib/systemd/systemd-udevd
root       609  0.0  0.0      0     0 ?        S<   Dec02   0:00 [kvm-irqfd-clean]
root       647  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-buf/sda1]
root       648  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-data/sda1]
root       649  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-conv/sda1]
root       650  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-cil/sda1]
root       651  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-reclaim/sda]
root       652  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-log/sda1]
root       653  0.0  0.0      0     0 ?        S<   Dec02   0:00 [xfs-eofblocks/s]
root       654  0.0  0.0      0     0 ?        S    Dec02   0:00 [xfsaild/sda1]
root       704  0.0  0.0  55532   236 ?        S<sl Dec02   0:00 /sbin/auditd
polkitd    730  0.0  0.0 612332  2880 ?        Ssl  Dec02   0:00 /usr/lib/polkit-1/polkitd --no-debug
root       731  0.0  0.0  27992  1028 ?        Ss   Dec02   0:02 /usr/lib/systemd/systemd-logind
root       732  0.0  0.0  21536   368 ?        Ss   Dec02   0:03 /usr/sbin/irqbalance --foreground
dbus       739  0.0  0.0  66572  1204 ?        Ssl  Dec02   0:05 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
chrony     747  0.0  0.0  22536   592 ?        S    Dec02   0:00 /usr/sbin/chronyd
root       784  0.0  0.0 474312  3184 ?        Ssl  Dec02   0:02 /usr/sbin/NetworkManager --no-daemon
root       789  0.0  0.0 110204     8 tty1     Ss+  Dec02   0:00 /sbin/agetty --noclear tty1 linux
root       851  0.0  0.0 102988   868 ?        S    Dec02   0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-enp1s0.pid -lf /var/lib/NetworkManager/dhclient-a7e556ca-ab1c-4b40-92f8-34f8afd59b72-enp1s0.lease -cf /var/lib/NetworkManager/dhclient-enp1s0.conf enp1s0
root      1071  0.0  0.0 586428  2632 ?        Ssl  Dec02   0:06 /usr/bin/python2 -Es /usr/sbin/tuned -l -P
root      1074  0.0  0.0 112984   248 ?        Ss   Dec02   0:00 /usr/sbin/sshd -D
root      1076  0.0  0.0 216400  3060 ?        Ssl  Dec02   0:03 /usr/sbin/rsyslogd -n
root      1105  0.0  0.0 115808   376 ?        Ss   Dec02   0:00 /usr/bin/rhsmcertd
root      1636  0.0  0.0  89784   248 ?        Ss   Dec02   0:00 /usr/libexec/postfix/master -w
postfix   1639  0.0  0.0  90064   528 ?        S    Dec02   0:00 qmgr -l -t unix -u
root      2699  0.0  0.0 126420   452 ?        Ss   Dec02   0:01 /usr/sbin/crond -n
root      2727  0.0  0.0 108068   472 ?        Ss   Dec02   0:00 rhnsd
redis     8570  0.1  0.0 156616  5668 ?        Ssl  Dec02   1:32 /opt/rh/rh-redis5/root/usr/bin/redis-server 127.0.0.1:6379
postgres  8744  0.0  0.4 834368 46504 ?        Ss   Dec02   0:03 postmaster -D /var/opt/rh/rh-postgresql12/lib/pgsql/data
postgres  8749  0.0  0.0 252084   980 ?        Ss   Dec02   0:00 postgres: logger
postgres  8751  0.0  1.2 834640 130420 ?       Ss   Dec02   0:02 postgres: checkpointer
postgres  8752  0.0  0.1 834504 16312 ?        Ss   Dec02   0:01 postgres: background writer
postgres  8753  0.0  0.1 834368 17008 ?        Ss   Dec02   0:07 postgres: walwriter
postgres  8754  0.0  0.0 835324  1644 ?        Ss   Dec02   0:01 postgres: autovacuum launcher
postgres  8755  0.0  0.0 252936  1384 ?        Ss   Dec02   0:27 postgres: stats collector
postgres  8756  0.0  0.0 835212  1100 ?        Ss   Dec02   0:00 postgres: logical replication launcher
tomcat    8907  0.4  9.5 8279496 956400 ?      Ssl  Dec02   4:03 /usr/lib/jvm/jre-11/bin/java -Xms1024m -Xmx4096m -Djava.security.auth.login.config=/usr/share/tomcat/conf/login.config -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
postgres  9120  0.0  0.0 836796  9068 ?        Ss   Dec02   0:00 postgres: candlepin candlepin 127.0.0.1(35578) idle
postgres  9121  0.0  0.0 835848  2528 ?        Ss   Dec02   0:00 postgres: candlepin candlepin 127.0.0.1(35580) idle
postgres  9122  0.0  0.0 836916  9180 ?        Ss   Dec02   0:00 postgres: candlepin candlepin 127.0.0.1(35582) idle
root      9202  0.0  0.0 184228  1052 ?        Ss   Dec02   0:02 /usr/sbin/httpd -DFOREGROUND
apache    9203  0.0  0.0 184928  3552 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9204  0.0  0.0 185064  3560 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9205  0.0  0.0 184896  3664 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9206  0.0  0.0 184936  3452 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9207  0.0  0.0 184992  3748 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9208  0.0  0.0 185216  3960 ?        S    Dec02   0:03 /usr/sbin/httpd -DFOREGROUND
apache    9209  0.0  0.0 184944  3816 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9210  0.0  0.0 184892  3732 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
foreman   9282  0.0  2.9 947964 293884 ?       Ssl  Dec02   0:48 puma 5.3.2 (unix:///run/foreman.sock) [foreman]
postgres  9304  0.0  0.0 836856  5652 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
foreman   9308  0.0  4.1 992672 418952 ?       Sl   Dec02   0:39 puma: cluster worker 0: 9282 [foreman]
foreman   9314  0.1  5.3 1143056 536972 ?      Sl   Dec02   1:15 puma: cluster worker 1: 9282 [foreman]
postgres  9320  0.0  0.0 835716  2448 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
postgres  9321  0.0  0.0 835716  2552 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
postgres  9325  0.0  0.0 836620  9884 ?        Ss   Dec02   0:02 postgres: foreman foreman [local] idle
postgres  9326  0.0  0.1 836600 10060 ?        Ss   Dec02   0:03 postgres: foreman foreman [local] idle
postgres  9330  0.0  0.0 836348  6308 ?        Ss   Dec02   0:02 postgres: foreman foreman [local] idle
postgres  9331  0.0  0.0 836488  6532 ?        Ss   Dec02   0:02 postgres: foreman foreman [local] idle
postgres  9332  0.0  0.0 837024  6704 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
postgres  9333  0.0  0.0 837024  6064 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
foreman   9389  0.2  4.0 988712 411220 ?       Ssl  Dec02   1:59 sidekiq 5.2.7  [0 of 1 busy]
postgres  9415  0.0  0.0 835720  3436 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
postgres  9419  0.0  0.1 836672 17556 ?        Ss   Dec02   0:02 postgres: foreman foreman [local] idle
postgres  9420  0.0  0.1 836624 17676 ?        Ss   Dec02   0:02 postgres: foreman foreman [local] idle
foreman   9459  0.6  6.7 1216080 676400 ?      Ssl  Dec02   6:26 sidekiq 5.2.7  [0 of 2 busy]
postgres  9484  0.0  0.0 835720  3452 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
postgres  9488  0.0  0.1 836672 11648 ?        Ss   Dec02   0:01 postgres: foreman foreman [local] idle
postgres  9489  0.0  0.1 836624 11468 ?        Ss   Dec02   0:01 postgres: foreman foreman [local] idle
foreman   9536  0.1  3.8 984064 388352 ?       Ssl  Dec02   1:20 sidekiq 5.2.7  [0 of 2 busy]
postgres  9563  0.0  0.0 835720  2408 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
postgres  9567  0.0  0.0 836004  5276 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
postgres  9568  0.0  0.0 835836  5184 ?        Ss   Dec02   0:00 postgres: foreman foreman [local] idle
pulp      9721  0.0  0.4 498476 46828 ?        Ss   Dec02   0:17 /usr/bin/python3 /usr/bin/pulpcore-worker
pulp      9757  0.0  0.6 498416 68936 ?        Ss   Dec02   0:17 /usr/bin/python3 /usr/bin/pulpcore-worker
pulp      9834  0.0  0.0 218772  8580 ?        Ss   Dec02   0:08 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
pulp      9838  0.0 10.4 1468268 1047116 ?     S    Dec02   0:32 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
pulp      9848  0.0  4.1 836188 417376 ?       S    Dec02   0:08 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
pulp      9857  0.0  2.3 637972 231672 ?       S    Dec02   0:13 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
pulp      9874  0.0  2.0 626308 207076 ?       S    Dec02   0:07 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
pulp      9875  0.0  3.7 792996 372400 ?       S    Dec02   0:25 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
pulp      9920  0.0  0.0 267680  8280 ?        Ss   Dec02   0:09 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9924  0.0  0.9 668120 91432 ?        Sl   Dec02   0:39 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9928  0.0  0.7 665208 74972 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9931  0.0  0.7 665204 79428 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9938  0.0  0.7 665204 75816 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9939  0.0  0.8 668160 88020 ?        Sl   Dec02   0:40 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9941  0.0  0.7 665204 78296 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9945  0.0  0.7 665208 73064 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9946  0.0  0.7 665204 72844 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9947  0.0  0.7 665204 75728 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
foreman+  9968  0.0  0.6 1374256 64416 ?       Ssl  Dec02   0:25 ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize
postgres  9989  0.0  0.2 836544 21572 ?        Ss   Dec02   0:04 postgres: pulp pulpcore ::1(52986) idle
postgres  9990  0.0  0.2 836544 21288 ?        Ss   Dec02   0:04 postgres: pulp pulpcore ::1(52988) idle
postgres 10136  0.0  0.1 835692 19056 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(53018) idle
postgres 10137  0.0  0.0 835696  5352 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53020) idle
postgres 10141  0.0  0.0 835696  8544 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53022) idle
postgres 10142  0.0  0.1 835836 10244 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53024) idle
postgres 10143  0.0  0.1 835696 16464 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(53026) idle
postgres 10147  0.0  0.1 835692 18776 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(53028) idle
postgres 10148  0.0  0.1 835692 19064 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(53030) idle
postgres 10149  0.0  0.0 835692  8052 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53032) idle
postgres 10293  0.0  0.1 835696 13340 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53070) idle
postgres 10294  0.0  0.1 835696 18932 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(53072) idle
postgres 10493  0.0  0.0 835696  5292 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53078) idle
postgres 10494  0.0  0.0 835696  5140 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53080) idle
postgres 10495  0.0  0.1 835696 18412 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(53082) idle
postgres 10496  0.0  0.0 835836  8052 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53084) idle
postgres 10497  0.0  0.1 835696 18136 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(53086) idle
postgres 10498  0.0  0.0 835696  5132 ?        Ss   Dec02   0:02 postgres: pulp pulpcore ::1(53088) idle
apache   10955  0.0  0.0 184764  3492 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache   10956  0.0  0.0 184900  3836 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache   10957  0.0  0.0 184940  3816 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
postgres 11265  0.0  0.1 836612 17672 ?        Ss   Dec02   0:02 postgres: foreman foreman [local] idle
postgres 11430  0.0  0.1 838604 14712 ?        Ss   Dec02   0:08 postgres: foreman foreman [local] idle
postgres 11635  0.0  0.2 836984 24128 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(54086) idle
postgres 11637  0.0  0.2 836984 23580 ?        Ss   Dec02   0:03 postgres: pulp pulpcore ::1(54092) idle
root     15604  0.0  0.0      0     0 ?        S    Dec02   0:02 [kworker/3:3]
root     16896  0.0  0.0      0     0 ?        S    01:00   0:01 [kworker/3:0]
root     20190  0.0  0.0      0     0 ?        S    05:47   0:00 [kworker/u32:2]
postfix  21312  0.0  0.0  89888  4096 ?        S    07:18   0:00 pickup -l -t unix -u
root     21363  0.0  0.0      0     0 ?        S    07:27   0:00 [kworker/u32:0]
root     21942  0.0  0.0      0     0 ?        S    08:05   0:00 [kworker/2:1]
root     21961  0.0  0.0      0     0 ?        S    08:09   0:00 [kworker/0:2]
root     21992  0.0  0.0 153120  5948 ?        Ds   08:14   0:00 sshd: root@pts/0
root     21995  0.0  0.0 115544  2048 pts/0    Ss   08:14   0:00 -bash
root     22021  0.0  0.0      0     0 ?        S    08:16   0:00 [kworker/2:2]
root     22043  0.0  0.0      0     0 ?        S    08:19   0:00 [kworker/1:1]
root     22044  0.0  0.0      0     0 ?        S    08:19   0:00 [kworker/0:0]
root     22046  0.0  0.0      0     0 ?        S    08:19   0:00 [kworker/3:2]
postgres 22048  0.0  0.0 835720  4340 ?        Ss   08:19   0:00 postgres: candlepin candlepin 127.0.0.1(38846) idle
postgres 22054  0.0  0.0 835720  4344 ?        Ss   08:21   0:00 postgres: candlepin candlepin 127.0.0.1(38848) idle
postgres 22060  0.0  0.0 835720  4340 ?        Ss   08:22   0:00 postgres: candlepin candlepin 127.0.0.1(38850) idle
postgres 22067  0.0  0.0 835720  4344 ?        Ss   08:23   0:00 postgres: candlepin candlepin 127.0.0.1(38852) idle
root     22069  0.0  0.0      0     0 ?        S    08:24   0:00 [kworker/2:0]
postgres 22075  0.0  0.0 835720  4344 ?        Ss   08:24   0:00 postgres: candlepin candlepin 127.0.0.1(38854) idle
root     22076  0.0  0.0      0     0 ?        R    08:25   0:00 [kworker/0:1]
root     22081  0.0  0.0 155448  1876 pts/0    R+   08:25   0:00 ps axuwww
root     29790  0.0  0.4 384844 49468 ?        Ssl  Dec02   0:06 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent --no-daemonize
root     30884  0.0  0.0      0     0 ?        S<   Dec02   0:00 [bioset]
root     31357  0.0  0.0      0     0 ?        S    Dec02   0:04 [kworker/1:0]
1 Like
/usr/lib/jvm/jre-11/bin/java -Xms1024m -Xmx4096m -Djava.security.auth.login.config=/usr/share/tomcat/conf/login.config -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

This is Tomcat hosting Candlepin, I am not running Puppet Server on this host. Although Java processes are ugly (many arguments) these are typically single-process multi-threaded deployments so it is not an issue. Skipping.

root      9202  0.0  0.0 184228  1052 ?        Ss   Dec02   0:02 /usr/sbin/httpd -DFOREGROUND
apache    9203  0.0  0.0 184928  3552 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9204  0.0  0.0 185064  3560 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9205  0.0  0.0 184896  3664 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9206  0.0  0.0 184936  3452 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9207  0.0  0.0 184992  3748 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9208  0.0  0.0 185216  3960 ?        S    Dec02   0:03 /usr/sbin/httpd -DFOREGROUND
apache    9209  0.0  0.0 184944  3816 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND
apache    9210  0.0  0.0 184892  3732 ?        S    Dec02   0:00 /usr/sbin/httpd -DFOREGROUND

Apache httpd, not much to see here. I am assuming we only have one instance, if we have multiple then we should consider tagging them.

foreman   9282  0.0  2.9 947964 293884 ?       Ssl  Dec02   0:48 puma 5.3.2 (unix:///run/foreman.sock) [foreman]
foreman   9308  0.0  4.1 992672 418952 ?       Sl   Dec02   0:39 puma: cluster worker 0: 9282 [foreman]
foreman   9314  0.1  5.3 1143056 536972 ?      Sl   Dec02   1:15 puma: cluster worker 1: 9282 [foreman]

Puma, obviously. A nice example how I would like to see the other processes too.

foreman   9389  0.2  4.0 988712 411220 ?       Ssl  Dec02   1:59 sidekiq 5.2.7  [0 of 1 busy]
foreman   9459  0.6  6.7 1216080 676400 ?      Ssl  Dec02   6:26 sidekiq 5.2.7  [0 of 2 busy]
foreman   9536  0.1  3.8 984064 388352 ?       Ssl  Dec02   1:20 sidekiq 5.2.7  [0 of 2 busy]

Sidekiq is the first problem - although it tries to nicely inform about how many workers are busy, we appear to spawn three instances without any identifier. Quickly looking into the CLI code, it looks like sidekiq appears to support --tag option. I suggest to tag them with something like sidekiq-dynflow or whatever we use sidekiq for.

Can you help me to identify the three instances? What are these?

pulp      9721  0.0  0.4 498476 46828 ?        Ss   Dec02   0:17 /usr/bin/python3 /usr/bin/pulpcore-worker
pulp      9757  0.0  0.6 498416 68936 ?        Ss   Dec02   0:17 /usr/bin/python3 /usr/bin/pulpcore-worker

Pulp worker, not bad.

gfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
pulp      9838  0.0 10.4 1468268 1047116 ?     S    Dec02   0:32 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
pulp      9848  0.0  4.1 836188 417376 ?       S    Dec02   0:08 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
pulp      9857  0.0  2.3 637972 231672 ?       S    Dec02   0:13 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
pulp      9874  0.0  2.0 626308 207076 ?       S    Dec02   0:07 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
pulp      9875  0.0  3.7 792996 372400 ?       S    Dec02   0:25 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application --timeout 90 -w 5 --access-logfile - --access-logformat pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"

Pulp API, not bad, little bit cryptic. I wonder if it can be configured for something better and it can and it is easy! From the docs:

If you install the Python package setproctitle Gunicorn will set the process names to something a bit more meaningful. This will affect the output you see in tools like ps and top . This helps for distinguishing the master process as well as between masters when running more than one app on a single machine. See the proc_name setting for more information.

Created tickets:

pulp      9920  0.0  0.0 267680  8280 ?        Ss   Dec02   0:09 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9924  0.0  0.9 668120 91432 ?        Sl   Dec02   0:39 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9928  0.0  0.7 665208 74972 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9931  0.0  0.7 665204 79428 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9938  0.0  0.7 665204 75816 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9939  0.0  0.8 668160 88020 ?        Sl   Dec02   0:40 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9941  0.0  0.7 665204 78296 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9945  0.0  0.7 665208 73064 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9946  0.0  0.7 665204 72844 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
pulp      9947  0.0  0.7 665204 75728 ?        Sl   Dec02   0:37 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --timeout 90 --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -

Pulp content, the same story.

foreman+  9968  0.0  0.6 1374256 64416 ?       Ssl  Dec02   0:25 ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize

Smart proxy, I will look into what can be done here. Perhaps just renaming this to foreman-proxy could do it, although that is just a change for nothing.