Removing MongoDB After Pulp3 Migration

fresh-pie · January 12, 2022, 10:05pm

Hello!

I’m running Katello 3.17 with Foreman 2.2.3 which seems to had come with MongoDB 3.4. Unfortunately, our security scanner reported that 3.4 is EOL and is coming up on our vulnerability report.

So with that, after running the Pulp2 to Pulp3 migration process for Katello 3.17, I’m wondering if and how I can now safely remove MongoDB?

I read in the Katello - Saying goodbye to Pulp 2 document that after moving to Pulp 3, MondoDB will no longer be needed. However, after executing the Pulp migration steps, I noticed that MongoDB is still installed and the service is enabled and running.

# foreman-maintain service list
Running Service List
================================================================================
List applicable services:
foreman-proxy.service                         enabled
foreman.service                               enabled
goferd.service                                enabled
httpd.service                                 enabled
postgresql.service                            enabled
pulp_celerybeat.service                       enabled
pulp_resource_manager.service                 enabled
pulp_streamer.service                         enabled
pulp_workers.service                          enabled
pulpcore-api.service                          enabled
pulpcore-content.service                      enabled
pulpcore-resource-manager.service             enabled
pulpcore-worker@.service                      enabled
puppetserver.service                          enabled
qdrouterd.service                             enabled
qpidd.service                                 enabled
rh-mongodb34-mongod.service                   enabled
rh-redis5-redis.service                       enabled
squid.service                                 enabled
tomcat.service                                enabled

All services listed                                                   [OK]
--------------------------------------------------------------------------------

# systemctl status rh-mongodb34-mongod.service
● rh-mongodb34-mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/rh-mongodb34-mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-01-12 21:17:00 UTC; 28min ago
  Process: 19699 ExecStart=/opt/rh/rh-mongodb34/root/usr/libexec/mongodb-scl-helper enable $RH_MONGODB34_SCLS_ENABLED -- /opt/rh/rh-mongodb34/root/usr/bin/mongod $OPTIONS run (code=exited, status=0/SUCCESS)
 Main PID: 19704 (mongod)
    Tasks: 48
   CGroup: /system.slice/rh-mongodb34-mongod.service
           └─19704 /opt/rh/rh-mongodb34/root/usr/bin/mongod -f /etc/opt/rh/rh-mongodb34/mongod.conf run

Jan 12 21:17:00 foreman.example.net mongod.27017[19704]: [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
Jan 12 21:17:00 foreman.example.net mongod.27017[19704]: [initandlisten] **        We suggest setting it to 'never'
Jan 12 21:17:00 foreman.example.net mongod.27017[19704]: [initandlisten]
Jan 12 21:17:00 foreman.example.net mongod.27017[19704]: [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 31193 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files.
Jan 12 21:17:00 foreman.example.net mongod.27017[19704]: [initandlisten]
Jan 12 21:17:00 foreman.example.net mongod.27017[19704]: [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'
Jan 12 21:17:00 foreman.example.net mongod.27017[19704]: [thread1] waiting for connections on port 27017
Jan 12 21:17:00 foreman.example.net mongodb-scl-helper[19699]: child process started successfully, parent exiting
Jan 12 21:17:00 foreman.example.net systemd[1]: Started High-performance, schema-free document-oriented database.
Jan 12 21:30:28 foreman.example.net mongod.27017[19704]: [conn53] received client metadata from 127.0.0.1:41242 conn53: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.9" }, os...
Hint: Some lines were ellipsized, use -l to show in full.

I know there is a foreman-maintain content remove-pulp2 command used to remove Pulp2, which may end up disabling/removing MongoDB? However it doesn’t seem the version of foreman-maintain that comes with Katello 3.17/Foreman 2.2.3 has that option. Not too sure, but I believe I’d need to upgrade to Katello 4 first?

I do plan to upgrade to Katello 4 as soon as possible (planning on finishing up an OS deployment first) and wondering if I can somehow drop MongoDB before then.

My main question I suppose is how should I go about removing MongoDB completely from my Katello host? If upgrading to Katello 4 is really the way to go about that, that is cool, but if I can do so prior, that would be really, really sweet. Just trying to review my options

Thanks!

quba42 · January 13, 2022, 8:53am

Katello 3.17 is really old, and most of the 2to3 migration fixes went into 3.18 only. As a result, I can’t really recommend running 2to3 migrations on 3.17, I would upgrade to 3.18 first.

@Justin_Sherrill Should there be a big warning box in documentation somewhere that 3.18 is the preferred version for running 2to3 migrations?

I don’t know exactly what version, the foreman-maintain content remove-pulp2 command is added, but I would not recommend doing so before having updated all the way to a 4.* Katello version. Even Katello 3.18 has a really old version of Pulp 3 in it, and I would recommend using it only for the 2to3 migration, and then upgrading to 4.* as soon as the content switch over is done. I would also consider skipping 4.0 and going straight for 4.1 at the least…

fresh-pie · January 13, 2022, 1:27pm

Thank you for the insight!

So if I’m understanding correctly, a good path to take is:

Upgrade to 3.18.
Execute the 2 to 3 migration steps.
Upgrade straight to 4.1.
Execute foreman-maintain content remove-pulp2.

I suppose my only follow up question is, at which of the above steps is MongoDB removed, or is there perhaps some additional steps I need to take to completely remove it from the system?

Thanks again

quba42 · January 13, 2022, 2:10pm

Yes, that sounds like a good plan to me. My understanding is that foreman-maintain content remove-pulp2 will remove Mongo (amongst other things), but it is the step of the process I know least about. I am not sure if this will completely remove Mongo (uninstall the RPM packages), or just remove all Pulp 2 content from it. I do think it should be safe to remove Mongo manually after running that command (if the command did not do so already).

Also note that it is not quite clear from your previous post if you have already run foreman-maintain content switchover. I am not sure if it is possible/advisable to run the 2to3 migration steps again if you have already switched. In that case it might be better to just hope your Katello 3.17 based 2to3 migration went fine, and go straight to step 3 in your list…

fresh-pie · January 13, 2022, 7:50pm

Hey there Quirin, I just wanted to report back that I followed your advice on my lab Foreman server and everything went according to plan. I documented the process and will execute the same steps on my production system.

To confirm, foreman-maintain content remove-pulp2 does indeed remove MongoDB, which is awesome.

And to clarify, I did execute the Pulp migration steps on 3.17. It wasn’t necessary to run them again or do anything special after upgrading thankfully.

Thanks for your help. Once again this community has proven invaluable.