ModuleNotFoundError: No module named 'pulpcore'

Support
1

I just built a brand new Foreman/Katello server. Foreman 3.11.2. I setup all the products/Lifecycles/Content Views. I tried to run through a lifecycle and got errors that the pulpcore-workers were not running. Looking at foreman-tail I see this:

pulpcore-worker-3[20567]: Traceback (most recent call last):
pulpcore-worker-3[20567]:  File "/usr/local/bin/pulpcore-worker", line 5, in <module>
pulpcore-worker-3[20567]:    from pulpcore.tasking.entrypoint import worker
pulpcore-worker-3[20567]: ModuleNotFoundError: No module named 'pulpcore'

For all 8 pulpcore-workers.

I ran pulp status and see this:

Error: HTTPSConnectionPool(host='fm01-main02a', port=443): Max retries exceeded with url: /pulp/api/v3/docs/api.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)')))
2

Just to be clear I am not using any custom certs. Its all default.

3

How was that system installed?
Our packages don’t touch /usr/local that’s visible in your error message.

4

I followed the install manual and even documented all the commands I ran. This is on Rocky Linux 9. But I have installed a previous version of Foreman on Rocky Linux 8 with no issues.

5

Here is what I ran:
echo “Prepping for Foreman!!!”
dnf clean all
dnf -y upgrade
dnf -y install https://yum.puppet.com/puppet7-release-el-9.noarch.rpm
dnf -y install https://yum.theforeman.org/releases/3.11/el9/x86_64/foreman-release.rpm
dnf -y install https://yum.theforeman.org/katello/4.13/katello/el9/x86_64/katello-repos-latest.rpm
dnf -y install ruby
dnf -y install foreman-installer
dnf clean all
dnf -y upgrade
dnf -y install foreman-installer-katello

foreman-installer --scenario katello
–tuning default
–enable-foreman
–enable-katello
–enable-foreman-cli
–enable-foreman-cli-ansible
–enable-foreman-cli-katello
–enable-foreman-cli-remote-execution
–enable-foreman-cli-webhooks
–enable-foreman-proxy
–foreman-proxy-content-pulpcore-mirror
–enable-foreman-proxy-content

6

Come On GIF by Max|100%x100%

Can you show us:

dnf repolist -v
rpm -qa |grep pulp
rpm -qf /usr/local/bin/pulpcore-worker
systemctl cat pulp\*
cat /etc/pulp/cli.toml
cat ~/.config/pulp/cli.toml
7

This should not be used on a Katello system (but on a proxy, and there it will be enabled already).

Which docs were you following?

8

That option was not original for me but something I found while trying to trouble shoot the problem.

If its not needed on this host whats the simple way to disable it?

9 
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, kpatch, needs-restarting, notify-packagekit, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, system-upgrade, uploadprofile
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

DNF version: 4.14.0
cachedir: /var/cache/dnf
Last metadata expiration check: 1:22:38 ago on Wed 25 Sep 2024 05:23:24 PM UTC.
Repo-id            : appstream
Repo-name          : Rocky Linux 9 - AppStream
Repo-revision      : 1727121317
Repo-updated       : Mon 23 Sep 2024 07:55:17 PM UTC
Repo-pkgs          : 5,842
Repo-available-pkgs: 5,604
Repo-size          : 7.9 G
Repo-mirrors       : https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=AppStream-9
Repo-baseurl       : https://repo.uccs.edu/rocky-linux/9.4/AppStream/x86_64/os/ (44 more)
Repo-expire        : 21,600 second(s) (last: Wed 25 Sep 2024 05:23:23 PM UTC)
Repo-filename      : /etc/yum.repos.d/rocky.repo

Repo-id            : baseos
Repo-name          : Rocky Linux 9 - BaseOS
Repo-revision      : 1726691854
Repo-updated       : Wed 18 Sep 2024 08:37:34 PM UTC
Repo-pkgs          : 1,164
Repo-available-pkgs: 1,164
Repo-size          : 1.3 G
Repo-mirrors       : https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=BaseOS-9
Repo-baseurl       : https://sftp.hpc.fau.edu/mirror/rocky-linux/9.4/BaseOS/x86_64/os/ (41 more)
Repo-expire        : 21,600 second(s) (last: Wed 25 Sep 2024 05:23:23 PM UTC)
Repo-filename      : /etc/yum.repos.d/rocky.repo

Repo-id            : candlepin
Repo-name          : Candlepin: an open source entitlement management system.
Repo-revision      : 1722960359
Repo-updated       : Tue 06 Aug 2024 04:05:59 PM UTC
Repo-pkgs          : 12
Repo-available-pkgs: 12
Repo-size          : 431 M
Repo-baseurl       : https://yum.theforeman.org/candlepin/4.4/el9/x86_64/
Repo-expire        : 172,800 second(s) (last: Wed 25 Sep 2024 05:23:23 PM UTC)
Repo-filename      : /etc/yum.repos.d/katello.repo

Repo-id            : extras
Repo-name          : Rocky Linux 9 - Extras
Repo-revision      : 1723416257
Repo-updated       : Sun 11 Aug 2024 10:44:17 PM UTC
Repo-pkgs          : 48
Repo-available-pkgs: 48
Repo-size          : 3.0 M
Repo-mirrors       : https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=extras-9
Repo-baseurl       : https://us.mirrors.cicku.me/rocky/9.4/extras/x86_64/os/ (43 more)
Repo-expire        : 21,600 second(s) (last: Wed 25 Sep 2024 05:23:24 PM UTC)
Repo-filename      : /etc/yum.repos.d/rocky-extras.repo

Repo-id            : foreman
Repo-name          : Foreman 3.11
Repo-revision      : 1725479581
Repo-updated       : Wed 04 Sep 2024 07:53:01 PM UTC
Repo-pkgs          : 646
Repo-available-pkgs: 646
Repo-size          : 348 M
Repo-baseurl       : https://yum.theforeman.org/releases/3.11/el9/x86_64
Repo-expire        : 172,800 second(s) (last: Wed 25 Sep 2024 05:23:22 PM UTC)
Repo-filename      : /etc/yum.repos.d/foreman.repo

Repo-id            : foreman-plugins
Repo-name          : Foreman plugins 3.11
Repo-revision      : 1727211730
Repo-updated       : Tue 24 Sep 2024 09:02:10 PM UTC
Repo-pkgs          : 382
Repo-available-pkgs: 382
Repo-size          : 444 M
Repo-baseurl       : https://yum.theforeman.org/plugins/3.11/el9/x86_64
Repo-expire        : 172,800 second(s) (last: Wed 25 Sep 2024 05:23:22 PM UTC)
Repo-filename      : /etc/yum.repos.d/foreman-plugins.repo

Repo-id            : katello
Repo-name          : Katello 4.13
Repo-revision      : 1726820311
Repo-updated       : Fri 20 Sep 2024 08:18:31 AM UTC
Repo-pkgs          : 58
Repo-available-pkgs: 58
Repo-size          : 148 M
Repo-baseurl       : https://yum.theforeman.org/katello/4.13/katello/el9/x86_64/
Repo-expire        : 172,800 second(s) (last: Wed 25 Sep 2024 05:23:22 PM UTC)
Repo-filename      : /etc/yum.repos.d/katello.repo

Repo-id            : pulpcore
Repo-name          : pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.
Repo-revision      : 1726076740
Repo-updated       : Wed 11 Sep 2024 05:45:40 PM UTC
Repo-pkgs          : 328
Repo-available-pkgs: 328
Repo-size          : 90 M
Repo-baseurl       : https://yum.theforeman.org/pulpcore/3.49/el9/x86_64/
Repo-expire        : 172,800 second(s) (last: Wed 25 Sep 2024 05:23:23 PM UTC)
Repo-filename      : /etc/yum.repos.d/katello.repo

Repo-id            : puppet7
Repo-name          : Puppet 7 Repository el 9 - x86_64
Repo-revision      : 1726062098
Repo-updated       : Wed 11 Sep 2024 01:41:38 PM UTC
Repo-pkgs          : 82
Repo-available-pkgs: 82
Repo-size          : 2.4 G
Repo-baseurl       : http://yum.puppet.com/puppet7/el/9/x86_64
Repo-expire        : 172,800 second(s) (last: Wed 25 Sep 2024 05:23:23 PM UTC)
Repo-filename      : /etc/yum.repos.d/puppet7-release.repo
Total packages: 8,562
python3.11-pulp-glue-0.27.2-1.el9.noarch
python3.11-pulpcore-3.49.19-1.el9.noarch
pulpcore-selinux-2.0.1-1.el9.x86_64
rubygem-pulp_ansible_client-0.21.3-1.el9.noarch
rubygem-pulp_certguard_client-3.49.6-1.el9.noarch
rubygem-pulp_container_client-2.20.0-1.el9.noarch
rubygem-pulp_deb_client-3.2.0-1.el9.noarch
rubygem-pulp_file_client-3.49.6-1.el9.noarch
rubygem-pulp_ostree_client-2.3.0-1.el9.noarch
rubygem-pulp_python_client-3.11.1-1.el9.noarch
rubygem-pulp_rpm_client-3.26.1-1.el9.noarch
rubygem-pulpcore_client-3.49.6-1.el9.noarch
python3.11-pulp-cli-0.27.2-1.el9.noarch
python3.11-pulp-container-2.20.3-1.el9.noarch
python3.11-pulp-rpm-3.26.1-1.el9.noarch
python3.11-pulp-deb-3.2.1-1.el9.noarch
python3.11-pulp-ansible-0.21.8-1.el9.noarch
python3.11-pulp-python-3.11.3-1.el9.noarch
rubygem-smart_proxy_pulp-3.3.0-1.el9.noarch
file /usr/local/bin/pulpcore-worker is not owned by any package
# /etc/systemd/system/pulpcore-worker@.service
[Unit]
Description=Pulp Worker
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/default/pulp-workers
EnvironmentFile=-/etc/default/pulp-workers-%i
Environment="DJANGO_SETTINGS_MODULE=pulpcore.app.settings"
Environment="PULP_SETTINGS=/etc/pulp/settings.py"
User=pulp
Group=pulp
WorkingDirectory=/var/lib/pulp
RuntimeDirectory=pulpcore-worker-%i
ExecStart=/usr/libexec/pulpcore/pulpcore-worker
SyslogIdentifier=pulpcore-worker-%i

# This provides reconnect support for PostgreSQL and Redis. Without reconnect support, if either
# is not available at startup or becomes disconnected, this process will die and not respawn.
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target


# /etc/systemd/system/pulpcore-content.service
[Unit]
Description=Pulp Content App
Requires=pulpcore-content.socket
After=network.target

[Service]
Type=notify
Environment="DJANGO_SETTINGS_MODULE=pulpcore.app.settings"
Environment="PULP_SETTINGS=/etc/pulp/settings.py"
User=pulp
Group=pulp
WorkingDirectory=/var/lib/pulp
RuntimeDirectory=pulpcore-content
ExecStart=/usr/bin/pulpcore-content \
          --preload \
          --timeout 90 \
          --workers 17 \
          --access-logfile -
ExecReload=/bin/kill -s HUP $MAINPID
SyslogIdentifier=pulpcore-content

# This provides reconnect support for PostgreSQL and Redis. Without reconnect support, if either
# is not available at startup or becomes disconnected, this process will die and not respawn.
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target


# /etc/systemd/system/pulpcore-worker@.service
[Unit]
Description=Pulp Worker
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/default/pulp-workers
EnvironmentFile=-/etc/default/pulp-workers-%i
Environment="DJANGO_SETTINGS_MODULE=pulpcore.app.settings"
Environment="PULP_SETTINGS=/etc/pulp/settings.py"
User=pulp
Group=pulp
WorkingDirectory=/var/lib/pulp
RuntimeDirectory=pulpcore-worker-%i
ExecStart=/usr/libexec/pulpcore/pulpcore-worker
SyslogIdentifier=pulpcore-worker-%i

# This provides reconnect support for PostgreSQL and Redis. Without reconnect support, if either
# is not available at startup or becomes disconnected, this process will die and not respawn.
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target


# /etc/systemd/system/pulpcore-api.socket
[Unit]
Description=Pulp API Server socket

[Socket]
ListenStream=/run/pulpcore-api.sock
SocketUser=apache
SocketMode=0600

[Install]
WantedBy=sockets.target

# /etc/systemd/system/pulpcore-worker@.service
[Unit]
Description=Pulp Worker
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/default/pulp-workers
EnvironmentFile=-/etc/default/pulp-workers-%i
Environment="DJANGO_SETTINGS_MODULE=pulpcore.app.settings"
Environment="PULP_SETTINGS=/etc/pulp/settings.py"
User=pulp
Group=pulp
WorkingDirectory=/var/lib/pulp
RuntimeDirectory=pulpcore-worker-%i
ExecStart=/usr/libexec/pulpcore/pulpcore-worker
SyslogIdentifier=pulpcore-worker-%i

# This provides reconnect support for PostgreSQL and Redis. Without reconnect support, if either
# is not available at startup or becomes disconnected, this process will die and not respawn.
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target


# /etc/systemd/system/pulpcore-worker@.service
[Unit]
Description=Pulp Worker
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/default/pulp-workers
EnvironmentFile=-/etc/default/pulp-workers-%i
Environment="DJANGO_SETTINGS_MODULE=pulpcore.app.settings"
Environment="PULP_SETTINGS=/etc/pulp/settings.py"
User=pulp
Group=pulp
WorkingDirectory=/var/lib/pulp
RuntimeDirectory=pulpcore-worker-%i
ExecStart=/usr/libexec/pulpcore/pulpcore-worker
SyslogIdentifier=pulpcore-worker-%i

# This provides reconnect support for PostgreSQL and Redis. Without reconnect support, if either
# is not available at startup or becomes disconnected, this process will die and not respawn.
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target


# /etc/systemd/system/pulpcore-worker@.service
[Unit]
Description=Pulp Worker
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/default/pulp-workers
EnvironmentFile=-/etc/default/pulp-workers-%i
Environment="DJANGO_SETTINGS_MODULE=pulpcore.app.settings"
Environment="PULP_SETTINGS=/etc/pulp/settings.py"
User=pulp
Group=pulp
WorkingDirectory=/var/lib/pulp
RuntimeDirectory=pulpcore-worker-%i
ExecStart=/usr/libexec/pulpcore/pulpcore-worker
SyslogIdentifier=pulpcore-worker-%i

# This provides reconnect support for PostgreSQL and Redis. Without reconnect support, if either
# is not available at startup or becomes disconnected, this process will die and not respawn.
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target


# /etc/systemd/system/pulpcore-worker@.service
[Unit]
Description=Pulp Worker
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/default/pulp-workers
EnvironmentFile=-/etc/default/pulp-workers-%i
Environment="DJANGO_SETTINGS_MODULE=pulpcore.app.settings"
Environment="PULP_SETTINGS=/etc/pulp/settings.py"
User=pulp
Group=pulp
WorkingDirectory=/var/lib/pulp
RuntimeDirectory=pulpcore-worker-%i
ExecStart=/usr/libexec/pulpcore/pulpcore-worker
SyslogIdentifier=pulpcore-worker-%i

# This provides reconnect support for PostgreSQL and Redis. Without reconnect support, if either
# is not available at startup or becomes disconnected, this process will die and not respawn.
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target


# /etc/systemd/system/pulpcore-content.socket
[Unit]
Description=Pulp Content App socket

[Socket]
ListenStream=/run/pulpcore-content.sock
SocketUser=apache
SocketMode=0600

[Install]
WantedBy=sockets.target

# /etc/systemd/system/pulpcore-worker@.service
[Unit]
Description=Pulp Worker
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/default/pulp-workers
EnvironmentFile=-/etc/default/pulp-workers-%i
Environment="DJANGO_SETTINGS_MODULE=pulpcore.app.settings"
Environment="PULP_SETTINGS=/etc/pulp/settings.py"
User=pulp
Group=pulp
WorkingDirectory=/var/lib/pulp
RuntimeDirectory=pulpcore-worker-%i
ExecStart=/usr/libexec/pulpcore/pulpcore-worker
SyslogIdentifier=pulpcore-worker-%i

# This provides reconnect support for PostgreSQL and Redis. Without reconnect support, if either
# is not available at startup or becomes disconnected, this process will die and not respawn.
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target


# /etc/systemd/system/pulpcore-worker@.service
[Unit]
Description=Pulp Worker
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/default/pulp-workers
EnvironmentFile=-/etc/default/pulp-workers-%i
Environment="DJANGO_SETTINGS_MODULE=pulpcore.app.settings"
Environment="PULP_SETTINGS=/etc/pulp/settings.py"
User=pulp
Group=pulp
WorkingDirectory=/var/lib/pulp
RuntimeDirectory=pulpcore-worker-%i
ExecStart=/usr/libexec/pulpcore/pulpcore-worker
SyslogIdentifier=pulpcore-worker-%i

# This provides reconnect support for PostgreSQL and Redis. Without reconnect support, if either
# is not available at startup or becomes disconnected, this process will die and not respawn.
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target


# /etc/systemd/system/pulpcore-api.service
[Unit]
Description=Pulp API Server
After=network.target
Requires=pulpcore-api.socket

[Service]
Type=notify
Environment="DJANGO_SETTINGS_MODULE=pulpcore.app.settings"
Environment="PULP_SETTINGS=/etc/pulp/settings.py"
User=pulp
Group=pulp
WorkingDirectory=/var/lib/pulp
RuntimeDirectory=pulpcore-api
ExecStart=/usr/bin/pulpcore-api \
          --preload \
          --timeout 90 \
          --workers 5 \
          --max-requests 800 \
          --max-requests-jitter 100 \
          --access-logfile - \
          --access-logformat 'pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"'
ExecReload=/bin/kill -s HUP $MAINPID
ProtectSystem=full
PrivateTmp=yes
PrivateDevices=yes

SyslogIdentifier=pulpcore-api

# This provides reconnect support for PostgreSQL and Redis. Without reconnect support, if either
# is not available at startup or becomes disconnected, this process will die and not respawn.
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

[cli]
base_url = "https://gp2us1opsfm01-main02a.example.net"
verify_ssl = true
dry_run = true
[cli]
cert = "/etc/foreman/client_cert.pem"
key = "/etc/foreman/client_key.pem"
10

I am going to say this. I have zero issues running foreman-installer --reset-data. Nor do I have a problem just starting over from scratch. I have the entire build scripted. I have the entire config reduced to hammer commands that can be re-run. But before I go that far I need to know what went wrong and how to make sure it does not happen a second time.

But I will happily do a fix if there is a fix rather then do this over again.

11

Well, something is wrong with your system, and it shouldn’t be from the steps you posted (most of the flags you pass are either on anyway, or wouldn’t affect things much):

  • the /usr/local/bin/pulpcore-worker path
  • the pulp status error having fm01-main02a as the hostname while the config points at gp2us1opsfm01-main02a.example.net (or was that just a miss-redaction in the first case?)

Can you try redeploying the machine fresh, using only the following installer flags, and than re-evaluating what does and what doesn’t work?

foreman-installer --scenario katello

If you want Ansible, Remote Execution and Webhooks, you gotta pass --enable-foreman-plugin-<name> too, not only --enable-foreman-cli-<name>. You’ll also probably want their Proxy counterparts, but let’s first get Katello up and running?

12

The host name thing is just to mask my companies domain name. I just messed up the did not use the same on twice.

As for re-deploying. Yes I can. Unless there is a simple way to rip out all the foreman rpms and configs it will take a while.

13

There is none, plus I suspect something modified the system outside of RPMs (you wouldn’t have /usr/local paths otherwise)

14

Okay. This time as I install I will keep checking /usr/local to see if I
can figure out what puts things there.

I will report back with my findings.

1 Like
15

I have run the foreman-installer and all is good so far. Before I run these commands can you look at them and tell me if any of these might put things in /usr/local

dnf -y install rubygem-foreman_maintain \
    rubygem-foreman_maintain-doc

dnf -y install rubygem-foreman_webhooks-doc \
    rubygem-foreman_hooks \
    rubygem-foreman_hooks-doc

dnf -y install rubygem-foreman_remote_execution \
    rubygem-foreman_remote_execution-doc \
    rubygem-foreman-tasks \
    rubygem-foreman-tasks-doc

dnf -y install rubygem-foreman-tasks \
    rubygem-foreman-tasks-doc

echo "Add remaining ansible items"
pip install ansible-core
pip install ansible-lint ansible-runner
pip install ansible-builder
pip install pulp-ansible

dnf -y install ansible-collection-theforeman-foreman
dnf -y install ansible-collection-theforeman-operations

echo "Adding some docs"
dnf -y install rubygem-foreman_ansible-doc
dnf -y install rubygem-hammer_cli_foreman_ansible-doc
dnf -y install rubygem-pulp_ansible_client-doc
dnf -y install rubygem-smart_proxy_ansible-doc

pip install pulp-cli  # minimal dependencies
pip install pulp-cli[pygments]  # colorized output
pip install pulp-cli[shell]  # with interactive shell mode
16

This should already be installed (and the -doc package doesn’t contain much anyway)

foreman_hooks is deprecated, please don’t use that.

Use the installer (that way backup/restore will work too)

foreman-installer --enable-foreman-plugin-webhooks

Again, I recommend the installer :slight_smile:

This will touch /usr/local! Especially the last one. Why do you need these?

Same, we setup cli for you, no need for that.
(and might break things)

17

Our Foreman server is also our Ansbile server so at the very least I need these:

pip install ansible-core
pip install ansible-lint ansible-runner
pip install ansible-builder

And over the years we have had many issues running Ansible through Foreman. So when that fails we just run it via the CLI on the local Foreman server.

I will run via foreman-installer what you suggested.

18

Please don’t do this.

(But also, all of these are available from our repos as we integrate with Ansible)

19

OK. I will do as you suggested and install Ansible via RPMs in the repos.

20

I ran this as my foreman-installer command:

foreman-installer --scenario katello \
    --tuning default \
    --enable-foreman-cli-ansible \
    --enable-foreman-cli-remote-execution \
    --enable-foreman-cli-ssh \
    --enable-foreman-cli-tasks \
    --enable-foreman-compute-ec2 \
    --enable-foreman-plugin-ansible \
    --enable-foreman-plugin-default-hostgroup \
    --enable-foreman-plugin-remote-execution \
    --enable-foreman-plugin-statistics \
    --enable-foreman-plugin-tasks \
    --enable-foreman-plugin-templates

But this is failing on these 2 migrations:

You have 2 pending migrations:
  20221103122801 FixDefaultHostgroupSettingsCategoryToDsl
  20230416170000 EnableAnsibleCallbackForRexFeature

foreman-rake db:migrate --trace --verbose shows me this:

== 20221103122801 FixDefaultHostgroupSettingsCategoryToDsl: migrating =========
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::UndefinedColumn: ERROR:  column settings.category does not exist
LINE 1: ... IN (SELECT "settings"."id" FROM "settings" WHERE "settings"...

and

Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR:  column settings.category does not exist
LINE 1: ... IN (SELECT "settings"."id" FROM "settings" WHERE "settings"...

and

Caused by:
PG::UndefinedColumn: ERROR:  column settings.category does not exist
LINE 1: ... IN (SELECT "settings"."id" FROM "settings" WHERE "settings"...

How do I fix this?

Or how do I clear out the migrations so I can run a foreman-installer --reset-data and re-run foreman-installer with out the lines:
–enable-foreman-plugin-ansible
–enable-foreman-plugin-default-hostgroup
Which I think are the lines causing the issue.