IP suggestion is getting slow

Problem:
We have some /16 subnets managed by Foreman and we use the “Internal” IPAM for IP suggestion.

We notice /api/subnet/:id/freeip is taking a long time, somethings longer than 60s.

Expected outcome:

Could use some help troubleshooting why it is taking so long.

Where is the implementation of the Internal DB? My search-fu is failing me.

Would really appreciate any points to get to the bottom of this.

Foreman and Proxy versions:

v3.12

Foreman and Proxy plugin versions:

ansible-collection-theforeman-foreman-4.2.0-1.el8.noarch
candlepin-4.4.16-1.el8.noarch
candlepin-selinux-4.4.16-1.el8.noarch
dynflow-utils-1.6.3-1.el8.x86_64
foreman-3.12.0-1.el8.noarch
foreman-cli-3.12.0-1.el8.noarch
foreman-debug-3.12.0-1.el8.noarch
foreman-dynflow-sidekiq-3.12.0-1.el8.noarch
foreman-installer-3.12.0-1.el8.noarch
foreman-installer-katello-3.12.0-1.el8.noarch
foreman-libvirt-3.12.0-1.el8.noarch
foreman-obsolete-packages-1.10-1.el8.noarch
foreman-postgresql-3.12.0-1.el8.noarch
foreman-proxy-3.12.0-1.el8.noarch
foreman-redis-3.12.0-1.el8.noarch
foreman-release-3.12.0-1.el8.noarch
foreman-selinux-3.12.0-1.el8.noarch
foreman-service-3.12.0-1.el8.noarch
katello-4.14.0-1.el8.noarch
katello-certs-tools-2.10.0-1.el8.noarch
katello-client-bootstrap-1.7.9-2.el8.noarch
katello-common-4.14.0-1.el8.noarch
katello-debug-4.14.0-1.el8.noarch
katello-repos-4.14.0-1.el8.noarch
katello-selinux-5.0.2-1.el8.noarch
pulpcore-obsolete-packages-1.2.0-1.el8.noarch
pulpcore-selinux-2.0.1-1.el8.x86_64
python3.11-pulp-ansible-0.21.8-1.el8.noarch
python3.11-pulp-cli-0.27.2-1.el8.noarch
python3.11-pulp-container-2.20.3-1.el8.noarch
python3.11-pulp-deb-3.2.1-1.el8.noarch
python3.11-pulp-glue-0.27.2-1.el8.noarch
python3.11-pulp-python-3.11.3-1.el8.noarch
python3.11-pulp-rpm-3.26.1-1.el8.noarch
python3.11-pulpcore-3.49.21-1.el8.noarch
rubygem-dynflow-1.9.0-1.el8.noarch
rubygem-foreman-tasks-9.2.3-1.fm3_12.el8.noarch
rubygem-foreman_ansible-14.2.1-1.fm3_12.el8.noarch
rubygem-foreman_discovery-24.0.2-1.fm3_12.el8.noarch
rubygem-foreman_maintain-1.7.4-1.el8.noarch
rubygem-foreman_remote_execution-13.2.5-1.fm3_12.el8.noarch
rubygem-foreman_statistics-2.1.0-3.fm3_11.el8.noarch
rubygem-foreman_webhooks-3.2.3-1.fm3_12.el8.noarch
rubygem-hammer_cli-3.12.0-1.el8.noarch
rubygem-hammer_cli_foreman-3.12.0-1.el8.noarch
rubygem-hammer_cli_foreman_discovery-1.3.0-1.fm3_11.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.3.0-1.fm3_10.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.21-1.fm3_11.el8.noarch
rubygem-hammer_cli_foreman_webhooks-0.1.0-1.fm3_10.el8.noarch
rubygem-hammer_cli_katello-1.14.3-1.el8.noarch
rubygem-katello-4.14.0-1.el8.noarch
rubygem-pulp_ansible_client-0.21.7-1.el8.noarch
rubygem-pulp_certguard_client-3.49.17-1.el8.noarch
rubygem-pulp_container_client-2.20.2-1.el8.noarch
rubygem-pulp_deb_client-3.2.1-1.el8.noarch
rubygem-pulp_file_client-3.49.17-1.el8.noarch
rubygem-pulp_ostree_client-2.3.2-1.el8.noarch
rubygem-pulp_python_client-3.11.2-1.el8.noarch
rubygem-pulp_rpm_client-3.26.1-1.el8.noarch
rubygem-pulpcore_client-3.49.17-1.el8.noarch
rubygem-smart_proxy_dynflow-0.9.3-1.fm3_12.el8.noarch
rubygem-smart_proxy_pulp-3.3.0-1.fm3_10.el8.noarch

Distribution and version:

CentOS Stream 8.

Other relevant data:

According to curl https://foreman/api/subnets/:id/interfaces | jq .total there are 724 interfaces defined in that subnet.

production.log:
2024-10-14T05:37:12 [I|app|3ccdbaa8] Rendered api/v2/subnets/freeip.json.rabl (Duration: 70698.4ms | Allocations: 43759320)
2024-10-14T05:37:12 [I|app|3ccdbaa8] Completed 200 OK in 70822ms (Views: 64363.9ms | ActiveRecord: 6342.7ms | Allocations: 43768116)
:cry:

Hello in 2025!
Turns out our facts upload (Ansible mainly) created a lot of duplicated interface records (hundreds per host) and after cleaning them up systematically it got faster.

Could be my imagination but seems it got even faster after we upgraded to v3.13.1.