How many registered hosts can perform yum update operations at a single time on a default minimal Foreman / Katello server?

Problem: Unable to find documentation on how many hosts can perform a yum update against a default foreman katello install running on an AWS EC2 m4.xlarge.

Expected outcome: Locating a document that describes how many hosts can perform yum updates at a single time and options / methods to scale up to handling more hosts. Maybe a ratio .eg 1 foreman server can handle x amount of hosts and y amount of scaled up assets will be needed for each additional multiple of x amount of hosts.

Foreman and Proxy versions: Foreman 2.4.0 , foreman-proxy-2.4.0-1.el7.noarch

Foreman and Proxy plugin versions: katello-4.0.0-1.el7.noarch

Distribution and version: Amazon Linux 2

Other relevant data:

Installed Packages

candlepin-3.2.11-1.el7.noarch
candlepin-selinux-3.2.11-1.el7.noarch
foreman-2.4.0-1.el7.noarch
foreman-cli-2.4.0-1.el7.noarch
foreman-debug-2.4.0-1.el7.noarch
foreman-dynflow-sidekiq-2.4.0-1.el7.noarch
foreman-installer-2.4.0-1.el7.noarch
foreman-installer-katello-2.4.0-1.el7.noarch
foreman-postgresql-2.4.0-1.el7.noarch
foreman-proxy-2.4.0-1.el7.noarch
foreman-release-2.4.0-1.el7.noarch
foreman-selinux-2.4.0-1.el7.noarch
foreman-service-2.4.0-1.el7.noarch
foremankl00uemg.hhc.hilton.com-apache-1.0-1.noarch
foremankl00uemg.hhc.hilton.com-foreman-client-1.0-1.noarch
foremankl00uemg.hhc.hilton.com-foreman-proxy-1.0-1.noarch
foremankl00uemg.hhc.hilton.com-foreman-proxy-client-1.0-1.noarch
foremankl00uemg.hhc.hilton.com-puppet-client-1.0-1.noarch
foremankl00uemg.hhc.hilton.com-qpid-broker-1.0-1.noarch
foremankl00uemg.hhc.hilton.com-qpid-router-server-1.0-1.noarch
katello-4.0.0-1.el7.noarch
katello-certs-tools-2.7.3-1.el7.noarch
katello-client-bootstrap-1.7.6-1.el7.noarch
katello-common-4.0.0-1.el7.noarch
katello-debug-4.0.0-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-repos-4.0.0-1.el7.noarch
katello-selinux-4.0.0-1.el7.noarch
katello-server-ca-1.0-1.noarch
pulp-client-1.0-1.noarch
pulpcore-selinux-1.2.4-1.el7.x86_64
python2-qpid-1.37.0-5.el7.noarch
python2-qpid-proton-0.33.0-1.el7.x86_64
python2-qpid-qmf-1.39.0-1.el7.x86_64
python3-pulp-certguard-1.1.0-1.el7.noarch
python3-pulp-container-2.2.1-1.el7.noarch
python3-pulp-deb-2.9.1-1.el7.noarch
python3-pulp-file-1.5.0-1.el7.noarch
python3-pulp-rpm-3.10.0-1.el7.noarch
python3-pulpcore-3.9.1-2.el7.noarch
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-dispatch-router-1.15.0-1.el7.x86_64
qpid-proton-c-0.33.0-1.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch
qpid_router_katello_agent-qpid-router-client-1.0-1.noarch
rubygem-foreman_maintain-0.7.9-1.el7.noarch
tfm-rubygem-actioncable-6.0.3.4-1.el7.noarch
tfm-rubygem-actionmailbox-6.0.3.4-1.el7.noarch
tfm-rubygem-actionmailer-6.0.3.4-1.el7.noarch
tfm-rubygem-actionpack-6.0.3.4-1.el7.noarch
tfm-rubygem-actiontext-6.0.3.4-1.el7.noarch
tfm-rubygem-actionview-6.0.3.4-1.el7.noarch
tfm-rubygem-activejob-6.0.3.4-1.el7.noarch
tfm-rubygem-activemodel-6.0.3.4-1.el7.noarch
tfm-rubygem-activerecord-6.0.3.4-1.el7.noarch
tfm-rubygem-activerecord-import-1.0.0-2.el7.noarch
tfm-rubygem-activerecord-session_store-1.1.1-4.el7.noarch
tfm-rubygem-activestorage-6.0.3.4-1.el7.noarch
tfm-rubygem-activesupport-6.0.3.4-1.el7.noarch
tfm-rubygem-addressable-2.6.0-2.el7.noarch
tfm-rubygem-algebrick-0.7.3-7.el7.noarch
tfm-rubygem-amazing_print-1.1.0-1.el7.noarch
tfm-rubygem-ancestry-3.0.7-1.el7.noarch
tfm-rubygem-anemone-0.7.2-17.el7.noarch
tfm-rubygem-angular-rails-templates-1.1.0-1.el7.noarch
tfm-rubygem-ansi-1.5.0-2.el7.noarch
tfm-rubygem-apipie-bindings-0.4.0-1.el7.noarch
tfm-rubygem-apipie-dsl-2.3.0-1.el7.noarch
tfm-rubygem-apipie-params-0.0.5-4.el7.noarch
tfm-rubygem-apipie-rails-0.5.17-3.el7.noarch
tfm-rubygem-audited-4.9.0-3.el7.noarch
tfm-rubygem-bcrypt-3.1.12-3.el7.x86_64
tfm-rubygem-builder-3.2.4-1.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-5.el7.noarch
tfm-rubygem-clamp-1.1.2-6.el7.noarch
tfm-rubygem-concurrent-ruby-1.1.6-2.el7.noarch
tfm-rubygem-concurrent-ruby-edge-0.6.0-2.fm2_1.el7.noarch
tfm-rubygem-connection_pool-2.2.2-2.el7.noarch
tfm-rubygem-crass-1.0.6-1.el7.noarch
tfm-rubygem-css_parser-1.4.7-4.el7.noarch
tfm-rubygem-daemons-1.2.3-6.el7.noarch
tfm-rubygem-deacon-1.0.0-4.el7.noarch
tfm-rubygem-deep_cloneable-3.0.0-3.el7.noarch
tfm-rubygem-deface-1.5.3-2.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-4.el7.noarch
tfm-rubygem-dynflow-1.4.7-1.fm2_3.el7.noarch
tfm-rubygem-erubi-1.9.0-1.el7.noarch
tfm-rubygem-excon-0.76.0-1.el7.noarch
tfm-rubygem-facter-4.0.44-1.el7.x86_64
tfm-rubygem-faraday-0.17.3-1.el7.noarch
tfm-rubygem-fast_gettext-1.4.1-4.el7.noarch
tfm-rubygem-ffi-1.12.2-1.el7.x86_64
tfm-rubygem-fog-core-2.1.0-3.el7.noarch
tfm-rubygem-foreman-tasks-4.0.1-1.fm2_4.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-foreman_remote_execution-4.3.0-1.fm2_4.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.4.0-1.el7.noarch
tfm-rubygem-formatador-0.2.1-12.el7.noarch
tfm-rubygem-friendly_id-5.3.0-1.el7.noarch
tfm-rubygem-fx-0.5.0-1.el7.noarch
tfm-rubygem-get_process_mem-0.2.1-4.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.8.0-2.el7.noarch
tfm-rubygem-gitlab-sidekiq-fetcher-0.6.0-1.el7.noarch
tfm-rubygem-globalid-0.4.2-1.el7.noarch
tfm-rubygem-graphql-1.8.14-2.el7.noarch
tfm-rubygem-graphql-batch-0.3.10-2.el7.noarch
tfm-rubygem-gssapi-1.2.0-7.el7.noarch
tfm-rubygem-hammer_cli-2.4.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.4.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.7-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.2.1-1.fm2_4.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.15-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_katello-1.0-1.el7.noarch
tfm-rubygem-hashie-3.6.0-2.el7.noarch
tfm-rubygem-highline-2.0.3-1.el7.noarch
tfm-rubygem-hocon-1.3.1-1.el7.noarch
tfm-rubygem-http-cookie-1.0.2-4.el7.noarch
tfm-rubygem-i18n-1.8.2-1.el7.noarch
tfm-rubygem-jwt-2.2.1-2.el7.noarch
tfm-rubygem-kafo-6.2.1-1.el7.noarch
tfm-rubygem-kafo_parsers-1.1.0-3.el7.noarch
tfm-rubygem-kafo_wizards-0.0.2-1.el7.noarch
tfm-rubygem-katello-4.0.0-1.el7.noarch
tfm-rubygem-ldap_fluff-0.5.0-1.el7.noarch
tfm-rubygem-little-plugger-1.1.4-2.el7.noarch
tfm-rubygem-locale-2.0.9-14.el7.noarch
tfm-rubygem-logging-2.3.0-1.el7.noarch
tfm-rubygem-loofah-2.4.0-1.el7.noarch
tfm-rubygem-mail-2.7.1-1.el7.noarch
tfm-rubygem-marcel-0.3.3-1.el7.noarch
tfm-rubygem-method_source-0.9.2-2.el7.noarch
tfm-rubygem-mime-types-3.2.2-4.el7.noarch
tfm-rubygem-mime-types-data-3.2018.0812-4.el7.noarch
tfm-rubygem-mimemagic-0.3.5-1.el7.noarch
tfm-rubygem-mini_mime-1.0.2-1.el7.noarch
tfm-rubygem-mini_portile2-2.4.0-1.el7.noarch
tfm-rubygem-multi_json-1.14.1-2.el7.noarch
tfm-rubygem-multipart-post-2.0.0-2.el7.noarch
tfm-rubygem-mustermann-1.0.2-4.el7.noarch
tfm-rubygem-net-ldap-0.17.0-1.el7.noarch
tfm-rubygem-net-ping-2.0.1-4.el7.noarch
tfm-rubygem-net-scp-1.2.1-4.el7.noarch
tfm-rubygem-net-ssh-4.2.0-2.el7.noarch
tfm-rubygem-netrc-0.11.0-5.el7.noarch
tfm-rubygem-nio4r-2.5.4-1.el7.x86_64
tfm-rubygem-nokogiri-1.10.9-1.el7.x86_64
tfm-rubygem-oauth-0.5.4-4.el7.noarch
tfm-rubygem-parse-cron-0.1.4-4.fm2_1.el7.noarch
tfm-rubygem-pg-1.1.4-3.el7.x86_64
tfm-rubygem-polyglot-0.3.5-2.el7.noarch
tfm-rubygem-powerbar-2.0.1-2.el7.noarch
tfm-rubygem-promise.rb-0.7.4-2.el7.noarch
tfm-rubygem-public_suffix-3.0.3-2.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.7.0-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.6.0-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.3-1.el7.noarch
tfm-rubygem-pulp_container_client-2.2.0-1.el7.noarch
tfm-rubygem-pulp_deb_client-2.8.0-1.el7.noarch
tfm-rubygem-pulp_file_client-1.5.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.10.0-1.el7.noarch
tfm-rubygem-pulpcore_client-3.9.0-1.el7.noarch
tfm-rubygem-puma-5.1.1-1.el7.x86_64
tfm-rubygem-qpid_proton-0.33.0-1.el7.x86_64
tfm-rubygem-rabl-0.14.3-1.el7.noarch
tfm-rubygem-rack-2.2.3-1.el7.noarch
tfm-rubygem-rack-cors-1.0.2-2.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-9.el7.noarch
tfm-rubygem-rack-protection-2.1.0-1.el7.noarch
tfm-rubygem-rack-test-1.1.0-4.el7.noarch
tfm-rubygem-rails-6.0.3.4-1.el7.noarch
tfm-rubygem-rails-dom-testing-2.0.3-6.el7.noarch
tfm-rubygem-rails-html-sanitizer-1.3.0-1.el7.noarch
tfm-rubygem-rails-i18n-6.0.0-2.el7.noarch
tfm-rubygem-railties-6.0.3.4-1.el7.noarch
tfm-rubygem-rainbow-2.2.1-3.el7.noarch
tfm-rubygem-rb-inotify-0.9.7-5.el7.noarch
tfm-rubygem-record_tag_helper-1.0.1-3.el7.noarch
tfm-rubygem-redfish_client-0.5.2-1.el7.noarch
tfm-rubygem-redis-4.1.2-2.el7.noarch
tfm-rubygem-responders-3.0.0-3.el7.noarch
tfm-rubygem-rest-client-2.0.2-3.el7.noarch
tfm-rubygem-rkerberos-0.1.5-19.el7.x86_64
tfm-rubygem-roadie-3.4.0-3.el7.noarch
tfm-rubygem-roadie-rails-2.1.1-2.el7.noarch
tfm-rubygem-robotex-1.0.0-21.el7.noarch
tfm-rubygem-rsec-0.4.3-4.el7.noarch
tfm-rubygem-ruby-libvirt-0.7.1-1.el7.x86_64
tfm-rubygem-ruby2ruby-2.4.2-3.el7.noarch
tfm-rubygem-ruby_parser-3.10.1-3.el7.noarch
tfm-rubygem-rubyipmi-0.10.0-6.el7.noarch
tfm-rubygem-runcible-2.13.1-1.el7.noarch
tfm-rubygem-safemode-1.3.6-1.el7.noarch
tfm-rubygem-scoped_search-4.1.9-1.el7.noarch
tfm-rubygem-sd_notify-0.1.0-1.el7.noarch
tfm-rubygem-secure_headers-6.3.0-2.el7.noarch
tfm-rubygem-sequel-5.7.1-3.el7.noarch
tfm-rubygem-server_sent_events-0.1.2-1.el7.noarch
tfm-rubygem-sexp_processor-4.10.0-6.el7.noarch
tfm-rubygem-sidekiq-5.2.7-3.el7.noarch
tfm-rubygem-sinatra-2.1.0-1.el7.noarch
tfm-rubygem-smart_proxy_dynflow-0.3.0-2.fm2_4.el7.noarch
tfm-rubygem-smart_proxy_dynflow_core-0.3.2-1.fm2_4.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_remote_execution_ssh-0.3.1-1.fm2_4.el7.noarch
tfm-rubygem-sprockets-4.0.2-1.el7.noarch
tfm-rubygem-sprockets-rails-3.2.1-6.el7.noarch
tfm-rubygem-sqlite3-1.3.13-6.el7.x86_64
tfm-rubygem-sshkey-1.9.0-4.el7.noarch
tfm-rubygem-statsd-instrument-2.1.4-3.el7.noarch
tfm-rubygem-stomp-1.4.9-1.el7.noarch
tfm-rubygem-thor-1.0.1-2.el7.noarch
tfm-rubygem-thread_safe-0.3.6-5.el7.noarch
tfm-rubygem-tilt-2.0.8-4.el7.noarch
tfm-rubygem-tzinfo-1.2.6-1.el7.noarch
tfm-rubygem-unf-0.1.3-8.el7.noarch
tfm-rubygem-unf_ext-0.0.7.2-3.el7.x86_64
tfm-rubygem-unicode-0.4.4.4-3.el7.x86_64
tfm-rubygem-unicode-display_width-1.0.5-4.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-7.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-5.el7.noarch
tfm-rubygem-websocket-driver-0.7.1-1.el7.x86_64
tfm-rubygem-websocket-extensions-0.1.5-1.el7.noarch
tfm-rubygem-will_paginate-3.1.7-3.el7.noarch
tfm-rubygem-xmlrpc-0.3.0-2.el7.noarch
tfm-rubygem-zeitwerk-2.2.2-1.el7.noarch
tfm-runtime-6.1-4.el7.x86_64

@thepatchlife I think a remote execution job has a default concurrency of 100 which the default profile seems to handle just fine on the latest Foreman/Katello version for me. There have been older releases that I’ve had to set the concurrency to 50 but this seems to not be necessary any longer.

Thanks @Cassius_Clay

The end goal is to find a methodical way of scaling up. I’d imagine we can up the default concurrency of 100 to match the AWS hardware we’ve placed it on but I’m not sure of the software limits of a single out of the box foreman install. I’m left wondering at what point do we need to scale up foreman components and exactly what components needs to be added? I’m looking at smart proxy abilities and comparing this to what Satellite does with capsules but the documentation proves difficult to find in how we can look at foreman for more large scale use. .eg lets say we have 5000 total hosts and want to patch 1000 at a time. These hosts arent managed in foreman .eg not provisioned, they are just registered with an activation key to do patch management.

How do you scale foreman up?

I can’t recall the version that this was implemented, but since you are running 4.x then it should be available at you. There are several performance profiles that can be used with recommended specs to help fit your needs. This can be found in the installation documentation at docs.theforeman.org. These profiles are set using the foreman-installer with the --tuning option.