Unable to create Products from UI

Support
1

Problem:
I am testing out the repo discovery functionality using https://package.perforce.com/yum/rhel/8/x86_64/

I get the following error:

 Task 2438c719-7853-470b-9945-4265e23b6c4c: PulpRpmClient::ApiError: Error message: the server returns an error HTTP status code: 500 Response headers: {"date"=>"Fri, 04 Sep 2020 18:58:56 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"text/html; charset=UTF-8", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"27", "vary"=>"Cookie", "via"=>"1.1 dwhitfield-foreman.openlogic.local", "connection"=>"close"} Response body: <h1>Server Error (500)</h1>

Expected outcome:
Repo is created.

Foreman and Proxy versions:
Version 2.1.2

Foreman and Proxy plugin versions:
My guess is you don’t need all of these, but I don’t know which ones would be relevant.

candlepin-3.1.16-1.el7.noarch
candlepin-selinux-3.1.16-1.el7.noarch
dwhitfield-foreman.openlogic.local-apache-1.0-1.noarch
dwhitfield-foreman.openlogic.local-foreman-client-1.0-1.noarch
dwhitfield-foreman.openlogic.local-foreman-proxy-1.0-1.noarch
dwhitfield-foreman.openlogic.local-foreman-proxy-client-1.0-1.noarch
dwhitfield-foreman.openlogic.local-puppet-client-1.0-1.noarch
dwhitfield-foreman.openlogic.local-qpid-broker-1.0-1.noarch
dwhitfield-foreman.openlogic.local-qpid-client-cert-1.0-1.noarch
dwhitfield-foreman.openlogic.local-qpid-router-client-1.0-1.noarch
dwhitfield-foreman.openlogic.local-qpid-router-server-1.0-1.noarch
foreman-2.1.2-1.el7.noarch
foreman-cli-2.1.2-1.el7.noarch
foreman-debug-2.1.2-1.el7.noarch
foreman-dynflow-sidekiq-2.1.2-1.el7.noarch
foreman-installer-2.1.2-1.el7.noarch
foreman-installer-katello-2.1.2-1.el7.noarch
foreman-postgresql-2.1.2-1.el7.noarch
foreman-proxy-2.1.2-1.el7.noarch
foreman-release-2.1.2-1.el7.noarch
foreman-selinux-2.1.2-1.el7.noarch
foreman-service-2.1.2-1.el7.noarch
katello-3.16.0-1.el7.noarch
katello-certs-tools-2.7.0-1.el7.noarch
katello-client-bootstrap-1.7.5-1.el8.noarch
katello-common-3.16.0-1.el7.noarch
katello-debug-3.16.0-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-repos-3.16.0-1.el7.noarch
katello-selinux-3.4.0-1.el7.noarch
katello-server-ca-1.0-1.noarch
pulp-client-1.0-1.noarch
pulp-deb-plugins-1.10.1-1.el7.noarch
pulp-docker-plugins-3.2.6-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
pulp-puppet-plugins-2.21.3-1.el7.noarch
pulp-puppet-tools-2.21.3-1.el7.noarch
pulp-rpm-plugins-2.21.3-1.el7.noarch
pulp-selinux-2.21.3-1.el7.noarch
pulp-server-2.21.3-1.el7.noarch
pulpcore-selinux-1.0.0-2.el7.x86_64
python-gofer-qpid-2.12.5-3.el7.noarch
python-pulp-common-2.21.3-1.el7.noarch
python-pulp-deb-common-1.10.1-1.el7.noarch
python-pulp-docker-common-3.2.6-1.el7.noarch
python-pulp-oid_validation-2.21.3-1.el7.noarch
python-pulp-puppet-common-2.21.3-1.el7.noarch
python-pulp-repoauth-2.21.3-1.el7.noarch
python-pulp-rpm-common-2.21.3-1.el7.noarch
python-pulp-streamer-2.21.3-1.el7.noarch
python2-qpid-1.37.0-4.el7.noarch
python2-qpid-proton-0.31.0-3.el7.x86_64
python2-qpid-qmf-1.39.0-1.el7.x86_64
python3-pulp-2to3-migration-0.2.0-0.1.b7.el7.noarch
python3-pulp-certguard-0.1.0-0.2.rc5.el7.noarch
python3-pulp-container-1.4.2-1.el7.noarch
python3-pulp-file-1.0.1-1.el7.noarch
python3-pulp-rpm-3.5.0-3.el7.noarch
python3-pulpcore-3.4.1-1.el7.noarch
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-client-devel-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-dispatch-router-1.12.0-1.el7.x86_64
qpid-proton-c-0.31.0-3.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch
rubygem-foreman_maintain-0.6.9-1.el7.noarch
tfm-rubygem-actioncable-6.0.3.1-1.el7.noarch
tfm-rubygem-actionmailbox-6.0.3.1-1.el7.noarch
tfm-rubygem-actionmailer-6.0.3.1-1.el7.noarch
tfm-rubygem-actionpack-6.0.3.1-1.el7.noarch
tfm-rubygem-actiontext-6.0.3.1-1.el7.noarch
tfm-rubygem-actionview-6.0.3.1-1.el7.noarch
tfm-rubygem-activejob-6.0.3.1-1.el7.noarch
tfm-rubygem-activemodel-6.0.3.1-1.el7.noarch
tfm-rubygem-activerecord-6.0.3.1-1.el7.noarch
tfm-rubygem-activerecord-import-1.0.0-2.el7.noarch
tfm-rubygem-activerecord-session_store-1.1.1-4.el7.noarch
tfm-rubygem-activestorage-6.0.3.1-1.el7.noarch
tfm-rubygem-activesupport-6.0.3.1-1.el7.noarch
tfm-rubygem-addressable-2.6.0-2.el7.noarch
tfm-rubygem-algebrick-0.7.3-7.el7.noarch
tfm-rubygem-amazing_print-1.1.0-1.el7.noarch
tfm-rubygem-ancestry-3.0.7-1.el7.noarch
tfm-rubygem-anemone-0.7.2-17.el7.noarch
tfm-rubygem-angular-rails-templates-1.1.0-1.el7.noarch
tfm-rubygem-ansi-1.5.0-2.el7.noarch
tfm-rubygem-apipie-bindings-0.3.0-2.el7.noarch
tfm-rubygem-apipie-dsl-2.2.2-2.el7.noarch
tfm-rubygem-apipie-params-0.0.5-4.el7.noarch
tfm-rubygem-apipie-rails-0.5.17-3.el7.noarch
tfm-rubygem-audited-4.9.0-3.el7.noarch
tfm-rubygem-bcrypt-3.1.12-3.el7.x86_64
tfm-rubygem-builder-3.2.4-1.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-5.el7.noarch
tfm-rubygem-clamp-1.1.2-6.el7.noarch
tfm-rubygem-concurrent-ruby-1.1.6-2.el7.noarch
tfm-rubygem-concurrent-ruby-edge-0.6.0-2.fm2_1.el7.noarch
tfm-rubygem-connection_pool-2.2.2-2.el7.noarch
tfm-rubygem-crass-1.0.6-1.el7.noarch
tfm-rubygem-css_parser-1.4.7-4.el7.noarch
tfm-rubygem-daemons-1.2.3-6.el7.noarch
tfm-rubygem-deacon-1.0.0-4.el7.noarch
tfm-rubygem-deep_cloneable-3.0.0-3.el7.noarch
tfm-rubygem-deface-1.5.3-2.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-4.el7.noarch
tfm-rubygem-dynflow-1.4.6-1.fm2_1.el7.noarch
tfm-rubygem-erubi-1.9.0-1.el7.noarch
tfm-rubygem-excon-0.58.0-5.el7.noarch
tfm-rubygem-facter-2.4.0-7.el7.x86_64
tfm-rubygem-faraday-0.15.4-2.el7.noarch
tfm-rubygem-fast_gettext-1.4.1-4.el7.noarch
tfm-rubygem-ffi-1.12.2-1.el7.x86_64
tfm-rubygem-fog-core-2.1.0-3.el7.noarch
tfm-rubygem-foreman-tasks-2.0.2-1.fm2_1.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-foreman_remote_execution-3.3.5-1.fm2_1.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.3.0-1.el7.noarch
tfm-rubygem-foreman_salt-13.2.3-1.fm2_1.el7.noarch
tfm-rubygem-formatador-0.2.1-12.el7.noarch
tfm-rubygem-friendly_id-5.3.0-1.el7.noarch
tfm-rubygem-fx-0.5.0-1.el7.noarch
tfm-rubygem-get_process_mem-0.2.1-4.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.8.0-2.el7.noarch
tfm-rubygem-gitlab-sidekiq-fetcher-0.5.2-2.el7.noarch
tfm-rubygem-globalid-0.4.2-1.el7.noarch
tfm-rubygem-graphql-1.8.14-2.el7.noarch
tfm-rubygem-graphql-batch-0.3.10-2.el7.noarch
tfm-rubygem-gssapi-1.2.0-7.el7.noarch
tfm-rubygem-hammer_cli-2.1.1-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.1.2-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.5-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_salt-0.0.5-4.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.14-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_katello-0.22.2-1.el7.noarch
tfm-rubygem-hashie-3.6.0-2.el7.noarch
tfm-rubygem-highline-1.7.8-5.el7.noarch
tfm-rubygem-http-cookie-1.0.2-4.el7.noarch
tfm-rubygem-i18n-1.8.2-1.el7.noarch
tfm-rubygem-jwt-2.2.1-2.el7.noarch
tfm-rubygem-kafo-4.1.0-3.el7.noarch
tfm-rubygem-kafo_parsers-1.1.0-3.el7.noarch
tfm-rubygem-kafo_wizards-0.0.1-4.el7.noarch
tfm-rubygem-katello-3.16.0-1.el7.noarch
tfm-rubygem-ldap_fluff-0.4.7-5.el7.noarch
tfm-rubygem-little-plugger-1.1.4-2.el7.noarch
tfm-rubygem-locale-2.0.9-14.el7.noarch
tfm-rubygem-logging-2.2.2-5.el7.noarch
tfm-rubygem-loofah-2.4.0-1.el7.noarch
tfm-rubygem-mail-2.7.1-1.el7.noarch
tfm-rubygem-marcel-0.3.3-1.el7.noarch
tfm-rubygem-method_source-0.9.2-1.el7.noarch
tfm-rubygem-mime-types-3.2.2-4.el7.noarch
tfm-rubygem-mime-types-data-3.2018.0812-4.el7.noarch
tfm-rubygem-mimemagic-0.3.5-1.el7.noarch
tfm-rubygem-mini_mime-1.0.2-1.el7.noarch
tfm-rubygem-mini_portile2-2.4.0-1.el7.noarch
tfm-rubygem-multi_json-1.14.1-1.el7.noarch
tfm-rubygem-multipart-post-2.0.0-2.el7.noarch
tfm-rubygem-mustermann-1.0.2-4.el7.noarch
tfm-rubygem-net-ldap-0.16.1-2.el7.noarch
tfm-rubygem-net-ping-2.0.1-4.el7.noarch
tfm-rubygem-net-scp-1.2.1-4.el7.noarch
tfm-rubygem-net-ssh-4.2.0-2.el7.noarch
tfm-rubygem-netrc-0.11.0-5.el7.noarch
tfm-rubygem-nio4r-2.5.2-1.el7.x86_64
tfm-rubygem-nokogiri-1.10.9-1.el7.x86_64
tfm-rubygem-oauth-0.5.4-4.el7.noarch
tfm-rubygem-parse-cron-0.1.4-4.fm2_1.el7.noarch
tfm-rubygem-pg-1.1.4-3.el7.x86_64
tfm-rubygem-polyglot-0.3.5-2.el7.noarch
tfm-rubygem-powerbar-2.0.1-2.el7.noarch
tfm-rubygem-promise.rb-0.7.4-2.el7.noarch
tfm-rubygem-public_suffix-3.0.3-2.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.2.0-0.1.b6.el7.noarch
tfm-rubygem-pulp_ansible_client-0.2.0b13.dev01588546902-1.el7.noarch
tfm-rubygem-pulp_certguard_client-0.1.0rc5-1.el7.noarch
tfm-rubygem-pulp_container_client-1.4.1-1.el7.noarch
tfm-rubygem-pulp_file_client-1.0.1-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.5.0-1.el7.noarch
tfm-rubygem-pulpcore_client-3.4.1-1.el7.noarch
tfm-rubygem-puma-4.3.3-4.el7.x86_64
tfm-rubygem-rabl-0.14.3-1.el7.noarch
tfm-rubygem-rack-2.2.2-1.el7.noarch
tfm-rubygem-rack-cors-1.0.2-2.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-9.el7.noarch
tfm-rubygem-rack-protection-2.0.3-4.el7.noarch
tfm-rubygem-rack-test-1.1.0-4.el7.noarch
tfm-rubygem-rails-6.0.3.1-1.el7.noarch
tfm-rubygem-rails-dom-testing-2.0.3-6.el7.noarch
tfm-rubygem-rails-html-sanitizer-1.3.0-1.el7.noarch
tfm-rubygem-rails-i18n-6.0.0-2.el7.noarch
tfm-rubygem-railties-6.0.3.1-1.el7.noarch
tfm-rubygem-rainbow-2.2.1-3.el7.noarch
tfm-rubygem-rb-inotify-0.9.7-5.el7.noarch
tfm-rubygem-record_tag_helper-1.0.1-3.el7.noarch
tfm-rubygem-redis-4.1.2-2.el7.noarch
tfm-rubygem-responders-3.0.0-3.el7.noarch
tfm-rubygem-rest-client-2.0.2-3.el7.noarch
tfm-rubygem-rkerberos-0.1.5-18.el7.x86_64
tfm-rubygem-roadie-3.4.0-3.el7.noarch
tfm-rubygem-roadie-rails-2.1.1-2.el7.noarch
tfm-rubygem-robotex-1.0.0-21.el7.noarch
tfm-rubygem-rsec-0.4.3-4.el7.noarch
tfm-rubygem-ruby-libvirt-0.7.1-1.el7.x86_64
tfm-rubygem-ruby2ruby-2.4.2-3.el7.noarch
tfm-rubygem-ruby_parser-3.10.1-3.el7.noarch
tfm-rubygem-rubyipmi-0.10.0-6.el7.noarch
tfm-rubygem-runcible-2.13.1-1.el7.noarch
tfm-rubygem-safemode-1.3.5-3.el7.noarch
tfm-rubygem-scoped_search-4.1.8-1.el7.noarch
tfm-rubygem-secure_headers-6.3.0-2.el7.noarch
tfm-rubygem-sequel-5.7.1-3.el7.noarch
tfm-rubygem-sexp_processor-4.10.0-6.el7.noarch
tfm-rubygem-sidekiq-5.2.7-3.el7.noarch
tfm-rubygem-sinatra-2.0.3-4.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_1.el7.noarch
tfm-rubygem-sprockets-3.7.2-6.el7.noarch
tfm-rubygem-sprockets-rails-3.2.1-6.el7.noarch
tfm-rubygem-sshkey-1.9.0-4.el7.noarch
tfm-rubygem-statsd-instrument-2.1.4-3.el7.noarch
tfm-rubygem-stomp-1.4.9-1.el7.noarch
tfm-rubygem-thor-1.0.1-2.el7.noarch
tfm-rubygem-thread_safe-0.3.6-5.el7.noarch
tfm-rubygem-tilt-2.0.8-4.el7.noarch
tfm-rubygem-tzinfo-1.2.6-1.el7.noarch
tfm-rubygem-unf-0.1.3-8.el7.noarch
tfm-rubygem-unf_ext-0.0.7.2-3.el7.x86_64
tfm-rubygem-unicode-0.4.4.4-3.el7.x86_64
tfm-rubygem-unicode-display_width-1.0.5-4.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-7.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-5.el7.noarch
tfm-rubygem-websocket-driver-0.7.1-1.el7.x86_64
tfm-rubygem-websocket-extensions-0.1.4-1.el7.noarch
tfm-rubygem-will_paginate-3.1.7-3.el7.noarch
tfm-rubygem-x-editable-rails-1.5.5-5.el7.noarch
tfm-rubygem-xmlrpc-0.3.0-2.el7.noarch
tfm-rubygem-zeitwerk-2.2.2-1.el7.noarch
tfm-runtime-6.1-1.el7.x86_64

Distribution and version:
CentOS Linux release 7.8.2003 (Core)

Other relevant data:

I believe this is due to pulp not starting correctly, but I don’t know how to fix that issue.

That said, I really don’t know what would be useful, but here’s some of the log:

[dwhitfield@dwhitfield-foreman ~]$ sudo tail /var/log/foreman/production.log
2020-09-04T14:05:52 [I|app|577a26a2]   Rendered text template (Duration: 0.1ms | Allocations: 3)
2020-09-04T14:05:52 [I|app|577a26a2] Completed 200 OK in 278ms (Views: 2.4ms | ActiveRecord: 55.9ms | Allocations: 22292)
2020-09-04T14:05:53 [I|app|f2fca745] Started POST "/api/config_reports" for 127.0.0.1 at 2020-09-04 14:05:53 -0500
2020-09-04T14:05:53 [I|app|f2fca745] Processing by Api::V2::ConfigReportsController#create as JSON
2020-09-04T14:05:53 [I|app|f2fca745]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2020-09-04T14:05:54 [I|app|f2fca745] Scanning report with: Foreman::PuppetReportScanner
2020-09-04T14:05:54 [I|app|f2fca745] Imported report for dwhitfield-foreman.openlogic.local in 282.1 ms, status refreshed in 36.1 ms
2020-09-04T14:05:54 [I|app|f2fca745]   Rendering api/v2/config_reports/create.json.rabl
2020-09-04T14:05:54 [I|app|f2fca745]   Rendered api/v2/config_reports/create.json.rabl (Duration: 53.8ms | Allocations: 8321)
2020-09-04T14:05:54 [I|app|f2fca745] Completed 201 Created in 435ms (Views: 53.7ms | ActiveRecord: 48.9ms | Allocations: 23669)
[dwhitfield@dwhitfield-foreman ~]$ sudo tail -100 /var/log/foreman/production.log
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/etag.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/conditional_get.rb:40:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/session/abstract/id.rb:266:in `context'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/session/abstract/id.rb:260:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/cookies.rb:648:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:101:in `run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:37:in `call_app'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:28:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0/lib/katello/prevent_json_parsing.rb:12:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/method_override.rb:24:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/sendfile.rb:110:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/engine.rb:527:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:74:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/configuration.rb:228:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:682:in `handle_request'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:472:in `process_client'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:328:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/thread_pool.rb:134:in `block in spawn_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-09-04T13:58:56 [I|app|] Completed 500 Internal Server Error in 2834ms (Views: 1.1ms | ActiveRecord: 104.1ms | Allocations: 117746)
2020-09-04T13:59:02 [I|app|a243c561] Started GET "/notification_recipients" for 127.0.0.1 at 2020-09-04 13:59:02 -0500
2020-09-04T13:59:02 [I|app|a243c561] Processing by NotificationRecipientsController#index as JSON
2020-09-04T13:59:02 [I|app|a243c561] Completed 200 OK in 75ms (Views: 0.5ms | ActiveRecord: 19.8ms | Allocations: 4900)
2020-09-04T13:59:12 [I|app|cdd35b4c] Started GET "/notification_recipients" for 127.0.0.1 at 2020-09-04 13:59:12 -0500
2020-09-04T13:59:12 [I|app|cdd35b4c] Processing by NotificationRecipientsController#index as JSON
2020-09-04T13:59:12 [I|app|cdd35b4c] Completed 200 OK in 23ms (Views: 0.4ms | ActiveRecord: 3.3ms | Allocations: 2157)
2020-09-04T13:59:22 [I|app|476ac7e4] Started GET "/notification_recipients" for 127.0.0.1 at 2020-09-04 13:59:22 -0500
2020-09-04T13:59:22 [I|app|476ac7e4] Processing by NotificationRecipientsController#index as JSON
2020-09-04T13:59:22 [I|app|476ac7e4] Completed 200 OK in 25ms (Views: 0.3ms | ActiveRecord: 3.5ms | Allocations: 2157)
2020-09-04T13:59:32 [I|app|12abf57e] Started GET "/notification_recipients" for 127.0.0.1 at 2020-09-04 13:59:32 -0500
2020-09-04T13:59:32 [I|app|12abf57e] Processing by NotificationRecipientsController#index as JSON
2020-09-04T13:59:32 [I|app|12abf57e] Completed 200 OK in 22ms (Views: 0.3ms | ActiveRecord: 2.8ms | Allocations: 2168)
2020-09-04T13:59:42 [I|app|45ba9e76] Started GET "/notification_recipients" for 127.0.0.1 at 2020-09-04 13:59:42 -0500
2020-09-04T13:59:42 [I|app|45ba9e76] Processing by NotificationRecipientsController#index as JSON
2020-09-04T13:59:42 [I|app|45ba9e76] Completed 200 OK in 23ms (Views: 0.3ms | ActiveRecord: 2.9ms | Allocations: 2157)
2020-09-04T13:59:52 [I|app|9ddccd4c] Started GET "/notification_recipients" for 127.0.0.1 at 2020-09-04 13:59:52 -0500
2020-09-04T13:59:52 [I|app|9ddccd4c] Processing by NotificationRecipientsController#index as JSON
2020-09-04T13:59:52 [I|app|9ddccd4c] Completed 200 OK in 22ms (Views: 0.4ms | ActiveRecord: 3.0ms | Allocations: 2157)
2020-09-04T14:00:08 [I|app|] Rails cache backend: File
2020-09-04T14:00:08 [I|app|] Rails cache backend: File
2020-09-04T14:00:15 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2020-09-04T14:00:15 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2020-09-04T14:00:18 [W|app|] Scoped order is ignored, it's forced to be batch order.
2020-09-04T14:00:18 [W|app|] Scoped order is ignored, it's forced to be batch order.
2020-09-04T14:00:18 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2020-09-04T14:00:18 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2020-09-04T14:05:44 [I|app|c5ff3cea] Started GET "/node/dwhitfield-foreman.openlogic.local?format=yml" for 127.0.0.1 at 2020-09-04 14:05:44 -0500
2020-09-04T14:05:45 [I|app|c5ff3cea] Processing by HostsController#externalNodes as YML
2020-09-04T14:05:45 [I|app|c5ff3cea]   Parameters: {"name"=>"dwhitfield-foreman.openlogic.local"}
2020-09-04T14:05:45 [I|app|c5ff3cea]   Rendering text template
2020-09-04T14:05:45 [I|app|c5ff3cea]   Rendered text template (Duration: 0.1ms | Allocations: 3)
2020-09-04T14:05:45 [I|app|c5ff3cea] Completed 200 OK in 580ms (Views: 5.2ms | ActiveRecord: 118.3ms | Allocations: 26399)
2020-09-04T14:05:49 [I|app|62d7f07f] Started POST "/api/hosts/facts" for 127.0.0.1 at 2020-09-04 14:05:49 -0500
2020-09-04T14:05:49 [I|app|62d7f07f] Processing by Api::V2::HostsController#facts as JSON
2020-09-04T14:05:49 [I|app|62d7f07f]   Parameters: {"facts"=>"[FILTERED]", "name"=>"dwhitfield-foreman.openlogic.local", "certname"=>"dwhitfield-foreman.openlogic.local", "apiv"=>"v2", "host"=>{"certname"=>"dwhitfield-foreman.openlogic.local", "name"=>"dwhitfield-foreman.openlogic.local"}}
2020-09-04T14:05:51 [I|app|62d7f07f] Import facts for 'dwhitfield-foreman.openlogic.local' completed. Added: 0, Updated: 19, Deleted 0 facts
2020-09-04T14:05:51 [I|app|62d7f07f] Completed 201 Created in 2402ms (Views: 10.6ms | ActiveRecord: 544.4ms | Allocations: 212357)
2020-09-04T14:05:51 [I|app|577a26a2] Started GET "/node/dwhitfield-foreman.openlogic.local?format=yml" for 127.0.0.1 at 2020-09-04 14:05:51 -0500
2020-09-04T14:05:51 [I|app|577a26a2] Processing by HostsController#externalNodes as YML
2020-09-04T14:05:51 [I|app|577a26a2]   Parameters: {"name"=>"dwhitfield-foreman.openlogic.local"}
2020-09-04T14:05:52 [I|app|577a26a2]   Rendering text template
2020-09-04T14:05:52 [I|app|577a26a2]   Rendered text template (Duration: 0.1ms | Allocations: 3)
2020-09-04T14:05:52 [I|app|577a26a2] Completed 200 OK in 278ms (Views: 2.4ms | ActiveRecord: 55.9ms | Allocations: 22292)
2020-09-04T14:05:53 [I|app|f2fca745] Started POST "/api/config_reports" for 127.0.0.1 at 2020-09-04 14:05:53 -0500
2020-09-04T14:05:53 [I|app|f2fca745] Processing by Api::V2::ConfigReportsController#create as JSON
2020-09-04T14:05:53 [I|app|f2fca745]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2020-09-04T14:05:54 [I|app|f2fca745] Scanning report with: Foreman::PuppetReportScanner
2020-09-04T14:05:54 [I|app|f2fca745] Imported report for dwhitfield-foreman.openlogic.local in 282.1 ms, status refreshed in 36.1 ms
2020-09-04T14:05:54 [I|app|f2fca745]   Rendering api/v2/config_reports/create.json.rabl
2020-09-04T14:05:54 [I|app|f2fca745]   Rendered api/v2/config_reports/create.json.rabl (Duration: 53.8ms | Allocations: 8321)
2020-09-04T14:05:54 [I|app|f2fca745] Completed 201 Created in 435ms (Views: 53.7ms | ActiveRecord: 48.9ms | Allocations: 23669)`
2

in increased my RAM and CPU and got rid of the pulp issue, but the inability to create repos remains

3

well, it got rid of the pulp not starting issue. There are no logs in /var/log/pulp. I’m not sure if I should look elsewhere or increase logging.

4

Hi @douglasawh

The tail -100 was almost there but not quite. I see

[I|app|] Completed 500 Internal Server Error

at the end of the error stack. What I’d like to see is the beginning of that request, so we can see why it’s sending the internal server error.

Also check the output of sudo foreman-maintain service status – does it show the following?

All services are running                                            [OK]
1 Like
5

@jeremylenz so, eventually it shows the “All services are running”, but there is a lot of other information as well. I will attempt to get some more log information here in a bit.

1 Like
6

Idk if the error message from Friday is the most useful considering things have changed (adding more RAM and allowing pulp to start), but I figured consistent timestamps would be useful. I do want to note that the issue is the same whether I try to add a yum repo or a container repo. I tried a handful of both.

[ sidekiq ] [ concurrent-ruby ] 2020-09-04T13:58:56 [I|bac|] Task {label: Actions::Katello::Repository::CreateRoot, id: 2438c719-7853-470b-9945-4265e23b6c4c, execution_plan_id: 33f84cb7-ac39-43cf-9606-8cd00bd88750} state changed: paused result: error 2020-09-04T13:58:56 [E|app|] ForemanTasks::TaskError: Task 2438c719-7853-470b-9945-4265e23b6c4c: PulpRpmClient::ApiError: Error message: the server returns an error HTTP status code: 500 Response headers: {"date"=>"Fri, 04 Sep 2020 18:58:56 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"text/html; charset=UTF-8", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"27", "vary"=>"Cookie", "via"=>"1.1 dwhitfield-foreman.openlogic.local", "connection"=>"close"} Response body: <h1>Server Error (500)</h1> /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/lib/foreman_tasks.rb:59:inblock in sync_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/lib/foreman_tasks.rb:58:in tap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/lib/foreman_tasks.rb:58:in sync_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/lib/foreman_tasks/triggers.rb:22:in sync_task' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0/app/controllers/katello/api/v2/repositories_controller.rb:242:in create’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/basic_implicit_render.rb:6:in send_action' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:195:in process_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rendering.rb:30:in process_action' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:42:in block in process_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:112:in block in run_callbacks' /usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in disable_json_root’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in block in run_callbacks' /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in set_timezone’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in block in run_callbacks' /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in clear_thread’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in block in run_callbacks' /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in set_topbar_sweeper_controller’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in block in run_callbacks' /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in around’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in block in run_callbacks' /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in around’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in block in run_callbacks' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:139:in run_callbacks’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:41:in process_action' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rescue.rb:22:in process_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:33:in block in process_action' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in block in instrument’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications/instrumenter.rb:24:in instrument' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in instrument’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:32:in process_action' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/params_wrapper.rb:245:in process_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/railties/controller_runtime.rb:27:in process_action' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0/app/controllers/katello/concerns/api/api_controller.rb:40:in process_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:136:in process' /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.1/lib/action_view/rendering.rb:39:in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:190:in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:254:in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:50:in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:33:in serve’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:18:in block in <class:Constraints>' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:48:in serve’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:49:in block in serve' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in serve' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:834:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/engine.rb:527:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in public_send’
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in method_missing' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:19:in block in class:Constraints’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:48:in serve' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:49:in block in serve’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in serve’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:834:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0/lib/katello/middleware/event_daemon.rb:10:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.2.2/lib/apipie_dsl/static_dispatcher.rb:67:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in call' /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in call' /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/tempfile_reaper.rb:15:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/etag.rb:27:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/conditional_get.rb:40:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/head.rb:12:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/http/content_security_policy.rb:18:in call' /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/session/abstract/id.rb:266:in context' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/session/abstract/id.rb:260:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/cookies.rb:648:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in block in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:101:in run_callbacks' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:37:in call_app’
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:28:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in call’
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/remote_ip.rb:81:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/request_id.rb:27:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0/lib/katello/prevent_json_parsing.rb:12:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/method_override.rb:24:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/runtime.rb:22:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/sendfile.rb:110:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:76:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/engine.rb:527:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in public_send’
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in method_missing' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:74:in block in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/configuration.rb:228:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:682:in handle_request’
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:472:in process_client' /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:328:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/thread_pool.rb:134:in block in spawn_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’
2020-09-04T13:58:56 [I|app|] Completed 500 Internal Server Error in 2834ms (Views: 1.1ms | ActiveRecord: 104.1ms | Allocations: 117746)`

7

Thanks @douglasawh

Do you see anything interesting in /var/log/messages?

8

I did a little more testing, and it doesn’t matter whether I use repo discovery (what I was doing before), or if I try to go in and manually add a repo.

The following might be interesting from sudo grep -i err /var/log/messages

Sep  8 15:22:20 dwhitfield-foreman pulpcore-api: pulp: django.request:ERROR: Internal Server Error: /pulp/api/v3/repositories/rpm/rpm/
Sep  8 15:22:20 dwhitfield-foreman pulpcore-api: psycopg2.errors.UndefinedColumn: column "retain_package_versions" of relation "rpm_rpmrepository" does not exist
Sep  8 15:22:20 dwhitfield-foreman pulpcore-api: django.db.utils.ProgrammingError: column "retain_package_versions" of relation "rpm_rpmrepository" does not exist

Based off of the above output, decided to refine the grep, and got this:
[dwhitfield@dwhitfield-foreman yum.repos.d]$ sudo grep ERR /var/log/messages
Sep 6 22:00:28 dwhitfield-foreman pulpcore-api: pulp: django.request:ERROR: Internal Server Error: /pulp/api/v3/repositories/container/container/39f2a0fa-6de9-41a3-8746-2b3d45f8c876/versions/
Sep 8 15:22:20 dwhitfield-foreman pulpcore-api: pulp: django.request:ERROR: Internal Server Error: /pulp/api/v3/repositories/rpm/rpm/

9

I had the thought that at some point I was running a 3.16 RC and maybe some stuff had gotten update.

Perhaps this is unrelated to the initial issue, but I’m suspicious.

[dwhitfield@dwhitfield-foreman yum.repos.d]$ sudo foreman-installer
Preparing installation Done
Executing: foreman-rake upgrade:run
=============================================
Upgrade Step 1/3: katello:correct_repositories. This may take a long while.
Processing Repository 1/10: Perforce VCS Repo (1)
Repository 1 Missing
Recreating 1
Failed upgrade task: katello:correct_repositories, see logs for more information.
10

Can you try running:

sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' python3-django-admin migrate

then:

foreman-maintain service restart

I’m not sure why foreman-installer wouldn’t have taken care of this…

1 Like
11

The errors from yesterday are gone in /var/log/messages, but I am still getting unexpected errors when attempting to add a repo. The thing about this being unauthorized is it is a public repo.

[dwhitfield@dwhitfield-foreman ~]$ sudo grep error /var/log/foreman/production.log
2020-09-09T14:00:04 [I|app|37624340]   Rendering api/v2/errors/unauthorized.json.rabl within api/v2/layouts/error_layout
2020-09-09T14:00:04 [I|app|37624340]   Rendered api/v2/errors/unauthorized.json.rabl within api/v2/layouts/error_layout (Duration: 3.9ms | Allocations: 314)
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:475:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:475:in `with_error_handling'
2020-09-09T14:03:37 [I|bac|] Task {label: Actions::Katello::Repository::CreateRoot, id: ead28cc3-60fc-40b4-92ae-ac6d845617dd, execution_plan_id: 5599f32c-8919-463f-8de3-2eb421f419df} state changed: paused  result: error
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
12

can you look in monitor > tasks and look for a create repository task, its likely paused, see if you can spot an error in a traceback within that task

13

It is unclear the exact sequence of events that got this working, but it is working. I’m going to mark the PULP_SETTINGS change as the solution so it’s clear to future humans that this has been resolved, but it’s far from clear that that post contains the entire solution.

I did go ahead and test and make sure the repo actually synced to make sure there weren’t any other issues lurking about.

Thanks everyone for the help!

1 Like