Foreman 1.20.1/Katello 3.10 fails to create a repo for a product

Problem:
I have successfully created a new product and I’m trying to attache a repo to that product. In this case I have tried to add repo’s for Fedora and Ubuntu and both fail with the message “undefined method `pulp_api’ for nil:NilClass”. I followed the available Katello 3.10 doc’s to no avail - “katello-servce status” returns green across the board and no obvious errors are returned by pulp itself, but it’s possible I have missed something in a log file somewhere else from a different part of the suite (I’ll happily take suggestions as to where to look for such). I haven’t found any obvious returns for that error from google either. This setup is running on a RHEL7 server if that matters.

Expected outcome:
The repo gets added without complaint, so I can set about sync’ing it.

Foreman and Proxy versions:

candlepin.noarch                       2.5.7-1.el7         @katello-candlepin   
candlepin-selinux.noarch               2.5.7-1.el7         @katello-candlepin   
foreman.noarch                         1.20.1-1.el7        @foreman             
foreman-cli.noarch                     1.20.1-1.el7        @foreman             
foreman-compute.noarch                 1.20.1-1.el7        @foreman             
foreman-debug.noarch                   1.20.1-1.el7        @foreman             
foreman-installer.noarch               1:1.20.1-1.el7      @foreman             
foreman-installer-katello.noarch       3.10.0-1.el7        @katello             
foreman-postgresql.noarch              1.20.1-1.el7        @foreman             
foreman-proxy.noarch                   1.20.1-1.el7        @foreman             
foreman-release.noarch                 1.20.1-1.el7        installed            
foreman-selinux.noarch                 1.20.1-1.el7        @foreman             
gofer.noarch                           2.12.5-1.el7        @pulp                
is-nix-q01.uniwadev.uwa.edu.au-foreman-client.noarch
is-nix-q01.uniwadev.uwa.edu.au-foreman-proxy.noarch
is-nix-q01.uniwadev.uwa.edu.au-foreman-proxy-client.noarch
katello.noarch                         3.10.0-7.el7        @katello             
katello-agent.noarch                   3.3.5-3.el7sat      @rhel-7-server-satellite-tools-6.4-rpms
katello-ca-consumer-sat.is.uwa.edu.au.noarch
katello-certs-tools.noarch             2.4.0-1.el7         @katello             
katello-client-bootstrap.noarch        1.6.0-1.el7         @katello             
katello-common.noarch                  3.10.0-7.el7        @katello             
katello-debug.noarch                   3.10.0-7.el7        @katello             
katello-default-ca.noarch              1.0-1               installed            
katello-host-tools.noarch              3.3.5-3.el7sat      @rhel-7-server-satellite-tools-6.4-rpms
katello-host-tools-fact-plugin.noarch  3.3.5-3.el7sat      @rhel-7-server-satellite-tools-6.4-rpms
katello-installer-base.noarch          3.10.0-1.el7        @katello             
katello-repos.noarch                   3.10.0-2.el7        installed            
katello-selinux.noarch                 3.0.3-1.el7         @katello             
katello-server-ca.noarch               1.0-1               installed            
katello-service.noarch                 3.10.0-7.el7        @katello             
libmodulemd.x86_64                     1.6.3-1.el7         @pulp                
libsolv.x86_64                         0.6.34-2.pulp.el7   @pulp                
pulp-admin-client.noarch               2.18.0-2.el7        @pulp                
pulp-client.noarch                     1.0-1               installed            
pulp-deb-admin-extensions.noarch       1.8.0-1.el7         @pulp                
pulp-deb-plugins.noarch                1.8.0-1.el7         @pulp                
pulp-docker-plugins.noarch             3.2.2-1.el7         @pulp                
pulp-katello.noarch                    1.0.2-1.el7         @katello             
pulp-puppet-plugins.noarch             2.18.0-1.el7        @pulp                
pulp-puppet-tools.noarch               2.18.0-1.el7        @pulp                
pulp-rpm-admin-extensions.noarch       2.18.0-1.el7        @pulp                
pulp-rpm-plugins.noarch                2.18.0-1.el7        @pulp                
pulp-rpm-yumplugins.noarch             2.18.0-1.el7        @pulp                
pulp-selinux.noarch                    2.18.0-2.el7        @pulp                
pulp-server.noarch                     2.18.0-2.el7        @pulp                
python-bson.x86_64                     3.2-2.el7           @pulp                
python-crane.noarch                    3.3.0-1.el7         @pulp                
python-gofer.noarch                    2.12.5-1.el7        @pulp                
python-gofer-proton.noarch             2.12.5-1.el7        @pulp                
python-gofer-qpid.noarch               2.12.5-1.el7        @pulp                
python-isodate.noarch                  0.5.0-5.pulp.el7sat @rhel-7-server-satellite-tools-6.4-rpms
python-mongoengine.noarch              0.10.5-1.el7        @pulp                
python-nectar.noarch                   1.5.6-1.el7         @pulp                
python-pulp-bindings.noarch            2.18.0-2.el7        @pulp                
python-pulp-client-lib.noarch          2.18.0-2.el7        @pulp                
python-pulp-common.noarch              2.18.0-2.el7        @pulp                
python-pulp-deb-common.noarch          1.8.0-1.el7         @pulp                
python-pulp-docker-common.noarch       3.2.2-1.el7         @pulp                
python-pulp-oid_validation.noarch      2.18.0-2.el7        @pulp                
python-pulp-puppet-common.noarch       2.18.0-1.el7        @pulp                
python-pulp-repoauth.noarch            2.18.0-2.el7        @pulp                
python-pulp-rpm-common.noarch          2.18.0-1.el7        @pulp                
python-pulp-streamer.noarch            2.18.0-2.el7        @pulp                
python-pymongo.x86_64                  3.2-2.el7           @pulp                
python-pymongo-gridfs.x86_64           3.2-2.el7           @pulp                
python2-debpkgr.noarch                 1.1.0-1.el7         @pulp                
python2-solv.x86_64                    0.6.34-2.pulp.el7   @pulp                
rubygem-bundler_ext.noarch             0.4.1-3.el7         @foreman             
rubygem-clamp.noarch                   1.1.2-4.el7         @foreman             
rubygem-concurrent-ruby.noarch         1:1.0.3-6.el7       @foreman             
rubygem-foreman_maintain.noarch        0.3.0-1.el7         @foreman             
rubygem-hashie.noarch                  3.6.0-1.el7         @foreman             
rubygem-highline.noarch                1.7.8-4.el7         @foreman             
rubygem-jwt.noarch                     1.2.1-1.el7         @foreman             
rubygem-kafo.noarch                    2.1.0-1.el7         @foreman             
rubygem-kafo_parsers.noarch            0.1.6-1.el7         @foreman             
rubygem-kafo_wizards.noarch            0.0.1-2.el7         @foreman             
rubygem-little-plugger.noarch          1.1.3-23.el7        @foreman             
rubygem-logging.noarch                 2.2.2-3.el7         @foreman             
rubygem-multi_json.noarch              1.12.2-3.el7        @foreman             
rubygem-powerbar.noarch                2.0.1-1.el7         @foreman             
rubygem-rb-inotify.noarch              0.9.7-2.el7         @foreman             
rubygem-rsec.noarch                    0.4.2-1.el7         @foreman             
rubygem-rubyipmi.noarch                0.10.0-3.el7        @foreman             
rubygem-smart_proxy_pulp.noarch        1.3.0-1.el7         @foreman-plugins     
tfm-ror52-rubygem-actioncable.noarch   5.2.1-1.el7         @foreman-rails       
tfm-ror52-rubygem-actionmailer.noarch  5.2.1-1.el7         @foreman-rails       
tfm-ror52-rubygem-actionpack.noarch    5.2.1-1.el7         @foreman-rails       
tfm-ror52-rubygem-actionview.noarch    5.2.1-1.el7         @foreman-rails       
tfm-ror52-rubygem-activejob.noarch     5.2.1-1.el7         @foreman-rails       
tfm-ror52-rubygem-activemodel.noarch   5.2.1-1.el7         @foreman-rails       
tfm-ror52-rubygem-activerecord.noarch  5.2.1-1.el7         @foreman-rails       
tfm-ror52-rubygem-activestorage.noarch 5.2.1-1.el7         @foreman-rails       
tfm-ror52-rubygem-activesupport.noarch 5.2.1-1.el7         @foreman-rails       
tfm-ror52-rubygem-arel.noarch          9.0.0-1.el7         @foreman-rails       
tfm-ror52-rubygem-builder.noarch       3.2.3-1.el7         @foreman-rails       
tfm-ror52-rubygem-coffee-rails.noarch  4.2.2-1.el7         @foreman-rails       
tfm-ror52-rubygem-coffee-script.noarch 2.4.1-1.el7         @foreman-rails       
                                       1.12.2-1.el7        @foreman-rails       
                                       1.0.5-1.el7         @foreman-rails       
tfm-ror52-rubygem-crass.noarch         1.0.4-1.el7         @foreman-rails       
tfm-ror52-rubygem-erubi.noarch         1.7.1-1.el7         @foreman-rails       
tfm-ror52-rubygem-execjs.noarch        2.7.0-1.el7         @foreman-rails       
tfm-ror52-rubygem-globalid.noarch      0.4.1-3.el7         @foreman-rails       
tfm-ror52-rubygem-i18n.noarch          1.0.1-1.el7         @foreman-rails       
tfm-ror52-rubygem-loofah.noarch        2.2.2-2.el7         @foreman-rails       
tfm-ror52-rubygem-mail.noarch          2.7.0-1.el7         @foreman-rails       
tfm-ror52-rubygem-marcel.noarch        0.3.2-1.el7         @foreman-rails       
tfm-ror52-rubygem-method_source.noarch 0.9.0-1.el7         @foreman-rails       
tfm-ror52-rubygem-mime-types.noarch    3.2.2-1.el7         @foreman-rails       
                                       3.2018.0812-1.el7   @foreman-rails       
tfm-ror52-rubygem-mimemagic.noarch     0.3.2-1.el7         @foreman-rails       
tfm-ror52-rubygem-mini_mime.noarch     1.0.0-1.el7         @foreman-rails       
tfm-ror52-rubygem-mini_portile2.noarch 2.3.0-1.el7         @foreman-rails       
tfm-ror52-rubygem-multi_json.noarch    1.13.1-1.el7        @foreman-rails       
tfm-ror52-rubygem-mustermann.noarch    1.0.2-1.el7         @foreman-rails       
tfm-ror52-rubygem-nio4r.x86_64         2.3.1-1.el7         @foreman-rails       
tfm-ror52-rubygem-nokogiri.x86_64      1.8.4-2.el7         @foreman-rails       
tfm-ror52-rubygem-rack.noarch          2.0.5-2.el7         @foreman-rails       
                                       2.0.3-1.el7         @foreman-rails       
tfm-ror52-rubygem-rack-test.noarch     1.1.0-1.el7         @foreman-rails       
tfm-ror52-rubygem-rails.noarch         5.2.1-1.el7         @foreman-rails       
                                       2.0.3-3.el7         @foreman-rails       
                                       1.0.4-2.el7         @foreman-rails       
tfm-ror52-rubygem-railties.noarch      5.2.1-1.el7         @foreman-rails       
tfm-ror52-rubygem-sinatra.noarch       2.0.3-1.el7         @foreman-rails       
tfm-ror52-rubygem-sprockets.noarch     3.7.2-3.el7         @foreman-rails       
                                       3.2.1-1.el7         @foreman-rails       
tfm-ror52-rubygem-thor.noarch          0.20.0-3.el7        @foreman-rails       
tfm-ror52-rubygem-thread_safe.noarch   0.3.6-1.el7         @foreman-rails       
tfm-ror52-rubygem-tilt.noarch          2.0.8-1.el7         @foreman-rails       
tfm-ror52-rubygem-turbolinks.noarch    2.5.4-1.el7         @foreman-rails       
tfm-ror52-rubygem-tzinfo.noarch        1.2.5-1.el7         @foreman-rails       
                                       0.7.0-1.el7         @foreman-rails       
                                       0.1.3-1.el7         @foreman-rails       
tfm-ror52-runtime.x86_64               1.0-4.el7           @foreman-rails       
                                       1.1.1-2.el7         @foreman             
tfm-rubygem-addressable.noarch         2.3.6-6.el7         @foreman             
tfm-rubygem-algebrick.noarch           0.7.3-6.el7         @foreman             
tfm-rubygem-ancestry.noarch            3.0.0-3.el7         @foreman             
tfm-rubygem-anemone.noarch             0.7.2-16.el7        @katello             
                                       1:1.0.2-4.el7       @foreman-plugins     
tfm-rubygem-apipie-bindings.noarch     0.2.2-2.el7         @foreman             
tfm-rubygem-apipie-params.noarch       0.0.5-3.el7         @foreman             
tfm-rubygem-apipie-rails.noarch        0.5.14-1.el7        @foreman             
tfm-rubygem-audited.noarch             4.7.1-2.el7         @foreman             
tfm-rubygem-awesome_print.noarch       1.8.0-3.el7         @foreman             
tfm-rubygem-bastion.noarch             6.1.16-1.fm1_20.el7 @foreman-plugins     
tfm-rubygem-bundler_ext.noarch         0.4.1-3.el7         @foreman             
tfm-rubygem-clamp.noarch               1.1.2-4.el7         @foreman             
                                       1:0.2.4-2.el7       @foreman             
tfm-rubygem-css_parser.noarch          1.4.7-3.el7         @foreman             
tfm-rubygem-daemons.noarch             1.2.3-5.el7         @foreman             
tfm-rubygem-deacon.noarch              1.0.0-3.el7         @foreman             
tfm-rubygem-deep_cloneable.noarch      2.3.2-2.el7         @foreman             
tfm-rubygem-deface.noarch              1.3.2-1.el7         @foreman-plugins     
tfm-rubygem-docker-api.noarch          1.28.0-4.el7        @foreman-plugins     
tfm-rubygem-dynflow.noarch             1.1.2-1.fm1_20.el7  @foreman             
tfm-rubygem-excon.noarch               0.58.0-3.el7        @foreman             
tfm-rubygem-facter.x86_64              2.4.0-6.el7         @foreman             
tfm-rubygem-fast_gettext.noarch        1.4.1-3.el7         @foreman             
tfm-rubygem-fog.noarch                 1.42.1-2.el7        @foreman             
tfm-rubygem-fog-aws.noarch             1.3.0-3.el7         @foreman             
tfm-rubygem-fog-core.noarch            1.45.0-3.el7        @foreman             
tfm-rubygem-fog-digitalocean.noarch    0.3.0-3.el7         @foreman             
tfm-rubygem-fog-google.noarch          0.1.0-4.el7         @foreman             
tfm-rubygem-fog-json.noarch            1.0.2-6.el7         @foreman             
tfm-rubygem-fog-openstack.noarch       0.1.25-2.el7        @foreman             
tfm-rubygem-fog-ovirt.noarch           1.1.2-2.el7         @foreman             
tfm-rubygem-fog-rackspace.noarch       0.1.4-3.el7         @foreman             
tfm-rubygem-fog-vsphere.noarch         2.3.0-2.el7         @foreman             
tfm-rubygem-fog-xenserver.noarch       0.2.3-3.el7         @foreman             
tfm-rubygem-fog-xml.noarch             0.1.2-6.el7         @foreman             
tfm-rubygem-foreman-tasks.noarch       0.14.3-1.fm1_20.el7 @foreman-plugins     
tfm-rubygem-foreman-tasks-core.noarch  0.2.5-2.fm1_20.el7  @foreman-plugins     
tfm-rubygem-foreman_docker.noarch      4.1.0-2.fm1_20.el7  @foreman-plugins     
tfm-rubygem-formatador.noarch          0.2.1-11.el7        @foreman             
tfm-rubygem-friendly_id.noarch         5.2.4-2.el7         @foreman             
tfm-rubygem-get_process_mem.noarch     0.2.1-3.el7         @foreman             
tfm-rubygem-gettext_i18n_rails.noarch  1.2.1-5.el7         @foreman             
tfm-rubygem-hammer_cli.noarch          0.15.1-1.el7        @foreman             
tfm-rubygem-hammer_cli_foreman.noarch  0.15.1-1.el7        @foreman             
tfm-rubygem-hammer_cli_foreman_bootdisk.noarch
tfm-rubygem-hammer_cli_foreman_docker.noarch
tfm-rubygem-hammer_cli_foreman_tasks.noarch
                                       0.0.13-1.fm1_20.el7 @foreman-plugins     
tfm-rubygem-hammer_cli_katello.noarch  0.16.0-1.el7        @katello             
tfm-rubygem-hashie.noarch              3.6.0-1.el7         @foreman             
tfm-rubygem-highline.noarch            1.7.8-4.el7         @foreman             
tfm-rubygem-ipaddress.noarch           0.8.0-11.el7        @foreman             
tfm-rubygem-katello.noarch             3.10.0-1.el7        @katello             
tfm-rubygem-ldap_fluff.noarch          0.4.7-3.el7         @foreman             
tfm-rubygem-little-plugger.noarch      1.1.3-23.el7        @foreman             
tfm-rubygem-locale.noarch              2.0.9-13.el7        @foreman             
tfm-rubygem-net-ldap.noarch            0.15.0-3.el7        @foreman             
tfm-rubygem-net-ping.noarch            2.0.1-3.el7         @foreman             
tfm-rubygem-net-scp.noarch             1.2.1-3.el7         @foreman             
tfm-rubygem-net-ssh.noarch             4.0.1-5.el7         @foreman             
tfm-rubygem-netrc.noarch               0.11.0-3.el7        @foreman             
tfm-rubygem-oauth.noarch               0.5.4-3.el7         @foreman             
tfm-rubygem-ovirt-engine-sdk.x86_64    4.2.3-3.el7         @foreman             
tfm-rubygem-parse-cron.noarch          0.1.4-4.fm1_20.el7  @foreman-plugins     
tfm-rubygem-passenger.x86_64           4.0.18-10.12.el7    @foreman             
tfm-rubygem-passenger-native.x86_64    4.0.18-10.12.el7    @foreman             
                                       4.0.18-10.12.el7    @foreman             
tfm-rubygem-pg.x86_64                  0.21.0-3.el7        @foreman             
tfm-rubygem-polyglot.noarch            0.3.5-2.el7         @foreman-plugins     
tfm-rubygem-powerbar.noarch            2.0.1-1.el7         @foreman             
tfm-rubygem-qpid_messaging.x86_64      1.38.0-1.el7        @katello             
tfm-rubygem-rabl.noarch                0.13.1-2.el7        @foreman             
tfm-rubygem-rack-jsonp.noarch          1.3.1-7.el7         @foreman             
tfm-rubygem-rails-i18n.noarch          5.1.1-2.el7         @foreman             
tfm-rubygem-rainbow.noarch             2.2.1-3.el7         @foreman-plugins     
tfm-rubygem-rbovirt.noarch             0.1.7-2.el7         @foreman             
tfm-rubygem-rbvmomi.noarch             1.10.0-3.el7        @foreman             
tfm-rubygem-record_tag_helper.noarch   1.0.0-2.el7         @foreman             
tfm-rubygem-responders.noarch          2.4.0-2.el7         @foreman             
tfm-rubygem-rest-client.noarch         2.0.1-3.el7         @foreman             
tfm-rubygem-roadie.noarch              3.2.2-2.el7         @foreman             
tfm-rubygem-roadie-rails.noarch        1.3.0-2.el7         @foreman             
tfm-rubygem-robotex.noarch             1.0.0-21.el7        @katello             
tfm-rubygem-ruby2ruby.noarch           2.4.0-2.el7         @foreman             
tfm-rubygem-ruby_parser.noarch         3.10.1-2.el7        @foreman             
tfm-rubygem-runcible.noarch            2.10.1-1.el7        @katello             
tfm-rubygem-safemode.noarch            1.3.5-2.el7         @foreman             
tfm-rubygem-scoped_search.noarch       4.1.6-1.el7         @foreman             
tfm-rubygem-secure_headers.noarch      6.0.0-1.el7         @foreman             
tfm-rubygem-sequel.noarch              5.7.1-2.el7         @foreman             
tfm-rubygem-sexp_processor.noarch      4.10.0-5.el7        @foreman             
tfm-rubygem-sshkey.noarch              1.9.0-3.el7         @foreman             
tfm-rubygem-statsd-instrument.noarch   2.1.4-2.el7         @foreman             
tfm-rubygem-trollop.noarch             2.1.2-3.el7         @foreman             
tfm-rubygem-unf.noarch                 0.1.3-7.el7         @foreman             
                                       0.5.0-5.el7         @foreman             
tfm-rubygem-webpack-rails.noarch       0.9.8-3.el7         @foreman             
tfm-rubygem-wicked.noarch              1.3.3-1.el7         @foreman-plugins     
tfm-rubygem-will_paginate.noarch       3.1.5-3.el7         @foreman             
tfm-rubygem-x-editable-rails.noarch    1.5.5-3.el7         @foreman             
tfm-runtime.x86_64                     5.0-2.el7           @foreman             

Foreman and Proxy plugin versions:

Other relevant data:
Pulp reported as follows for my last attempt:

Feb 13 12:06:13 is-nix-q01 pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Feb 13 12:06:13 is-nix-q01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[8c53fa62-37a5-4065-85cd-148c51806202]
Feb 13 12:06:13 is-nix-q01 pulp: kombu.transport.qpid:INFO: [8c53fa62] Connected to qpid with SASL mechanism ANONYMOUS
Feb 13 12:06:13 is-nix-q01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[d6efc0d9-f354-47ee-a016-1a6b6c15965e]
Feb 13 12:06:13 is-nix-q01 pulp: py.warnings:WARNING: [8c53fa62] (7452-12256) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Feb 13 12:06:13 is-nix-q01 pulp: py.warnings:WARNING: [8c53fa62] (7452-12256)   "MongoClient opened before fork. Create MongoClient "
Feb 13 12:06:13 is-nix-q01 pulp: py.warnings:WARNING: [8c53fa62] (7452-12256)
Feb 13 12:06:13 is-nix-q01 pulp: celery.app.trace:INFO: [8c53fa62] Task pulp.server.controllers.repository.queue_download_deferred[8c53fa62-37a5-4065-85cd-148c51806202] succeeded in 0.0830746419999s: None
Feb 13 12:06:13 is-nix-q01 pulp: py.warnings:WARNING: [d6efc0d9] (7440-18656) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Feb 13 12:06:13 is-nix-q01 pulp: py.warnings:WARNING: [d6efc0d9] (7440-18656)   "MongoClient opened before fork. Create MongoClient "
Feb 13 12:06:13 is-nix-q01 pulp: py.warnings:WARNING: [d6efc0d9] (7440-18656)
Feb 13 12:06:13 is-nix-q01 pulp: celery.app.trace:INFO: [d6efc0d9] Task pulp.server.controllers.repository.download_deferred[d6efc0d9-f354-47ee-a016-1a6b6c15965e] succeeded in 0.025652239s: None
Feb 13 12:07:09 is-nix-q01 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS


Now that I have had the chance to returnt o this issue again… Inside the foreman logs after I attempt to save a repo, I find the following. Does this help narrow it down?

2019-02-22T14:16:00 [I|app|d9790] Processing by Katello::Api::V2::RepositoriesController#create as JSON
2019-02-22T14:16:00 [I|app|d9790]   Parameters: {"product_id"=>"1", "unprotected"=>true, "checksum_type"=>nil, "mirror_on_sync"=>true, "verify_ssl_on_sync"=>true, "download_policy"=>"on_demand", "arch"=>"noarch", "ostree_upstream_sync_policy"=>"latest", "name"=>"Fedora Core 29 Workstation_x86_64", "label"=>"Fedora_Core_29_Workstation_x86_64", "content_type"=>"yum", "url"=>"http://mirror.aarnet.edu.au/pub/fedora/linux/releases/29/Workstation/x86_64/", "ignorable_content"=>["srpm"], "organization_id"=>"1", "api_version"=>"v2", "repository"=>{"name"=>"Fedora Core 29 Workstation_x86_64", "arch"=>"noarch", "label"=>"Fedora_Core_29_Workstation_x86_64", "url"=>"http://mirror.aarnet.edu.au/pub/fedora/linux/releases/29/Workstation/x86_64/", "unprotected"=>true, "content_type"=>"yum", "product_id"=>"1", "checksum_type"=>nil, "mirror_on_sync"=>true, "download_policy"=>"on_demand", "verify_ssl_on_sync"=>true, "ostree_upstream_sync_policy"=>"latest", "ignorable_content"=>["srpm"]}}
2019-02-22T14:16:00 [I|app|d9790] Current user set to admin (admin)
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on name Fedora Core 29 Workstation_x86_64
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on major 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on minor 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on gpg_key_id 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on content_id 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on arch noarch
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on label Fedora_Core_29_Workstation_x86_64
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on url http://mirror.aarnet.edu.au/pub/fedora/linux/releases/29/Workstation/x86_64/
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on unprotected true
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on content_type yum
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on product_id 1
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on checksum_type 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on pulp_scratchpad_checksum_type 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on docker_upstream_name 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on mirror_on_sync true
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on download_policy on_demand
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on verify_ssl_on_sync true
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on upstream_username 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on upstream_password 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on ostree_upstream_sync_policy 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on ostree_upstream_sync_depth 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on deb_releases 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on deb_components 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on deb_architectures 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on ignore_global_proxy false
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on ssl_ca_cert_id 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on ssl_client_cert_id 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on ssl_client_key_id 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on ignorable_content ["srpm"]
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on docker_tags_whitelist 
2019-02-22T14:16:00 [I|aud|d9790] Katello::RootRepository (16) create event on description 
2019-02-22T14:16:00 [I|kat|d9790] GET: https://XXXXXXXXXX.uniwadev.uwa.edu.au/pulp/api/v2/users/: {"content_type"=>"application/json", "accept"=>"application/json"}
Response: 200: [{"_id": {"$oid": "5c5d3bff5ea3e75eab4d32da"}, "name": "admin", "roles": ["super-users"], "_ns": "users", "login": "admin", "_href": "/pulp/api/v2/users/admin/"}]
2019-02-22T14:16:00 [I|app|d9790] Current user set to admin (admin)
2019-02-22T14:16:00 [E|bac|d9790] undefined method `pulp_api' for nil:NilClass (NoMethodError)
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/services/katello/pulp/repository.rb:63:in `create'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/lib/actions/pulp/repository/create_in_plan.rb:8:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:493:in `block (3 levels) in execute_plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/lib/actions/middleware/backend_services_check.rb:23:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/lib/actions/middleware/remote_action.rb:9:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:35:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:35:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_user.rb:31:in `with_current_user'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:35:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/world.rb:30:in `execute'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:492:in `block (2 levels) in execute_plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:396:in `concurrence'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:491:in `block in execute_plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:451:in `block in with_error_handling'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:451:in `catch'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:451:in `with_error_handling'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:490:in `execute_plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:278:in `execute'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan/steps/abstract.rb:162:in `with_meta_calculation'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:424:in `plan_action'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/lib/actions/katello/repository/create.rb:14:in `block in plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:401:in `sequence'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/lib/actions/katello/repository/create.rb:13:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:493:in `block (3 levels) in execute_plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:35:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:35:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_user.rb:31:in `with_current_user'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:35:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/world.rb:30:in `execute'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:492:in `block (2 levels) in execute_plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:396:in `concurrence'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:491:in `block in execute_plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:451:in `block in with_error_handling'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:451:in `catch'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:451:in `with_error_handling'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:490:in `execute_plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:278:in `execute'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan/steps/abstract.rb:162:in `with_meta_calculation'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:424:in `plan_action'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/lib/actions/katello/repository/create_root.rb:13:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:493:in `block (3 levels) in execute_plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:35:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:35:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_user.rb:31:in `with_current_user'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:35:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/world.rb:30:in `execute'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:492:in `block (2 levels) in execute_plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:396:in `concurrence'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:491:in `block in execute_plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:451:in `block in with_error_handling'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:451:in `catch'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:451:in `with_error_handling'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:490:in `execute_plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/action.rb:278:in `execute'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan/steps/abstract.rb:162:in `with_meta_calculation'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan.rb:286:in `block (2 levels) in plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan.rb:374:in `with_planning_scope'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan.rb:285:in `block in plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:43:in `plan_phase'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:43:in `plan_phase'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:43:in `plan_phase'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:43:in `plan_phase'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:26:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware.rb:18:in `pass'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/common/transaction.rb:16:in `block in rollback_on_error'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/var/root-opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `transaction'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:212:in `transaction'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/transaction_adapters/active_record.rb:5:in `transaction'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/common/transaction.rb:15:in `rollback_on_error'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/common/transaction.rb:5:in `plan_phase'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/stack.rb:22:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/middleware/world.rb:30:in `execute'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/execution_plan.rb:284:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/world.rb:200:in `block in plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/world.rb:198:in `tap'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/world.rb:198:in `plan'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.2/lib/dynflow/world.rb:174:in `trigger'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/lib/foreman_tasks.rb:19:in `trigger'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/lib/foreman_tasks.rb:25:in `block in trigger_task'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/lib/foreman_tasks.rb:45:in `block in rails_safe_trigger_task'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/lib/foreman_tasks.rb:44:in `rails_safe_trigger_task'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/lib/foreman_tasks.rb:23:in `trigger_task'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/lib/foreman_tasks.rb:54:in `sync_task'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.3/lib/foreman_tasks/triggers.rb:22:in `sync_task'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/controllers/katello/api/v2/repositories_controller.rb:221:in `create'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/controllers/katello/concerns/api/api_controller.rb:39:in `process_action'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:38:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/lib/katello/prevent_json_parsing.rb:12:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
/var/root-opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
/usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-02-22T14:16:00 [I|app|d9790] Completed 500 Internal Server Error in 218ms (Views: 0.2ms | ActiveRecord: 29.0ms)

Hey @Phroggelator,

Sincere apologies for the lack of response on this for over a week. I’m looking into this now. The logs you shared today are definitely useful. Will let you know what I find!

@Phroggelator,

I’m still narrowing down the problem, but I see where something might be wrong (but not sure how it got to that state)

You can help me by providing some more details from your server. Using foreman-rake simply type SmartProxy.pulp_master and let me know what it returns. For example, mine looks like this:

irb(main):001:0> SmartProxy.pulp_master
=> #<SmartProxy id: 1, name: "centos7-katello-3-10.strangeways.example.com", url: "https://centos7-katello-3-10.strangeways.example.c...", created_at: "2019-02-05 22:26:55", updated_at: "2019-02-05 22:26:55", expired_logs: "0", puppet_path: "/etc/puppetlabs/code/environments", download_policy: "on_demand">

Results are:

]# foreman-rake SmartProxy.pulp_master
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
rake aborted!
Don't know how to build task 'SmartProxy.pulp_master' (see --tasks)
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
(See full trace by running task with --trace)

Unless I misunderstood how you wanted foreman-rake invoked - I dont have a lot of understanding about what happens under the hood of foreman as yet. I will note that our Satellite 6 server returns the same text (essentially - obviously the exact paths are a little different).

My mistake. I omitted something very important: the full command is foreman-rake console and at the prompt you can type SmartProxy.pulp_master Sorry about that!

I thought it looked something was missing! So foreman-rake console had some complaints on startup and since I don’t know if that’s normal I included them as well:

# foreman-rake console
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
Apipie cache enabled but not present yet. Run apipie:cache rake task to speed up API calls.
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/content_facet_host_extensions.rb:7: warning: already initialized constant Katello::Concerns::ContentFacetHostExtensions::ERRATA_STATUS_MAP
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/content_facet_host_extensions.rb:7: warning: previous definition of ERRATA_STATUS_MAP was here
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/content_facet_host_extensions.rb:14: warning: already initialized constant Katello::Concerns::ContentFacetHostExtensions::TRACE_STATUS_MAP
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/content_facet_host_extensions.rb:14: warning: previous definition of TRACE_STATUS_MAP was here
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/subscription_facet_host_extensions.rb:13: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::SUBSCRIPTION_STATUS_MAP
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/subscription_facet_host_extensions.rb:13: warning: previous definition of SUBSCRIPTION_STATUS_MAP was here
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/subscription_facet_host_extensions.rb:21: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::SLA_STATUS_MAP
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/subscription_facet_host_extensions.rb:21: warning: previous definition of SLA_STATUS_MAP was here
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/subscription_facet_host_extensions.rb:26: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::USAGE_STATUS_MAP
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/subscription_facet_host_extensions.rb:26: warning: previous definition of USAGE_STATUS_MAP was here
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/subscription_facet_host_extensions.rb:31: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::ROLE_STATUS_MAP
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/subscription_facet_host_extensions.rb:31: warning: previous definition of ROLE_STATUS_MAP was here
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/subscription_facet_host_extensions.rb:36: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::ADDONS_STATUS_MAP
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/subscription_facet_host_extensions.rb:36: warning: previous definition of ADDONS_STATUS_MAP was here
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/subscription_facet_host_extensions.rb:41: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::PURPOSE_STATUS_MAP
/var/root-opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0/app/models/katello/concerns/subscription_facet_host_extensions.rb:41: warning: previous definition of PURPOSE_STATUS_MAP was here
Loading production environment (Rails 5.2.1)
Failed to load console gems, starting anyway
irb(main):001:0> SmartProxy.pulp_master
=> nil
irb(main):002:0> 

Our Satellite 6 install returns fewer of those error, but the response to “SmartProxy.pulp_master” was different. I would guess that the foreman-installer missing installing something, but it’s not clear to me what that might be. I used the following command for the install if that also helps:

foreman-installer --scenario katello --foreman-configure-epel-repo=false --foreman-configure-scl-repo=false --puppet-ssldir=/etc/puppetlabs/puppet/ssl --foreman-proxy-content-enable-ostree=true -v

I enabled ostree support since our Satellite 6 seems to have it and it was one of the few clear differences between the package set in Sat 6 and Foreman/Katello

Hmmm… Now that I know to looks for SmartProxy I find that this package:

tfm-rubygem-smart_proxy_dynflow_core.noarch

appears to not be installed (whereas it is installed in Sat 6). Is this whats missing?

Interesting. Those warnings when you opened the console are safe to ignore, and I don’t think that the missing RPM has anything to do with it at the moment.

Check on this in your UI: Infrastructure -> Smart Proxies - do you see anything listed there when you are using the Default Organization?

If you do, select the smart proxy and let me know which features it has listed as active.

By the way, when you did run the installer, it completed successfully?

From the UI, the only thing shown under smart proxies is a button to create one.

The first (apparently) successful installer pass was 8 Feb 2019, the only reported errors were a cert clash (cert already existed, along with a warning about puppet cert being deprecated) and:

[ERROR 2019-02-08T16:23:58 verbose]  Could not find a suitable provider for foreman_smartproxy

which I ignored since it seemed like the UI was working correctly. The second installer run was on the 22 Feb 2019 to get the ostree stuff installed and it’s complaints were about being unable to start apache (because it creates a vhost to listen on that port which clashes with puppetserver - I nobbled that vhost and let puppetserver have it’s own way) and the same foreman_smartproxy error listed above from the first run. I have the full installer logs if they are needed.

I think the suitable provider error is getting us in the right direction of the cause. One more thing to confirm from the console: SmartProxy.all I suspect you’ll have 0 results at the moment - give it a look.

Full installation logs are probably going to be helpful here. Would you mind putting them into a pastebin? https://paste.fedoraproject.org/

You should be able to rerun the installer. If it can’t register itself, you will have an issue because it can’t find the right proxy.

SmartProxy.all returns:

irb(main):004:0> SmartProxy.all
=> #<ActiveRecord::Relation []>

Our Sat 6 install returns a rather different answer…

Pastebin link:

https://paste.fedoraproject.org/paste/c7x0CDJWP~eMpcy-c939pQ

I have rerun that installer many times (especially prior to the successful(-ish) install on the 8th Feb), it is (understandably) a complicated and imperfect beast. I had to correct a number of issues even before I could run the installer without getting failures around installing/running puppet & puppetserver. It also doesn’t seem to handle SELinux correctly and I’m sure I read somewhere in the doco that it handled that. In any case, my notes around that will be for a different ticket once I have this running enough to be able handle the patching for a slew of different workstation distro’s.

If I’m reading the SmartProxy.all returns from Sat 6 correctly it should be talking to a URL on port 9090. There is ruby process listening on the port in Sat 6 and likewise on my foreman/Katello box there is a similar ruby process listening on port 9090, so I’m guessing it has in fact not been registered for some reason, rather than not installed.

Thanks for sharing the install logs - something is definitely ‘off’ there and I think your puppet options may be playing into it somehow. I agree with your assessment - your server should be registered as a Smart Proxy to itself but we see that the database returns no record so that never happened.

@ekohl can you review the install logs here? I’m not very familiar with the puppet pieces and you can probably make a better assessment.