Provisioning - Subscription Manager Fails: Executing dynaflow

Problem:
Can’t use provisioning to automatically configure host with Subscription Manager and thus all subsequent actions fail I/E install katello-agent and puppet.

Running provisioning scripts -> Automatic registration to Subscription Manager Fails -> Task c28c9b1e-0e8a-43d9-b2e7-8f1f6a816bb9: RuntimeError: Executing dynflow action inside a transaction is not a good idea.

Even with Chef removed this seems to keep rearing it’s ugly head.

Am I considering this process wrong here and it shouldn’t be able to automatically register the host?

Expected outcome:
After provisioning a host it is subscribed to Katello Repos, Puppet is installed and Katello-agent is running. AKA a push button / fully automated host build. The goal for the final product is to have all services / code installed so the server is up and running once the automatic process is completed.

Foreman and Proxy versions:
1.16.2-1.el7

Other relevant data:
Bootstrap file on host

# Network Setup
`# interface` `real=`grep -l 00:50:56:a6:2e:03 /sys/class/net/*/{bonding_slave/perm_hwaddr,address} 2>/dev/null | `awk -F '/' '// {print $5}' | head -1 sanitized_real='echo $real | sed s/:/_/'`

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-$sanitized_real1
BOOTPROTO="dhcp"
DOMAIN="mydomain.dick.jane.com"
DEVICE=$real
HWADDR="00:50:56:a6:2e:03"
ONBOOT=yes
PEERDNS=yes
PEERROUTES=yes
DEFROUTE=yes
EOF

service network restart

# Subscription Manager

# add subscription manager

yum -t -y -e 0 install subscription-manager rpm -ivh http://tlistmrsnd07.mydomain.dick.jane.com/pub/katello-ca-consumer-latest.noarch.rpm

echo "Registering the System"
subscription-manager register --org="8675309" --name="gary-thixton.mydomain.dick.jane.com" --activationkey="CentOS7-Auto"

echo "Installing Katello Agent"
yum -t -y -e 0 install katello-agent
chkconfig goferd on

# update local time
echo "updating system time"
/usr/sbin/ntpdate -sub 0.fedora.pool.ntp.org
/usr/sbin/hwclock --systohc

# update all the base packages from the updates repository
####yum -t -y -e 0 update
# remote_execution_ssh_keys

# SSH keys setup snippet for Remote Execution plugin
#
# Parameters:
#
# remote_execution_ssh_keys: public keys to be put in ~/.ssh/authorized_keys
#
# remote_execution_ssh_user: user for which remote_execution_ssh_keys will be
# authorized
#
# remote_execution_create_user: create user if it not already existing
#
# remote_execution_effective_user_method: method to switch from ssh user to

# This template sets up SSH keys in any host so that as long as your public
# SSH key is in remote_execution_ssh_keys, you can SSH into a host. This only
# works in combination with Remote Execution plugin.

# The Remote Execution plugin queries smart proxies to build the
# remote_execution_ssh_keys array which is then made available to this template
# via the host's parameters. There is currently no way of supplying this
# parameter manually.
# See http://projects.theforeman.org/issues/16107 for details.

if [ -f /usr/bin/dnf ]; then
dnf -y install puppet-agent
else
yum -t -y install puppet-agent
fi

cat > /etc/puppetlabs/puppet/puppet.conf << EOF

[main]
vardir = /opt/puppetlabs/puppet/cache
logdir = /var/log/puppetlabs/puppet
rundir = /var/run/puppetlabs
ssldir = /etc/puppetlabs/puppet/ssl

[agent]
pluginsync = true
report = true
ignoreschedules = true
ca_server = tlistmrsnd07.mydomain.dick.jane.com
certname = gary-thixton.mydomain.dick.jane.com
environment = development
server = tlistmrsnd07.mydomain.dick.jane.com

EOF

puppet_unit=puppet
/usr/bin/systemctl list-unit-files | grep -q puppetagent && puppet_unit=puppetagent
/usr/bin/systemctl enable ${puppet_unit}
/sbin/chkconfig --level 345 puppet on

# export a custom fact called 'is_installer' to allow detection of the installer environment in Puppet modules
export FACTER_is_installer=true
# passing a non-existent tag like "no_such_tag" to the puppet agent only initializes the node /opt/puppetlabs/bin/puppet agent --config /etc/puppetlabs/puppet/puppet.conf --onetime --tags no_such_tag --server tlistmrsnd07.mydomain.dick.jane.com --no-daemonize

Bootstrap Output

Restarting network (via systemctl):  [  OK  ]
./bootstrap-50263117-f060-bdb3-ebae-68b516b8b795: line 27: Redhat: command not found
./bootstrap-50263117-f060-bdb3-ebae-68b516b8b795: line 28: CentOS: command not found
./bootstrap-50263117-f060-bdb3-ebae-68b516b8b795: line 29: tlistmrsnd07.cbs.ad.cbs.net: command not found
pm_set: true
Loaded plugins: fastestmirror, langpacks
Determining fastest mirrors
 * base: mirrors.sorengard.com
 * extras: mirror.wdc1.us.leaseweb.net
 * updates: mirror.siena.edu
Resolving Dependencies
--> Running transaction check
---> Package subscription-manager.x86_64 0:1.20.11-1.el7.centos will be installed
--> Processing Dependency: subscription-manager-rhsm = 1.20.11 for package: subscription-manager-1.20.11-1.el7.centos.x86_64
--> Processing Dependency: python-inotify for package: subscription-manager-1.20.11-1.el7.centos.x86_64
--> Processing Dependency: python-ethtool for package: subscription-manager-1.20.11-1.el7.centos.x86_64
--> Processing Dependency: python-dateutil for package: subscription-manager-1.20.11-1.el7.centos.x86_64
--> Running transaction check
---> Package python-dateutil.noarch 0:1.5-7.el7 will be installed
---> Package python-ethtool.x86_64 0:0.8-5.el7 will be installed
---> Package python-inotify.noarch 0:0.9.4-4.el7 will be installed
---> Package subscription-manager-rhsm.x86_64 0:1.20.11-1.el7.centos will be installed
--> Processing Dependency: subscription-manager-rhsm-certificates = 1.20.11-1.el7.centos for package: subscription-manager-rhsm-1.20.11-1.el7.centos.x86_64
--> Running transaction check
---> Package subscription-manager-rhsm-certificates.x86_64 0:1.20.11-1.el7.centos will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                                Arch   Version               Repository
                                                                           Size
================================================================================
Installing:
 subscription-manager                   x86_64 1.20.11-1.el7.centos  base 993 k
Installing for dependencies:
 python-dateutil                        noarch 1.5-7.el7             base  85 k
 python-ethtool                         x86_64 0.8-5.el7             base  33 k
 python-inotify                         noarch 0.9.4-4.el7           base  49 k
 subscription-manager-rhsm              x86_64 1.20.11-1.el7.centos  base 285 k
 subscription-manager-rhsm-certificates x86_64 1.20.11-1.el7.centos  base 195 k

Transaction Summary
================================================================================
Install  1 Package (+5 Dependent packages)

Total download size: 1.6 M
Installed size: 5.2 M
Downloading packages:
warning: /var/cache/yum/x86_64/7/base/packages/python-ethtool-0.8-5.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for python-ethtool-0.8-5.el7.x86_64.rpm is not installed
--------------------------------------------------------------------------------
Total                                              3.7 MB/s | 1.6 MB  00:00
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) "
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-2.1511.el7.centos.2.10.x86_64 (@anaconda)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : python-dateutil-1.5-7.el7.noarch                             1/6
  Installing : python-inotify-0.9.4-4.el7.noarch                            2/6
  Installing : python-ethtool-0.8-5.el7.x86_64                              3/6
  Installing : subscription-manager-rhsm-certificates-1.20.11-1.el7.cento   4/6
  Installing : subscription-manager-rhsm-1.20.11-1.el7.centos.x86_64        5/6
  Installing : subscription-manager-1.20.11-1.el7.centos.x86_64             6/6
  Verifying  : python-dateutil-1.5-7.el7.noarch                             1/6
  Verifying  : subscription-manager-rhsm-certificates-1.20.11-1.el7.cento   2/6
  Verifying  : subscription-manager-rhsm-1.20.11-1.el7.centos.x86_64        3/6
  Verifying  : python-ethtool-0.8-5.el7.x86_64                              4/6
  Verifying  : python-inotify-0.9.4-4.el7.noarch                            5/6
  Verifying  : subscription-manager-1.20.11-1.el7.centos.x86_64             6/6

Installed:
  subscription-manager.x86_64 0:1.20.11-1.el7.centos

Dependency Installed:
  python-dateutil.noarch 0:1.5-7.el7
  python-ethtool.x86_64 0:0.8-5.el7
  python-inotify.noarch 0:0.9.4-4.el7
  subscription-manager-rhsm.x86_64 0:1.20.11-1.el7.centos
  subscription-manager-rhsm-certificates.x86_64 0:1.20.11-1.el7.centos

Complete!
Retrieving http://tlistmrsnd07.cbs.ad.cbs.net/pub/katello-ca-consumer-latest.noarch.rpm
Preparing...                          ########################################
Updating / installing...
katello-ca-consumer-tlistmrsnd07.cbs.a########################################
hostname: Name or service not known
Registering the System

Taskc28c9b1e-0e8a-43d9-b2e7-8f1f6a816bb9: RuntimeError: Executing dynflow action inside a transaction is not a good idea

Installing Katello Agent
Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos,
              : subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
 * base: mirrors.sorengard.com
 * extras: mirror.wdc1.us.leaseweb.net
 * updates: mirror.siena.edu
No package katello-agent available.
Error: Nothing to do
error reading information on service goferd: No such file or directory
updating system time
Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos,
              : subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
 * base: mirrors.sorengard.com
 * extras: mirror.wdc1.us.leaseweb.net
 * updates: mirror.siena.edu
No package puppet-agent available.
Error: Nothing to do
./bootstrap-50263117-f060-bdb3-ebae-68b516b8b795: line 103: /etc/puppetlabs/puppet/puppet.conf: No such file or directory
Failed to execute operation: No such file or directory
error reading information on service puppet: No such file or directory
./bootstrap-50263117-f060-bdb3-ebae-68b516b8b795: line 131: /opt/puppetlabs/bin/puppet: No such file or directory

FYI, i did follow the steps suggesting to remove Chef from Foreman. Still no luck. I can delete the host, but the subscription manager keeps failing.

It also happens on command-line:

[root@gary-thixton ~]# subscription-manager register --org="8675309" --name="gary-thixton.cbs.ad.cbs.net" --activationkey="CentOS7-Auto"
Task df8bdfc8-99eb-48e6-a62f-0cc430fc3598: RuntimeError: Executing dynflow action inside a transaction is not a good idea

Did you restart both httpd and the foreman-tasks service after removing foreman_chef?