Forklifting a Development Environment

forklift

#1

I’ve had a few requests to do a write up of a Forklift workflow for development environments. While there are a few different ways to build a development environment, the forklift tool provides an isolated environment through Vagrant and allows for easy replication of environments with some customization. Forklift can also be used to spin up various flavors of production environments but for this post I am going to focus on development.

As part of writing this post, I went through these steps fresh myself so if you hit any errors please reply here and we’ll work through them.

This derives from the Forklift documentation that can be found in the development docs. Advanced configuration such as memory increasing or poor-man’s DNS can also be found in the README.

Forklift uses Vagrant to spin up virtual machines and configure an environment. Provisioning is handled by Ansible playbooks.

Dependencies

For the purposes of this, I will be assuming the use of a yum based host such as Fedora or CentOS with libvirt. The use of Virtualbox for those on a Mac is possible but largely untested. Ensure that you have the latest Vagrant and Ansible installed.

sudo yum -y install  ansible libvirt-daemon-kvm
sudo yum -y local-install https://releases.hashicorp.com/vagrant/2.0.2/vagrant_2.0.2_x86_64.rpm
sudo chkconfig libvirtd on

Setup Forklift

Now clone the Forklift repository:

git clone https://github.com/theforeman/forklift
cd forklift

There are a number of boxes (i.e. virtual machine configurations) that come batteries included with Forklift that can be viewed by doing an initial vagrant status. The definitions for boxes can be found in the boxes.d/ folder. The one we are interested in for development is boxes.d/99-local.yaml.example. Let’s copy that from being an example to a real box file:

cp boxes.d/99-local.yaml.example boxes.d/99-local.yaml

This copy of the file is not checked into git so we can modify it without worry of committing changes. Within this copied box file we are interested in the primary development box definition:

centos7-devel:
  primary: true
  box: centos7
  ansible:
    playbook: 'playbooks/devel.yml'
    group: 'devel'
    variables:
      ssh_forward_agent: true
      foreman_devel_github_push_ssh: True
      katello_devel_github_username: <REPLACE ME>

The first step is to edit boxes.d/99-local.yaml and set your github username in place of <REPLACE_ME>. Note that this assumes that you have forked both the Foreman and Katello repositories.

Now we are ready to spin up the development box (this takes around 20 minutes to run):

vagrant up centos7-devel

Post Install Activities

The development environment at present lacks complete NPM support and is in progress. Until that time users need to do the following:

sudo yum -y install npm
cd foreman/
npm install

cat > .env <<EOF
BIND=0.0.0.0
PORT=3000
EOF

This is needed due to the standard Rails 5+ port of 5000 is used by Pulp’s crane service. There is also an HTTPS reverse proxy running that looks for 3000.

At this point there are two options:

  • use the built in webpack devserver (recommended for those modifying UI code)
  • precompile webpack to a faster environment (recommend for those not modifying UI code)

Built in Webpack Server

Edit foreman/config/settings.yaml and set :webpack_dev_server: to true

Now start the server:

foreman start
Precompile Webpack

This option precompiles the webpack bundle and will need to be re-run anytime you make UI changes.

rake webpack:compile

Now start the server:

rails s

Accessing the Dev Server

The development server can now be accessed via http://<ip_of_virtual_machine:3000 and, when precompiling webpack, over HTTPS at https://<ip_of_virtual_machine/

What’s On the Box

After the installation completes, the development environment is configured with the following:

  • foreman and katello git checkout in /home/vagrant/
  • Foreman proxy running in the background on port 9090
  • Postgres with initial database created and migrated
  • Pulp and MongoDB running in the background
  • Candlepin running under tomcat in the background
  • Puppetserver running in the background
  • Apache running for Pulp and with an HTTPS reverse proxy pointing to the standard Rails development port (3000)
  • Pulp crane running on port 5000

Advanced Topics

Code via NFS

Code can be mounted over NFS to the Vagrant VM to use your favorite editor such as Rubymine when making changes. Please refer to the README for more information or ask in #theforemandev as a number of developers use this technique.

Adding Additional Foreman Plugins

If you’d like to have more than the base level Foreman and Katello checked out via git, then can be added to the box definition:

  variables:
      foreman_installer_options: "
        --katello-devel-extra-plugins theforeman/foreman_remote_execution
        --katello-devel-extra-plugins theforeman/foreman_discovery
        "
Customized Environments

Things like box memory and cpus can be increased by setting it on the box definition. For a comprehensive list of options see here.

Troubleshooting

If you run into issues with the initial setup or using NFS mounts please see our Troubleshooting doc first. We are happy to help with any issues either here or via Github issues on the repository.

Notes

There are a few steps that are manual in this process that are candidates for being added baseline to the development setup or are in progress. If you run into issues or have feature requests feel free to file a Github issue. Contributions are always welcome to enhance the setup for different user configurations.


#2

For vagrant-libvirt you also need to install the plugin. vagrant plugin install vagrant-libvirt should work. Personally I prefer Fedora packages: dnf install vagrant vagrant-libvirt.

For those on Linux using libvirt I’d recommend https://liquidat.wordpress.com/2017/03/03/howto-automated-dns-resolution-for-kvmlibvirt-guests-with-a-local-domain/ so you can the generated hostnames without any setup or ugly /etc/hosts solutions. By default forklift uses HOSTNAME.example.com as the domain but this can be changed in settings.yaml.


#3

If you don’t want to use vagrant, it’s also possible to only use ansible playbooks and roles which are provided in the repo. The downside is, you must know which parameters should be tweaked. I hope to provide few examples soon as I prefer to use forklift with beaker or my own VMs.


#4

I prefer the nss-module for libvirt for name resolution. It is really easily setup, and it requires no extra dnsmasq service (which also affects all other name resolution of your computer, and we have seen strange situations in combination with vpns). After being hooked into the name resolution process itself, you never need to think about it again.
https://libvirt.org/nss.html


#5

Hello, I cannot get the user&passwd form “foreman start” outputs,so I was blocked in login page of the development environment.
What do I need to do? Thank you.

The outputs:

[root@promote foreman]# foreman start
09:39:05 rails.1   | started with pid 8256
09:39:05 webpack.1 | started with pid 8257
09:39:58 webpack.1 | Project is running at http://0.0.0.0:3808/
09:39:58 webpack.1 | webpack output is served from /webpack/
09:40:05 rails.1   | => Booting Puma
09:40:05 rails.1   | => Rails 5.2.1 application starting in development 
09:40:05 rails.1   | => Run `rails server -h` for more startup options
09:40:16 rails.1   | The Apipie cache is turned off. Enable it and run apipie:cache rake task to speed up API calls.
09:40:25 rails.1   | /root/foreman/app/services/facets.rb:2: warning: already initialized constant Facets::SUPPORTED_CORE_OBJECTS
09:40:25 rails.1   | /root/foreman/app/services/facets.rb:2: warning: previous definition of SUPPORTED_CORE_OBJECTS was here
09:40:29 rails.1   | 2019-04-10T09:40:29 [W|app|] DEPRECATION WARNING: Leaving `ActiveRecord::ConnectionAdapters::SQLite3Adapter.represent_boolean_as_integer`
09:40:29 rails.1   |  | set to false is deprecated. SQLite databases have used 't' and 'f' to serialize
09:40:29 rails.1   |  | boolean values and must have old data converted to 1 and 0 (its native boolean
09:40:29 rails.1   |  | serialization) before setting this flag to true. Conversion can be accomplished
09:40:29 rails.1   |  | by setting up a rake task which runs
09:40:29 rails.1   |  | 
09:40:29 rails.1   |  |   ExampleModel.where("boolean_column = 't'").update_all(boolean_column: 1)
09:40:29 rails.1   |  |   ExampleModel.where("boolean_column = 'f'").update_all(boolean_column: 0)
09:40:29 rails.1   |  | 
09:40:29 rails.1   |  | for all models and all boolean columns, after which the flag must be set to
09:40:29 rails.1   |  | true by adding the following to your application.rb file:
09:40:29 rails.1   |  | 
09:40:29 rails.1   |  |   Rails.application.config.active_record.sqlite3.represent_boolean_as_integer = true
09:40:29 rails.1   |  |  (called from <main> at /root/foreman/config/environment.rb:8)
09:40:29 rails.1   | 2019-04-10T09:40:29 [W|app|] DEPRECATION WARNING: `secrets.secret_token` is deprecated in favor of `secret_key_base` and will be removed in Rails 6.0. (called from <main> at /root/foreman/config/environment.rb:8)
09:40:29 rails.1   | 2019-04-10T09:40:29 [W|app|] DEPRECATION WARNING: Using a dynamic :controller segment in a route is deprecated and will be removed in Rails 6.0. (called from block in <main> at /root/foreman/config/routes.rb:17)
09:40:31 rails.1   | Puma starting in single mode...
09:40:31 rails.1   | * Version 3.12.1 (ruby 2.5.3-p105), codename: Llamas in Pajamas
09:40:31 rails.1   | * Min threads: 0, max threads: 16
09:40:31 rails.1   | * Environment: development
09:40:31 rails.1   | * Listening on tcp://0.0.0.0:3000
09:40:31 rails.1   | Use Ctrl-C to stop

#6

If you are using forklift, it should be admin:changeme.
If that doesn’t work for you, you can try
rake permissions:reset # Create or reset "admin" user permissions to defaults