Dockerfile is now included in foreman core

containers

#1

Hello,

tl;dr

There is now a production docker container of foreman building automatically (on every merge to core) at https://quay.io/repository/foreman/foreman and a docker-compose ready to use production environment you can deploy by:

docker-compose run app bundle exec rake db:create db:migrate
docker-compose run app bundle exec rake db:seed
docker-compose up

and login to http://localhost:3000 (note that the password was printed during db:seed above)

if you like to upgrade your environment

docker-compose pull
docker-compose run app bundle exec rake db:migrate db:seed
docker-compose up

ATM if you would like to add additional plugins you should rebuild the image yourself (details below).

Please don’t use it for real production environments (yet)

Long version

Recently I’ve been working of adding dockerfile to foreman core, that an image can be built easily directly from source.

The aim of this change was not to introduce a new way for users to install Foreman via containers (even if its possible) rather more as a development (or developers) tool.

The motivation I had in mind was:

  • reliable / continues builds based on git / source
  • Build production environments from PR so people can easily test, reproduce bugs etc
  • Simple enough that everyone can just take it out for a quick spin
  • Being able to add E2E testing on top
  • Allow users to see latest features, translators to see the current state of strings etc.

Currently, there is a docker-compose file that enable production environment, in order to use it you need to ensure you have both docker and docker-compose installed locally.

Out of the box, it runs:

  • MySQL server (can be easily swapped to pg, I just happened to use my local DB for testing)
  • A single rails puma process
  • Dynflow worker.

If you look into the content of the file, its pretty simple to add additional services if needed.

Additionally, this image contains a report of the webpack build that can be found at http://localhost:3000/webpack/report.html, this is useful to see how the assets are complied and used within the application.

It supports the various languages (so translators and developers can see the current language state) and include hammer cached API binding that hammer tests can be done against it as well.

This by no means a complete work, nor I’m suggesting we should invest a lot into this before it proof itself useful to us.

How to build your own image

  • Using quay.io (or similar service), I’ve found its easiest simply to configure quay.io to build the images for me, the workflow is simple enough (e.g. on every git push to my repo an image will be built) and it doesn’t slow down my machine while building it (it takes about 7-8 minutes or so).
  • Building locally - The easiest is simply to run
    docker-compose build app
    or
    docker build . -t mytag
    if you have any plugins configured in your bundler.d/Gemfile.local.rb it will pick them up and include them in your generated container, you would then might need to change the line pointing to the container image to point to the image you just created.
  • Note that you need a recent version of docker (e.g. from the last couple of years, not what ships in fedora by default) to build it, you might consider using podman to build it instead.

If you happen to have gem in a local checkout - e.g.

gem 'foreman_memcache', path: '../foreman_memcache'

and don’t want to mess up with paths, I would suggest to create directory called “dev” or something under your foreman directory, so in your bundle file you have something like

gem 'foreman_memcache', path: 'dev/foreman_memcache'

I personally just did

mkdir dev
cd dev
ln -s ../../foreman_memcache .
cd ..

and then, let tar follow symlinks, e.g.

tar -chf - --exclude-from=.dockerignore | docker build -

I did not verify that all features work, I’m pretty sure stuff breaks, a list of stuff I’m aware of are:

  • Logging doesn’t work to STDOUT (not sure why)
  • audit logs contains references to the container hostname (and many times they swap over because of multiple containers setting the value)
  • No way to pass settings via environment variables (to common way to solve configuration with containers).

Further work

  • investigate base image should we use, maybe UBI - I did do initially a POC here but felt it was too much to do in one PR, maybe better use ubi-minimal…
  • move the base image to a foreman-container-base, so builds could be a bit faster to build
  • E2E testing, perhaps similar to the suggestion by @amirfefer
  • Add foreman-proxy
  • Add LB / SSL Termination
  • Enable more specific services (such as the one needed for Katello)

I would be happy to see continuation of this for developer environments, perhaps similar to

I know this is not 100% aligned with whats in RFC @ehelms suggested, I hope its viewed as a positive first step .

Thanks for reading…


Foreman in Containers on Openshift
Foreman Dockerfile proposal: publish base and builder-base images
#2

settings.yaml is ran through ERB which means you can use #{ENV['MY_ENV_VAR']} in it to use env vars. That means you can include a config/settings.yaml.container which sets the right env vars to be configurable and let the Dockerfile use that (either symlink or copy it during the build process).


#3

I’ve followed up with more official documentation at https://github.com/theforeman/foreman/pull/6786


#4

Thanks, I guess then it would be easy to use the upstream image as base and add a settings file, but the whole point of using env variables would be to skip the need of building a customized image just for configuration, ideally foreman it self should accept ENV variables as valid values that override the settings by default (similar to RAILS_ENV or other rails specific environment variables).


#5

@ohadlevy: Are you also planning on adding a Dockerfile to smart-proxy?
I’ve been toying with the idea of adding configuration via ENV variables to smart-proxy.


#6

Massive +1 to this, I’m going to look into using it for tests with foreman-ansible-modules


#7

I didnt, but perhaps its too easy :- ) something like https://github.com/shlomizadok/foreman-docker-compose/blob/master/proxy/Dockerfile


#8

I’ve also been toying with a proof of concept. For a demo I decided to rewrite the proxy from scratch in Python (just the Puppet module for now). The framework I’m using defaults to reading env vars. It does type casting since env vars are strings. It’s going to be a challenge to do that right in the current proxy which doesn’t really know about data types and trusts it to come from YAML. That may also be an issue in the Foreman configuration but probably easier since it already has a model around settings.

@TimoGoebel (and others present next week): let’s talk about this next week and see if we can come up with a plan/proposal


#9

may I ask why? just wondering what kind of issues are you having with the current proxy code base?


#10

I want to demonstrate the interaction with Foreman and that it’s really straight forward. The codebase is big and complicated so it’s hard to show a very minimal version. In about 100 lines I could implement a functional proxy including an implementation of the Puppet feature.

It was also a lot of fun to teach myself modern Python web frameworks and I always teach myself by implementing something semi-serious. Automatically generating an openapi schema was also an interesting experiment.

While doing so, I even found a bug in the Smart Proxy Puppet module for which I still need to open an issue. When a Puppet parameter defaults to false, it’s represented as a string instead of a boolean (where true is a boolean).


#11

OK, I did an initial POC at https://github.com/ohadlevy/smart-proxy/commit/3caf0f1a9ffcc16d46514d0d26f4121156ed20e4 - if you like it, please continue based on it, the main part that breaks is the settings.


#12

Wow! Congratulations! Great job! :+1:

This looks like some serious hard work to have that gotten to work. Especially the Dockerfile.

While I’ve not yet reviewed everything I’ve started to note down a few thoughts somewhere else. (I’m not quoting it here, it would be unnecessary distraction.)


#13

up2date, “official” documentation can now be found at https://github.com/theforeman/foreman/blob/develop/developer_docs/containers.asciidoc thanks for everyone who helped to review.


#14

I’m working on a project for Foreman and Puppet, but would rather contribute to this instead of working solo. Do you have a parent issue (epic) for this? For starters, what integration testing framework are you planning on using? I’m using Fabric8’s docker-maven-plugin which seems like it would fit nicely into Redhat’s stack, but maybe you want to use something more Ruby-ish? Maybe a good start would be to move in some testing and go from there? Or maybe you already have a roadmap staked out?


#15

@luksi1 what are you trying to achieve? its hard to understand from your comment above…


#16

We’re running Foreman + Puppet in production (I know that you’re offering
it as a development option) and it would be nice to contribute. I guess my
convoluted question is simply: is there an issue, epic, roadmap that we
could look at to grab issues, bugs, and feature requests so that we could
help with this? We’re interested especially in contributing test cases (I
don’t see any docker tests in the master branch) and maybe some smart proxy
images if you’re accepting them.


#17

I would guess the parent “epic” (we use redmine which doesn’t have a concept of epics) is at Refactor #18732: Make Foreman Containerizable - Foreman.

Feel free to add issues and link them to that one? it would be good to understand your plans, and of course, patches are welcomed :slight_smile:


#18

Thanks! That’s exactly what I’m looking for. I’ll take a look at what’s
already on there and see if I can start making some merge requests.

We would like to help make foreman production ready and robust.

Would you be OK with some syntax and unit testing to start with? Hadolint.
Shellchecker. Some kind of scanner… maybe Trivy. Any other ideas?

Do you have any ideas around integration testing? Or maybe you’re simply
thinking about overlaying your current integration tests?

Den tis 16 juli 2019 08:41Ohad Levy via TheForeman <
noreply@community.theforeman.org> skrev: