New Forklift feature merged - customize devel environment config from git repo

John_Mitsch · November 18, 2019, 2:23pm

Hey all,

Just want to give a heads up that a new forklift feature was merged to help with devel workflows.

A quick straw poll of devs revealed that we all do similar things to customize a devel environment after spinning one up. This usually means copying over dotfiles like .bashrc and installing packages.

The change adds a way to copy over custom files from a git repo and execute a specific script. So you can host your dotfiles in version control along with your custom script that installs packages, makes your bash prompt neon-pink, posts on twitter that you are spinning up a new environment, or whatever you wish

There is also a way to copy from a local folder, which is helpful for more sensitive information.

The documentation is here so I won’t repeat it, but let me know if you have any questions or any part can be clarified. Currently the ansible role is automatically added to both centos7-katello-devel and centos7-katello-devel-stable box provisioning, but can be used independently as well.

This along with the stable devel box are my efforts to improve the developer experience around creating a development environment. This is not only to make our lives easier, but to make it an easy decision to spin up a new box that has the latest configuration, packages, and back-end changes. Hopefully these changes improve things.

Let me know if you have any feedback or questions! Thanks

Jonathon_Turel · November 19, 2019, 8:16pm

This is cool! I took a quick glance so I hope I didn’t miss the spirit of your PR. I’m wondering what you think about a future enhancement to provide a kind of map file to provide source and destination paths for desired files to be copied? Or is that what you envision the bootstrap script should do?

It might look something like this:

{
  files: [
     {
        src: '/home/jturel/.vimrc',
        dest: '/home/vagrant/.vimrc',
      }
  ]
}

This would allow finer control without having to write a custom script. The same could be done for any additional packages that someone may want to install. I figure those cover most of the dev environment configuration needs.

Did you consider any other approaches to the bootstrap script? Perhaps augmenting the run with a custom playbook?

John_Mitsch · November 20, 2019, 2:23pm

Thanks, I’m glad find it useful and appreciate the feedback! Are you thinking this for the local file copy strategy or from the git repo one? If I’m interpreting correctly, it sounds like you want to copy dotfiles from your hypervisor, do I have that correct? I could see this being useful for the local copy strategy

I am hoping that this is a flexible solution, it really is centered around the git repo strategy. You can put dotfiles in the root level of the repo so they wind up in ~/vagrant and you can make the bootstrap script do whatever you want. This could include installing ansible and calling playbooks in your repo. It also gently nudges devs to keep configuration files in version control and not just on their machines.

I went with this approach as it seemed like it had the least assumptions about how things should be done and what should be done. A prescriptive approach like creating a list of packages to install wouldn’t be as flexible and I’m sure someone would have $MY_SPECIAL_SETUP that they wouldn’t fit that workflow.

So the thought was to keep things as flexible as possible to start, if there is an ask for a different approach we can definitely consider that, I’m really looking for feedback as I may assume that we all want to do things this way and maybe we don’t. If you have any broader goals, like “I want to use files from X to do Y”, that can help come up with an approach.

John_Mitsch · November 20, 2019, 4:16pm

@Jonathon_Turel I just remembered something, this did come up in the PR. You can symlink files on your hypervisor to the user_devel_env_files directory and they will be copied. I just tested it out to make sure it works:

I have a symlinked file in the directory in forklift:

[jomitsch@jomitsch forklift{master}]$ ls -l user_devel_env_files/
total 0
-rw-rw-r--. 1 jomitsch jomitsch  0 Nov 20 11:03 hi.txt
lrwxrwxrwx. 1 jomitsch jomitsch 20 Nov 20 10:59 iamasymlink.txt -> /tmp/iamasymlink.txt

I spin up the box, and I see it present in the home directory:

[vagrant@centos7-katello-devel-stable ~]$ ls -al iamasymlink.txt 
-rw-rw-r--. 1 vagrant vagrant 3 Nov 20 16:12 iamasymlink.txt

Does this help your use case? If so, we should probably add this technique to the documentation.

ehelms · November 20, 2019, 5:57pm

Jonathon_Turel:

It might look something like this:
{
  files: [
     {
        src: '/home/jturel/.vimrc',
        dest: '/home/vagrant/.vimrc',
      }
  ]
}
This would allow finer control without having to write a custom script. The same could be done for any additional packages that someone may want to install. I figure those cover most of the dev environment configuration needs.

For me, this is a bit of inventing our own format when a custom bootstrap script would suffice (e.g. user can write bash to do this, or use ansible like I do – https://github.com/ehelms/vim-settings/blob/master/bootstrap.yml

Jonathon_Turel · November 20, 2019, 7:16pm

Let me explain the reasoning for my suggestion a little more.

When I think about the customization of my own dev environment there are two things that I do:

copy various config files
install a few extra packages

For most this is likely 99% of what it means to customize their environment. The current solution does give a way to address those cases which is great. Although, it’s two steps at a minimum. Someone must put their files in a git repo, or symlink them into a specific folder that Forklift knows about. Then they need to write a script to do other things like install extra packages or arrange those files in a particular way.

We can very easily provide a simple recipe for doing the above without the need to write a single line of code. That’s what I was trying to convey with the example JSON that I gave (though YAML would probably make more sense). By all means there should be a way to do some extra work (currently via the bootstrap mechanism), but the most basic environment customizations shouldn’t require that in my opinion. The folks new to the project who are spinning up for the first time and wanting to get started in a mostly familiar environment would find an advantage in the simplicity of what I’m suggesting.

John_Mitsch · November 20, 2019, 9:27pm

I’m not strictly opposed, and I considered this approach when I first started the feature. The reason I went with a more flexible git/local folder approach is I feel this feature is most beneficial for the power-user crowd, those who live in Forklift every day and like to destroy + recreate boxes multiple times and have opinionated setups.

Its meant as something to set + forget and have your configuration be the same across development boxes. This is most beneficial for those who recreate them frequently as you are not repeating the same steps across many box creations. Of course, it is not just for power-users, there is nothing stopping new users from using it, but I see your point that there could be a more new-user-friendly strategy with a nice YAML or JSON template.

I am wondering if you are making this suggestion because it will work best for you as a developer or to benefit those new to Forklift? This discussion taps into a bit of a larger conversation of Forklift serving many masters and who should design its features for (I saw this a similar discussion on another PR). I don’t think there is a right or wrong way, it just depends who we want to prioritize for.

Jonathon_Turel · November 20, 2019, 10:05pm

Understood, and I see my suggestion as entirely within the scope of those goals.

I use a small ansible script to move some files and install some packages, so I probably wouldn’t even use it myself. My angle is to consider what is going to be most useful to most people with minimal effort and I think scribbling a few lines of YAML to automate the most common use cases is a good way to accomplish that.

It’s hard to get a sense of what is going to make the most people happy. At a minimum we’ll need your work to bake-in for a while so more folks can test it out, and for them to weigh in.

John_Mitsch · November 21, 2019, 12:41am

It just doesn’t seem as flexible as scripting and as Eric put it “inventing our own format” so I’m just trying to figure out who its most meant to benefit if we were to go this direction.

One thing we can consider is replacing the local file strategy with what you are suggesting, so you have the easy template of packages and files to copy over and also have the git repo + bootstrap script workflow. I’m hesitant to go this direction, but if this is the workflow devs prefer and it will accomodate the different types of users, its worth considering.

Agreed, I would love to hear from more devs, get some feedback from those using the feature, and also hear if anything is preventing devs from using it. I really would like to do whats best for developers overall, I’m just trying to understand the different perspective.

John_Mitsch · November 21, 2019, 1:35am

I realized there is no reason we can’t execute a boostrap file that came from the local directory too, so I opened up this PR to add that functionality:

This will make it easier to just run some simple commands, hopefully giving the existing workflow a bit lower barrier to use for executing some simple tasks.