Infrastructure SIG Meeting Notes 9/22

Development Infra SIG
1

Infrastructure SIG

Agenda

  • Introduction
  • State of Initiatives
    • Finish Webserver migration
    • Migrate all docs
    • Move jenkins-jobs
  • New Business
    • Archiving old Debian releases
    • Use of Jenkinsfiles
    • Rackspace migrations
    • Auto-building Debian on PR merge
  • Next steps

Areas of Care

  • Underlying Infrastructure management
  • Jenkins
  • Jenkins Jobs
  • Redmine
  • Website and webservers
  • DNS
  • Foreman and puppetserver
  • Koji

Initiatives

Documentation

Owners: ehelms, ewoud

  • Where to move and store documentation for infrastructure?
    • docs/ directory in foreman-infra written in markdown
      • Source that is outside of our infrastructure
    • auto-publish to github pages to publish docs
  • Action Item
    • Create docs/ directory [ehelms]
    • Migrate wiki pages from Redmine [ehelms]
    • Reviews

Rackspace migration

  • Currently needs migration
    • Jenkins
    • Webserver
    • Foreman/puppetserver
  • Webserver
    • Owner: Evgeni
    • web02 on Rackspace
    • New machine running in OSUOSL
      • Receives mirrors of yum content
      • Debian content mirroring in progress
    • Action Item
      • Final sync of content
        • Copy over Tomer’s homedir
      • Switchover
        • Target Date: 9/28 - EMEA morning
      • Shutdown web02
        • Target Date: 9/29
      • Destroy
        • Taget Date: 10/5
  • Jenkins
    • New hostname: controller01.jenkins.osuosl.theforeman.org
    • Action Items
      • Create new machine in OSUOSL with CentOS 8
      • Add it to Foreman
      • Assign the right Hostgroup
      • Take an outage window
        • Mark nodes as in maintenance mode in old Jenkins
        • Sync over /var/lib/jenkins
        • Take all but one node out of maintenance mode on old Jenkins
      • Turn on new Jenkins
        • Turn on one node
      • Run a test job
        • Run a nightly pipeline
      • Pick switchover date
        • Target Date:
        • Lower TTL day or two before target date
        • Update DNS
  • Foreman/puppetserver
    • Migrate to EL8

Redmine migration

Owner: ??

  • Run on Scaleways currently
    • Sponsoring ceased
    • Migrate to OSUOSL
  • Current Redmine version: 3.Y
  • Redmine Git Instance
  • Action Items
    • Build out migration plan
    • Test Redmine upgrade locally
    • Upgrade to EL8
    • Upgrade Redmine to 4.Y

ARM Builders

Owner: ??

  • Two currently running on Scaleways
  • Community member raised sponsoring new ARM servers on AWS
    • Access controls a concern due to Debian push
  • ARM builds disabled as of 2.1
  • Action Item
    • Decide if keeping ARM

foreman-infra cleanup, ci/ directory

Owner: ewoud

  • Jenkins job locations in foreman-infra
    • Jenkins jobs deployed via puppet
      • max 30 minute delay in getting job updates
      • contained inside a puppet module
    • Should we deploy jobs via Jenkins itself from a script?
      • Yes
    • Action Items
      • Move puppet/modules/jenkins_job_builder/files to jenkins-jobs/
      • Job in Jenkins that runs JJB to populate all jobs and updates
      • puppet module that deploys the Jenkins job that populates that builds the JJB

Use of Jenkinsfiles

Owner: ewoud

  • Giving projects control of building their own Jenkins jobs through a Jenkinsfile in the repository
  • Discussion
  • Will require moving to shared libraries instead of composed JJB
    • Still requires storing job definitions in JJB in foreman-infra
  • How to deal with secrets?
    • Does Jenkinsfile or Multi-branch PR have builtin for this?
  • Idea
    • Spin up a Jenkins server on OSUOSL and test the workflow

Archiving Old Debian Releases

Owner: evgeni

  • Discussion
  • Freight scans old archives on every run back to Foreman 1.2; increase speed of Debian builds
  • Proposal
    • Pick a date, and archive everything up to Foreman 2.0
    • Continue to expose the archives on an archive site
  • Action Items
    • Build archive site up to Foreman 2.0
    • Pick an archive date

Auto-building Debian on PR merge

Targets for Next Meeting

  • Webserver migration completion
  • Docs migration completion
  • Jenkins migration
  • Archiving old Debian builds
  • Moving JJB files to ci/ directory in foreman-infra

Completed Items

  • Where to track infrastructure updates? [DONE]
    • Development discourse topic?
      • Sub-topic “Infrastructure”
  • Schedule Next Meeting [DONE]
  • Post Discourse tracking posts for each initiative [DONE]
    • Track updates
2 Likes
2

Recording of today’s meeting is here, for those who like to watch/listen: