Continuous Delivery on OpenShift

layout: true
name: top-bar
.left[![HealthPartners](/rhug-sept-2018/images/hplogo.gif)]
.twitter-handle[
  `@jmcshane`
]
---

class: center

# Continuous Delivery on OpenShift

<hr />

## Building a functional build/deploy system at HealthPartners

---

# Philosophies

1. High availability
2. Secure credentials
3. Good default behavior
4. CI/CD Principles
5. Source control
6. Ephemeral build systems

---

# Definitions

* Continuous integration (CI) is the practice of merging all developer working copies to a shared mainline several times a day.
  * A breaking build should spring teams to action
* Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time.
  * The application "can" be deployed on demand, when application teams approve
---

# Cluster Design

* Build tools run on bld1 cluster
* Applications have three environments (dev, uat, prod)
.center[![Clusters](/rhug-sept-2018/images/cluster-design.png)]

---

# Tools

* Jenkins
* Bitbucket and Gitlab (long story)
* Nexus and Artifactory (long story)
* Jira & HipChat
* Hashicorp Vault
* Fortify
* Sonarqube
* ServiceNow

---

# OpenShift Concepts

* `BuildConfig` object integrates with Jenkins using `openshift-sync` plugin
* Jenkins pod running in each OpenShift namespace
* Each team owns one or more namespaces

---

# Jenkins Concepts

* Shared libraries create functions that can be called from Jenkinsfile
* Groovy scripts stored in `vars/` directory
* Groovy classes stored in `src/` directory
* Resources stored in `resources/` directory
* More info at https://jenkins.io/doc/book/pipeline/shared-libraries/

---

# Developer Viewpoint

* Single entrypoint to administrative actions
* Building more capable administration:
  * New namespaces
  * Jenkins machine management
  * Pod diagnostic actions
  * Creating new credentials
  * Cleanup jobs
  * Temporary access

---

# Developer Viewpoint

.left-column[![NamespaceManagement](/rhug-sept-2018/images/namespaceManagement.png)]
.right-column[![ToolManagement](/rhug-sept-2018/images/toolManagement.png)]

---

# Developer Viewpoint

* Create your git repository with Jenkinsfile and pipeline.yaml at the root
* `oc login https://openshift-bld1.hp.com && oc project MYPROJECT`
* `oc create -f pipeline.yaml`
* Jenkins pod in the namespace runs the build

.center[![BlueOcean](/rhug-sept-2018/images/jenkinsBO.png)]

---

.center[![ApplicationView](/rhug-sept-2018/images/deploymentExample.png)]

---

# Developer Viewpoint

```groovy
library identifier: 'hp-pipeline-library@master', retriever: modernSCM(
    [$class: 'GitSCMSource',
    remote: 'ssh://git@git.healthpartners.com/paas/hp-pipeline-library.git',
    credentialsId: "${env.NAMESPACE}-bitbucketsecret"]
)

hpPipeline([
  appName: "my-service",
  deployPom: "my-service/pom.xml",
  team: "Cloud"
])
```

---

# The Pipeline

1. Prepare
2. Build
3. Test
4. Pre-deploy
5. Deploy
6. Verify
7. Repeat 4-6

---

# The Code

hpPipeline.groovy
```groovy
def call(pipelineConfig) {
	new PipelineWrapper(this).wrapPipeline(pipelineConfig) { config ->
        def target = [namespace: env.NAMESPACE,
          environment: OpenshiftEnvironments.getBuildEnvironment()]
		DefaultEvaluator.evaluateDefault(config,
            'buildStage')?.call(config, target, this)
		def envArray = DefaultEvaluator.evaluateDefault(config,
            'deploymentEnvironments').call(config)
		for (String environment : envArray) {
            target << [environment: environment]
			beginEnvironment(environment, config)
			deployToEnvironment(environment, config)
			postDeployStages(environment, config)
		}
	}
}
```

---

# Preparation

* Set some base variables

```groovy
def wrapPipeline(pipelineConfig, closure) {
    pipelineConfig.audit = [:]
    pipelineConfig.defaults = new DefaultEvaluator().initialize()
    if (!pipelineConfig.openshift) {
       pipelineConfig.openshift = [:]
    }
    if (!pipelineConfig.skipNotify) {
        pipelineConfig.skipNotify = false
    }
    pipelineConfig.pipelineJobId = UUID.randomUUID().toString()
    ...
```

---

# Preparation

* Tell people what is happening

```groovy
context.notifyHipChatRoom([
  team: pipelineConfig.team,
  message: "Jenkins build started: ${detail}",
  skipNotify: pipelineConfig.skipNotify
])
context.audit([
        message: "Jenkins build started",
        build: context.env.BUILD_DISPLAY_NAME,
        job: context.env.JOB_NAME
], pipelineConfig)
```

---

# Preparation

* Enforce specific properties

```groovy
if (!pipelineConfig.appName) {
    error "You must define an appName property"
}
if(!pipelineConfig.team) {
    error "You must define a team property"
}
```

---

# Preparation

* Set global environment variables for the job

```groovy
env.DOCKER_HOST = 'docker.healthpartners.com'
env.FORTIFY_CLD_CTRL_HOST = 'https://fortify.healthpartners.com/cloud-ctrl'
env.FORTIFY_SSC_HOST = 'https://fortify.healthpartners.com/ssc'
env.FORTIFY_TOKEN = 'my-fav-token'
env.MAVEN_IMAGE_HASH = 'sha256:abc123'
env.FORTIFY_IMAGE_HASH = 'sha256:def456'
env.NODE_IMAGE_HASH = 'sha256:ghi789'
env.JIRA_HOST = 'jira.healthpartners.com'
env.SONARQUBE_URL = "http://sonarqube.${env.NAMESPACE}.svc:9000".toString()
env.AUDIT_URL = 'https://auditing-service.healthpartners.com/'
```

---

# Defaults and Overrides

* Be able to override anything
* Sensible defaults in a place you can find them
* Created a structure `Evaluator`/`Configurer`
  * Single place where defaults are stored and evaluated
  * Each default contributor is specific to a category
    * e.g. `MavenDefaultContributors`
---

# Defaults and Overrides: Evaluator

* Use groovy truthiness to look for default values

```groovy
def evaluateDefault(config, String key) {
    def overrides = config?.overrides?.get(key)
    //single argument function
    def metaClassArgs = overrides?.metaClass?.respondsTo(overrides,
        'call', [Object.class].toArray())
    if (overrides && metaClassArgs && !metaClassArgs.isEmpty()) {
        return overrides.call(config)
    } else if (overrides){
        //its not a closure, so it must be a value: return it
        return overrides
    } else {
        //attempt the default closure, then the default value
        return defaultClosures?.get(key)?.call(config) ?:
                defaultValues[key]
    }
}
```

---

# Defaults and Overrides: Contributor

```groovy
class MavenDefaultContributors implements DefaultContributor {
    @Override
    LinkedHashMap getClosures() {
        return [
                packaging: {it.packaging},
                compilePom: {it.pomPath ?: it.appDir ? it.appDir + "/pom.xml" : null},
                packageFlags: {it.packageFlags ?: ''},
                deployFlags: {it.deployFlags ?: ''},
                integrationTestFlags: {it.integrationTestFlags ?: ''},
				webTestFlags: {it.webTest?.mavenFlags ?: '' }
        ]
    }

@Override
    LinkedHashMap getValues() {
        return [
                packaging: 'jar'
        ]
    }
}
```

---

# Defaults and Overrides

* This structure allows developers to override anything that is keyed by a contributor

```groovy

hpPipeline([
  ...
  overrides: [
    packageFlags: "-PmyFavoriteProfile -Dprop=value"
  ]
])
```

---

# Build Stage

* Goal: Get a validated docker image to deploy
* Steps:
  * Package artifact
  * Run unit/component tests
  * Run s2i build from base image
  * Publish image to external registry

---

# Testing

* Unit & Component tests run before pre-deployment begins
* Generally JUnit tests, building `@SpringBootTest` component tests to load parts of the application
* H2 Database testing

---

# Pre-Deployment

* Approve the environment
* Check that the cluster is up
* Change management (will go into detail later)

```groovy
def call(envKey, config) {

def envConfig = OpenshiftEnvironments.getEnv(envKey)

getEnvironmentApproval(envConfig, config)

checkEnvironmentAvailability(envConfig)

jiraValidator(config, envConfig)

changeStart(config, envConfig)
}
```

---

# Deployment

* Use the OpenShift API to `apply` objects to the environment
* Two strategies
  * Built in "happy path" - use Jenkins to generate objects
  * More flexible templated path - developers provide OpenShift `Template` object

```groovy
def objects = [
    'BuildConfig' : new BuildConfig().create(config, target),
    'ImageStream' : new ImageStream().create(config, target),
    'DeploymentConfig' : new DeploymentConfig().create(config, target),
    'Service' : new Service().create(config, target),
]
```

```groovy
configurationApply(target) {
    sh "oc process ${parameters} -f ../${filePath} > ./template-output.yaml"
}
```

---

# Deployment

* Store OpenShift serviceaccount tokens in Vault
* Load those tokens in a common interface to make API actions

```groovy
vaultToken([environment: envKey, namespace: tokenNamespace]) {
  def ocCmd = """oc --server=${envConfig.serverUrl} \
    --insecure-skip-tls-verify=true --token ${TOKEN}"""

if(commandConfig.namespace) {
    ocCmd += " --namespace=${commandConfig.namespace}"
  }
  ocCmd += "  --config=/dev/null"
  sh "${ocCmd} ${commandConfig.command}"
}
```

```groovy
ocCommand(config.environment, [
        "namespace"   : "${config.namespace}",
        "command"     : "deploy ${deploymentConfig} --latest"
])
```
---

# Verification

* Post deployment separated into "Phases" and run in parallel
* SonarQube scan phase
* Fortify scan phase
  * Separate scanning from retriving results for better user experience
* Integration tests in source code repository
* Zephyr tests from JIRA for manual test phases

---

# Verification

```groovy
if(!config.buildOnly) {
    addPhase(postDeployMap, new IntegrationTestPhase(this),
      environment, config, null)
}
if (config.zephyr) {
    addPhase(postDeployMap, new ZephyrTestPhase(this),
      environment, config, null)
}
if (project.buildType == 'maven') {
    if (environment.contains('dev')) {
        addPhase(postDeployMap,
          new StaticCodeScanPhase(this), environment, config,
          [new SonarQubeReporting(this), new FortifyReporting(this)])
    }
    else if(environment.contains('uat'))
    {
        addPhase(postDeployMap,
          new StaticCodeValidationPhase(this),
          environment, config, null)
    }
}
```

---

# Verification

* Integration tests use the OpenShift proxy API to target new pods
* Record test results for change management
* Three automated test types:
  * API tests
  * Browser tests (Selenium)
  * Shell scripting validation

---

# Change Management

* ServiceNow is our company-wide audit store
  * Test results
  * JIRA ticket number
  * Git commit info
  * Scan results
  * Deployment time
  * Deployment results

---

# Change Management

* ServiceNow TableAPI
  * Much of the actions in the ServiceNow UI are available via API
* Standard change request process
  * Change request templates that can be instantiated to changes
  * Simplifies approval process for low priority changes
* Opt in:
  ```groovy
  hpPipeline([
      change : [
         cmdb_item: "HealthPartners.com Services",
         autoCreate: true
      ]
  ])
  ```

---

# Change Management

* Created a Jenkins plugin to carry out standard actions
* https://github.com/jenkinsci/service-now-plugin
  * Manage tables
  * Attach files
  * Update state
* Provides a simple entrypoint for making the necessary API calls

---

# Change Management: Results

* Can go from commit to production in 18 minutes
* Have completed over 10 deploys in a single day
* More than 100 audited production actions per week
* Multiple other companies using our plugin
* Validated our change management approach with our audit/compliance groups

---

# Visualizing the Data

* Created our own audit datastore and send data to Splunk
* Splunk: Short term event view
* Data store: Long term auditing view of things that happen in the pipeline
  * Approvals
  * Jobs
  * Deployments
  * Validations

---

# Visualizing the Data

.center[![Deployments](/rhug-sept-2018/images/deploymentLog.png)]

---

# Multi-Faceted Approach

* This is one of many of these libraries we have written:
  * Liquibase pipelines
  * ConfigMap and Secret management
  * Namespace management
  * Orchestrate Ansible Tower jobs for OpenShift infrastructure
  * Even manage our weblogic servers through these pipelines(!)

---

# Building for the future

* Canary deployments and advanced deployment strategy
* More complete testing strategies
* Read data from metrics stores to know about deployment success
* Tracking historical data
* Overlay application data with deployment events

---

# Open Source

* Working to open-source hp-pipeline-library
* Open sourced our testing library
* Our team manages two Jenkins plugins (service-now and prometheus)
* Open sourced a library that translates Prometheus alerts to ServiceNow (Snogo)
* Looking for more ways to contribute to the community

---

# Testing our Code

* The pipeline library code is used by multiple teams in our organization
* Breaking people's builds is a very poor customer experience
* Important to provide safe structures for updating this code
* Again, taking a layered approach here

---

# Testing our Code

* Unit tests using the shared library test helper
* Integration tests creating small "snippets" of Jenkinsfile
* End to end tests creating real pipelines

---

# Testing our Code

```groovy
def call(config) {
    node('master') {
        vaultToken([vaultSecrets: HipChatRooms.getHipchatRoomInfo(config.team)]) {
            if (HIPCHAT_ROOM_ID) {
                def notify = config.warn ?: false
                def color = notify ? (config.failed ? 'RED' : 'GREEN') : 'GRAY'
                hipchatSend message: config.message.toString(),
                  color: color, notify: notify, room: HIPCHAT_ROOM_ID,
                  server: "hipchat.healthpartners.com",
                  token: HIPCHAT_ROOM_TOKEN, v2enabled: true
            }
        }
    }
}

return this
```

---

# Testing our Code

```groovy
class NotifyHipChatRoomTest extends BasePipelineTest {
    def 'When call to notifyHipChatRoom, send a message with correct configuration' ()  {
        given:
        def binding = initializeMocks('hipchatSend')
        def notifyHipChatRoom = getShell(binding, 'vars/notifyHipChatRoom.groovy')
        def config = [
                warn: true,
                message: 'mock message',
                target: 'eng1'
        ]
        when:
        notifyHipChatRoom(config)
        then:
        1*binding.hipchatSend.call(*_) >> { arguments ->
            assert arguments[0][0].message == 'mock message - eng1'
            assert arguments[0][0].notify == true
            assert arguments[0][0].color == 'GREEN'
        }
    }
}
```
---

# Testing our Code

```groovy
node('master') {
    checkout scm
    parallel(
      'validate post deploy stages' : {
        load "integration/maven-build-jar/Jenkinsfile.postdeploystages"
      }, 'validate integrationTest step' : {
        load "integration/maven-test-step/Jenkinsfile.integrationtest"
      },'validate ocDeploy step' :  {
        load "integration/sb-app/Jenkinsfile.ocdeploy"
      }, 'validate mavenBuild step' : {
        load "integration/maven-build-jar/Jenkinsfile.mavenbuild"
        load "integration/maven-build-war/Jenkinsfile.mavenbuildwar"
      }
    )
}
```

---

# Testing our Code

```groovy
mavenNode([appName:'sb-app-integrationtest']) {
    def verify = { ->
        def testFile = readFile('target/surefire-reports/TestThatTargetIsCorrect.txt')
        if (testFile.contains("FAILURE!")) {
            sh "exit 1"
        }
    }
    checkout scm
    def config = [
            appName: 'jenkins',
            defaults: new com.healthpartners.pipeline.defaults.DefaultEvaluator(),
            overrides: [
              integrationTestDirectory: 'integration/maven-test-step'
            ]
    ]
    config.defaults.initialize()
    integrationTest(config,
        [environment: 'bld1', namespace: 'artifact-builder'],
        verify)
}
```
---

# Testing our Code

* This creates a pipeline to test our pipeline
* Continuous updates for our source code
* Developers reference `master`
* Integration and E2E test against `develop`
* Unit test on `feature/` branches

---

# Takeaways

* Jenkins & OpenShift can be a powerful pairing
* Optimizing for delivery has sped adoption
* Supporting testing is critical
* Audit & compliance are our friends
* Open source is great!
* Come work with us: https://healthpartners.com/careers

---

# References

1. https://github.com/jenkinsci/service-now-plugin
2. https://github.com/HealthPartners/jenkins-shared-library-test-helper
3. https://github.com/HealthPartners/snogo
4. https://jenkins.io/doc/book/pipeline/shared-libraries/
5. https://github.com/HealthPartners/rhug-sept-2018