Jenkins + Spinnaker 기반 웹 애플리케이션 배포

주요개념

• Docker: 컨테이너 기반 오픈소스 가상화 플랫폼
• Docker Hub: docker image repository
• Kubernetes: 컨테이너 배포/관리 오픈소스 플랫폼
• Helm: Kubernetes 패키지 관리 도구
• Chartmuseum: helm chart registry
• Spinnaker: 멀티클라우드를 지원하는 Continuous Delivery 툴
• Jenkins: Continuous Integration 툴
• Minio: 오픈소스 오브젝트 스토리지 서버. (Demo에서는 Spinnaker metadata 저장용도로 사용)

---

Demo 환경

• Ubuntu v20.04.1 (Master, Worker node 각각 1대)
• Docker v19.03.13
• Kubernetes v1.19.4
• Helm v3.4.2
• Chartmuseum v0.14.0
• Jenkins v2.332.1
• Minio
• Spinnaker v1.24.1
• Gitlab v13.5.3
• Docker Hub
• Minio
• Maven project

---

CI/CD 프로세스

1. Jenkins

  ①     Git pull

  ②     Maven Build

  ③     Docker image build & push to docker hub

  ④     Helm packaging & push to chartmuseum

  ⑤     Spinnaker webhook 실행

2. Spinnaker

  ①     Bake manifest

  ②     Deploy manifest

---

Web application 배포 관련 주요 구성

helm chart는 '$ helm create'명령어로 기본 템플릿을 생성이 가능하며, 필요한 부분은 사용하고, 필요 없는부분은 삭제하며 사용가능하다.

Dockerfile			Docker image를 생성하기위한 파일
boot-demo(helm chart)	templates	helpers.tpl	템플릿에 반복되어 사용되는 표현을 정의
		deployment.yaml
		service.yaml
	Chart.yaml		chart에 대한 정보 정의
	values.yaml		템플릿에서 사용하는 각종 값들을 정의

Dockerfile

FROM openjdk:11

ADD target/able-base-jpa-0.1.0.jar /usr/share/demo/demo.jar
ENTRYPOINT ["java", "-jar", "/usr/share/demo/demo.jar"]

Chart.yaml

apiVersion: v2
name: boot-demo
description: A Helm chart for Kubernetes

# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.0.1

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
appVersion: 0.0.2

values.yaml

# Default values for hello-helm.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

image:
  repository: [Docker repository]/boot-demo
  pullPolicy: Always
  # Overrides the image tag whose default is the chart appVersion.
  tag: "0.0.1"

service:
  type: NodePort
  port: 30001


resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

podAnnotations: {}

_helpers.tpl

{{/*
Expand the name of the chart.
*/}}
{{- define "hello-helm.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "hello-helm.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "hello-helm.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Common labels
*/}}
{{- define "hello-helm.labels" -}}
helm.sh/chart: {{ include "hello-helm.chart" . }}
{{ include "hello-helm.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{/*
Selector labels
*/}}
{{- define "hello-helm.selectorLabels" -}}
app.kubernetes.io/name: {{ include "hello-helm.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

deployment.yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: {{ include "hello-helm.fullname" . }}
  namespace: boot-demo-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      {{- include "hello-helm.selectorLabels" . | nindent 6 }}
  template:
    metadata:
    {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
    {{- end }}
      labels:
        {{- include "hello-helm.selectorLabels" . | nindent 8 }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
          - containerPort: 8000
            name: {{ .Chart.Name }}
            protocol: TCP

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: {{ include "hello-helm.fullname" . }}
  namespace: boot-demo-ns
spec:
  type: {{ .Values.service.type }}
  ports:
    - nodePort: {{ .Values.service.port }}
      port: 8000
      targetPort: 8000
  selector:
    {{- include "hello-helm.selectorLabels" . | nindent 4 }}

 

---

Jenkins 구성

1. Release tag를 매개변수로 받아 빌드를 시작한다.

2. git 연동

 

3. Docker hub, chartmuseum, spinnaker token(Spinnaker에서 설정하며, 서로 약속된 token 사용) 바인딩

 

4. Maven build

5. Excute shell

helm version
docker --version
docker login -u $HUB_ID -p $HUB_PW
docker build -t [Docker repository]/boot-demo:$RELEASE_TAG .
docker push [Docker repository]/boot-demo:$RELEASE_TAG
docker rmi [Docker repository]/boot-demo:$RELEASE_TAG

helm package boot-demo
curl -u "$CHART_MUSEUM_ID:$CHART_MUSEUM_PW" --data-binary "@boot-demo-0.0.1.tgz" http://[chartmuseum IP 또는 domain]/api/charts


ENDPOINT="http://[spinnaker IP 또는 domain]/webhooks/webhook/demo.json"
DATA="{\"token\": \"${SPINNAKER_TOEKN}\", \"parameters\": { \"RELEASE_TAG\": \"${RELEASE_TAG}\"}}"
curl --header "Content-Type: application/json" --request POST --data "${DATA}" -k ${ENDPOINT}

---

Spinnaker 구성

- Configuration

Trigger 설정

Parameters 설정

- Deploy NS

Namespace 생성

- Bake(Manifest)

Bakering 과정에서 필요한 설정들을 한다. 엔진은 HELM3를 사용하고, Name과 Namespace도 지정 가능하다. chartmuseum 설정을 해두었기 때문에 자동으로 Account, Name, Version이 연동되어 나타나게 된다.

그리고 Overrides 기능이 유용한데, helm chart의 values를 override하여 사용가능하다.

-> Jenkins에서 빌드 시 입력하는 Release tag를 image tag로 override 할 수 있도록 하였다.

-> excution['id']는 spinnaker가 pipeline을 실행할때마다 생성되는 고유 id이다. 기본적으로 helm chart에 변화된 것이 없으면 pod가 재배포되지 않는다. 문제는 개발환경에서는 image tag를 업데이트하지 않고 계속해서 배포를 진행하게 되는데, 그러면 helm chart는 수정이 되지않으므로 pod가 새로 배포가 되지 않는 형상이 발생한다. 그래서 excution id를 values에 추가하여 pipeline을 실행할때마다 helm chart를 변경해주어 매번 배포가 되게하였다.

Bake 설정

Artifacts 설정

- Deploy (Manifest)

어떤 cloud에 배포할 지 account를 선택하고, 앞서 생성된 artifact가 배포되도록 한다.

Deploy 설정

---

CI/CD 데모 실행

1. Jenkins build 실행

2. Spinnaker pipeline webhook 실행

3. 서비스 확인