환경
Spinnaker 설치 및 설정 (online)
gajy
2022. 4. 6. 22:06
728x90
설치 환경
- Ubuntu v20.04.1 (Master, Worker node 각각 1대)
- Docker v19.03.13
- Kubernetes v1.19.4
- Chartmuseum v0.14.0
- Minio
- Spinnaker v1.24.1
Chartmuseum 설치
※ ALLOW_OVERWRITE=true 설정을 해주지 않으면, 같은 버전의 helm chart를 push하더라도 overwrite되지 않는다. 기본 false이므로 overwrite가 필요한 경우 반드시 설정해주어야한다.
Docker, On-premise 두가지로 설치해보았고, 한가지를 선택하면 된다. 개인적으로는 저장소는 안정성 측면에서 On-premise로 설치하는 것을 선호하는 편이다.
- Docker
$ docker run -d --rm -it \
-p 8090:8080 \
-e DEBUG=1 \
-e STORAGE=local \
-e STORAGE_LOCAL_ROOTDIR=/charts \
-e ALLOW_OVERWRITE=true\
-v /home/sunju/spinnaker/chartmuseum/charts:/charts \
ghcr.io/helm/chartmuseum:v0.14.0### chartmuseum 다운로드
$ curl -LO https://s3.amazonaws.com/chartmuseum/release/latest/bin/linux/amd64/chartmuseum
$ chmod +x chartmuseum
$ mv chartmuseum /usr/local/bin/chartmuseum
### chartmuseum 데몬 등록
# 1. 환경변수 파일 등록
vi /etc/default/chartmuseum
ARGS=\
--port=8090 \
--storage="local" \
--storage-local-rootdir="/home/sunju/chartmuseum/charts" \
--depth=1 \
--log-json \
--deisable-api="false" \
--allow-overwrite \
--basic-auth-user=myChartmuseumId \
--basic-auth-pass=myChartmuseumPass
### 2. chartmuseum service 파일 등록
vi /etc/systemd/system/chartmuseum.service
Requires=network-online.target
After=network-online.target
[Service]
EnvironmentFile=/etc/default/chartmuseum
User=root
Restart=always
ExecStart=/usr/local/bin/chartmuseum $ARGS
ExecStop=/usr/local/bin/chartmuseum step-down
[Install]
WantedBy=multi-user.target
### 3. chartmuseum service 실행
$ sudo systemctl start chartmuseum
$ sudo systemctl enable chartmuseum
$ sudo systemctl status chartmuseum
...
● chartmuseum.service - chartmuseum
Loaded: loaded (/etc/systemd/system/chartmuseum.service; enabled; vendor preset: disabled)
Active: active (running) since 5 2022-04-06 18:20:45 CST; 19h ago
...Minio 설치
Kubernetes, On-premise 두가지로 설치해보았고, 한가지를 선택하면 된다. 개인적으로는 저장소는 안정성 측면에서 On-premise로 설치하는 것을 선호하는 편이다.
- Kubernetes
### 1.spinnaker namespace 생성
$ vi spinnaker_ns.yaml
apiVersion: v1
kind: Namespace
metadata:
name: spinnaker
$ kubectl apply -f spinnaker_ns.yaml
### 2.minio PersistentVolume 생성
$ vi minio_pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: minio-pv
namespace: spinnaker
labels:
app: minio
spec:
storageClassName: minio-sc
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: /home/sunju/spinnaker/minio/data
$ kubectl apply -f minio_pv.yaml
### 3.minio 배포
$ vi minio.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: minio-pvc
labels:
app: minio
namespace: spinnaker
spec:
storageClassName: minio-sc
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: minio
namespace: spinnaker
spec:
replicas: 1
serviceName: minio
selector:
matchLabels:
app: minio
template:
metadata:
labels:
app: minio
spec:
containers:
- name: minio
image: minio/minio
args:
- server
- /storage
env:
# MinIO access key and secret key
- name: MINIO_ROOT_USER
value: myMinioRootUser
- name: MINIO_ROOT_PASSWORD
value: myMinioRootPassword
ports:
- containerPort: 9000
volumeMounts:
- name: storage
mountPath: "/storage"
securityContext:
runAsUser: 1000
runAsGroup: 65535
fsGroup: 65535
volumes:
- name: storage
persistentVolumeClaim:
claimName: minio-pvc
---
apiVersion: v1
kind: Service
metadata:
name: minio
namespace: spinnaker
spec:
ports:
- port: 9000
targetPort: 9000
protocol: TCP
selector:
app: minio
$ kubectl apply -f minio.yaml
$ kubectl get po -n spinnaker
NAME READY STATUS RESTARTS AGE
minio-0 1/1 Running 0 145m- On-premise (https://teamsmiley.github.io/2019/01/28/minio/)
### minio 다운로드
$ wget https://dl.min.io/server/minio/release/linux-amd64/minio
$ chmod +x minio
$ mv minio /usr/local/bin/minio
### minio 데몬 등록
# 1. 환경변수 파일 등록
vi /etc/default/minio
# Volume to be used for Minio server.
MINIO_VOLUMES="/home/hkmc/minio/data/"
# Use if you want to run Minio on a custom port.
MINIO_OPTS="--address :9000"
# Access Key of the server.
MINIO_ROOT_USER=myMinioRootUser
# Secret key of the server.
MINIO_ROOT_PASSWORD=myMinioRootPassword
### 2. minio service 파일 등록
vi /etc/systemd/system/minio.service
[Unit]
Description=MinIO
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio
[Service]
WorkingDirectory=/usr/local/
User=root
Group=root
EnvironmentFile=/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES
# Let systemd restart this service always
Restart=always
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536
# Specifies the maximum number of threads this process can create
TasksMax=infinity
# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no
[Install]
WantedBy=multi-user.target
# Built for ${project.name}-${project.version} (${project.name})
### 3. minio service 실행
$ sudo systemctl start minio
$ sudo systemctl enable minio
$ sudo systemctl status minio
...
● minio.service - MinIO
Loaded: loaded (/etc/systemd/system/minio.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2022-04-06 14:14:13 KST; 14min ago
...Spinnaker 구성
- Halyard 설치 (https://spinnaker.io/docs/setup/install/halyard/)
- halyard에서는 기본적으로 몇가지 디렉터리 및 파일을 볼륨으로 저장하고 있어야한다. 예를 들면, Spinnaker 배포를 위한 /home/spinnaker/.hal 디렉터리, cloud provider 접근을 위한 config 파일 등이 필요하다. 일단 volume 설정을 해놓으면 spinnaker를 배포하면서 필요한 디렉터리들이 volume에 함께 생성되므로 처음부터 해당 파일들을 생성할 필요는 없다.
- 주요 디렉토리 구조
| /home/spinnaker/.hal | config | spinnaker configuration 파일 | ||
| default | profiles | front50-local.yaml | minio는 versioning을 지원하지 않기때문에 versioning false 설정을 위해 생성한 파일 | |
| .kube | config | cloud provider 접근을 위한 config 파일 (파일과 디렉터리 이름은 변경 가능하다) |
$ vi halyard.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: halyard
namespace: spinnaker
spec:
replicas: 1
serviceName: halyard
selector:
matchLabels:
app: halyard
template:
metadata:
labels:
app: halyard
spec:
containers:
- name: halyard
image: gcr.io/spinnaker-marketplace/halyard:stable
volumeMounts:
- name: hal
mountPath: "/home/spinnaker/.hal"
- name: kube
mountPath: "/home/spinnaker/.kube"
env:
- name: HOME
value: "/home/spinnaker"
ports:
- containerPort: 8064
protocol: TCP
readinessProbe:
exec:
command:
- wget
- --no-check-certificate
- --spider
- -q
- http://localhost:8064/health
securityContext:
runAsUser: 1000
runAsGroup: 65535
volumes:
- name: hal
hostPath:
path: /home/sunju/spinnaker/halyard
type: DirectoryOrCreate
- name: kube
hostPath:
path: /home/sunju/spinnaker/halyard/.kube
type: DirectoryOrCreate
$ kubectl apply -f halyard.yaml
$ kubectl get po -n spinnaker
NAME READY STATUS RESTARTS AGE
halyard-0 1/1 Running 0 142m- Spinnaker 배포
halyard 커맨드 hal을 통해 config 설정부터 배포까지 가능하다 (https://spinnaker.io/docs/reference/halyard/commands/)
### 1. halyard 컨테이너 접속
$ kubectl exec -it halyard-0 -n spinnaker -- /bin/bash
bash-5.0$
### 2. Cloud Providers 설정
# Choose Cloud Providers
bash-5.0$ hal config provider kubernetes enable
# account 추가
bash-5.0$ hal config provider kubernetes account add my-k8s-account \
--provider-version v2\
--context $(kubectl config current-context)
# artifacts 사용 enable
bash-5.0$ hal config features edit --artifacts true
### 3. Choose your Environment → Distributed installation
bash-5.0$ hal config deploy edit --type distributed --account-name my-k8s-account
### 4. Choose a Storage Service → Minio
# Minio는 versioning을 지원하지 않으므로, versioning disable
bash-5.0$ mkdir ~/.hal/default/profiles && \
touch ~/.hal/default/profiles/front50-local.yaml
bash-5.0$ vi ~/.hal/default/profiles/front50-local.yaml
bash-5.0$ spinnaker.s3.versioning: false
# storage provider 설정
bash-5.0$ export MINIO_SECRET_KEY=myMinioRootUser
bash-5.0$ export MINIO_ACCESS_KEY=myMinioRootPassword
bash-5.0$ export ENDPOINT=http://minio:9000 # ENDPOINT는 minio 설치 방법에 따라 minio가 설치된 IP가 될 수 있다.
bash-5.0$ echo $MINIO_SECRET_KEY | \
hal config storage s3 edit --endpoint $ENDPOINT \
--access-key-id $MINIO_ACCESS_KEY \
--secret-access-key
# s3 storage provider enable 설정
bash-5.0$ hal config storage edit --type s3
### 5. Configure Helm Artifact Account -> chartmuseum
bash-5.0$ echo myChartmuseumId:myChartmuseumPass > chartmuseumAccount
bash-5.0$ hal config artifact helm enable
bash-5.0$ hal config artifact helm account add my-helm-account \
--username-password-file chartmuseumAccount
### 6. Deploy Spinnaker
# version 선택
bash-5.0$ hal config version edit --version 1.24.1
# spinnaker 배포
bash-5.0$ hal deploy apply
### 7. NodePort로 외부 통신을 열어준다. (실제 운영에서는 ingress를 통해 tls 및 도메인 설정등이 가능하다)
bash-5.0$ kubectl edit svc spin-deck -n spinnaker
...
type: NodePort
ports:
- port: 9000
protocol: TCP
targetPort: 9000
nodePort: 30300
bash-5.0$ kubectl edit svc spin-gate -n spinnaker
...
type: NodePort
ports:
- port: 8084
protocol: TCP
targetPort: 8084
nodePort: 30400
bash-5.0$ hal config security ui edit --override-base-url "http://IP address:30300"
bash-5.0$ hal config security api edit --override-base-url "http://IP address:30400"
bash-5.0$ hal deploy apply
$ kubectl get po -n spinnaker
NAME READY STATUS RESTARTS AGE
halyard-0 1/1 Running 0 142m
spin-clouddriver-5b8b599bbd-h7m5b 1/1 Running 0 138m
spin-deck-7558dbc646-cpwqf 1/1 Running 0 138m
spin-echo-556bb45877-vxgpj 1/1 Running 0 138m
spin-front50-7988979784-bk79d 1/1 Running 0 138m
spin-gate-5cb9f45947-qgc7k 1/1 Running 0 138m
spin-orca-67ff8dc76-g5xv6 1/1 Running 0 138m
spin-redis-c8f9995cc-gmd62 1/1 Running 0 138m
spin-rosco-5d5955d954-qjtr4 1/1 Running 0 138mhttp://IP address 또는 domain:30300으로 접속
728x90