Intro
Static Agent
Jenkins를 오래전부터 사용했다면 대부분 Baremetal 또는 VM 기반의 Remote Agent를 마스터에 추가해 본 경험이 있을 것이다. 관리자가 직접 패키지, 컴파일러, 환경변수 등의 빌드 환경을 Agent로 사용할 장비에 미리 세팅해놓고, JNLP 또는 SSH를 통해 연결하는 방식이다.
Job의 갯수나 관리할 Agent가 많지 않다면 큰 어려움이 없겠지만, 점점 규모가 커질 수록 아래와 같은 문제를 겪을 수 밖에 없다.
-
Agent가 장애 등의 원인으로 Offline 상태가 될 경우
-
Agent 하나에 여러 Job을 빌드하기 위해 구성한 환경이 서로 충돌하는 경우
-
짧은 시간 동안 많은 빌드 요청이 발생하는 경우
On-demand Agent
Jenkins Kubernetes 플러그인은 빌드를 일종의 서비스로 제공하는 접근법에서 시작한다. 빌드 요청이 발생하면 이를 위한 Agent를 생성, 지정된 빌드를 수행하고 완료된 후엔 자동으로 제거되는 On-demand 방식이다.
Agent의 스펙은 Pod Template에 정의할 수 있고 하나의 Pod은 다시 여러개의 Container로 구성된다. 기본적으로 inbound-agent 이미지로부터 생성된 JNLP 컨테이너가 마스터와의 연결을 담당하고 다음과 같은 환경변수가 자동으로 삽입된다.
-
JENKINS_URL: Jenkins 웹 URL
-
JENKINS_SECRET: 인증용 Secret
-
JENKINS_AGENT_NAME: Agent 이름
그리고 JNLP 컨테이너 외 실제 빌드에 사용할 컨테이너(ex: docker, maven, go 등)를 1개 이상 정의하며, 컨테이너 간 Workspace Volume을 공유하기 때문에 서로의 작업물이 계속해서 이어질 수 있다. 예를 들어 git 컨테이너에서 Repository를 체크아웃했다면 바톤을 넘겨받은 maven 컨테이너에서 이를 빌드하는 식이다.
Pipeline 타입의 Job에선 Jenkinsfile 안에 Pod Template을 통해 파이프라인에서 사용할 Agent를 사용자가 직접 정의할 수 있다. 그 외 경우, Jenkins 관리 메뉴에서 미리 Pod Template을 정의해두고 Job 설정의 Label을 통해 사용 가능하다.
사용법
Pipeline에서 Pod Template을 정의하고, 이를 사용하는 방법에 대해서 알아보자. 우선 Scripted Pipeline과 Declarative Pipeline의 Pod Template 작성 방식에 큰 차이가 있다.
Scripted Pipeline
podTemplate(label: 'hello',
containers: [
containerTemplate(name: 'maven', image: 'maven:alpine', ttyEnabled: true, command: 'cat'),
containerTemplate(name: 'busybox', image: 'busybox', ttyEnabled: true, command: 'cat')
]) {
node('hello') {
stage('Maven') {
container('maven') {
sh 'mvn -version'
}
}
stage('Busybox') {
container('busybox') {
sh '/bin/busybox'
}
}
}
}파이프라인 수행 결과는 다음과 같다.
Started by user Admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Created Pod: kubernetes ci/hello-qrbjz-3q54h
[Normal][ci/hello-qrbjz-3q54h][Scheduled] Successfully assigned ci/hello-qrbjz-3q54h to worker-1
[Normal][ci/hello-qrbjz-3q54h][Pulled] Container image "busybox" already present on machine
[Normal][ci/hello-qrbjz-3q54h][Created] Created container busybox
[Normal][ci/hello-qrbjz-3q54h][Started] Started container busybox
[Normal][ci/hello-qrbjz-3q54h][Pulling] Pulling image "maven:alpine"
[Normal][ci/hello-qrbjz-3q54h][Pulled] Successfully pulled image "maven:alpine" in 7.784163963s
[Normal][ci/hello-qrbjz-3q54h][Created] Created container maven
[Normal][ci/hello-qrbjz-3q54h][Started] Started container maven
[Normal][ci/hello-qrbjz-3q54h][Pulled] Container image "jenkins/inbound-agent:4.3-4" already present on machine
[Normal][ci/hello-qrbjz-3q54h][Created] Created container jnlp
[Normal][ci/hello-qrbjz-3q54h][Started] Started container jnlp
Still waiting to schedule task
‘hello-qrbjz-3q54h’ is offline
Agent hello-qrbjz-3q54h is provisioned from template hello-qrbjz
---
apiVersion: "v1"
kind: "Pod"
metadata:
annotations:
buildUrl: "http://jenkins:8080/job/test2/3/"
runUrl: "job/test2/3/"
labels:
jenkins: "slave"
jenkins/label-digest: "b6d795fbd58cc7592d955a219374339a323801a9"
jenkins/label: "hello"
name: "hello-qrbjz-3q54h"
spec:
containers:
- command:
- "cat"
image: "busybox"
imagePullPolicy: "IfNotPresent"
name: "busybox"
resources:
limits: {}
requests: {}
tty: true
volumeMounts:
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
- command:
- "cat"
image: "maven:alpine"
imagePullPolicy: "IfNotPresent"
name: "maven"
resources:
limits: {}
requests: {}
tty: true
volumeMounts:
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
- env:
- name: "JENKINS_SECRET"
value: "********"
- name: "JENKINS_TUNNEL"
value: "jenkins-jnlp:50000"
- name: "JENKINS_AGENT_NAME"
value: "hello-qrbjz-3q54h"
- name: "JENKINS_NAME"
value: "hello-qrbjz-3q54h"
- name: "JENKINS_AGENT_WORKDIR"
value: "/home/jenkins/agent"
- name: "JENKINS_URL"
value: "http://jenkins:8080/"
image: "jenkins/inbound-agent:4.3-4"
name: "jnlp"
resources:
limits: {}
requests:
memory: "256Mi"
cpu: "100m"
volumeMounts:
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
nodeSelector:
kubernetes.io/os: "linux"
restartPolicy: "Never"
volumes:
- emptyDir:
medium: ""
name: "workspace-volume"
Running on hello-qrbjz-3q54h in /home/jenkins/agent/workspace/test2
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Maven)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
+ mvn -version
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-04T19:00:29Z)
Maven home: /usr/share/maven
Java version: 1.8.0_212, vendor: IcedTea, runtime: /usr/lib/jvm/java-1.8-openjdk/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.4.0-66-generic", arch: "amd64", family: "unix"
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Busybox)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
+ /bin/busybox
BusyBox v1.32.1 (2021-02-18 00:40:10 UTC) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2015.
Licensed under GPLv2. See source distribution for detailed
copyright notices.
Usage: busybox [function [arguments]...]
or: busybox --list[-full]
or: busybox --show SCRIPT
or: busybox --install [-s] [DIR]
or: function [arguments]...
BusyBox is a multi-call binary that combines many common Unix
utilities into a single executable. Most people will create a
link to busybox for each function they wish to use and BusyBox
will act like whatever it was invoked as.
Currently defined functions:
[, [[, acpid, add-shell, addgroup, adduser, adjtimex, ar, arch, arp,
arping, ash, awk, base64, basename, bc, beep, blkdiscard, blkid,
blockdev, bootchartd, brctl, bunzip2, bzcat, bzip2, cal, cat, chat,
chattr, chgrp, chmod, chown, chpasswd, chpst, chroot, chrt, chvt,
cksum, clear, cmp, comm, conspy, cp, cpio, crond, crontab, cryptpw,
cttyhack, cut, date, dc, dd, deallocvt, delgroup, deluser, depmod,
devmem, df, dhcprelay, diff, dirname, dmesg, dnsd, dnsdomainname,
dos2unix, dpkg, dpkg-deb, du, dumpkmap, dumpleases, echo, ed, egrep,
eject, env, envdir, envuidgid, ether-wake, expand, expr, factor,
fakeidentd, fallocate, false, fatattr, fbset, fbsplash, fdflush,
fdformat, fdisk, fgconsole, fgrep, find, findfs, flock, fold, free,
freeramdisk, fsck, fsck.minix, fsfreeze, fstrim, fsync, ftpd, ftpget,
ftpput, fuser, getopt, getty, grep, groups, gunzip, gzip, halt, hd,
hdparm, head, hexdump, hexedit, hostid, hostname, httpd, hush, hwclock,
i2cdetect, i2cdump, i2cget, i2cset, i2ctransfer, id, ifconfig, ifdown,
ifenslave, ifplugd, ifup, inetd, init, insmod, install, ionice, iostat,
ip, ipaddr, ipcalc, ipcrm, ipcs, iplink, ipneigh, iproute, iprule,
iptunnel, kbd_mode, kill, killall, killall5, klogd, last, less, link,
linux32, linux64, linuxrc, ln, loadfont, loadkmap, logger, login,
logname, logread, losetup, lpd, lpq, lpr, ls, lsattr, lsmod, lsof,
lspci, lsscsi, lsusb, lzcat, lzma, lzop, makedevs, makemime, man,
md5sum, mdev, mesg, microcom, mim, mkdir, mkdosfs, mke2fs, mkfifo,
mkfs.ext2, mkfs.minix, mkfs.vfat, mknod, mkpasswd, mkswap, mktemp,
modinfo, modprobe, more, mount, mountpoint, mpstat, mt, mv, nameif,
nanddump, nandwrite, nbd-client, nc, netstat, nice, nl, nmeter, nohup,
nologin, nproc, nsenter, nslookup, ntpd, nuke, od, openvt, partprobe,
passwd, paste, patch, pgrep, pidof, ping, ping6, pipe_progress,
pivot_root, pkill, pmap, popmaildir, poweroff, powertop, printenv,
printf, ps, pscan, pstree, pwd, pwdx, raidautorun, rdate, rdev,
readahead, readlink, readprofile, realpath, reboot, reformime,
remove-shell, renice, reset, resize, resume, rev, rm, rmdir, rmmod,
route, rpm, rpm2cpio, rtcwake, run-init, run-parts, runlevel, runsv,
runsvdir, rx, script, scriptreplay, sed, sendmail, seq, setarch,
setconsole, setfattr, setfont, setkeycodes, setlogcons, setpriv,
setserial, setsid, setuidgid, sh, sha1sum, sha256sum, sha3sum,
sha512sum, showkey, shred, shuf, slattach, sleep, smemcap, softlimit,
sort, split, ssl_client, start-stop-daemon, stat, strings, stty, su,
sulogin, sum, sv, svc, svlogd, svok, swapoff, swapon, switch_root,
sync, sysctl, syslogd, tac, tail, tar, taskset, tc, tcpsvd, tee,
telnet, telnetd, test, tftp, tftpd, time, timeout, top, touch, tr,
traceroute, traceroute6, true, truncate, ts, tty, ttysize, tunctl,
ubiattach, ubidetach, ubimkvol, ubirename, ubirmvol, ubirsvol,
ubiupdatevol, udhcpc, udhcpc6, udhcpd, udpsvd, uevent, umount, uname,
unexpand, uniq, unix2dos, unlink, unlzma, unshare, unxz, unzip, uptime,
users, usleep, uudecode, uuencode, vconfig, vi, vlock, volname, w,
wall, watch, watchdog, wc, wget, which, who, whoami, whois, xargs, xxd,
xz, xzcat, yes, zcat, zcip
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS실제 로그를 확인하면 Pod Template에 정의한 대로 Agent를 생성한 뒤, maven 컨테이너로부터 'maven -version', busybox 컨테이너로부터 '/bin/busybox' 명령어의 결과를 출력하고 있음을 알 수 있다.
Declarative Pipeline
위 Scripted Pipeline과 동일한 동작을 하는 Declarative Pipeline을 아래와 같이 정의하였다.
pipeline {
agent {
kubernetes {
label 'hello'
yaml """\
apiVersion: v1
kind: Pod
metadata:
your-label: some-additional-label
spec:
containers:
- name: maven
image: maven:alpine
command:
- cat
tty: true
- name: busybox
image: busybox
command:
- cat
tty: true
""".stripIndent()
}
}
stages {
stage('Maven') {
steps {
container('maven') {
sh 'mvn -version'
}
}
}
stage('Busybox') {
steps {
container('busybox') {
sh '/bin/busybox'
}
}
}
}
}Scripted Pipeline에서 정의한 Pod Template과는 달리, 실제 k8s의 Pod의 Manifest 형식(yaml)으로 작성해야 한다. 물론 모든 항목을 정의하는 대신, 추가할 컨테이너나 볼륨 등이면 충분하다. Namespace를 따로 지정하지 않았기 때문에 'default' Namespace에 pod이 생성됨에 유의해야 한다.
다만 IDE 환경에서 yaml 필드를 작성할 때, 가끔씩 Indentation 구조가 깨져서 에러가 발생할 수 있으니 아래와 같이 'yamlFile'로 대체하는 방식도 고려해볼만 하다.
pipeline {
agent {
kubernetes {
label 'hello'
yamlFile 'hello-pod-template.yaml'
}
}
stages {
stage('Maven') {
steps {
container('maven') {
sh 'mvn -version'
}
}
}
stage('Busybox') {
steps {
container('busybox') {
sh '/bin/busybox'
}
}
}
}
}apiVersion: v1
kind: Pod
metadata:
your-label: some-additional-label
spec:
containers:
- name: maven
image: maven:alpine
command:
- cat
tty: true
- name: busybox
image: busybox
command:
- cat
tty: true<hello-pod-template.yaml>
이 경우, Pod Template 파일은 Jenkinsfile이 있는 Repository에 같이 위치해야 한다.
이제 빌드 결과를 확인해보자.
Started by user Admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] echo
[WARNING] label option is deprecated. To use a static pod template, use the 'inheritFrom' option.
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Created Pod: kubernetes ci/hello-kzw8g-xwlwv
[Normal][ci/hello-kzw8g-xwlwv][Scheduled] Successfully assigned ci/hello-kzw8g-xwlwv to worker-1
[Normal][ci/hello-kzw8g-xwlwv][Pulling] Pulling image "busybox"
[Normal][ci/hello-kzw8g-xwlwv][Pulled] Successfully pulled image "busybox" in 2.478996134s
[Normal][ci/hello-kzw8g-xwlwv][Created] Created container busybox
[Normal][ci/hello-kzw8g-xwlwv][Started] Started container busybox
[Normal][ci/hello-kzw8g-xwlwv][Pulled] Container image "maven:alpine" already present on machine
[Normal][ci/hello-kzw8g-xwlwv][Created] Created container maven
[Normal][ci/hello-kzw8g-xwlwv][Started] Started container maven
[Normal][ci/hello-kzw8g-xwlwv][Pulled] Container image "jenkins/inbound-agent:4.3-4" already present on machine
[Normal][ci/hello-kzw8g-xwlwv][Created] Created container jnlp
[Normal][ci/hello-kzw8g-xwlwv][Started] Started container jnlp
Still waiting to schedule task
‘hello-kzw8g-xwlwv’ is offline
Agent hello-kzw8g-xwlwv is provisioned from template hello-kzw8g
---
apiVersion: "v1"
kind: "Pod"
metadata:
annotations:
buildUrl: "http://jenkins:8080/job/test2/6/"
runUrl: "job/test2/6/"
labels:
jenkins: "slave"
jenkins/label-digest: "b6d795fbd58cc7592d955a219374339a323801a9"
jenkins/label: "hello"
name: "hello-kzw8g-xwlwv"
spec:
containers:
- command:
- "cat"
image: "busybox"
name: "busybox"
tty: true
volumeMounts:
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
- command:
- "cat"
image: "maven:alpine"
name: "maven"
tty: true
volumeMounts:
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
- env:
- name: "JENKINS_SECRET"
value: "********"
- name: "JENKINS_TUNNEL"
value: "jenkins-jnlp:50000"
- name: "JENKINS_AGENT_NAME"
value: "hello-kzw8g-xwlwv"
- name: "JENKINS_NAME"
value: "hello-kzw8g-xwlwv"
- name: "JENKINS_AGENT_WORKDIR"
value: "/home/jenkins/agent"
- name: "JENKINS_URL"
value: "http://jenkins:8080/"
image: "jenkins/inbound-agent:4.3-4"
name: "jnlp"
resources:
limits: {}
requests:
memory: "256Mi"
cpu: "100m"
volumeMounts:
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
nodeSelector:
kubernetes.io/os: "linux"
restartPolicy: "Never"
volumes:
- emptyDir:
medium: ""
name: "workspace-volume"
Running on hello-kzw8g-xwlwv in /home/jenkins/agent/workspace/test2
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Maven)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
+ mvn -version
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-04T19:00:29Z)
Maven home: /usr/share/maven
Java version: 1.8.0_212, vendor: IcedTea, runtime: /usr/lib/jvm/java-1.8-openjdk/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.4.0-66-generic", arch: "amd64", family: "unix"
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Busybox)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
+ /bin/busybox
BusyBox v1.32.1 (2021-02-18 00:40:10 UTC) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2015.
Licensed under GPLv2. See source distribution for detailed
copyright notices.
Usage: busybox [function [arguments]...]
or: busybox --list[-full]
or: busybox --show SCRIPT
or: busybox --install [-s] [DIR]
or: function [arguments]...
BusyBox is a multi-call binary that combines many common Unix
utilities into a single executable. Most people will create a
link to busybox for each function they wish to use and BusyBox
will act like whatever it was invoked as.
Currently defined functions:
[, [[, acpid, add-shell, addgroup, adduser, adjtimex, ar, arch, arp,
arping, ash, awk, base64, basename, bc, beep, blkdiscard, blkid,
blockdev, bootchartd, brctl, bunzip2, bzcat, bzip2, cal, cat, chat,
chattr, chgrp, chmod, chown, chpasswd, chpst, chroot, chrt, chvt,
cksum, clear, cmp, comm, conspy, cp, cpio, crond, crontab, cryptpw,
cttyhack, cut, date, dc, dd, deallocvt, delgroup, deluser, depmod,
devmem, df, dhcprelay, diff, dirname, dmesg, dnsd, dnsdomainname,
dos2unix, dpkg, dpkg-deb, du, dumpkmap, dumpleases, echo, ed, egrep,
eject, env, envdir, envuidgid, ether-wake, expand, expr, factor,
fakeidentd, fallocate, false, fatattr, fbset, fbsplash, fdflush,
fdformat, fdisk, fgconsole, fgrep, find, findfs, flock, fold, free,
freeramdisk, fsck, fsck.minix, fsfreeze, fstrim, fsync, ftpd, ftpget,
ftpput, fuser, getopt, getty, grep, groups, gunzip, gzip, halt, hd,
hdparm, head, hexdump, hexedit, hostid, hostname, httpd, hush, hwclock,
i2cdetect, i2cdump, i2cget, i2cset, i2ctransfer, id, ifconfig, ifdown,
ifenslave, ifplugd, ifup, inetd, init, insmod, install, ionice, iostat,
ip, ipaddr, ipcalc, ipcrm, ipcs, iplink, ipneigh, iproute, iprule,
iptunnel, kbd_mode, kill, killall, killall5, klogd, last, less, link,
linux32, linux64, linuxrc, ln, loadfont, loadkmap, logger, login,
logname, logread, losetup, lpd, lpq, lpr, ls, lsattr, lsmod, lsof,
lspci, lsscsi, lsusb, lzcat, lzma, lzop, makedevs, makemime, man,
md5sum, mdev, mesg, microcom, mim, mkdir, mkdosfs, mke2fs, mkfifo,
mkfs.ext2, mkfs.minix, mkfs.vfat, mknod, mkpasswd, mkswap, mktemp,
modinfo, modprobe, more, mount, mountpoint, mpstat, mt, mv, nameif,
nanddump, nandwrite, nbd-client, nc, netstat, nice, nl, nmeter, nohup,
nologin, nproc, nsenter, nslookup, ntpd, nuke, od, openvt, partprobe,
passwd, paste, patch, pgrep, pidof, ping, ping6, pipe_progress,
pivot_root, pkill, pmap, popmaildir, poweroff, powertop, printenv,
printf, ps, pscan, pstree, pwd, pwdx, raidautorun, rdate, rdev,
readahead, readlink, readprofile, realpath, reboot, reformime,
remove-shell, renice, reset, resize, resume, rev, rm, rmdir, rmmod,
route, rpm, rpm2cpio, rtcwake, run-init, run-parts, runlevel, runsv,
runsvdir, rx, script, scriptreplay, sed, sendmail, seq, setarch,
setconsole, setfattr, setfont, setkeycodes, setlogcons, setpriv,
setserial, setsid, setuidgid, sh, sha1sum, sha256sum, sha3sum,
sha512sum, showkey, shred, shuf, slattach, sleep, smemcap, softlimit,
sort, split, ssl_client, start-stop-daemon, stat, strings, stty, su,
sulogin, sum, sv, svc, svlogd, svok, swapoff, swapon, switch_root,
sync, sysctl, syslogd, tac, tail, tar, taskset, tc, tcpsvd, tee,
telnet, telnetd, test, tftp, tftpd, time, timeout, top, touch, tr,
traceroute, traceroute6, true, truncate, ts, tty, ttysize, tunctl,
ubiattach, ubidetach, ubimkvol, ubirename, ubirmvol, ubirsvol,
ubiupdatevol, udhcpc, udhcpc6, udhcpd, udpsvd, uevent, umount, uname,
unexpand, uniq, unix2dos, unlink, unlzma, unshare, unxz, unzip, uptime,
users, usleep, uudecode, uuencode, vconfig, vi, vlock, volname, w,
wall, watch, watchdog, wc, wget, which, who, whoami, whois, xargs, xxd,
xz, xzcat, yes, zcat, zcip
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS
참고
Kubernetes plugin for Jenkins
'CI-CD' 카테고리의 다른 글
| Argo Workflows - (1) Introduction (0) | 2021.07.01 |
|---|---|
| Jenkins - Approval Stage 구현 (0) | 2021.03.05 |
| Jenkins Pipeline (0) | 2021.03.01 |
| kustomize를 활용한 Manifest 관리 (0) | 2021.02.24 |
| CD를 위한 Jenkins, Argo CD 연계 (0) | 2021.02.21 |