K8s 安裝筆記 (ubuntu16.04)

by Rick Hwang

Complete Think / 2019-03-18 02:06

整理安裝 Kubernetes 的筆記,主要是以 kubeadm 為主。

準備

  • VMWare on macOS
  • ubuntu 16.04 (ubuntu-16.04.6-server-amd64.iso)

制作 Base Image

這個 Base Image 會用在安裝 Master / Worker Node.

  • 安裝 ubuntu 16.04 (ubuntu-16.04.6-server-amd64.iso)
  • 在 VMWare 上,將 Guess OS 的網路設定為 Bridge Mode
  • 關閉 swap: swapoff -a
  • 註解 /etc/fstab

安裝 Docker

因為 CRI 最近已經從 CNCF 畢業了,所以除了 Docker,還有其他可以選擇。但這裡還是先以 docker 為主。

以下參考自:https://kubernetes.io/docs/setup/cri/

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# Install Docker CE
## Set up the repository:
### Update the apt package index
apt-get update

### Install packages to allow apt to use a repository over HTTPS
apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common

### Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

### Add docker apt repository.
add-apt-repository
"deb [arch=amd64] https://download.docker.com/linux/ubuntu
$(lsb_release -cs)
stable"

## Install docker ce.
apt-get update && apt-get install docker-ce=18.06.2~ce~3-0~ubuntu

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker

確認版本:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
~# docker version
Client:
Version: 18.06.2-ce
API version: 1.38
Go version: go1.10.3
Git commit: 6d37f41
Built: Sun Feb 10 03:48:06 2019
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.06.2-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: 6d37f41
Built: Sun Feb 10 03:46:30 2019
OS/Arch: linux/amd64
Experimental: false

安裝 kubeadm, kubectl, kubelet

主要參考自 官方文件 ,不過最新版 kubeadm 無法順利初始化,所以這個紀錄以 1.11.3 為範例。

準備 repository:

1
2
3
4
5
6
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update

指定 kubeadm 版本:

1
2
3
apt-get install -y kubelet=1.11.3-00
apt-get install -y kubectl=1.11.3-00
apt-get install -y kubeadm=1.11.3-00

安裝新版本 kubeadm 如下,但是本次 Lab 無法運作:

1
2
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

確認安裝版本:kubeadm / kubectl

如果是安裝指定版本:

1
2
3
4
5
~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:59:42Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

~# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

最新版的確認:

1
2
3
4
5
~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:35:32Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

~# kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

安裝 Kubernetes Cluster

  • 安裝 Master Node
  • 安裝 Worker Node
  • 安裝 Dashboard
  • 安裝 PV / PVC 插件

安裝 Master Node

改機器名字:

  1. /etc/hostname: k8s-master01-u1604
  2. /etc/hosts: k8s-master01-u1604
  3. reboot

初始化 kubeadm

建立 kubeadm 配置檔:

kubeadm.yaml

1
2
3
4
5
6
7
8
9
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
controllerManagerExtraArgs:
horizontal-pod-autoscaler-use-rest-clients: "true"
horizontal-pod-autoscaler-sync-period: "10s"
node-monitor-grace-period: "10s"
apiServerExtraArgs:
runtime-config: "api/all=true"
kubernetesVersion: "stable-1.11"

這份詳細的資訊,請參閱 原始碼

執行初始化配置,過程中會根據指訂的 CRI 抓取 image,預設的 CRI 是 Docker.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
kubeadm init --config kubeadm.yaml

## 順利的話,執行過程約 2-3 分鐘,最後會出現以下訊息,
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

kubeadm join 192.168.2.16:6443 --token s8o9wi.dylbvs735sy53mmq --discovery-token-ca-cert-hash sha256:0c16a05978533ca8f44af6e779162a1c99516fa2a4acd81915f0379755a856bc

查看 docker ps,會出現一堆 container 已經在跑:

1
2
3
4
5
6
7
8
~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ab574cbeb1f2 2ed65dca1a98 "/usr/local/bin/kube…" 33 seconds ago Up 33 seconds k8s_kube-proxy_kube-proxy-7cfkg_kube-system_83851117-48bf-11e9-b533-000c29d7e00b_0
5bb5811d11d4 k8s.gcr.io/pause:3.1 "/pause" 34 seconds ago Up 33 seconds k8s_POD_kube-proxy-7cfkg_kube-system_83851117-48bf-11e9-b533-000c29d7e00b_0
acc7433382dc b8df3b177be2 "etcd --advertise-cl…" 56 seconds ago Up 55 seconds k8s_etcd_etcd-k8s-master01-u1604_kube-system_f09a86c0e59bd660bdd359cf6d46e2be_0
bf812ade4168 14028d7dcbf9 "kube-scheduler --ad…" 56 seconds ago Up 55 seconds k8s_kube-scheduler_kube-scheduler-k8s-master01-u1604_kube-system_cbb979db2eb698a42e58c4ca7edd7b16_0
3951b23da250 abbc2fa179b7 "kube-controller-man…" 56 seconds ago Up 55 seconds k8s_kube-controller-manager_kube-controller-manager-k8s-master01-u1604_kube-system_fc391fbab6130026480db4a97e595c16_0
9c146a4aae4b 6de771eabf8c "kube-apiserver --au…" 56 seconds ago Up 55 seconds k8s_kube-apiserver_kube-apiserver-k8s-master01-u1604_kube-system_f09f833b5c32ac560364b59f58055df6_0

同樣的,查看 docker images,應該已經抓了一堆東西。

1
2
3
4
5
6
7
8
9
10
11
~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy-amd64 v1.11.8 2ed65dca1a98 2 weeks ago 98.1MB
k8s.gcr.io/kube-apiserver-amd64 v1.11.8 6de771eabf8c 2 weeks ago 187MB
k8s.gcr.io/kube-controller-manager-amd64 v1.11.8 abbc2fa179b7 2 weeks ago 155MB
k8s.gcr.io/kube-scheduler-amd64 v1.11.8 14028d7dcbf9 2 weeks ago 56.9MB
weaveworks/weave-npc 2.5.1 789b7f496034 7 weeks ago 49.6MB
weaveworks/weave-kube 2.5.1 1f394ae9e226 7 weeks ago 148MB
k8s.gcr.io/coredns 1.1.3 b3b94275d97c 9 months ago 45.6MB
k8s.gcr.io/etcd-amd64 3.2.18 b8df3b177be2 11 months ago 219MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 15 months ago 742kB

確認 Master Node 狀態

取得 node 狀態,k8s-master01-u1604 還沒有 ready。可以從 describe 中看到這段關鍵訊息:network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

1
2
3
4
5
6
7
8
9
10
11
12
13
14

~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01-u1604 NotReady master 2m v1.11.3

~# kubectl describe node k8s-master01-u1604
... 略 ...

Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
... 略 ...
Ready False Sun, 17 Mar 2019 22:20:59 +0800 Sun, 17 Mar 2019 22:18:02 +0800 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

... 略 ...

取得 kube-system pods 狀態,可以看到 coredns 還沒 ready

1
2
3
4
5
6
7
8
9
~# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcdf6894-j88px 0/1 Pending 0 4m
coredns-78fcdf6894-lvlf7 0/1 Pending 0 4m
etcd-k8s-master01-u1604 1/1 Running 0 3m
kube-apiserver-k8s-master01-u1604 1/1 Running 0 3m
kube-controller-manager-k8s-master01-u1604 1/1 Running 0 3m
kube-proxy-7cfkg 1/1 Running 0 4m
kube-scheduler-k8s-master01-u1604 1/1 Running 0 3m

部署 weave

網路架構

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# 部署網路插件
kubectl apply -f https://git.io/weave-kube-1.6

# 再次檢查狀態:wave 部署中
~# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcdf6894-j88px 0/1 Pending 0 8m
coredns-78fcdf6894-lvlf7 0/1 Pending 0 8m
etcd-k8s-master01-u1604 1/1 Running 0 7m
kube-apiserver-k8s-master01-u1604 1/1 Running 0 7m
kube-controller-manager-k8s-master01-u1604 1/1 Running 0 8m
kube-proxy-7cfkg 1/1 Running 0 8m
kube-scheduler-k8s-master01-u1604 1/1 Running 0 7m
weave-net-4gbxq 0/2 ContainerCreating 0 19s

## 已經完成部署 wave
~# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcdf6894-j88px 1/1 Running 0 9m
coredns-78fcdf6894-lvlf7 1/1 Running 0 9m
etcd-k8s-master01-u1604 1/1 Running 0 8m
kube-apiserver-k8s-master01-u1604 1/1 Running 0 8m
kube-controller-manager-k8s-master01-u1604 1/1 Running 0 8m
kube-proxy-7cfkg 1/1 Running 0 9m
kube-scheduler-k8s-master01-u1604 1/1 Running 0 8m
weave-net-4gbxq 2/2 Running 0 39s

## 再次取得 node 狀態
~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01-u1604 Ready master 10m v1.11.3

安裝 Worker Node

改機器名字:

  1. /etc/hostname: k8s-worker01-u1604
  2. /etc/hosts: k8s-worker01-u1604
  3. reboot

安裝 kubeadm, kubectl, kubelet

1
~# kubeadm join 192.168.2.16:6443 --token s8o9wi.dylbvs735sy53mmq --discovery-token-ca-cert-hash sha256:0c16a05978533ca8f44af6e779162a1c99516fa2a4acd81915f0379755a856bc

k8s-master01-u1604 這台機器上,下 kubectl get nodes,就會看到 worker node 已經加入 cluster.

1
2
3
4
~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01-u1604 Ready master 19m v1.11.3
k8s-worker01-u1604 Ready <none> 2m v1.11.3

部署 Dashboard

1
2
3
4
5
6
~# kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended/kubernetes-dashboard.yaml

~# kubectl get pods -n kube-system
... 略 ...
kubernetes-dashboard-5dd89b9875-5mx9n 1/1 Running 0 16s
... 略 ...

部署 Persistent Volume(PV)和 Persistent Volume Claim (PVC) 插件

主要針對 K8s 的 PV / PVC 套件安裝,安裝之後,才能夠跑 StatefulSet。這邊安裝的是 rook

1
2
3
4
5
6
7
8
9
10
11
12
13
14
~# kubectl apply -f https://raw.githubusercontent.com/rook/rook/master/cluster/examples/kubernetes/ceph/operator.yaml
~# kubectl apply -f https://raw.githubusercontent.com/rook/rook/master/cluster/examples/kubernetes/ceph/cluster.yaml
~# kubectl get pods -n rook-ceph-system
NAME READY STATUS RESTARTS AGE
rook-ceph-agent-pjrld 1/1 Running 0 15s
rook-ceph-operator-5496d44d7c-9jrpb 1/1 Running 0 1m
rook-discover-zr7d8 1/1 Running 0 15s

~# kubectl get pods -n rook-ceph
NAME READY STATUS RESTARTS AGE
rook-ceph-mgr-a-7c9f847c77-rqhft 1/1 Running 0 16s
rook-ceph-mon-a-cc78585f-4d6mg 1/1 Running 0 47s
rook-ceph-mon-b-7c645b6547-b9p7r 1/1 Running 0 41s
rook-ceph-mon-c-689766dd4c-blqv8 1/1 Running 0 33s

參考資料

Shared via Inoreader