但是,在实际的场景中,并不能满足所有应用,尤其是分布式应用,会部署多个实例,这些实例之间往往有依赖关系,例如主从关系、主备关系,这种应用称为“有状态”,例如MySQL主从、Etcd集群、redis-cluster 等等
2、StatefulSet 控制器概述
StatefulSet控制器用于部署有状态应用,满足一些有状态应用的需求:
- Pod有序的部署、扩容、删除和停止
- Pod分配一个稳定的且唯一的网络标识
- Pod分配一个独享的存储
3、StatefulSet 控制器:网络标识
稳定的网络标识:
使用无头服务 Headless Service(相比普通Service只是将spec.clusterIP定义为None,也就是没有clusterIP,使用endport 来通信)来维护Pod网络身份,会为每个Pod分配一个数字编号并且按照编号顺序部署。还需要在StatefulSet添加serviceName: “nginx”字段指定StatefulSet控制器要使用这个Headless Service。
稳定主要体现在主机名和Pod A记录:
- 主机名:
-<编号> - Pod DNS A记录:
. . .svc.cluster.local (POD 之间通过DNS A 记录通信)
例如: web-0.web.default.svc.cluster.local
- 备注:
-
- A记录: 将域名指向一个IPv4地址(例如:100.100.100.100),需要增加A记录
- CNAME记录: 如果将域名指向一个域名,实现与被指向域名相同的访问效果,需要增加CNAME记录。这个域名一般是主机服务商提供的一个域名
- MX记录: 建立电子邮箱服务,将指向邮件服务器地址,需要设置MX记录。建立邮箱时,一般会根据邮箱服务商提供的MX记录填写此记录
- NS记录: 域名解析服务器记录,如果要将子域名指定某个域名服务器来解析,需要设置NS记录
- TXT记录: 可任意填写,可为空。一般做一些验证记录时会使用此项,如:做SPF(反垃圾邮件)记录
- AAAA记录: 将主机名(或域名)指向一个IPv6地址(例如:ff03:0:0:0:0:0:0:c1),需要添加AAAA记录
案例:
通过创建一个 nginx 应用的statefluset 控制器
创建 Headless Service ,定义 clusterIP: None (表示K8S 不会在给这个service 去颁发一个clusterIP 了;相比 deployment 控制器 的每个POD 都是相同的,而 statefuset 控制器的每个POD 都是有状态的
需要单独去访问 )
- [root@master-1 statefulset]# vim service.yaml
- apiVersion: v1
- kind: Service
- metadata:
- name: web
- spec:
- clusterIP: None
- ports:
- - protocol: TCP
- port: 80
- selector:
- app: nginx
-
- [root@master-1 statefulset]# kubectl apply -f service.yaml
-
- [root@master-1 statefulset]# kubectl get service
- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
- web ClusterIP None
80/TCP 8m18s
#创建statefulset,指定serviceName
- [root@master-1 statefulset]# cat statefulset.yaml
- apiVersion: apps/v1
- kind: StatefulSet
- metadata:
- name: web
- spec:
- serviceName: "web"
- replicas: 3
- selector:
- matchLabels:
- app: nginx
- template:
- metadata:
- labels:
- app: nginx
- spec:
- containers:
- - name: nginx
- image: nginx
- ports:
- - containerPort: 80
- name: web
-
-
- [root@master-1 statefulset]# kubectl apply -f statefulset.yaml
- statefulset.apps/web created
-
-
- #发现pod名称 带有序号
- [root@master-1 statefulset]# kubectl get pods
- NAME READY STATUS RESTARTS AGE
- web-0 1/1 Running 0 16s
- web-1 1/1 Running 0 13s
- web-2 1/1 Running 0 6s
-
-
-
-
- #查了POD的主机名
- 主机名 默认与 POD 名一致,即使POD 飘逸到其他node 上 或者删除后重建 主机名 都是和POD 名一致。有个稳定的主机名
-
-
- [root@master-1 statefulset]# kubectl exec -it web-0 -- hostname
- web-0
- [root@master-1 statefulset]# kubectl exec -it web-1 -- hostname
- web-1
- [root@master-1 statefulset]# kubectl exec -it web-2 -- hostname
- web-2
-
-
-
- #临时启动一个busybox pod ,测试dns 解析(注意这里的busybox版本为1.28.4 最新版的busybox nslookup 会有问题)
- [root@master-1 statefulset]# kubectl run -it dns-test --rm --image=busybox:1.28.4 -- sh
- / # nslookup web
- Server: 10.0.0.2
- Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local
-
- Name: web
- Address 1: 10.244.2.114 web-2.web.default.svc.cluster.local
- Address 2: 10.244.2.113 web-0.web.default.svc.cluster.local
- Address 3: 10.244.1.65 web-1.web.default.svc.cluster.local
-
-
- 可以看到解析出3条记录出来,解析出对应的三个Pod IP记录,其他Pod可使用这个名称访问
-
-
- 模拟测试删除这些 pod ,升级镜像版本,发现pod ip虽然发生变化,但是 主机名,Pod DNS A记录 不会发生变化
- / # nslookup web
- Server: 10.0.0.2
- Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local
-
- Name: web
- Address 1: 10.244.2.116 web-2.web.default.svc.cluster.local
- Address 2: 10.244.2.115 web-0.web.default.svc.cluster.local
- Address 3: 10.244.1.66 web-1.web.default.svc.cluster.local
这个就验证了 statefulset 的 Pod是 有序的部署、扩容、删除和停止 且 给每一个POD 分配一个稳定的且唯一的网络标识
4、StatefulSet 控制器:独享存储
独享存储:StatefulSet的存储卷使用VolumeClaimTemplate创建,称为卷申请模板,当StatefulSet使用VolumeClaimTemplate创建一个PersistentVolume时,
同样也会为每个Pod分配并创建一个编号的PVC,每个PVC绑定对应的PV,从而保证每个Pod都拥有独立的存储。
在创建StatefulSet 控制器 独享存储前,需要先定义好存储卷,使用pv 作为持久化存储卷,后端存储为NFS
这里采用动态PV 的方式(NFS server 搭建的过程省略)
一、部署NFS服务器
- #服务器安装nfs服务,提供nfs存储功能
- 1、安装nfs-utils
- yum install nfs-utils (centos)
- 或者 apt-get install nfs-kernel-server (ubuntu)
-
- 2、启动服务
- systemctl enable nfs-server
- systemctl start nfs-server
-
- 3、创建共享目录完成共享配置
- mkdir /home/nfs #创建共享目录
-
- 4、编辑共享配置
- vim /etc/exports
- #语法格式: 共享文件路径 客户机地址(权限) #这里的客户机地址可以是IP,网段,域名,也可以是任意*
- /home/nfs *(rw,async,no_root_squash)
- 服务自检命令
- exportfs -arv
-
-
- 5、重启服务
- systemctl restart nfs-server
-
-
- 6、本机查看nfs 共享目录
- #showmount -e 服务器IP地址 (如果提示命令不存在,则需要yum install showmount)
-
- showmount -e 127.0.0.1
- /home/nfs/nginx *
-
-
-
- 7、客户端模拟挂载[所有k8s的节点都需要安装客户端]
- [root@master-1 ~]# yum install nfs-utils (centos)
- 或者 apt-get install nfs-common (ubuntu)
- [root@master-1 ~]# mkdir /test
- [root@master-1 ~]# mount -t nfs 172.16.201.209:/home/nfs /test
-
- #取消挂载
- [root@master-1 ~]# umount /test
二、配置PV 动态供给(NFS StorageClass),创建pvc
#部署NFS实现自动创建PV插件: 一共设计到4个yaml 文件 ,官方的文档有详细的说明
https://github.com/kubernetes-incubator/external-storage
- root@k8s-master1:~ # mkdir /root/pvc
- root@k8s-master1:~ # cd /root/pvc
创建rbac.yaml 文件
- root@k8s-master1:pvc # cat rbac.yaml
- kind: ServiceAccount
- apiVersion: v1
- metadata:
- name: nfs-client-provisioner
- ---
- kind: ClusterRole
- apiVersion: rbac.authorization.k8s.io/v1
- metadata:
- name: nfs-client-provisioner-runner
- rules:
- - apiGroups: [""]
- resources: ["persistentvolumes"]
- verbs: ["get", "list", "watch", "create", "delete"]
- - apiGroups: [""]
- resources: ["persistentvolumeclaims"]
- verbs: ["get", "list", "watch", "update"]
- - apiGroups: ["storage.k8s.io"]
- resources: ["storageclasses"]
- verbs: ["get", "list", "watch"]
- - apiGroups: [""]
- resources: ["events"]
- verbs: ["create", "update", "patch"]
- ---
- kind: ClusterRoleBinding
- apiVersion: rbac.authorization.k8s.io/v1
- metadata:
- name: run-nfs-client-provisioner
- subjects:
- - kind: ServiceAccount
- name: nfs-client-provisioner
- namespace: default
- roleRef:
- kind: ClusterRole
- name: nfs-client-provisioner-runner
- apiGroup: rbac.authorization.k8s.io
- ---
- kind: Role
- apiVersion: rbac.authorization.k8s.io/v1
- metadata:
- name: leader-locking-nfs-client-provisioner
- rules:
- - apiGroups: [""]
- resources: ["endpoints"]
- verbs: ["get", "list", "watch", "create", "update", "patch"]
- ---
- kind: RoleBinding
- apiVersion: rbac.authorization.k8s.io/v1
- metadata:
- name: leader-locking-nfs-client-provisioner
- subjects:
- - kind: ServiceAccount
- name: nfs-client-provisioner
- # replace with namespace where provisioner is deployed
- namespace: default
- roleRef:
- kind: Role
- name: leader-locking-nfs-client-provisioner
- apiGroup: rbac.authorization.k8s.io
创建deployment.yaml 文件
#官方默认的镜像地址,国内可能无法下载,可以使用 image:
fxkjnj/nfs-client-provisioner:latest
#定义NFS 服务器的地址,共享目录名称
- root@k8s-master1:pvc # cat deployment.yaml
- apiVersion: v1
- kind: ServiceAccount
- metadata:
- name: nfs-client-provisioner
- ---
- kind: Deployment
- apiVersion: apps/v1
- metadata:
- name: nfs-client-provisioner
- spec:
- replicas: 1
- strategy:
- type: Recreate
- selector:
- matchLabels:
- app: nfs-client-provisioner
- template:
- metadata:
- labels:
- app: nfs-client-provisioner
- spec:
- serviceAccountName: nfs-client-provisioner
- containers:
- - name: nfs-client-provisioner
- image: fxkjnj/nfs-client-provisioner:latest
- volumeMounts:
- - name: nfs-client-root
- mountPath: /persistentvolumes
- env:
- - name: PROVISIONER_NAME
- value: fuseim.pri/ifs
- - name: NFS_SERVER
- value: 172.16.201.209
- - name: NFS_PATH
- value: /home/nfs
- volumes:
- - name: nfs-client-root
- nfs:
- server: 172.16.201.209
- path: /home/nfs
创建class.yaml
# archiveOnDelete: "true" 表示当PVC 删除后,后端数据不直接删除,而是归档
- root@k8s-master1:pvc # cat class.yaml
- apiVersion: storage.k8s.io/v1
- kind: StorageClass
- metadata:
- name: managed-nfs-storage
- provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
- parameters:
- archiveOnDelete: "true"
查看存储类
- root@k8s-master1:~/kubernetes/redis# kubectl get sc
- NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
- managed-nfs-storage fuseim.pri/ifs Delete Immediate false 6s
基于上面的内容创建 statefulset.yaml 文件
- root@k8s-master1:~ # mkdir /root/statefulset
- root@k8s-master1:~ # cd /root/statefulset
- root@k8s-master1:statefulset # vim statefulset.yaml
- apiVersion: apps/v1
- kind: StatefulSet
- metadata:
- name: web
- spec:
- serviceName: "web"
- replicas: 3
- selector:
- matchLabels:
- app: nginx
- template:
- metadata:
- labels:
- app: nginx
- spec:
- containers:
- - name: nginx
- image: nginx:1.16
- ports:
- - containerPort: 80
- name: web
- volumeMounts:
- - name: nginx-pvc #指定PVC名称
- mountPath: /usr/share/nginx/html
- volumeClaimTemplates: #相当于pvc模板
- - metadata:
- name: nginx-pvc #创建的PVC名称
- spec:
- storageClassName: "managed-nfs-storage" #指定动态PV名称
- accessModes:
- - ReadWriteOnce #访问模式,读写在单台机器
- resources:
- requests:
- storage: 1Gi
-
-
- root@k8s-master1:statefulset # kubectl apply -f statefulset.yaml
- statefulset.apps/web created
-
- root@k8s-master1:~/kubernetes/statefulset# kubectl get pv
- NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
- pvc-8eacbe25-3875-4f78-91ca-ba83b6967a8a 100Gi RWX Delete Bound redis/nfs-redis managed-nfs-storage 6d
- pvc-935033b7-9ac8-4346-8543-1f95492dcde9 1Gi RWO Delete Bound default/nginx-pvc-web-1 managed-nfs-storage 39s
- pvc-bd3a8c59-b66d-457b-a6f2-90f3b7f9ebf0 1Gi RWO Delete Bound default/nginx-pvc-web-2 managed-nfs-storage 19s
- pvc-be5cf42a-aeaa-4667-901c-77e1d2350f49 1Gi RWO Delete Bound default/nginx-pvc-web-0 managed-nfs-storage 61s
-
-
-
- root@k8s-master1:~/kubernetes/statefulset# kubectl get pvc
- NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
- nginx-pvc-web-0 Bound pvc-be5cf42a-aeaa-4667-901c-77e1d2350f49 1Gi RWO managed-nfs-storage 82s
- nginx-pvc-web-1 Bound pvc-935033b7-9ac8-4346-8543-1f95492dcde9 1Gi RWO managed-nfs-storage 61s
- nginx-pvc-web-2 Bound pvc-bd3a8c59-b66d-457b-a6f2-90f3b7f9ebf0 1Gi RWO managed-nfs-storage 40s
-
- oot@k8s-master1:~/kubernetes/statefulset# kubectl get pods --show-labels
- NAME READY STATUS RESTARTS AGE LABELS
- web-0 1/1 Running 0 4m50s app=nginx,controller-revision-hash=web-b56c497b,statefulset.kubernetes.io/pod-name=web-0
- web-1 1/1 Running 0 4m29s app=nginx,controller-revision-hash=web-b56c497b,statefulset.kubernetes.io/pod-name=web-1
- web-2 1/1 Running 0 4m8s app=nginx,controller-revision-hash=web-b56c497b,statefulset.kubernetes.io/pod-name=web-2
-
- #分别进入到3个pod 中,写入一个数据,验证各自的独享存储
- root@k8s-master1:~/kubernetes/statefulset# kubectl get pods -o wide --selector app=nginx
- NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
- web-0 1/1 Running 0 7m6s 10.244.169.179 k8s-node2
- web-1 1/1 Running 0 6m45s 10.244.107.228 k8s-node3
- web-2 1/1 Running 0 6m24s 10.244.169.180 k8s-node2
-
-
- [root@master-1 ~]# kubectl exec -it web-0 -- bash -c "echo 'congratulations web-0 for k8s' > /usr/share/nginx/html/index.html"
- [root@master-1 ~]# kubectl exec -it web-1 -- bash -c "echo 'congratulations web-1 for k8s' > /usr/share/nginx/html/index.html"
- [root@master-1 ~]# kubectl exec -it web-2 -- bash -c "echo 'congratulations web-2 for k8s' > /usr/share/nginx/html/index.html"
-
-
- #直接访问pod IP 测试内容:
- root@k8s-master1:~/kubernetes/statefulset# curl 10.244.169.179
- congratulations web-0 for k8s
-
- root@k8s-master1:~/kubernetes/statefulset# curl 10.244.107.228
- congratulations web-1 for k8s
-
- root@k8s-master1:~/kubernetes/statefulset# curl 10.244.169.180
- congratulations web-2 for k8s
删除statefulset
删除statefulset 有两张方法,级联删除 和 非级联删除
- 使用非级联删除 statefulset 时,statefulset 的POD 不会被删除
- 使用级联删除时,statefulset 和 pod 都会被删除
- (1)、非级联删除
-
- 使用kubectl delete statefulset XXXX 删除 statefulset ,只需要提供 --cascade=false 参数,就会采用非联机删除,此时删除statefulset 不会删除pod
-
- kubectl delete statefulset web --cascade=false
-
-
- (2)、级联删除
- 省略 --cascade=false 参数 即可
-
- kubectl delete statefulset web