[TOC] ### **架构图** 本次我们性能测试的架构图如下：  ### **安装Kubemark集群** Kubemark集群就是被压测的集群，它有三个Master。另外有一个External集群，它负责创建hollow pod，来模拟节点，注册到Kubemark集群中。虚拟节点通过LVS与Kubemark-Master通信。 对于kubemark集群： - 使用kubeadm安装 - 需要安装网络插件，这里我们安装的是calico - calico、kube-proxy、coredns这些工作负载需要手动更改，让它们只运行在master节点上，coredns可能需要添加toleration，让其能运行在master节点上 - master节点不用去掉taint（注意不用去掉，后面压测过程中，大量创建Pod时，那些Pod才不会调度在master节点上） 安装完成后， 如下： ``` $ kubectl get node | grep master 10.35.20.2 Ready control-plane,master 84d v1.23.3 10.35.20.3 Ready control-plane,master 84d v1.23.3 10.35.20.4 Ready control-plane,master 84d v1.23.3 $ kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-kube-controllers-7cf567797-l8q8k 1/1 Running 2 (51d ago) 84d 172.28.200.11 10.35.20.4 <none> <none> calico-node-4s6gw 1/1 Running 2 (51d ago) 84d 10.35.20.4 10.35.20.4 <none> <none> calico-node-grt64 1/1 Running 2 (51d ago) 84d 10.35.20.3 10.35.20.3 <none> <none> calico-node-r5t5t 1/1 Running 3 (39h ago) 84d 10.35.20.2 10.35.20.2 <none> <none> coredns-7466d9c549-85ss4 1/1 Running 2 (51d ago) 84d 172.28.200.10 10.35.20.4 <none> <none> coredns-7466d9c549-br4km 1/1 Running 2 (51d ago) 84d 172.28.200.12 10.35.20.4 <none> <none> etcd-10.35.20.2 1/1 Running 2 (51d ago) 84d 10.35.20.2 10.35.20.2 <none> <none> etcd-10.35.20.3 1/1 Running 2 (51d ago) 84d 10.35.20.3 10.35.20.3 <none> <none> etcd-10.35.20.4 1/1 Running 2 (51d ago) 84d 10.35.20.4 10.35.20.4 <none> <none> kube-apiserver-10.35.20.2 1/1 Running 3 (51d ago) 84d 10.35.20.2 10.35.20.2 <none> <none> kube-apiserver-10.35.20.3 1/1 Running 2 (51d ago) 84d 10.35.20.3 10.35.20.3 <none> <none> kube-apiserver-10.35.20.4 1/1 Running 4 (51d ago) 84d 10.35.20.4 10.35.20.4 <none> <none> kube-controller-manager-10.35.20.2 1/1 Running 3 (51d ago) 84d 10.35.20.2 10.35.20.2 <none> <none> kube-controller-manager-10.35.20.3 1/1 Running 3 (51d ago) 84d 10.35.20.3 10.35.20.3 <none> <none> kube-controller-manager-10.35.20.4 1/1 Running 7 (51d ago) 84d 10.35.20.4 10.35.20.4 <none> <none> kube-proxy-rw58b 1/1 Running 2 (51d ago) 84d 10.35.20.4 10.35.20.4 <none> <none> kube-proxy-tq4gk 1/1 Running 2 (51d ago) 84d 10.35.20.3 10.35.20.3 <none> <none> kube-proxy-wz6zz 1/1 Running 2 (51d ago) 84d 10.35.20.2 10.35.20.2 <none> <none> kube-scheduler-10.35.20.2 1/1 Running 3 (51d ago) 84d 10.35.20.2 10.35.20.2 <none> <none> kube-scheduler-10.35.20.3 1/1 Running 5 (51d ago) 84d 10.35.20.3 10.35.20.3 <none> <none> kube-scheduler-10.35.20.4 1/1 Running 6 (51d ago) 84d 10.35.20.4 10.35.20.4 <none> <none> ``` ### **安装External集群** 对于External集群，需要注意： 1、使用kubeadm安装 2、需要安装网络插件，这里我们安装calico 安装好后如下： ``` $ kubectl get node NAME STATUS ROLES AGE VERSION 10.35.20.10 Ready <none> 84d v1.23.3 10.35.20.12 Ready <none> 84d v1.23.3 10.35.20.13 Ready <none> 84d v1.23.3 10.35.20.14 Ready <none> 84d v1.23.3 10.35.20.8 Ready control-plane,master 84d v1.23.3 10.35.20.9 Ready <none> 84d v1.23.3 $ kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-kube-controllers-7cf567797-59xhg 1/1 Running 2 (51d ago) 84d 10.10.138.139 10.35.20.8 <none> <none> calico-node-8jx4l 1/1 Running 2 (51d ago) 84d 10.35.20.8 10.35.20.8 <none> <none> calico-node-ls2z5 1/1 Running 2 (51d ago) 84d 10.35.20.10 10.35.20.10 <none> <none> calico-node-plnjs 1/1 Running 2 (51d ago) 84d 10.35.20.14 10.35.20.14 <none> <none> calico-node-tdcxx 1/1 Running 2 (51d ago) 84d 10.35.20.13 10.35.20.13 <none> <none> calico-node-twbpc 1/1 Running 2 (51d ago) 84d 10.35.20.9 10.35.20.9 <none> <none> calico-node-xtzfm 1/1 Running 2 (51d ago) 84d 10.35.20.12 10.35.20.12 <none> <none> coredns-7466d9c549-gx7ht 1/1 Running 2 (51d ago) 84d 10.10.138.155 10.35.20.8 <none> <none> coredns-7466d9c549-mfnxd 1/1 Running 2 (51d ago) 84d 10.10.138.170 10.35.20.8 <none> <none> etcd-10.35.20.8 1/1 Running 4 (51d ago) 84d 10.35.20.8 10.35.20.8 <none> <none> kube-apiserver-10.35.20.8 1/1 Running 14 (51d ago) 84d 10.35.20.8 10.35.20.8 <none> <none> kube-controller-manager-10.35.20.8 1/1 Running 14 (51d ago) 84d 10.35.20.8 10.35.20.8 <none> <none> kube-proxy-hfv4m 1/1 Running 3 (51d ago) 84d 10.35.20.12 10.35.20.12 <none> <none> kube-proxy-hnhgw 1/1 Running 3 (51d ago) 84d 10.35.20.8 10.35.20.8 <none> <none> kube-proxy-j4f62 1/1 Running 3 (51d ago) 84d 10.35.20.14 10.35.20.14 <none> <none> kube-proxy-r9qrr 1/1 Running 3 (51d ago) 84d 10.35.20.10 10.35.20.10 <none> <none> kube-proxy-smlwk 1/1 Running 3 (51d ago) 84d 10.35.20.13 10.35.20.13 <none> <none> kube-proxy-zg8tx 1/1 Running 3 (51d ago) 84d 10.35.20.9 10.35.20.9 <none> <none> kube-scheduler-10.35.20.8 1/1 Running 13 (51d ago) 84d 10.35.20.8 10.35.20.8 <none> <none> ``` 然后，在External集群中创建kubemark集群的kubeconfig文件。拷贝Kubemark集群的admin.conf文件到External集群的Master节点上，重命名为kubemark-kubeconfig。然后执行以下命令把kubeconfig创建在kubemark命名空间的kubeconfig这个secret中 ``` $ kubectl create ns kubemark $ kubectl create secret generic kubeconfig --type=Opaque --namespace=kubemark --from-file=kubelet.kubeconfig=kubemark-kubeconfig --from-file=kubeproxy.kubeconfig=kubemark-kubeconfig ``` 接着，我们创建如下的statefulset，向kubemark集群注册虚拟节点（我们使用statefulset，这里节点名字更有规律一点）： ``` apiVersion: apps/v1 kind: StatefulSet metadata: name: k8s-worker namespace: kubemark spec: replicas: 2 podManagementPolicy: Parallel serviceName: k8s-worker selector: matchLabels: name: k8s-worker template: metadata: labels: name: k8s-worker spec: initContainers: - name: init-inotify-limit image: harbor.ccse.io:8021/library/busybox:latest imagePullPolicy: IfNotPresent command: ['sysctl', '-w', 'fs.inotify.max_user_instances=200'] securityContext: privileged: true volumes: - name: kubeconfig-volume secret: secretName: kubeconfig - name: logs-volume hostPath: path: /var/log containers: - name: hollow-kubelet image: harbor.ccse.io:8021/kube-build/kubemark:v1.23.3 ports: - containerPort: 4194 - containerPort: 10250 - containerPort: 10255 env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: metadata.name command: - /kubemark args: - --morph=kubelet - --use-host-image-service=false - --name=$(NODE_NAME) - --kubeconfig=/kubeconfig/kubelet.kubeconfig - --alsologtostderr - --v=2 volumeMounts: - name: kubeconfig-volume mountPath: /kubeconfig readOnly: true - name: logs-volume mountPath: /var/log resources: requests: cpu: 20m memory: 50M securityContext: privileged: true - name: hollow-proxy image: harbor.ccse.io:8021/kube-build/kubemark:v1.23.3 env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: metadata.name command: - /kubemark args: - --morph=proxy - --name=$(NODE_NAME) - --use-real-proxier=false - --kubeconfig=/kubeconfig/kubeproxy.kubeconfig - --alsologtostderr - --v=2 volumeMounts: - name: kubeconfig-volume mountPath: /kubeconfig readOnly: true - name: logs-volume mountPath: /var/log resources: requests: cpu: 20m memory: 50M tolerations: - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists ``` 然后，我们在kubemark集群中就可以看到这两个虚拟节点了 ``` $ kubectl get node NAME STATUS ROLES AGE VERSION 10.35.20.2 Ready control-plane,master 84d v1.23.3 10.35.20.3 Ready control-plane,master 84d v1.23.3 10.35.20.4 Ready control-plane,master 84d v1.23.3 k8s-worker-0 Ready <none> 87m v1.23.3 k8s-worker-1 Ready <none> 87m v1.23.3 ``` ### **参考** * https://github.com/kubernetes/community/blob/452f681d92d98d6d62dfb24fbc9c8da10935632c/contributors/devel/sig-scalability/kubemark-setup-guide.md * https://segmentfault.com/a/1190000020516070 * https://docs.google.com/presentation/d/1AOfPEvAPcu31WxEZcMj8Kt5MlYXE-TRUrWvsFbkKp5g/htmlpresent * [使用kubemark进行大规模Kubernetes集群性能测试 | Coding](https://ihac.xyz/2018/08/05/%E5%A4%A7%E8%A7%84%E6%A8%A1Kubernetes%E9%9B%86%E7%BE%A4%E6%80%A7%E8%83%BD%E6%B5%8B%E8%AF%95/) * [community/contributors/devel/sig-scalability at master · kubernetes/community](https://github.com/kubernetes/community/tree/master/contributors/devel/sig-scalability) * [kubemark模拟k8s计算节点，测试k8s组件性能 - shininglight - 博客园](https://www.cnblogs.com/janeysj/p/12580454.html) * [kubemark + perf\_test(clusterloader) 本地性能测试的注意事项 | 演好自己的戏，走好自己的路](http://www.jungler.cn/2021/09/21/kubemark-perf-test-clusterloader-%E6%9C%AC%E5%9C%B0%E6%80%A7%E8%83%BD%E6%B5%8B%E8%AF%95%E7%9A%84%E6%B3%A8%E6%84%8F%E4%BA%8B%E9%A1%B9/)