[TOC]
kubelet可以对磁盘进行管控，但是只能对nodefs与imagefs这两个分区进行管控。其中
* imagefs: docker安装目录所在的分区
* nodefs: kubelet的启动参数--root-dir所指定的目录（默认/var/lib/kubelet）所在的分区
接下来，我们来验证一下我们对imagefs与nodefs的理解。
### **前置条件**
k8s集群使用1.8.6版本
```
$ kubectl get node
NAME STATUS ROLES AGE VERSION
10.142.232.161 Ready <none> 263d v1.8.6
10.142.232.162 NotReady <none> 263d v1.8.6
10.142.232.163 Ready,SchedulingDisabled <none> 227d v1.8.6
```
10.142.232.161上docker安装在/app/docker目录下，kubelet的--root-dir没有设置，使用默认的/var/lib/kubelet。/app是一块盘，使用率为70%；/是一块盘，使用率为57%；而imagesfs与nodefs此时设置的阈值都为80%，如下：
```
$ df -hT
文件系统 类型 容量 已用 可用 已用% 挂载点
devtmpfs devtmpfs 16G 0 16G 0% /dev
tmpfs tmpfs 16G 0 16G 0% /dev/shm
tmpfs tmpfs 16G 1.7G 15G 11% /run
tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 45G 26G 20G 57% /
/dev/xvda1 xfs 497M 254M 243M 52% /boot
/dev/xvde xfs 150G 105G 46G 70% /app
$ ps -ef | grep kubelet
root 125179 1 37 17:50 ? 00:00:01 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<20%,imagefs.available<20% --network-plugin=cni
```
此时，10.142.232.161该node没有报磁盘的错
```
$ kubectl describe node 10.142.232.161
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 18s kubelet, 10.142.232.161 Starting kubelet.
Normal NodeAllocatableEnforced 18s kubelet, 10.142.232.161 Updated Node Allocatable limit across pods
Normal NodeHasSufficientDisk 18s kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 18s kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 18s kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasNoDiskPressure
Normal NodeNotReady 18s kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeNotReady
Normal NodeReady 8s kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady
```
### **验证方案**
* 验证imagefs是/app/docker目录所在分区（/app分区使用率为70%）
* 修改imagefs的阈值为60%，node应该报imagefs超标
* 修改imagefs的阈值为80%，node应该正常
* 验证nodefs是/var/lib/kubelet目录所在的分区（/分区使用率为57%）
* 修改nodefs的阈值为50%，node应该报nodefs超标
* 修改nodefs的阈值为60%，node应该正常
* 修改kubelet启动参数--root-dir，将值设成/app/kubelet
* 修改让imagefs的阈值为80%，nodefs的阈值为60%；此时应该报nodefs超标
* 修改让imagefs的阈值为60%，nodefs的阈值为80%；此时应该报imagefs超标
* 修改让imagefs的阈值为60%，nodefs的阈值为60%；此时应该报两个都超标
* 修改让imagefs的阈值为80%，nodefs的阈值为80%；此时node应该正常
### **验证步骤**
一、验证imagefs是/app/docker目录所在分区
1.1 修改imagefs的阈值为60%，node应该imagefs超标
如下，我们把imagefs的阈值设为60%
```
$ ps -ef | grep kubelet
root 41234 1 72 18:17 ? 00:00:02 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<20%,imagefs.available<40% --network-plugin=cni
```
然后我们查看节点的状态，Attempting to reclaim imagefs，意思为尝试回收imagefs
```
$ kubectl describe node 10.142.232.161
...
Normal NodeAllocatableEnforced 1m kubelet, 10.142.232.161 Updated Node Allocatable limit across pods
Normal Starting 1m kubelet, 10.142.232.161 Starting kubelet.
Normal NodeHasSufficientDisk 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasNoDiskPressure
Normal NodeNotReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeNotReady
Normal NodeHasDiskPressure 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasDiskPressure
Normal NodeReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady
Warning EvictionThresholdMet 18s (x4 over 1m) kubelet, 10.142.232.161 Attempting to reclaim imagefs
```
1.2 修改imagefs的阈值为80%，node应该正常
我们把imagefs的阈值为80%
```
$ ps -ef | grep kubelet
root 51402 1 19 18:24 ? 00:00:06 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<20%,imagefs.available<20% --network-plugin=cni
```
然后再来查看node的状态，NodeHasNoDiskPressure，说明imagefs使用率没有超过阈值了
```
$ kubectl describe node 10.142.232.161
...
Warning EvictionThresholdMet 6m (x22 over 11m) kubelet, 10.142.232.161 Attempting to reclaim imagefs
Normal Starting 5m kubelet, 10.142.232.161 Starting kubelet.
Normal NodeAllocatableEnforced 5m kubelet, 10.142.232.161 Updated Node Allocatable limit across pods
Normal NodeHasSufficientDisk 5m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 5m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 5m (x2 over 5m) kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasNoDiskPressure
Normal NodeNotReady 5m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeNotReady
Normal NodeReady 4m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady
```
二、验证nodefs是/var/lib/kubelet目录所在的分区（/分区使用率为57%）
2.1 修改nodefs的阈值为50%，node应该报nodefs超标
修改nodefs的阈值为50%
```
$ ps -ef | grep kubelet
root 72575 1 59 18:35 ? 00:00:04 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<50%,imagefs.available<20% --network-plugin=cni
```
查看node的状态，报Attempting to reclaim nodefs，意思是尝试回收nodefs，也就是nodefs超标了
```
$ kubectl describe node 10.142.232.161
...
Normal Starting 1m kubelet, 10.142.232.161 Starting kubelet.
Normal NodeAllocatableEnforced 1m kubelet, 10.142.232.161 Updated Node Allocatable limit across pods
Normal NodeHasSufficientDisk 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasNoDiskPressure
Normal NodeNotReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeNotReady
Normal NodeHasDiskPressure 53s kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasDiskPressure
Normal NodeReady 53s kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady
Warning EvictionThresholdMet 2s (x5 over 1m) kubelet, 10.142.232.161 Attempting to reclaim nodefs
```
2.2 修改nodefs的阈值为60%，node应该正常
修改nodefs的阈值为60%
```
$ ps -ef | grep kubelet
root 78664 1 31 18:38 ? 00:00:02 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<40%,imagefs.available<20% --network-plugin=cni
```
此时查看node的状态，已正常
```
$ kubectl describe node 10.142.232.161
...
Normal Starting 2m kubelet, 10.142.232.161 Starting kubelet.
Normal NodeReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady
```
三、修改kubelet启动参数--root-dir，将值设成/app/kubelet
以下几个参数的默认值都与/var/lib/kubelet有关
```
--root-dir # 默认值为 /var/lib/kubelet
--seccomp-profile-root # 默认值为 /var/lib/kubelet/seccomp
--cert-dir # 默认值为 /var/lib/kubelet/pki
--kubeconfig # 默认值为 /var/lib/kubelet/kubeconfig
```
为了能够不再使用/var/lib/kubelet这个目录，我们需要对这四个参数显示设置。设置如下：
```
--root-dir=/app/kubelet
--seccomp-profile-root=/app/kubelet/seccomp
--cert-dir=/app/kubelet/pki
--kubeconfig=/etc/kubernetes/kubeconfig
```
3.1 修改让imagefs的阈值为80%，nodefs的阈值为60%；此时应该报nodefs超标
```
$ ps -ef | grep kubelet
root 14423 1 10 19:28 ? 00:00:34 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<40%,imagefs.available<20% --root-dir=/app/kubelet --seccomp-profile-root=/app/kubelet/seccomp --cert-dir=/app/kubelet/pki --network-plugin=cni
```
查看节点的状态，只报Attempting to reclaim nodefs，也就是说nodefs超标
```
$ kubectl describe node 10.142.232.161
...
Normal NodeHasDiskPressure 3m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasDiskPressure
Normal NodeReady 3m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady
Normal Starting 3m kube-proxy, 10.142.232.161 Starting kube-proxy.
Warning EvictionThresholdMet 27s (x15 over 3m) kubelet, 10.142.232.161 Attempting to reclaim nodefs
```
3.2 修改让imagefs的阈值为60%，nodefs的阈值为80%；此时应该报imagefs超标
```
$ ps -ef |grep kubelet
root 21381 1 30 19:36 ? 00:00:02 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<20%,imagefs.available<40% --root-dir=/app/kubelet --seccomp-profile-root=/app/kubelet/seccomp --cert-dir=/app/kubelet/pki --network-plugin=cni
```
我们查看node的状态，只报imagefs超标
```
$ kubectl describe node 10.142.232.161
...
Normal Starting 1m kubelet, 10.142.232.161 Starting kubelet.
Normal NodeAllocatableEnforced 1m kubelet, 10.142.232.161 Updated Node Allocatable limit across pods
Normal NodeHasSufficientDisk 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientDisk
Normal NodeNotReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeNotReady
Normal NodeHasNoDiskPressure 1m (x2 over 1m) kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientMemory 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientMemory
Normal NodeReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady
Normal NodeHasDiskPressure 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasDiskPressure
Warning EvictionThresholdMet 11s (x5 over 1m) kubelet, 10.142.232.161 Attempting to reclaim imagefs
```
3.3 修改让imagefs的阈值为60%，nodefs的阈值为60%；此时应该报两个都超标
```
$ ps -ef | grep kubelet
root 24524 1 33 19:39 ? 00:00:01 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<40%,imagefs.available<40% --root-dir=/app/kubelet --seccomp-profile-root=/app/kubelet/seccomp --cert-dir=/app/kubelet/pki --network-plugin=cni
```
我们查看node的状态，果然imagefs与nodefs都超标了
```
$ kubectl describe node 10.142.232.161
...
Normal Starting 1m kubelet, 10.142.232.161 Starting kubelet.
Normal NodeAllocatableEnforced 1m kubelet, 10.142.232.161 Updated Node Allocatable limit across pods
Normal NodeHasSufficientDisk 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 1m (x2 over 1m) kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasNoDiskPressure
Normal NodeNotReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeNotReady
Normal NodeHasDiskPressure 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasDiskPressure
Normal NodeReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady
Warning EvictionThresholdMet 14s kubelet, 10.142.232.161 Attempting to reclaim imagefs
Warning EvictionThresholdMet 4s (x8 over 1m) kubelet, 10.142.232.161 Attempting to reclaim nodefs
```
3.4 修改让imagefs的阈值为80%，nodefs的阈值为80%；此时node应该正常
```
$ ps -ef | grep kubelet
root 27869 1 30 19:43 ? 00:00:01 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<20%,imagefs.available<20% --root-dir=/app/kubelet --seccomp-profile-root=/app/kubelet/seccomp --cert-dir=/app/kubelet/pki --network-plugin=cni
```
我们查看node的状态，果然没有报imagefs与nodefs的错了
```
$ kubectl decribe node 10.142.232.161
...
Normal Starting 1m kubelet, 10.142.232.161 Starting kubelet.
Normal NodeHasSufficientDisk 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientMemory
Normal NodeNotReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeNotReady
Normal NodeAllocatableEnforced 1m kubelet, 10.142.232.161 Updated Node Allocatable limit across pods
Normal NodeReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady
```
### **总结**
1、nodefs是--root-dir目录所在分区，imagefs是docker安装目录所在的分区
2、建议nodefs与imagefs共用一个分区，但是这个分区要设置的大一些。
3、当nodefs与imagefs共用一个分区时，kubelet中的其他几个参数--root-dir、--cert-dir
- 常用命令
- 安装
- 安装Kubeadm
- 安装单Master集群
- 安装高可用集群(手动分发证书)
- 安装高可用集群(自动分发证书)
- 启动参数解析
- certificate-key
- ETCD相关参数
- Kubernetes端口汇总
- 安装IPv4-IPv6双栈集群
- 下载二进制文件
- 使用Kata容器
- 快速安装shell脚本
- 存储
- 实践
- Ceph-RBD实践
- CephFS实践
- 对象存储
- 阿里云CSI
- CSI
- 安全
- 认证与授权
- 认证
- 认证-实践
- 授权
- ServiceAccount
- NodeAuthorizor
- TLS bootstrapping
- Kubelet的认证
- 准入控制
- 准入控制示例
- Pod安全上下文
- Selinux-Seccomp-Capabilities
- 给容器配置安全上下文
- PodSecurityPolicy
- K8S-1.8手动开启认证与授权
- Helm
- Helm命令
- Chart
- 快速入门
- 内置对象
- 模板函数与管道
- 模板函数列表
- 流程控制
- Chart依赖
- Repository
- 开源的Chart包
- CRD
- CRD入门
- 工作负载
- Pod
- Pod的重启策略
- Container
- 探针
- 工作负载的状态
- 有状态服务
- 网络插件
- Multus
- Calico+Flannel
- 容器网络限速
- 自研网络插件
- 设计文档
- Cilium
- 安装Cilium
- Calico
- Calico-FAQ
- IPAM
- Whereabouts
- 控制平面与Pod网络分开
- 重新编译
- 编译kubeadm
- 编译kubeadm-1.23
- 资源预留
- 资源预留简介
- imagefs与nodefs
- 资源预留 vs 驱逐 vs OOM
- 负载均衡
- 灰度与蓝绿
- Ingress的TLS
- 多个NginxIngressController实例
- Service的会话亲和
- CNI实践
- CNI规范
- 使用cnitool模拟调用
- CNI快速入门
- 性能测试
- 性能测试简介
- 制作kubemark镜像
- 使用clusterloader2进行性能测试
- 编译clusterloader2二进制文件
- 搭建性能测试环境
- 运行density测试
- 运行load测试
- 参数调优
- Measurement
- TestMetrics
- EtcdMetrics
- SLOMeasurement
- PrometheusMeasurement
- APIResponsivenessPrometheus
- PodStartupLatency
- FAQ
- 调度
- 亲和性与反亲和性
- GPU
- HPA
- 命名规范
- 可信云认证
- 磁盘限速
- Virtual-kubelet
- VK思路整理
- Kubebuilder
- FAQ
- 阿里云日志服务SLS