本文翻译自nfs-client-provisioner的[说明文档](https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client)，本文将介绍使用nfs-client-provisioner这个应用，利用NFS Server给Kubernetes作为持久存储的后端，并且动态提供PV。前提条件是有已经安装好的NFS服务器，并且NFS服务器与Kubernetes的Slave节点都能网络连通。 所有下文用到的文件来自于`git clone https://github.com/kubernetes-incubator/external-storage.git`的nfs-client目录。
### nfs-client-provisioner
nfs-client-provisioner 是一个Kubernetes的简易NFS的外部provisioner，本身不提供NFS，需要现有的NFS服务器提供存储
* PV以 `${namespace}-${pvcName}-${pvName}`的命名格式提供（在NFS服务器上）
* PV回收的时候以 `archieved-${namespace}-${pvcName}-${pvName}` 的命名格式（在NFS服务器上）
----------------------
### 安装部署
修改deployment文件并部署 `deploy/deployment.yaml`
需要修改的地方只有NFS服务器所在的IP地址（10.10.10.60），以及NFS服务器共享的路径（`/ifs/kubernetes`），两处都需要修改为你实际的NFS服务器和共享目录
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 10.10.10.60
- name: NFS_PATH
value: /ifs/kubernetes
volumes:
- name: nfs-client-root
nfs:
server: 10.10.10.60
path: /ifs/kubernetes
修改StorageClass文件并部署 `deploy/class.yaml`
此处可以不修改，或者修改provisioner的名字，需要与上面的deployment的PROVISIONER\_NAME名字一致。
~~~
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-nfs-storage
provisioner: fuseim.pri/ifs
~~~
### 如果启用了RBAC
需要执行如下的命令来授权。
nfs-rbac.yaml 权限管理
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
namespace: default
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
授权
~~~
$ kubectl create -f deploy/auth/serviceaccount.yaml
serviceaccount "nfs-client-provisioner" created
$ kubectl create -f deploy/auth/clusterrole.yaml
clusterrole "nfs-client-provisioner-runner" created
$ kubectl create -f deploy/auth/clusterrolebinding.yaml
clusterrolebinding "run-nfs-client-provisioner" created
$ kubectl patch deployment nfs-client-provisioner -p '{"spec":{"template":{"spec":{"serviceAccount":"nfs-client-provisioner"}}}}'
~~~
如果您的集群启用了RBAC，或者您正在运行OpenShift，则必须授权provisioner。 如果你在非默认的“default”名称空间/项目之外部署，可以编辑`deploy/auth/clusterrolebinding.yaml`或编辑`oadm policy“指令。
----
### 问题描述:
kubernetes挂载nfs发现发现pod一直处于containercreating状态，describe pod 发现报错如下：
[root@master nginx-ip\]# kubectl describe pod nginx-r67f7
Name: nginx-r67f7
Namespace: default
Node: master/10.168.xx.xx
Start Time: Sun, 10 Jun 2018 09:47:50 +0800
Labels: name=nginx
Annotations:
Status: Pending
IP:
Controlled By: ReplicationController/nginx
Containers:
nginx:
Container ID:
Image: reg.xx.xx/library/nginx-ip:0.4
Image ID:
Port: 80/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
Mounts:
/usr/share/nginx/html from httpd-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-drh57 (ro)
Conditions:
Type Status
Initialized True
Ready False
Volumes:
httpd-storage:
Type: NFS (an NFS mount that lasts the lifetime of a pod)
Server: 10.168.xx.xx
Path: /data/web
ReadOnly: false
default-token-drh57:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-drh57
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
—- —— —- —- ——-
Warning FailedMount 18s kubelet, master MountVolume.SetUp failed for volume "httpd-storage" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: –description=Kubernetes transient mount for /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage –scope — mount -t nfs 10.168.xx.xx:/data/web /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage
Output: Running scope as unit run-34311.scope.
mount: wrong fs type, bad option, bad superblock on 10.168.xx.xx:/data/web,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount. helper program)
In some cases useful info is found in syslog – try
dmesg | tail or so.
Normal SuccessfulMountVolume 18s kubelet, master MountVolume.SetUp succeeded for volume "default-token-drh57"
Warning FailedMount 17s kubelet, master MountVolume.SetUp failed for volume "httpd-storage" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: –description=Kubernetes transient mount for /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage –scope — mount -t nfs 10.168.xx.xx:/data/web /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage
Output: Running scope as unit run-34316.scope.
mount: wrong fs type, bad option, bad superblock on 10.168.xx.xx:/data/web,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount. helper program)
In some cases useful info is found in syslog – try
dmesg | tail or so.
Warning FailedMount 16s kubelet, master MountVolume.SetUp failed for volume "httpd-storage" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: –description=Kubernetes transient mount for /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage –scope — mount -t nfs 10.168.xx.xx:/data/web /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage
Output: Running scope as unit run-34325.scope.
mount: wrong fs type, bad option, bad superblock on 10.168.xx.xx:/data/web,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount. helper program)
In some cases useful info is found in syslog – try
dmesg | tail or so.
Warning FailedMount 14s kubelet, master MountVolume.SetUp failed for volume "httpd-storage" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: –description=Kubernetes transient mount for /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage –scope — mount -t nfs 10.168.xx.xx:/data/web /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage
Output: Running scope as unit run-34341.scope.
mount: wrong fs type, bad option, bad superblock on 10.168.xx.xx:/data/web,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount. helper program)
In some cases useful info is found in syslog – try
dmesg | tail or so.
Warning FailedMount 10s kubelet, master MountVolume.SetUp failed for volume "httpd-storage" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: –description=Kubernetes transient mount for /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage –scope — mount -t nfs 10.168.xx.xx:/data/web /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage
Output: Running scope as unit run-34364.scope.
mount: wrong fs type, bad option, bad superblock on 10.168.xx.xx:/data/web,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount. helper program)
In some cases useful info is found in syslog – try
dmesg | tail or so.
Warning FailedMount 2s kubelet, master MountVolume.SetUp failed for volume "httpd-storage" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: –description=Kubernetes transient mount for /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage –scope — mount -t nfs 10.168.xx.xx:/data/web /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage
Output: Running scope as unit run-34394.scope.
mount: wrong fs type, bad option, bad superblock on 10.168.xx.xx:/data/web,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount. helper program)
In some cases useful info is found in syslog – try
dmesg | tail or so.
看这个报错想是容器没有安装nfs-common nfs-utils软件包安装导致，容器安装完这两个包发现，还是报这个错误，经过不懈的google，终于发现需要将这两个软件包安装到宿主机上即可。
yum install nfs-common nfs-utils -y
- 方案设计
- 使用kubespray部署k8s集群
- 使用ingress 访问dashboard v1.8.3
- 在 Kubernetes 中配置私有 DNS 和上游域名服务器(coredns forward)
- master 节点添加taint
- limitrang
- 配置私有仓库
- 利用NFS动态提供Kubernetes后端存储卷
- ingress 添加https 客户端双向认证
- 应用日采集
- eureka 迁移 k8s 集群
- 镜像下载地址
- helm install
- Install istio
- zookeeper 集群部署
- 基于prometheus自定义rabbitmq—exporter指标HPA弹性伸缩
- nacos huaweicloud