TLS bootstrapping · kubernetes

[TOC] 在创建一个集群的时候，我们需要为每个kubelet配置签名证书，用来作为访问API的凭证。在刚开始创建集群的时候，我们可以使用脚本的方法批量地执行这个任务。但是当我们需要为集群添加新的节点的时候，集群的管理员不得不手动为新节点创建密钥对、然后进行签名，然后再配置好kubelet的参数，然后再启动这个kubelet。整个过程手动操作太多，过于复杂。那么有没有简化的方法呢？答案当然是有，k8s设计了TLS-bootstrapping这套机制，kubeadm使用这个机制大大简化了集群的安装以及扩容。我们知道每个kubelet都需要一个签名证书，TLS-bootstrapping的核心思想就是：由kubelet自已生成及向apiserver提交自已的证书签名请求文件（CSR），k8s-master对CSR签发后，kubelet再向apiserver获取自已的签名证书，然后再正常访问apiserver。基于此，我们来讲一讲kubelet启动的完整流程 1、kubelet开始启动 2、kubelet检查`kubeconfig`文件是否已配置好（apiserver地址、签名证书等），若否，则继续第3步 3、kubelet搜索是否存在`bootstrap-kubeconfig`文件 4、kubelet读取`bootstrap-kubeconfig`文件中的信息，包括apiserver的地址、以及token 5、kubelet携带token向apiserver发起API请求，在集群中为自已创建一个CSR对象 6、对CSR对象进行审批，可以使用kubectl命令手动批准或者由controller-manager的`csr-auto-approver`自动批准；审批通过的CSR的状态会修改为`approved` 7、controller-manager中的`signing-controller`会对状态为`approved`的CSR对象进行签名，然后将签名证书放在CSR对象的`status.certificate`字段中 8、kubelet携带token向apiserver发起请求，获取`CertificateSigningRequest`对象，从`status.certificate`字段得到签名证书 9、kubelet将私钥与签名证书等信息放入到`kubeconfig`文件中 10、kubelet启动成功 TLS-bootstrapping是从上面的第3步开始的。接下来，我们针对每一步做出详细的介绍。 ### bootstrap-kubeconfig文件当kubelet发现kubeconfig文件还没有配置好时，于是开始tls-bootstrapping的启动流程。kubelet去搜索`bootstrap-kubeconfig`文件（由kubelet的参数`--bootstrap-kubeconfig`指定），该文件的模板如下，它包含访问apiserver需要的三要素：apiserverUrl、CA证书和token都在文件中了。 ``` apiVersion: v1 kind: Config preferences: {} clusters: - name: kubernetes cluster: certificate-authority-data: {ca.crt} # ca.crt which is base64 encoded server: https://192.168.1.103:6443 contexts: - name: tls-bootstrap-token-user@kubernetes context: cluster: kubernetes user: tls-bootstrap-token-user current-context: tls-bootstrap-token-user@kubernetes users: - name: tls-bootstrap-token-user user: token: {token} ``` ### bootstrap-token的认证与授权接下来，kubelet携带bootstrap-token去向apiserver发起API请求，在集群中为自已创建一个CSR对象（`POST /apis/certificates.k8s.io/v1beta1/certificatesigningrequests`）。那么apiserver是如何对这个token进行认证的呢？ kubelet在创建CSR对象时，可以使用两种类型的token：`bootstrap-token`和`static-token`，我们这里只介绍`bootstrap-token`。`bootstrap-token`形如`fxj0d5.qhtvyaseds22o96y`，由两部分组成，前部分叫`token-id`，占6位，后部分叫`token-secret`，占16位；两部分都只能由小写字母或数字组成。apiserver要开启`bootstrap-token`认证器（通过设置启动参数`---enable-bootstrap-token-auth=true`），才能对`bootstrap-token`进行进行认证。当kubelet携带`bootstrap-token`（假设为`fxj0d5.qhtvyaseds22o96y`）发起API请求时，`bootstrap-token`认证器会在集群的`kube-system`命名空间下查找类型为`bootstrap.kubernetes.io/token`的Secret，如果找到一个如下的Secret，其`data.token-id`与`data.token-secret`与kubelet携带的`bootstrap-token`一样，则认证通过。`bootstrap-token`认证器会把token`fxj0d5.qhtvyaseds22o96y`认证为用户`system:bootstrap:fxj0d5`（`system:bootstrap:<token-id>`），该用户属于组`system:bootstrappers`。如果匹配`bootstrap-token`匹配的Secret的`data.auth-extra-groups`字段中还有值，则该用户同时属于`data.auth-extra-groups`中列出的组。比如token`fxj0d5.qhtvyaseds22o96y`匹配的如下的Secret的`data.auth-extra-groups`字段中有`system:bootstrappers:kubeadm:default-node-token`，那么认证出来的用户`system:bootstrap:fxj0d5`也属于组`system:bootstrappers:kubeadm:default-node-token`。 ``` apiVersion: v1 kind: Secret type: bootstrap.kubernetes.io/token metadata: name: bootstrap-token-fxj0d5 namespace: kube-system data: auth-extra-groups: base64encode(system:bootstrappers:kubeadm:default-node-token) token-id: base64encode(fxj0d5) token-secret: base64encode(qhtvyaseds22o96y) usage-bootstrap-authentication: base64encode(true) usage-bootstrap-signing: base64encode(true) ``` 现在，token`fxj0d5.qhtvyaseds22o96y`已经被认证通过且User为`system:bootstrap:fxj0d5`、Group为`system:bootstrappers`和`system:bootstrappers:kubeadm:default-node-token`。那么，如何对它们进行授权呢？其实如果细想一下，我们会发现，就算每个kubelet携带的`bootstrap-token`不一样，只要它们认证通过，那么它们认证出来的用户一定属于组`system:bootstrappers`。而每个kubelet都要能够创建CSR对象（其实还要获取CSR对象，后面讲），所以我们只需要给组`system:bootstrappers`授权，让它能够创建CSR对象。当我们使用RBAC授权模式时，我们在集群中创建如下的ClusterRole以及ClusterRoleBinding，授权`system:bootstrappers`组能够`create`、`get`、`list`、`watch` CSR对象。有了`create`权限，通过认证的kubelet就能够创建CSR对象了。 ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: create-get-csr rules: - apiGroups: - certificates.k8s.io resources: - certificatesigningrequests verbs: - create - get - list - watch ``` ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: approve-bootstrappers-create-get-csr roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kubelet-bootstrap subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:bootstrappers ``` ### CSR的批准与签名 OK，apiserver对kubelet创建CSR对象的请求进行了认证与授权，CSR对象创建成功。刚创建还未批准的CSR对象如下，介绍一下该对象中的数据来源 * `spec.request`：经过base64编码的`PKCS#10 CSR`数据（通俗理解为通过openssl生成的CSR文件的内容），它是kubelet发起API请求创建k8s中的CSR对象（`POST /apis/certificates.k8s.io/v1beta1/certificatesigningrequests`）时需要携带的参数；如果把它base64解码后再用openssl工具查看其中的内容，会发现两项重要的信息：`O=system:nodes`和`CN=system:node:<node-name>` * `spec.groups`与`spec.username`是apiserver加进去的，因为上面`bootstrap-token`认证器把`fxj0d5.qhtvyaseds22o96y`认证成了用户`system:bootstrap:fxj0d5`以及组`system:bootstrappers`和`system:bootstrappers:kubeadm:default-node-token`。 ``` apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: node-csr-u7CZYAhzUS3lpqJVeFlRh9jCA2weDATogQ0G-9itcsU spec: groups: - system:bootstrappers - system:bootstrappers:kubeadm:default-node-token - system:authenticated request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlIMU1JR2JBZ0VBTURreEZUQVRCZ05WQkFvVERITjVjM1JsYlRwdWIyUmxjekVnTUI0R0ExVUVBeE1YYzNsegpkR1Z0T201dlpHVTZjV2xoYjJwcGJtY3hNREl3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVQwCjhGeXhDWExmN0FKVE15bTNVbkp4dFdzSkJ2SjJHbXk0ZVFYTVRMYU9QUDRESmpQQXVTZ2o2ZU10d3JObWxLV2UKeklRQ05YeXNxY25pTDI0QVFnUTBvQUF3Q2dZSUtvWkl6ajBFQXdJRFNRQXdSZ0loQUltRzBNU1ZtSzdrRjh2cgpYQW1OendwREFkNjYvV3Vtekc5UzV4VXFFdjRnQWlFQXdyNTV5UnpOVkdOVVRkcTA3dHl4WGZpRC9ZZTdvWmFYCm94SnFXZzhFWnVzPQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K usages: - digital signature - key encipherment - client auth username: system:bootstrap:fxj0d5 status: {} ``` 此时CSR对象还没有被批准，所以status为空。接下来，就该对该CSR对象进行批准了。批准的方法有两种，一是通过kubectl命令手动批准，二是由controller-manager的`csr-auto-approver`控制器自动批准。手动批准的命令如下 ``` kubectl certificate approve node-csr-u7CZYAhzUS3lpqJVeFlRh9jCA2weDATogQ0G-9itcsU ``` 当然，为了更自动化的管理，我们可以利用控制器`csr-auto-approver`进行自动批准。如果要达到对某个CSR进行自动批准，必须告诉`csr-auto-approver`控制器该CSR对象是可以批准的。为此，我们需要在集群中创建如下的ClusterRole与ClusterRoleBinding，当`csr-auto-approver`发现CSR对象的`spec.groups`或`spec.username`可以`create` apiGroup`certificates.k8s.io`下的资源`certificatesigningrequests/nodeclient`时，便会批准该CSR对象。比如如下的ClusterRole与ClusterBinding刚好授权Group`system:bootstrappers`可以`create`资源`certificatesigningrequests/nodeclient`，而上面的CSR对象的`spec.groups`有`system:bootstrappers`字段。 ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: create-nodeclient rules: - apiGroups: - certificates.k8s.io resources: - certificatesigningrequests/nodeclient verbs: - create ``` ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: approve-bootrappers-create-nodeclient roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: create-nodeclient subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:bootstrappers ``` 当我们创建好上面的ClusterRole与ClusterRoleBinding后，前面的CSR对象就会被`csr-auto-approver`自动批准。批准后的CSR对象的`status`中就会多了`approved`的状态信息。当CSR被批准后，controller-manager的`csr-signing`控制器便会对CSR进行签名。签名所使用的CA套件由controller-manager的启动参数`--cluster-signing-cert-file`和`--cluster-signing-key-file`指定。CSR的签名证书有效期默认为1年，可以通过controller-manager的启动参数`--experimental-cluster-signing-duration`设置其他的值。我们来看一下签名之后的CSR对象是什么样的，如下。我们发现，该对象多了一个`status.certificate`字段，而该字段的值正是签名证书经过base64编码之后的内容。 ``` apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: node-csr-u7CZYAhzUS3lpqJVeFlRh9jCA2weDATogQ0G-9itcsU spec: groups: - system:bootstrappers - system:bootstrappers:kubeadm:default-node-token - system:authenticated request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlIMU1JR2JBZ0VBTURreEZUQVRCZ05WQkFvVERITjVjM1JsYlRwdWIyUmxjekVnTUI0R0ExVUVBeE1YYzNsegpkR1Z0T201dlpHVTZjV2xoYjJwcGJtY3hNREl3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVQwCjhGeXhDWExmN0FKVE15bTNVbkp4dFdzSkJ2SjJHbXk0ZVFYTVRMYU9QUDRESmpQQXVTZ2o2ZU10d3JObWxLV2UKeklRQ05YeXNxY25pTDI0QVFnUTBvQUF3Q2dZSUtvWkl6ajBFQXdJRFNRQXdSZ0loQUltRzBNU1ZtSzdrRjh2cgpYQW1OendwREFkNjYvV3Vtekc5UzV4VXFFdjRnQWlFQXdyNTV5UnpOVkdOVVRkcTA3dHl4WGZpRC9ZZTdvWmFYCm94SnFXZzhFWnVzPQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K usages: - digital signature - key encipherment - client auth username: system:bootstrap:fxj0d5 status: certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNaVENDQVUyZ0F3SUJBZ0lVYUgyWXordkcySFNQRjZBOTZqL2xWMm8wSWJvd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0ZURVRNQkVHQTFVRUF4TUthM1ZpWlhKdVpYUmxjekFlRncweE9ERXlNRFV4TWpNNU1EQmFGdzB4T0RFeQpNRFV4TWpRNU1EQmFNRGt4RlRBVEJnTlZCQW9UREhONWMzUmxiVHB1YjJSbGN6RWdNQjRHQTFVRUF4TVhjM2x6CmRHVnRPbTV2WkdVNmNXbGhiMnBwYm1jeE1ESXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBVDAKOEZ5eENYTGY3QUpUTXltM1VuSnh0V3NKQnZKMkdteTRlUVhNVExhT1BQNERKalBBdVNnajZlTXR3ck5tbEtXZQp6SVFDTlh5c3FjbmlMMjRBUWdRMG8xUXdVakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJCkt3WUJCUVVIQXdJd0RBWURWUjBUQVFIL0JBSXdBREFkQmdOVkhRNEVGZ1FVMGowbGppanhCTVBwSTN4bzQxSjgKWnRPN0FmSXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBTEV2M0o3REpGK0FFZnN6Tyt5OElBeDZTMW1uQXNYVwpSUVk5TnYxZW1qUVhIc2ZsSzM3bHFMMytyQjVZZHEvcjRjSlV5dmNRR043NzRtNmN3NFhmaUdBVTdrQk9mQnFaCi93T0lYL1VHZG1nNnA4ZDE0WndobW95SVFOdHdMZCtFYmRLdmtnZ0tMaGlYOXZLMUtaWDNNMUYraEl0cTJYckMKeGx2U1FBSVliN3ZCYWpER2trbEVqeGdjc2ROejVMK09xSzFyL2Z3MEZqUStRRHplWDl2RTJXUkZyS2h0dEt5NQpycGJibnBTWEh1d1NRSkpnZ2tvZUxTcWRFNFF1b2xQS0Q4ZlNudHEyNW82WlpDdm1Zcjl6Ni9PWVhPL2Vwc0tUClhyQXlWZFdZait1UjZRNCt0NVFrMUFwZGNRNCtqWmRwOVI1Z1g5eGtPN1FOYzlubmsyaENBV2M9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K conditions: - lastUpdateTime: 2018-12-05T12:43:53Z message: Auto approving kubelet client certificate after SubjectAccessReview. reason: AutoApproved type: Approved ``` ### kubelet获取签名证书当kubelet创建完CSR对象后，它便会不停地向apiserver发请求，查询CSR的状态。当CSR对象被签名之后，kubelet便会向apiserver请求签名后的CSR对象。此时kubelet使用的仍然是`bootstrap-token`，根据前面的介绍，该token有权限`create`、`get`、`list`、`watch` CSR资源，所以就能够拿到签名后的CSR对象。当kubelet拿到签名后的CSR对象后，便可以从中拿到签名证书，然后把签名证书放在启动参数`--cert-dir`指定的对应位置，并完善`--kubeconfig`文件。完善后的`--kubeconfig`文件的内容如下，发现访问apiserver的三要素都有了：apiserverUrl，ca.crt以及client-certificate。其中apiserverUrl与ca.crt都是从`--bootstrap-kubeconfig`中拿到的，client-certificate是从签名后的CSR对象拿到的 ``` apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNE1URXlNVEEzTkRBMU0xb1hEVEk0TVRFeE9EQTNOREExTTFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTVQxCmRMTm9TYWQwK1J3MjEzdWhiZjNXM0l5UE5hZUxmQ3dDWko3bnVNVGZlSzBqeGdveHNnczQybVF5MkNkbEdsdzQKVWJPejhTcFhlSW4yRC9MREFmeEtqZGc0cWh5YllISmJjK1ZNVGVRRXNQb2VBejdBVWJDQ2hqMmlWdjlHK3p2QgpHWmtobXIzOUI4NWxuSklJTSsvRlliazBTbjBNdVhPc3BGZVJLK3lqbklZeXpxSEtTWVk4aW5SSDhhaExlUlNNCkFCTUVtRndyME1EdWdCR2YrbzFlQU5hamI0QzZ2K3NXV3B6Z01EelhEV1VoSTRpNzRVWHE5KzZOTDlMUGVFOTcKYTY3Yit1SE1DSTl1emg5WjlPcTV3QkY1OS8ram1QT1BzY2lrTnVSVkNEZHZ6enF2VzdjRERLUU1YcXpCYUxKQgpVVENZcDl2aml6c3J1bkpFTDNFQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFGMThHR3JvQmRCNytxa0xvM1RHOEJqTDE2SmcKTm9tL1E0cWJGdHZsaTQwbjZ5Z001cUdqNmI3VTRVT2NaRjl4aDd6cGxWVXJkM0Z0OGhKYjhnOThRaFYxbDVHNwo4a1RQWThQcnAxN2UxWlRjOVdGMXUzZXM4Y2p6VmVFOFAxMEE5c3JqY3ZDNldONnhiK2FNc2E0aEFJbEJ3WjNwCmErSFhhTVhLbVRGaE9KZHJ5QTBNcFJLdTB2eEl5VlNVVU5BdUtYZlk4L3d0bk1VR0JLK0FJTDF3OTRUZ2wxb2sKQ1J1MGhkWEtmek0xWnU0d0hzQVQrcVhWcWR2cUg4UGFWaHc0U2NRcmRlL2o5anBFckwvMUJaSXlBU0J2MndSdwpBZ2VOc3pnZDlRaVRmTWh6QXBLU3gyN2VseHhYRStObGJKbmJBeFJXdkQ2WHZJTmlKUDZuUFQ2eUF3UT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= server: https://192.168.1.103:6443 name: default-cluster contexts: - context: cluster: default-cluster namespace: default user: default-auth name: default-context current-context: default-context kind: Config preferences: {} users: - name: default-auth user: client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem client-key: /var/lib/kubelet/pki/kubelet-client-current.pem ``` 我们去`--cert-dir`的默认位置`/var/lib/kubelet/pki/`下看一下内容，有如下文件 ``` -rw-------. 1 root root 1118 Dec 5 20:43 kubelet-client-2018-12-05-20-43-48.pem lrwxrwxrwx. 1 root root 59 Dec 5 20:43 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2018-12-05-20-43-48.pem -rw-r--r--. 1 root root 2185 Dec 5 19:43 kubelet.crt -rw-------. 1 root root 1675 Dec 5 19:43 kubelet.key ``` 我们发现，签名证书就是`kubelet-client-current.pem`，它是一个符号链接，真实文件为`kubelet-client-2018-12-05-20-43-48.pem`。我们还会发现该目录下有`kubelet.key`与`kubelet.crt`这两个文件，它们又是什么呢？它们其实和`kubelet-client-current.pem`一点关系都没有，是由kubelet自已创建一个私钥与自签名证书，目前不知道有什么用，猜测是来给kubelet启动https服务用的。 ### client证书的替换我们上面提到了，`csr-signing`控制器签署的证书的有效期默认为1年。当签名证书快要到期的时候（一般证书有效期过了百分之九十的时候），kubelet就会向apiserver发起请求，请求置换证书。kubelet请求置换证书的流程是 1、kubelet发起API请求，在集群中创建一个新的CSR对象 2、手动或自动对CSR进行批准 3、`csr-signing`控制器发现CSR的状态为`approved`后，便对其进行签名 4、kubelet检测到CSR已经签名后，下载下来，置换节点上的相关配置第1步中，kubelet发起API请求时，携带的身份凭证是快要过期的签名证书，所以走的认证为client证书认证（将会被认证为用户`system:node:<node name>`以及组`system:nodes`），授权为`Node Authorization`。创建出来的新的CSR对象如下：`spec.request`仍然是kubelet发起请求时需要携带的信息，里面包含了`O=system:nodes`与`CN=system:node:<node-name>`（这里为`CN=system:node:qiaojing102`）；`spec.groups`变成了`system:nodes`，`spec.username`变成了`system:node:qiaojing102`。 ``` apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: csr-6g4v4 spec: groups: - system:nodes - system:authenticated request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlIME1JR2JBZ0VBTURreEZUQVRCZ05WQkFvVERITjVjM1JsYlRwdWIyUmxjekVnTUI0R0ExVUVBeE1YYzNsegpkR1Z0T201dlpHVTZjV2xoYjJwcGJtY3hNREl3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVJLCjdldENVcTkrZStzdW95Z1JBV2I0dkJZdWRRSmRpeHREQitSK0lXeG1PSEZnYkYwdDUzSVJ5Z29mQVlXS3Y0ZkQKeHBHOHg2a2hXM0dNRnlSWjZRclBvQUF3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnSko2a3d4MEJ2bmVldXpsdgpheUZrVVFWVEUrMEtBa3pjamFOQ0dLeTZUVGdDSVFEYWNxNzRrQjNlZnZLR1BzYlJJYUtxNjlLQk5mMS9memVPClhWbHl1TG1uVWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K usages: - digital signature - key encipherment - client auth username: system:node:qiaojing102 status: {} ``` CSR对象创建成功之后，就需要对它进行批准。同样，可以通过kubectl命令手动进行批准。我们现在来讨论一下如何由controller-manager的`csr-auto-approver`进行自动批准。在前面我们介绍了，只要CSR中的`spec.groups`或`spec.username`有权限`create`资源`certificatesigningrequests/nodeclient`，那么该CSR就会被自动批准。那么这个新的CSR是否也一样呢？答案为否。在这里，我们要介绍一下k8s中CSR的类型 * `nodeclient`：kubelet第一次申请client签名证书时创建的CSR * `selfnodeclient`：kubelet置换client签名证书时创建的CSR * `selfnodeserver`：（这种类型先不做介绍）我们知道，kubelet第一次申请client签名证书时，其创建的CSR中`spec.username`为`system:bootstrap:<token id>`，`spec.groups`为`system:bootstrappers`等。而`spec.request`中的内容包含的内容为`O=system:nodes`、`CN=system:node:<node-name>`。我们发现，该种类型的CSR的`spec.request`中的`O`的值没有包含在`spec.groups`中，`spec.request`中的`CN`的值不等于`spec.username`。 kubelet置换client签名证书时，其创建的CSR中`spec.username`为`system:node:<node-name>`，`spec.groups`为`system:nodes`。而`spec.request`中的内容包含的内容为`O=system:nodes`、`CN=system:node:<node-name>`。我们发现，该种类型的CSR的`spec.request`中的`O`的值包含在`spec.groups`中，`spec.request`中的`CN`的值刚好等于`spec.username`。 `csr-auto-approver`就是根据上面的原理来判断CSR是属于`nodeclient`类型还是`selfnodeclient`类型的。那么对于这两种类型的CSR的自动批准，需要不同的ClusterRole与ClusterRoleBinding。前面我们已经介绍了对于`nodeclient`类型的CSR，它的`spec.username`或`spec.groups`需要有权限`create`资源`certificatesigningrequests/nodeclient`，才能被自动批准。而对于`selfnodeclient`类型的CSR，它的`spec.username`或`spec.groups`需要有权限`create`资源`certificatesigningrequests/selfnodeclient`，才能被自动批准。于是，我们需要在集群中创建如下的ClusterRole与ClusterRoleBinding ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: create-selfnodeclient rules: - apiGroups: - certificates.k8s.io resources: - certificatesigningrequests/nodeclient verbs: - create ``` ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: approve-nodes-create-nodeclient roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: create-selfnodeclient subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:nodes ``` 当我们创建了如上的ClusterRole与ClusterRoleBinding之后，`csr-auto-approver`就会自动批准`selfnodeclient`类型的CSR。批准之后，`csr-signing`控制器就可以对CSR进行签名了。签名之后的CSR如下，签名证书已经放在了字段`status.certificate`中 ``` apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: csr-6g4v4 spec: groups: - system:nodes - system:authenticated request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlIME1JR2JBZ0VBTURreEZUQVRCZ05WQkFvVERITjVjM1JsYlRwdWIyUmxjekVnTUI0R0ExVUVBeE1YYzNsegpkR1Z0T201dlpHVTZjV2xoYjJwcGJtY3hNREl3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVJLCjdldENVcTkrZStzdW95Z1JBV2I0dkJZdWRRSmRpeHREQitSK0lXeG1PSEZnYkYwdDUzSVJ5Z29mQVlXS3Y0ZkQKeHBHOHg2a2hXM0dNRnlSWjZRclBvQUF3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnSko2a3d4MEJ2bmVldXpsdgpheUZrVVFWVEUrMEtBa3pjamFOQ0dLeTZUVGdDSVFEYWNxNzRrQjNlZnZLR1BzYlJJYUtxNjlLQk5mMS9memVPClhWbHl1TG1uVWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K usages: - digital signature - key encipherment - client auth username: system:node:qiaojing102 status: certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNaVENDQVUyZ0F3SUJBZ0lVSjRIZTVEZ1h2dVhaSUJSQ1J4cDVzQ3ZuRWZVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0ZURVRNQkVHQTFVRUF4TUthM1ZpWlhKdVpYUmxjekFlRncweE9ERXlNRFl3TXpBeE1EQmFGdzB4T0RFeQpNRFl3TXpFeE1EQmFNRGt4RlRBVEJnTlZCQW9UREhONWMzUmxiVHB1YjJSbGN6RWdNQjRHQTFVRUF4TVhjM2x6CmRHVnRPbTV2WkdVNmNXbGhiMnBwYm1jeE1ESXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUksKN2V0Q1VxOStlK3N1b3lnUkFXYjR2Qll1ZFFKZGl4dERCK1IrSVd4bU9IRmdiRjB0NTNJUnlnb2ZBWVdLdjRmRAp4cEc4eDZraFczR01GeVJaNlFyUG8xUXdVakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJCkt3WUJCUVVIQXdJd0RBWURWUjBUQVFIL0JBSXdBREFkQmdOVkhRNEVGZ1FVQVppSmZOWC9BYks2bUhFbTlROEwKdkx4LzZUUXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBRUVmYWpUejZLNlBXL1JlZnJGSENFa3lRMXV3WDdsWgpTOHBYb0pqZVpJcDNXcElNVGczMXRCVnJ3QlR6c0VNckRNZXdpTjNBRGZDbGdoY1Y2cGUxYWZWQ2FlWmZFUUxZCjRRR3pZTW1JK3N0eEpueWpWNTJERWhZcDVnNnFha3ZMdnpmUC9NZ2gwcFNROFF6c2ErL1VQZmgzaE8vUUVBNnYKY2tlS0tOYWczMnJhZjViOWhDWjUxT09iU3UwRkN4ZU1MRWZudk5nbnNsVk9KYXZpZGRFWG9aV0htNW5OaTV3NQpYb1d3SDFwUUUvbDJlNW0wOWxieko5eE9OdThxbHc1ZlN2am11dE14ZjVZSm1oKzZrYmwyYmdQMlg1MHdBcVdPCjBuTURyUFE3ZTUybVRodkNUTnlxeGNGcEo5bnVnUFdJS2E1UXhjUS91TG1FaERUQVFQeStKREE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K conditions: - lastUpdateTime: 2018-12-06T03:05:35Z message: Auto approving self kubelet client certificate after SubjectAccessReview. reason: AutoApproved type: Approved ``` 然后，kubelet就可以下载CSR，从中获取到证书，更新自已主机上的证书了。 ### Reference [1] https://mritd.me/2018/01/07/kubernetes-tls-bootstrapping-note/