ThinkChat2.0新版上线,更智能更精彩,支持会话、画图、阅读、搜索等,送10W Token,即刻开启你的AI之旅 广告
[TOC] 在创建一个集群的时候,我们需要为每个kubelet配置签名证书,用来作为访问API的凭证。在刚开始创建集群的时候,我们可以使用脚本的方法批量地执行这个任务。但是当我们需要为集群添加新的节点的时候,集群的管理员不得不手动为新节点创建密钥对、然后进行签名,然后再配置好kubelet的参数,然后再启动这个kubelet。整个过程手动操作太多,过于复杂。那么有没有简化的方法呢?答案当然是有,k8s设计了TLS-bootstrapping这套机制,kubeadm使用这个机制大大简化了集群的安装以及扩容。 我们知道每个kubelet都需要一个签名证书,TLS-bootstrapping的核心思想就是:由kubelet自已生成及向apiserver提交自已的证书签名请求文件(CSR),k8s-master对CSR签发后,kubelet再向apiserver获取自已的签名证书,然后再正常访问apiserver。 基于此,我们来讲一讲kubelet启动的完整流程 1、kubelet开始启动 2、kubelet检查`kubeconfig`文件是否已配置好(apiserver地址、签名证书等),若否,则继续第3步 3、kubelet搜索是否存在`bootstrap-kubeconfig`文件 4、kubelet读取`bootstrap-kubeconfig`文件中的信息,包括apiserver的地址、以及token 5、kubelet携带token向apiserver发起API请求,在集群中为自已创建一个CSR对象 6、对CSR对象进行审批,可以使用kubectl命令手动批准或者由controller-manager的`csr-auto-approver`自动批准;审批通过的CSR的状态会修改为`approved` 7、controller-manager中的`signing-controller`会对状态为`approved`的CSR对象进行签名,然后将签名证书放在CSR对象的`status.certificate`字段中 8、kubelet携带token向apiserver发起请求,获取`CertificateSigningRequest`对象,从`status.certificate`字段得到签名证书 9、kubelet将私钥与签名证书等信息放入到`kubeconfig`文件中 10、kubelet启动成功 TLS-bootstrapping是从上面的第3步开始的。接下来,我们针对每一步做出详细的介绍。 ### bootstrap-kubeconfig文件 当kubelet发现kubeconfig文件还没有配置好时,于是开始tls-bootstrapping的启动流程。kubelet去搜索`bootstrap-kubeconfig`文件(由kubelet的参数`--bootstrap-kubeconfig`指定),该文件的模板如下,它包含访问apiserver需要的三要素:apiserverUrl、CA证书和token都在文件中了。 ``` apiVersion: v1 kind: Config preferences: {} clusters: - name: kubernetes cluster: certificate-authority-data: {ca.crt} # ca.crt which is base64 encoded server: https://192.168.1.103:6443 contexts: - name: tls-bootstrap-token-user@kubernetes context: cluster: kubernetes user: tls-bootstrap-token-user current-context: tls-bootstrap-token-user@kubernetes users: - name: tls-bootstrap-token-user user: token: {token} ``` ### bootstrap-token的认证与授权 接下来,kubelet携带bootstrap-token去向apiserver发起API请求,在集群中为自已创建一个CSR对象(`POST /apis/certificates.k8s.io/v1beta1/certificatesigningrequests`)。那么apiserver是如何对这个token进行认证的呢? kubelet在创建CSR对象时,可以使用两种类型的token:`bootstrap-token`和`static-token`,我们这里只介绍`bootstrap-token`。`bootstrap-token`形如`fxj0d5.qhtvyaseds22o96y`,由两部分组成,前部分叫`token-id`,占6位,后部分叫`token-secret`,占16位;两部分都只能由小写字母或数字组成。apiserver要开启`bootstrap-token`认证器(通过设置启动参数`---enable-bootstrap-token-auth=true`),才能对`bootstrap-token`进行进行认证。 当kubelet携带`bootstrap-token`(假设为`fxj0d5.qhtvyaseds22o96y`)发起API请求时,`bootstrap-token`认证器会在集群的`kube-system`命名空间下查找类型为`bootstrap.kubernetes.io/token`的Secret,如果找到一个如下的Secret,其`data.token-id`与`data.token-secret`与kubelet携带的`bootstrap-token`一样,则认证通过。`bootstrap-token`认证器会把token`fxj0d5.qhtvyaseds22o96y`认证为用户`system:bootstrap:fxj0d5`(`system:bootstrap:<token-id>`),该用户属于组`system:bootstrappers`。如果匹配`bootstrap-token`匹配的Secret的`data.auth-extra-groups`字段中还有值,则该用户同时属于`data.auth-extra-groups`中列出的组。比如token`fxj0d5.qhtvyaseds22o96y`匹配的如下的Secret的`data.auth-extra-groups`字段中有`system:bootstrappers:kubeadm:default-node-token`,那么认证出来的用户`system:bootstrap:fxj0d5`也属于组`system:bootstrappers:kubeadm:default-node-token`。 ``` apiVersion: v1 kind: Secret type: bootstrap.kubernetes.io/token metadata: name: bootstrap-token-fxj0d5 namespace: kube-system data: auth-extra-groups: base64encode(system:bootstrappers:kubeadm:default-node-token) token-id: base64encode(fxj0d5) token-secret: base64encode(qhtvyaseds22o96y) usage-bootstrap-authentication: base64encode(true) usage-bootstrap-signing: base64encode(true) ``` 现在,token`fxj0d5.qhtvyaseds22o96y`已经被认证通过且User为`system:bootstrap:fxj0d5`、Group为`system:bootstrappers`和`system:bootstrappers:kubeadm:default-node-token`。那么,如何对它们进行授权呢? 其实如果细想一下,我们会发现,就算每个kubelet携带的`bootstrap-token`不一样,只要它们认证通过,那么它们认证出来的用户一定属于组`system:bootstrappers`。而每个kubelet都要能够创建CSR对象(其实还要获取CSR对象,后面讲),所以我们只需要给组`system:bootstrappers`授权,让它能够创建CSR对象。 当我们使用RBAC授权模式时,我们在集群中创建如下的ClusterRole以及ClusterRoleBinding,授权`system:bootstrappers`组能够`create`、`get`、`list`、`watch` CSR对象。有了`create`权限,通过认证的kubelet就能够创建CSR对象了。 ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: create-get-csr rules: - apiGroups: - certificates.k8s.io resources: - certificatesigningrequests verbs: - create - get - list - watch ``` ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: approve-bootstrappers-create-get-csr roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kubelet-bootstrap subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:bootstrappers ``` ### CSR的批准与签名 OK,apiserver对kubelet创建CSR对象的请求进行了认证与授权,CSR对象创建成功。刚创建还未批准的CSR对象如下,介绍一下该对象中的数据来源 * `spec.request`:经过base64编码的`PKCS#10 CSR`数据(通俗理解为通过openssl生成的CSR文件的内容),它是kubelet发起API请求创建k8s中的CSR对象(`POST /apis/certificates.k8s.io/v1beta1/certificatesigningrequests`)时需要携带的参数;如果把它base64解码后再用openssl工具查看其中的内容,会发现两项重要的信息:`O=system:nodes`和`CN=system:node:<node-name>` * `spec.groups`与`spec.username`是apiserver加进去的,因为上面`bootstrap-token`认证器把`fxj0d5.qhtvyaseds22o96y`认证成了用户`system:bootstrap:fxj0d5`以及组`system:bootstrappers`和`system:bootstrappers:kubeadm:default-node-token`。 ``` apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: node-csr-u7CZYAhzUS3lpqJVeFlRh9jCA2weDATogQ0G-9itcsU spec: groups: - system:bootstrappers - system:bootstrappers:kubeadm:default-node-token - system:authenticated request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlIMU1JR2JBZ0VBTURreEZUQVRCZ05WQkFvVERITjVjM1JsYlRwdWIyUmxjekVnTUI0R0ExVUVBeE1YYzNsegpkR1Z0T201dlpHVTZjV2xoYjJwcGJtY3hNREl3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVQwCjhGeXhDWExmN0FKVE15bTNVbkp4dFdzSkJ2SjJHbXk0ZVFYTVRMYU9QUDRESmpQQXVTZ2o2ZU10d3JObWxLV2UKeklRQ05YeXNxY25pTDI0QVFnUTBvQUF3Q2dZSUtvWkl6ajBFQXdJRFNRQXdSZ0loQUltRzBNU1ZtSzdrRjh2cgpYQW1OendwREFkNjYvV3Vtekc5UzV4VXFFdjRnQWlFQXdyNTV5UnpOVkdOVVRkcTA3dHl4WGZpRC9ZZTdvWmFYCm94SnFXZzhFWnVzPQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K usages: - digital signature - key encipherment - client auth username: system:bootstrap:fxj0d5 status: {} ``` 此时CSR对象还没有被批准,所以status为空。接下来,就该对该CSR对象进行批准了。批准的方法有两种,一是通过kubectl命令手动批准,二是由controller-manager的`csr-auto-approver`控制器自动批准。手动批准的命令如下 ``` kubectl certificate approve node-csr-u7CZYAhzUS3lpqJVeFlRh9jCA2weDATogQ0G-9itcsU ``` 当然,为了更自动化的管理,我们可以利用控制器`csr-auto-approver`进行自动批准。如果要达到对某个CSR进行自动批准,必须告诉`csr-auto-approver`控制器该CSR对象是可以批准的。为此,我们需要在集群中创建如下的ClusterRole与ClusterRoleBinding,当`csr-auto-approver`发现CSR对象的`spec.groups`或`spec.username`可以`create` apiGroup`certificates.k8s.io`下的资源`certificatesigningrequests/nodeclient`时,便会批准该CSR对象。比如如下的ClusterRole与ClusterBinding刚好授权Group`system:bootstrappers`可以`create`资源`certificatesigningrequests/nodeclient`,而上面的CSR对象的`spec.groups`有`system:bootstrappers`字段。 ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: create-nodeclient rules: - apiGroups: - certificates.k8s.io resources: - certificatesigningrequests/nodeclient verbs: - create ``` ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: approve-bootrappers-create-nodeclient roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: create-nodeclient subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:bootstrappers ``` 当我们创建好上面的ClusterRole与ClusterRoleBinding后,前面的CSR对象就会被`csr-auto-approver`自动批准。批准后的CSR对象的`status`中就会多了`approved`的状态信息。 当CSR被批准后,controller-manager的`csr-signing`控制器便会对CSR进行签名。签名所使用的CA套件由controller-manager的启动参数`--cluster-signing-cert-file`和`--cluster-signing-key-file`指定。CSR的签名证书有效期默认为1年,可以通过controller-manager的启动参数`--experimental-cluster-signing-duration`设置其他的值。 我们来看一下签名之后的CSR对象是什么样的,如下。我们发现,该对象多了一个`status.certificate`字段,而该字段的值正是签名证书经过base64编码之后的内容。 ``` apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: node-csr-u7CZYAhzUS3lpqJVeFlRh9jCA2weDATogQ0G-9itcsU spec: groups: - system:bootstrappers - system:bootstrappers:kubeadm:default-node-token - system:authenticated request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlIMU1JR2JBZ0VBTURreEZUQVRCZ05WQkFvVERITjVjM1JsYlRwdWIyUmxjekVnTUI0R0ExVUVBeE1YYzNsegpkR1Z0T201dlpHVTZjV2xoYjJwcGJtY3hNREl3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVQwCjhGeXhDWExmN0FKVE15bTNVbkp4dFdzSkJ2SjJHbXk0ZVFYTVRMYU9QUDRESmpQQXVTZ2o2ZU10d3JObWxLV2UKeklRQ05YeXNxY25pTDI0QVFnUTBvQUF3Q2dZSUtvWkl6ajBFQXdJRFNRQXdSZ0loQUltRzBNU1ZtSzdrRjh2cgpYQW1OendwREFkNjYvV3Vtekc5UzV4VXFFdjRnQWlFQXdyNTV5UnpOVkdOVVRkcTA3dHl4WGZpRC9ZZTdvWmFYCm94SnFXZzhFWnVzPQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K usages: - digital signature - key encipherment - client auth username: system:bootstrap:fxj0d5 status: certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNaVENDQVUyZ0F3SUJBZ0lVYUgyWXordkcySFNQRjZBOTZqL2xWMm8wSWJvd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0ZURVRNQkVHQTFVRUF4TUthM1ZpWlhKdVpYUmxjekFlRncweE9ERXlNRFV4TWpNNU1EQmFGdzB4T0RFeQpNRFV4TWpRNU1EQmFNRGt4RlRBVEJnTlZCQW9UREhONWMzUmxiVHB1YjJSbGN6RWdNQjRHQTFVRUF4TVhjM2x6CmRHVnRPbTV2WkdVNmNXbGhiMnBwYm1jeE1ESXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBVDAKOEZ5eENYTGY3QUpUTXltM1VuSnh0V3NKQnZKMkdteTRlUVhNVExhT1BQNERKalBBdVNnajZlTXR3ck5tbEtXZQp6SVFDTlh5c3FjbmlMMjRBUWdRMG8xUXdVakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJCkt3WUJCUVVIQXdJd0RBWURWUjBUQVFIL0JBSXdBREFkQmdOVkhRNEVGZ1FVMGowbGppanhCTVBwSTN4bzQxSjgKWnRPN0FmSXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBTEV2M0o3REpGK0FFZnN6Tyt5OElBeDZTMW1uQXNYVwpSUVk5TnYxZW1qUVhIc2ZsSzM3bHFMMytyQjVZZHEvcjRjSlV5dmNRR043NzRtNmN3NFhmaUdBVTdrQk9mQnFaCi93T0lYL1VHZG1nNnA4ZDE0WndobW95SVFOdHdMZCtFYmRLdmtnZ0tMaGlYOXZLMUtaWDNNMUYraEl0cTJYckMKeGx2U1FBSVliN3ZCYWpER2trbEVqeGdjc2ROejVMK09xSzFyL2Z3MEZqUStRRHplWDl2RTJXUkZyS2h0dEt5NQpycGJibnBTWEh1d1NRSkpnZ2tvZUxTcWRFNFF1b2xQS0Q4ZlNudHEyNW82WlpDdm1Zcjl6Ni9PWVhPL2Vwc0tUClhyQXlWZFdZait1UjZRNCt0NVFrMUFwZGNRNCtqWmRwOVI1Z1g5eGtPN1FOYzlubmsyaENBV2M9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K conditions: - lastUpdateTime: 2018-12-05T12:43:53Z message: Auto approving kubelet client certificate after SubjectAccessReview. reason: AutoApproved type: Approved ``` ### kubelet获取签名证书 当kubelet创建完CSR对象后,它便会不停地向apiserver发请求,查询CSR的状态。当CSR对象被签名之后,kubelet便会向apiserver请求签名后的CSR对象。此时kubelet使用的仍然是`bootstrap-token`,根据前面的介绍,该token有权限`create`、`get`、`list`、`watch` CSR资源,所以就能够拿到签名后的CSR对象。 当kubelet拿到签名后的CSR对象后,便可以从中拿到签名证书,然后把签名证书放在启动参数`--cert-dir`指定的对应位置,并完善`--kubeconfig`文件。 完善后的`--kubeconfig`文件的内容如下,发现访问apiserver的三要素都有了:apiserverUrl,ca.crt以及client-certificate。其中apiserverUrl与ca.crt都是从`--bootstrap-kubeconfig`中拿到的,client-certificate是从签名后的CSR对象拿到的 ``` apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNE1URXlNVEEzTkRBMU0xb1hEVEk0TVRFeE9EQTNOREExTTFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTVQxCmRMTm9TYWQwK1J3MjEzdWhiZjNXM0l5UE5hZUxmQ3dDWko3bnVNVGZlSzBqeGdveHNnczQybVF5MkNkbEdsdzQKVWJPejhTcFhlSW4yRC9MREFmeEtqZGc0cWh5YllISmJjK1ZNVGVRRXNQb2VBejdBVWJDQ2hqMmlWdjlHK3p2QgpHWmtobXIzOUI4NWxuSklJTSsvRlliazBTbjBNdVhPc3BGZVJLK3lqbklZeXpxSEtTWVk4aW5SSDhhaExlUlNNCkFCTUVtRndyME1EdWdCR2YrbzFlQU5hamI0QzZ2K3NXV3B6Z01EelhEV1VoSTRpNzRVWHE5KzZOTDlMUGVFOTcKYTY3Yit1SE1DSTl1emg5WjlPcTV3QkY1OS8ram1QT1BzY2lrTnVSVkNEZHZ6enF2VzdjRERLUU1YcXpCYUxKQgpVVENZcDl2aml6c3J1bkpFTDNFQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFGMThHR3JvQmRCNytxa0xvM1RHOEJqTDE2SmcKTm9tL1E0cWJGdHZsaTQwbjZ5Z001cUdqNmI3VTRVT2NaRjl4aDd6cGxWVXJkM0Z0OGhKYjhnOThRaFYxbDVHNwo4a1RQWThQcnAxN2UxWlRjOVdGMXUzZXM4Y2p6VmVFOFAxMEE5c3JqY3ZDNldONnhiK2FNc2E0aEFJbEJ3WjNwCmErSFhhTVhLbVRGaE9KZHJ5QTBNcFJLdTB2eEl5VlNVVU5BdUtYZlk4L3d0bk1VR0JLK0FJTDF3OTRUZ2wxb2sKQ1J1MGhkWEtmek0xWnU0d0hzQVQrcVhWcWR2cUg4UGFWaHc0U2NRcmRlL2o5anBFckwvMUJaSXlBU0J2MndSdwpBZ2VOc3pnZDlRaVRmTWh6QXBLU3gyN2VseHhYRStObGJKbmJBeFJXdkQ2WHZJTmlKUDZuUFQ2eUF3UT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= server: https://192.168.1.103:6443 name: default-cluster contexts: - context: cluster: default-cluster namespace: default user: default-auth name: default-context current-context: default-context kind: Config preferences: {} users: - name: default-auth user: client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem client-key: /var/lib/kubelet/pki/kubelet-client-current.pem ``` 我们去`--cert-dir`的默认位置`/var/lib/kubelet/pki/`下看一下内容,有如下文件 ``` -rw-------. 1 root root 1118 Dec 5 20:43 kubelet-client-2018-12-05-20-43-48.pem lrwxrwxrwx. 1 root root 59 Dec 5 20:43 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2018-12-05-20-43-48.pem -rw-r--r--. 1 root root 2185 Dec 5 19:43 kubelet.crt -rw-------. 1 root root 1675 Dec 5 19:43 kubelet.key ``` 我们发现,签名证书就是`kubelet-client-current.pem`,它是一个符号链接,真实文件为`kubelet-client-2018-12-05-20-43-48.pem`。我们还会发现该目录下有`kubelet.key`与`kubelet.crt`这两个文件,它们又是什么呢?它们其实和`kubelet-client-current.pem`一点关系都没有,是由kubelet自已创建一个私钥与自签名证书,目前不知道有什么用,猜测是来给kubelet启动https服务用的。 ### client证书的替换 我们上面提到了,`csr-signing`控制器签署的证书的有效期默认为1年。当签名证书快要到期的时候(一般证书有效期过了百分之九十的时候),kubelet就会向apiserver发起请求,请求置换证书。kubelet请求置换证书的流程是 1、kubelet发起API请求,在集群中创建一个新的CSR对象 2、手动或自动对CSR进行批准 3、`csr-signing`控制器发现CSR的状态为`approved`后,便对其进行签名 4、kubelet检测到CSR已经签名后,下载下来,置换节点上的相关配置 第1步中,kubelet发起API请求时,携带的身份凭证是快要过期的签名证书,所以走的认证为client证书认证(将会被认证为用户`system:node:<node name>`以及组`system:nodes`),授权为`Node Authorization`。创建出来的新的CSR对象如下:`spec.request`仍然是kubelet发起请求时需要携带的信息,里面包含了`O=system:nodes`与`CN=system:node:<node-name>`(这里为`CN=system:node:qiaojing102`);`spec.groups`变成了`system:nodes`,`spec.username`变成了`system:node:qiaojing102`。 ``` apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: csr-6g4v4 spec: groups: - system:nodes - system:authenticated request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlIME1JR2JBZ0VBTURreEZUQVRCZ05WQkFvVERITjVjM1JsYlRwdWIyUmxjekVnTUI0R0ExVUVBeE1YYzNsegpkR1Z0T201dlpHVTZjV2xoYjJwcGJtY3hNREl3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVJLCjdldENVcTkrZStzdW95Z1JBV2I0dkJZdWRRSmRpeHREQitSK0lXeG1PSEZnYkYwdDUzSVJ5Z29mQVlXS3Y0ZkQKeHBHOHg2a2hXM0dNRnlSWjZRclBvQUF3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnSko2a3d4MEJ2bmVldXpsdgpheUZrVVFWVEUrMEtBa3pjamFOQ0dLeTZUVGdDSVFEYWNxNzRrQjNlZnZLR1BzYlJJYUtxNjlLQk5mMS9memVPClhWbHl1TG1uVWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K usages: - digital signature - key encipherment - client auth username: system:node:qiaojing102 status: {} ``` CSR对象创建成功之后,就需要对它进行批准。同样,可以通过kubectl命令手动进行批准。我们现在来讨论一下如何由controller-manager的`csr-auto-approver`进行自动批准。在前面我们介绍了,只要CSR中的`spec.groups`或`spec.username`有权限`create`资源`certificatesigningrequests/nodeclient`,那么该CSR就会被自动批准。那么这个新的CSR是否也一样呢?答案为否。 在这里,我们要介绍一下k8s中CSR的类型 * `nodeclient`:kubelet第一次申请client签名证书时创建的CSR * `selfnodeclient`:kubelet置换client签名证书时创建的CSR * `selfnodeserver`:(这种类型先不做介绍) 我们知道,kubelet第一次申请client签名证书时,其创建的CSR中`spec.username`为`system:bootstrap:<token id>`,`spec.groups`为`system:bootstrappers`等。而`spec.request`中的内容包含的内容为`O=system:nodes`、`CN=system:node:<node-name>`。我们发现,该种类型的CSR的`spec.request`中的`O`的值没有包含在`spec.groups`中,`spec.request`中的`CN`的值不等于`spec.username`。 kubelet置换client签名证书时,其创建的CSR中`spec.username`为`system:node:<node-name>`,`spec.groups`为`system:nodes`。而`spec.request`中的内容包含的内容为`O=system:nodes`、`CN=system:node:<node-name>`。我们发现,该种类型的CSR的`spec.request`中的`O`的值包含在`spec.groups`中,`spec.request`中的`CN`的值刚好等于`spec.username`。 `csr-auto-approver`就是根据上面的原理来判断CSR是属于`nodeclient`类型还是`selfnodeclient`类型的。那么对于这两种类型的CSR的自动批准,需要不同的ClusterRole与ClusterRoleBinding。前面我们已经介绍了对于`nodeclient`类型的CSR,它的`spec.username`或`spec.groups`需要有权限`create`资源`certificatesigningrequests/nodeclient`,才能被自动批准。而对于`selfnodeclient`类型的CSR,它的`spec.username`或`spec.groups`需要有权限`create`资源`certificatesigningrequests/selfnodeclient`,才能被自动批准。于是,我们需要在集群中创建如下的ClusterRole与ClusterRoleBinding ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: create-selfnodeclient rules: - apiGroups: - certificates.k8s.io resources: - certificatesigningrequests/nodeclient verbs: - create ``` ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: approve-nodes-create-nodeclient roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: create-selfnodeclient subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:nodes ``` 当我们创建了如上的ClusterRole与ClusterRoleBinding之后,`csr-auto-approver`就会自动批准`selfnodeclient`类型的CSR。批准之后,`csr-signing`控制器就可以对CSR进行签名了。签名之后的CSR如下,签名证书已经放在了字段`status.certificate`中 ``` apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: csr-6g4v4 spec: groups: - system:nodes - system:authenticated request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlIME1JR2JBZ0VBTURreEZUQVRCZ05WQkFvVERITjVjM1JsYlRwdWIyUmxjekVnTUI0R0ExVUVBeE1YYzNsegpkR1Z0T201dlpHVTZjV2xoYjJwcGJtY3hNREl3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVJLCjdldENVcTkrZStzdW95Z1JBV2I0dkJZdWRRSmRpeHREQitSK0lXeG1PSEZnYkYwdDUzSVJ5Z29mQVlXS3Y0ZkQKeHBHOHg2a2hXM0dNRnlSWjZRclBvQUF3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnSko2a3d4MEJ2bmVldXpsdgpheUZrVVFWVEUrMEtBa3pjamFOQ0dLeTZUVGdDSVFEYWNxNzRrQjNlZnZLR1BzYlJJYUtxNjlLQk5mMS9memVPClhWbHl1TG1uVWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K usages: - digital signature - key encipherment - client auth username: system:node:qiaojing102 status: certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNaVENDQVUyZ0F3SUJBZ0lVSjRIZTVEZ1h2dVhaSUJSQ1J4cDVzQ3ZuRWZVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0ZURVRNQkVHQTFVRUF4TUthM1ZpWlhKdVpYUmxjekFlRncweE9ERXlNRFl3TXpBeE1EQmFGdzB4T0RFeQpNRFl3TXpFeE1EQmFNRGt4RlRBVEJnTlZCQW9UREhONWMzUmxiVHB1YjJSbGN6RWdNQjRHQTFVRUF4TVhjM2x6CmRHVnRPbTV2WkdVNmNXbGhiMnBwYm1jeE1ESXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUksKN2V0Q1VxOStlK3N1b3lnUkFXYjR2Qll1ZFFKZGl4dERCK1IrSVd4bU9IRmdiRjB0NTNJUnlnb2ZBWVdLdjRmRAp4cEc4eDZraFczR01GeVJaNlFyUG8xUXdVakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJCkt3WUJCUVVIQXdJd0RBWURWUjBUQVFIL0JBSXdBREFkQmdOVkhRNEVGZ1FVQVppSmZOWC9BYks2bUhFbTlROEwKdkx4LzZUUXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBRUVmYWpUejZLNlBXL1JlZnJGSENFa3lRMXV3WDdsWgpTOHBYb0pqZVpJcDNXcElNVGczMXRCVnJ3QlR6c0VNckRNZXdpTjNBRGZDbGdoY1Y2cGUxYWZWQ2FlWmZFUUxZCjRRR3pZTW1JK3N0eEpueWpWNTJERWhZcDVnNnFha3ZMdnpmUC9NZ2gwcFNROFF6c2ErL1VQZmgzaE8vUUVBNnYKY2tlS0tOYWczMnJhZjViOWhDWjUxT09iU3UwRkN4ZU1MRWZudk5nbnNsVk9KYXZpZGRFWG9aV0htNW5OaTV3NQpYb1d3SDFwUUUvbDJlNW0wOWxieko5eE9OdThxbHc1ZlN2am11dE14ZjVZSm1oKzZrYmwyYmdQMlg1MHdBcVdPCjBuTURyUFE3ZTUybVRodkNUTnlxeGNGcEo5bnVnUFdJS2E1UXhjUS91TG1FaERUQVFQeStKREE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K conditions: - lastUpdateTime: 2018-12-06T03:05:35Z message: Auto approving self kubelet client certificate after SubjectAccessReview. reason: AutoApproved type: Approved ``` 然后,kubelet就可以下载CSR,从中获取到证书,更新自已主机上的证书了。 ### Reference [1] https://mritd.me/2018/01/07/kubernetes-tls-bootstrapping-note/