[TOC]
本文介绍如何将ETCD从http迁移到https。本文主要参考[etcd-live-http-to-https-migration](https://github.com/coreos/docs/blob/master/etcd/etcd-live-http-to-https-migration.md)。不过如果只是按照此文操作，会也问题。接下来我们介绍详细步骤
### **准备条件**
本文准备了一个两节点的ETCD集群，版本为3.2.9，如下：
```
$ etcdctl member list
7b86fda650706ac0: name=192.168.92.121 peerURLs=http://192.168.92.121:2380 clientURLs=http://192.168.92.121:2379 isLeader=true
c1c299e07e050927: name=192.168.92.122 peerURLs=http://192.168.92.122:2380 clientURLs=http://192.168.92.122:2379 isLeader=false
$ etcdctl --version
etcdctl version: 3.2.9
API version: 2
```
ETCD使用二进制安装，并使用systemd进行管理，`/usr/lib/systemd/system/etcd.service`文件内容如下（两个节点都一样）：
```
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE}
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
```
配置文件`/etc/etcd/etcd.conf`的内容如下（以下为121节点的内容）：
```
# [member]
ETCD_NAME=192.168.92.121
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_LISTEN_PEER_URLS="http://192.168.92.121:2380"
#[cluster]
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.92.121:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.92.121:2380"
ETCD_INITIAL_CLUSTER="192.168.92.121=http://192.168.92.121:2380,192.168.92.122=http://192.168.92.122:2380"
ETCD_INITIAL_CLUSTER_TOKEN="mycluster"
ETCD_INITIAL_CLUSTER_STATE="new"
```
### **操作步骤**
**1、创建自签名证书**
首先创建根证书`etcd_ca.key`和`etcd_ca.crt`
```
$ openssl genrsa -out etcd_ca.key 4096
$ openssl req -x509 -new -nodes -key etcd_ca.key -subj "/CN=ETCD_CA" -days 36500 -out etcd_ca.crt
```
然后，把这两个文件分别拷贝到121、122节点的某个目录下，假设放在`/etc/kubernetes/key/etcd/`目录下。
然后分别为121、122节点创建2379与2380端口的服务证书。
这里，我们讲解为121节点创建服务证书的流程。首先，进入到121节点的`/etc/kubernetes/key/etcd/`目录下。
然后，为2379和2380端口创建key，假设2379端口的key为client.key，2380端口的key为peer.key：
```
$ openssl genrsa -out client.key 4096
$ openssl genrsa -out peer.key 4096
```
接着，创建一个etcd.csr.conf文件，用来生成csr文件，内容如下（注意，122节点上要更改IP）：
```
[ req ]
default_bits = 4096
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C = CN
ST = GuangDong
L = GuangZhou
O = SYSU
OU = SIST
CN = 192.168.92.121
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
IP.1 = 192.168.92.121
IP.2 = 127.0.0.1
[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names
```
然后，生成client.csr与peer.csr文件
```
$ openssl req -new -key client.key -out client.csr -config etcd.csr.conf
$ openssl req -new -key peer.key -out peer.csr -config etcd.csr.conf
```
最后，用CA进行签名：
```
$ openssl x509 -req -in client.csr -CA etcd_ca.crt -CAkey etcd_ca.key -CAcreateserial -out client.crt -days 36500 -extensions v3_ext -extfile etcd.csr.conf
$ openssl x509 -req -in peer.csr -CA etcd_ca.crt -CAkey etcd_ca.key -CAcreateserial -out peer.crt -days 36500 -extensions v3_ext -extfile etcd.csr.conf
```
此时，121节点上的`/etc/kubernetes/key/etcd/`目录下应该用如下几个文件：etcd_ca.key和etcd_ca.crt、client.key和client.crt、peer.key和peer.crt。
122节点也按如上的操作得到六个证书文件。
**2、配置TLS**
接下来我们给ETCD配置TLS（两个节点都要操作，可以同时操作）。
创建目录`/usr/lib/systemd/system/etcd.service.d`，然后在该目录下创建文件certs.conf，文件内容如下：
```
[Service]
Environment="ETCD_CERT_FILE=/etc/kubernetes/key/etcd/client.crt"
Environment="ETCD_KEY_FILE=/etc/kubernetes/key/etcd/client.key"
Environment="ETCD_TRUSTED_CA_FILE=/etc/kubernetes/key/etcd/etcd_ca.crt"
Environment="ETCD_CLIENT_CERT_AUTH=true"
Environment="ETCD_PEER_CERT_FILE=/etc/kubernetes/key/etcd/peer.crt"
Environment="ETCD_PEER_KEY_FILE=/etc/kubernetes/key/etcd/peer.key"
Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/kubernetes/etcd/etcd_ca.crt"
Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
```
然后，重启etcd服务
```
$ systemctl daemon-reload
$ systemctl restart etcd
```
**3、更新peerUrls**
我们先更新peerUrls。因为如果先更新clientUrls，那么在更新peerUrls的时候，etcdctl命令就得带上证书，比较麻烦。
更新peerUrls有三步：（1）etcdctl命令更新（2）更改etcd.conf配置文件（3）重启etcd
> 特别注意：[etcd-live-http-to-https-migration](https://github.com/coreos/docs/blob/master/etcd/etcd-live-http-to-https-migration.md)这个文档中只有第一步，[etcd.io的update-advertise-peer-urls](https://etcd.io/docs/v3.2/op-guide/runtime-configuration/#update-advertise-peer-urls)文档中只有第一步和第三步，在实践中发现会有问题。
另外：节点之间要串行操作
以下，我们以121节点的操作为例
（1）etcd命令更新
首先，我们根据用以下命令拿到etcdctl命令更新的语句：
```
$ etcdctl member list | awk -F'[: =]' '{print "etcdctl member update "$1" https:"$7":"$8}'
etcdctl member update 7b86fda650706ac0 https://192.168.92.121:2380
etcdctl member update c1c299e07e050927 https://192.168.92.122:2380
```
然后执行命令进行更新
```
$ etcdctl member update 7b86fda650706ac0 https://192.168.92.121:2380
```
（2）更改etcd.conf配置文件
根据[此文](https://github.com/coreos/docs/blob/master/etcd/tls-etcd-clients.md#remove-legacy-etcd-ports-configuration)及[此文](https://stackoverflow.com/questions/42922444/etcd2-cluster-not-communicating-after-https-migration)，`listen-peer-urls`最终配置要为https。所以，我们需要更改该配置文件中`listen-peer-urls`从http改成https，如下：
```
ETCD_LISTEN_PEER_URLS="https://192.168.92.121:2380"
```
（3）重启etcd
```
$ systemctl restart etcd
```
然后查看一下etcd集群状态：
```
$ etcdctl member list
7b86fda650706ac0: name=192.168.92.121 peerURLs=https://192.168.92.121:2380 clientURLs=http://192.168.92.121:2379 isLeader=false
c1c299e07e050927: name=192.168.92.122 peerURLs=http://192.168.92.122:2380 clientURLs=http://192.168.92.122:2379 isLeader=true
$ etcdctl cluster-health
member 7b86fda650706ac0 is healthy: got healthy result from http://192.168.92.121:2379
member c1c299e07e050927 is healthy: got healthy result from http://192.168.92.122:2379
cluster is healthy
```
根据上面的步骤操作122节点。
**4、更新clientUrls**
该步骤所有节点可以同步操作。这里以121节点为例。
编辑`/etc/etcd/etcd.conf`文件，更改`listen-client-urls`及`advertise-client-urls`参数，把http改成https：
```
ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.92.121:2379"
```
然后重启etcd
```
$ systemctl restart etcd
```
然后查看集群状态，可以看到，121的clientUrls已经变成了https
```
$ etcdctl --endpoints https://192.168.92.121:2379 --ca-file /etc/kubernetes/key/etcd/etcd_ca.crt --key-file /etc/kubernetes/key/etcd/peer.key --cert-file /etc/kubernetes/key/etcd/peer.crt member list
7b86fda650706ac0: name=192.168.92.121 peerURLs=https://192.168.92.121:2380 clientURLs=https://192.168.92.121:2379 isLeader=true
c1c299e07e050927: name=192.168.92.122 peerURLs=https://192.168.92.122:2380 clientURLs=http://192.168.92.122:2379 isLeader=false
```
按照上面的方法操作122节点。
### **参考**
* https://github.com/coreos/docs/blob/master/etcd/etcd-live-http-to-https-migration.md
* https://github.com/coreos/docs/blob/master/etcd/tls-etcd-clients.md#remove-legacy-etcd-ports-configuration
* https://stackoverflow.com/questions/42922444/etcd2-cluster-not-communicating-after-https-migration
* https://etcd.io/docs/v3.2/op-guide/runtime-configuration/#update-a-member
