## 背景

为了尽可能与社区原生k8s保持一致与同步，TKEStack使用社区的kubeadm新建独立集群。使用kubeadm创建的集群存在一个比较烦人的问题就是证书的有效期只有一年。kubeadm这么做有着其充分的理由：安全，但是这也就意味着每年至少要更新一次k8s的证书。好在kubeadm提供了方便更新证书的功能，本文将以TKEStack的global集群及创建的独立集群为例，介绍如何使用kubeadm更新证书。

## 限制条件

1. 本文只针对1.2.5+版本TKEStack的global集群及创建的独立集群，对导入集群并不适用；
2. kubeadm的版本要与k8s版本保持一致，例如1.16.x版本的k8s，请使用1.16.x的kubeadm进行证书更新操作，否则可能导致读取配置失败而使用kubeadm默认配置覆盖集群配置;
3. 证书更新后需要重启kube-apiserver、kube-controller、kube-scheduler和etcd，如是生产环境需要评估这些组件重启造成的影响。

## 波及节点

需要操作更新证书的节点包括global集群的master节点，以及通过TKEStack创建的独立集群的master节点。如果安装时配置了HA，需要对每个master节点上都执行更新证书操作。尽管kubelet也有一个client端证书需要更新维护，但是不需要我们进行手动维护，因为自1.8版本以后kubeadm已经将kubelet配置为自动更新证书，所以无论是master节点还是worker节点上的kubelet客户端证书都不需要使用kubeadm进行手动更新。

## 更新流程

### 下载kubeadm

一般情况下，TKEStcak创建的集群节点上的`/usr/bin/`文件夹下会存在kubeadm二进制文件，如果发现master节点上没有kubeadm，可以从官方下载。以amd64架构1.16.9版本的kubeadm为例子，可以通过`curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/v1.16.9/bin/linux/amd64/kubeadm`下载到本地，如果需要其他版本请将URL中的版本号修改为其他版本号即可。使用`chmod +x kubeadm`命令为二进制加权限后，将kubeadm发送到集群master节点。

### 备份文件

可以通过`cp -r /etc/kubernetes /etc/kubernetes.bak`和`cp -r /var/lib/etcd /var/lib/etcd.bak`将k8s和tecd相关文件做备份。

### 执行证书更新

执行`kubeadm alpha certs renew all`即可出发证书更新操作，如果使用的1.16.x版本，其打印内容如下：

```sh
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
```

如果是1.18.x版本，打印内容会多一些提示：

```sh
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
```

### 更新kubeconfig

执行`kubeadm init phase kubeconfig all`可使用新证书生成新的kubeconfig文件。执行`cp -r /root/.kube /root/.kube.bak`备份后，可以执行`cp -f /etc/kubernetes/admin.conf /root/.kube/config`覆盖掉原本的kubeconfig文件。

### 验证与组件重启

执行完上述更新证书操作后可以通过`echo | openssl s_client -showcerts -connect 127.0.0.1:6443 -servername api 2>/dev/null | openssl x509 -noout -enddate`验证apiserver证书有效期是否更新，会发现证书有效期没有更新。此原因是k8s组件如果没有重启就不会使用新证书导致的。

执行

```sh
docker rm -f $(docker ps -q -f label=io.kubernetes.container.name=kube-apiserver) && \
docker rm -f $(docker ps -q -f label=io.kubernetes.container.name=kube-controller-manager) && \
docker rm -f $(docker ps -q -f label=io.kubernetes.container.name=kube-scheduler) && \
docker rm -f $(docker ps -q -f label=io.kubernetes.container.name=etcd)
```

以重启k8s及etcd相关组件后，再次执行`echo | openssl s_client -showcerts -connect 127.0.0.1:6443 -servername api 2>/dev/null | openssl x509 -noout -enddate`，可以观察到证书有效期已经发生变化。

## 执行命令小结

下面把升级过程中在master节点执行命令的总结在下面：

```sh
cp -r /etc/kubernetes /etc/kubernetes.bak
cp -r /var/lib/etcd /var/lib/etcd.bak

kubeadm alpha certs renew all

kubeadm init phase kubeconfig all
cp -r /root/.kube /root/.kube.bak
cp -f /etc/kubernetes/admin.conf /root/.kube/config

docker rm -f $(docker ps -q -f label=io.kubernetes.container.name=kube-apiserver) && \
docker rm -f $(docker ps -q -f label=io.kubernetes.container.name=kube-controller-manager) && \
docker rm -f $(docker ps -q -f label=io.kubernetes.container.name=kube-scheduler) && \
docker rm -f $(docker ps -q -f label=io.kubernetes.container.name=etcd)

echo | openssl s_client -showcerts -connect 127.0.0.1:6443 -servername api 2>/dev/null | openssl x509 -noout -enddate
```

如果你更新的集群是TKEStack的global集群，为了保证TKEStack系统组件正常工作还需要更新`tke namespace`下的`certs configmap`里的`etcd.crt`和`etcd.key`:

```sh
# 执行更新操作前请先做备份
kubectl get -n tke cm certs -o yaml > certs.bak.yaml
# 使用/etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/apiserver-etcd-client.key的内容
# 替换configmap data中的etcd.crt和etcd.key下的内容
kubectl edit -n tke cm certs -o yaml
```

<!-- markdown for nginx, see https://phus.lu -->
<script>
!function(){
	var marked_js = 'https://cdnjs.cloudflare.com/ajax/libs/marked/0.5.2/marked.min.js'
	var github_markdown_css = 'https://cdnjs.cloudflare.com/ajax/libs/github-markdown-css/5.1.0/github-markdown-light.min.css'

	var dom = {
		element: null,
		get: function (o) {
			var obj = Object.create(this)
			obj.element = (typeof o == "object") ? o : document.createElement(o)
			return obj
		},
		add: function (o) {
			var obj = dom.get(o)
			this.element.appendChild(obj.element)
			return obj
		},
		text: function (t) {
			this.element.appendChild(document.createTextNode(t))
			return this
		},
		attr: function (k, v) {
			this.element.setAttribute(k, v)
			return this
		}
	}

	if (!document.head) return
	head = dom.get(document.head)
	head.add('meta').attr('charset', 'utf-8')
	head.add('meta').attr('name', 'viewport').attr('content', 'width=device-width,initial-scale=1')

	if (!document.body) {
		document.write(["<div class=\"container\">",
		"<h3>nginx.conf</h3>",
		"<textarea rows=6 cols=50>",
		"# download markdown.html to /wwwroot",
		"location ~ \\.md$ {",
		"    default_type text/html;",
		"    add_after_body /markdown.html;",
		"}",
		"</textarea>",
		"</div>"].join("\n"))
		return
	}

	var bodytext = document.body.innerHTML
	document.body.innerHTML = ''

	div = dom.get('div').attr('class', 'container')

	div.add('script').attr('src', marked_js)
	div.add('link').attr('rel', 'stylesheet').attr('href', github_markdown_css)

	title = decodeURIComponent(document.location.pathname.replace(/.*\//, '').replace(/\.html$/, ''))
	document.title = '《' + title.replace(/\.md$/, '') + '》'
	tbody = div.add('table').add('tbody')
	tbody.add('tr').add('th').text(title).attr('class', 'octicon-book')
	tbody.add('tr').add('td').add('div').attr('id', 'readme').attr('class', 'markdown-body')

	wait = function (name, callback) {
		var interval = 10; // ms
		window.setTimeout(function() {
			if (window[name]) {
				callback(window[name])
			} else {
				window.setTimeout(arguments.callee, interval)
			}
		}, interval)
	}
	wait('marked', function() {
		document.getElementById("readme").innerHTML = marked.parse(bodytext)
	})

	document.body.appendChild(div.element)
}()
</script>

<style>
body {
	margin: 0;
	font-family: "ubuntu", "Tahoma", "Microsoft YaHei", Arial, Serif;
}
.markdown-body {
	float: left;
	font-family: "ubuntu", "Tahoma", "Microsoft YaHei", Arial, Serif;
}
.container {
	padding-right: 15px;
	padding-left: 15px;
	margin-right: auto;
	margin-left: auto;
}
@media (min-width: 768px) {
	.container {
		max-width: 80%;
	}
}
@media (min-width: 992px) {
	.container {
		max-width: 70%;
	}
}
@media (min-width: 1200px) {
	.container {
		max-width: 60%;
	}
}
table {
	width: 100%;
	max-width: 100%;
	margin-bottom: 20px;
	border: 1px solid #ddd;
	padding: 0;
	border-collapse: collapse;
}
table th {
	font-size: 14px;
}
table tr {
	border: 1px solid #ddd;
	padding: 5px;
}
table th, table td {
	border: 1px solid #ddd;
	font-size: 14px;
	line-height: 20px;
	padding: 3px;
	text-align: left;
}
.octicon-book {
	background-position: center left;
	background-repeat: no-repeat;
	padding-left: 20px;
	background-image: url("data:image/svg+xml;charset=utf8,%3Csvg xmlns='http://www.w3.org/2000/svg' width='16' height='16' viewBox='0 0 16 16'%3E%3Cpath d='M3,5 L7,5 L7,6 L3,6 L3,5 L3,5 Z M3,8 L7,8 L7,7 L3,7 L3,8 L3,8 Z M3,10 L7,10 L7,9 L3,9 L3,10 L3,10 Z M14,5 L10,5 L10,6 L14,6 L14,5 L14,5 Z M14,7 L10,7 L10,8 L14,8 L14,7 L14,7 Z M14,9 L10,9 L10,10 L14,10 L14,9 L14,9 Z M16,3 L16,12 C16,12.55 15.55,13 15,13 L9.5,13 L8.5,14 L7.5,13 L2,13 C1.45,13 1,12.55 1,12 L1,3 C1,2.45 1.45,2 2,2 L7.5,2 L8.5,3 L9.5,2 L15,2 C15.55,2 16,2.45 16,3 L16,3 Z M8,3.5 L7.5,3 L2,3 L2,12 L8,12 L8,3.5 L8,3.5 Z M15,3 L9.5,3 L9,3.5 L9,12 L15,12 L15,3 L15,3 Z' /%3E%3C/svg%3E");
}
</style>