1.1 准备环境
1.1.1 安装相关程序
yum install -y conntrack ipvsadm ipset jq iptables curl sysstat libseccomp
1.1.2 配置内核参数
cat > /etc/sysctl.d/kubernetes.conf <<EOF net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward=1 vm.swappiness=0 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_watches=89100 EOF sysctl -p /etc/sysctl.d/kubernetes.conf
1.1.3 开启内核模块
modprobe br_netfilter modprobe ip_vs
1.1.4 设置防火墙规则
iptables -P FORWARD ACCEPT
1.1.5 创建程序工作目录
mkdir -p /var/lib/{kubelet,kube-proxy}
1.2 部署docker
1.2.1 安装docker程序
cd /server/tools/ yum localinstall -y docker-ce-18.03.1.ce-1.el7.centos.x86_64.rpm
1.2.2 创建docker配置文件
mkdir -p /etc/docker/ cat > /etc/docker/docker-daemon.json <<EOF { "registry-mirrors": ["https://hub-mirror.c.163.com", "https://docker.mirrors.ustc.edu.cn"], "max-concurrent-downloads": 20 } EOF
1.2.3 配置docker启动文件
vim /usr/lib/systemd/system/docker.service [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service Wants=network-online.target [Service] Type=notify # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker EnvironmentFile=-/run/flannel/docker ExecStart=/usr/bin/dockerd --log-level=error $DOCKER_NETWORK_OPTIONS ExecReload=/bin/kill -s HUP $MAINPID # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity # Uncomment TasksMax if your systemd version supports it. # Only systemd 226 and above support this version. #TasksMax=infinity TimeoutStartSec=0 # set delegate yes so that systemd does not reset the cgroups of docker containers Delegate=yes # kill only the docker process, not all processes in the cgroup KillMode=process # restart the docker process if it exits prematurely Restart=on-failure StartLimitBurst=3 StartLimitInterval=60s [Install] WantedBy=multi-user.target
- 说明:
- flanneld 启动时将网络配置写入/run/flannel/docker 文件中,dockerd 启动前读取该文件中的环境变量 DOCKER_NETWORK_OPTIONS ,然后设置 docker0 网桥网段
- 如果指定了多个EnvironmentFile 选项,则必须将 /run/flannel/docker 放在最后(确保 docker0 使用 flanneld 生成的 bip 参数)
- docker 需要以 root 用于运行
- docker 从13 版本开始,可能将iptables FORWARD chain的默认策略设置为DROP,从而导致 ping 其它 Node 上的 Pod IP 失败,遇到这种情况时,需要手动设置策略为 ACCEPT:iptables -P FORWARD ACCEPT
1.2.4 启动docker
systemctl daemon-reload systemctl enable docker systemctl restart docker systemctl status docker
1.2.5 导入相关镜像
提示:由于国内网络原因无法直接从google下载镜像,所以需要手动导入相关镜像
docker load -i /tmp/coredns-1.0.6.tar docker load -i /tmp/heapster-1.5.3.tar docker load -i /tmp/heapster-grafana.tar docker load -i /tmp/heapster-influxdb-amd64.tar docker load -i /tmp/kubernetes-dashboard-amd64-v1.8.3.tar
1.3 为每个node节点创建token和kubeconfig文件
1.3.1 node01节点
kubeadm token create \ --description kubelet-bootstrap-token \ --groups system:bootstrappers:k8s-node01 \ --kubeconfig ~/.kube/config kpte71.575cug5m6gyc7rpn
cd /opt/kubernetes/cfg/ kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=https://192.168.10.160:8443 \ # 此处为master节点的VIP --kubeconfig=kubelet-bootstrap.kubeconfig kubectl config set-credentials kubelet-bootstrap \ --token=kpte71.575cug5m6gyc7rpn \ --kubeconfig=kubelet-bootstrap.kubeconfig kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=kubelet-bootstrap.kubeconfig kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig
1.3.2 node02节点
kubeadm token create \ --description kubelet-bootstrap-token \ --groups system:bootstrappers:k8s-node02 \ --kubeconfig ~/.kube/config ygc8ct.gpeuia26ezd467nl
kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=https://192.168.10.160:8443 \ --kubeconfig=kubelet-bootstrap.kubeconfig kubectl config set-credentials kubelet-bootstrap \ --token=ygc8ct.gpeuia26ezd467nl \ --kubeconfig=kubelet-bootstrap.kubeconfig kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=kubelet-bootstrap.kubeconfig kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig
1.3.3 检查token状态
[root@k8s-node01 ~]# kubeadm token list --kubeconfig ~/.kube/config TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS mp3y9m.7l0f27t3vj6wa3kq 23h 2018-07-13T10:30:15+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-node02 yviv83.0iceglokz056snkl 23h 2018-07-13T10:22:17+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-node01 [root@k8s-master ~]# kubectl get secrets -n kube-system NAME TYPE DATA AGE bootstrap-token-mp3y9m bootstrap.kubernetes.io/token 7 3m bootstrap-token-yviv83 bootstrap.kubernetes.io/token 7 11m
1.4 部署kubelet
1.4.1 编写kubelet配置文件
1.4.1.1 node01节点
cat > /opt/kubernetes/cfg/kubelet.config.json <<EOF { "kind": "KubeletConfiguration", "apiVersion": "kubelet.config.k8s.io/v1beta1", "authentication": { "x509": { "clientCAFile": "/opt/kubernetes/ssl/ca.pem" }, "webhook": { "enabled": true, "cacheTTL": "2m0s" }, "anonymous": { "enabled": false } }, "authorization": { "mode": "Webhook", "webhook": { "cacheAuthorizedTTL": "5m0s", "cacheUnauthorizedTTL": "30s" } }, "address": "192.168.10.163", "port": 10250, "readOnlyPort": 0, "cgroupDriver": "cgroupfs", "hairpinMode": "promiscuous-bridge", "serializeImagePulls": false, "featureGates": { "RotateKubeletClientCertificate": true, "RotateKubeletServerCertificate": true }, "clusterDomain": "cluster.local.", "clusterDNS": ["10.254.0.2"] } EOF
- 说明:
- address:API 监听地址,不能为0.0.1,否则 kube-apiserver、heapster 等不能调用 kubelet 的 API
- readOnlyPort=0:关闭只读端口(默认 10255),等效为未指定
- anonymous.enabled:设置为 false,不允许匿名访问 10250 端口
- x509.clientCAFile:指定签名客户端证书的 CA 证书,开启 HTTP 证书认证
- webhook.enabled=true:开启 HTTPs bearer token 认证
- 对于未通过 x509 证书和 webhook 认证的请求(kube-apiserver 或其他客户端),将被拒绝,提示 Unauthorized
- mode=Webhook:kubelet 使用 SubjectAccessReview API 查询 kube-apiserver 某 user、group 是否具有操作资源的权限(RBAC)
- RotateKubeletClientCertificate、featureGates.RotateKubeletServerCertificate:自动 rotate 证书,证书的有效期取决于 kube-controller-manager 的 --experimental-cluster-signing-duration 参数
- 需要 root 账户运行
1.4.1.2 node02节点
cat > /opt/kubernetes/cfg/kubelet.config.json <<EOF { "kind": "KubeletConfiguration", "apiVersion": "kubelet.config.k8s.io/v1beta1", "authentication": { "x509": { "clientCAFile": "/opt/kubernetes/ssl/ca.pem" }, "webhook": { "enabled": true, "cacheTTL": "2m0s" }, "anonymous": { "enabled": false } }, "authorization": { "mode": "Webhook", "webhook": { "cacheAuthorizedTTL": "5m0s", "cacheUnauthorizedTTL": "30s" } }, "address": "192.168.10.164", "port": 10250, "readOnlyPort": 0, "cgroupDriver": "cgroupfs", "hairpinMode": "promiscuous-bridge", "serializeImagePulls": false, "featureGates": { "RotateKubeletClientCertificate": true, "RotateKubeletServerCertificate": true }, "clusterDomain": "cluster.local.", "clusterDNS": ["10.254.0.2"] } EOF
1.4.2 编写kubelet启动文件
1.4.2.1 node01节点
cat > /usr/lib/systemd/system/kubelet.service <<EOF [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet ExecStart=/opt/kubernetes/bin/kubelet \\ --bootstrap-kubeconfig=/opt/kubernetes/cfg/kubelet-bootstrap.kubeconfig \\ --cert-dir=/opt/kubernetes/ssl \\ --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\ --config=/opt/kubernetes/cfg/kubelet.config.json \\ --hostname-override=k8s-node01 \\ --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest \\ --allow-privileged=true \\ --alsologtostderr=true \\ --logtostderr=false \\ --log-dir=/var/log/kubernetes \\ --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF
- 说明:
- 如果设置了--hostname-override 选项,则 kube-proxy 也需要设置该选项,否则会出现找不到 Node 的情况
- --bootstrap-kubeconfig:指向 bootstrap kubeconfig 文件,kubelet 使用该文件中的用户名和 token 向 kube-apiserver 发送 TLS Bootstrapping 请求
- K8S approve kubelet 的 csr 请求后,在--cert-dir 目录创建证书和私钥文件,然后写入 --kubeconfig 文件
1.4.2.2 node02节点
cat > /usr/lib/systemd/system/kubelet.service <<EOF [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet ExecStart=/opt/kubernetes/bin/kubelet \\ --bootstrap-kubeconfig=/opt/kubernetes/cfg/kubelet-bootstrap.kubeconfig \\ --cert-dir=/opt/kubernetes/ssl \\ --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\ --config=/opt/kubernetes/cfg/kubelet.config.json \\ --hostname-override=k8s-node02 \\ --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest \\ --allow-privileged=true \\ --alsologtostderr=true \\ --logtostderr=false \\ --log-dir=/var/log/kubernetes \\ --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF
1.4.3 Bootstrap Token Auth 和授予权限
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers
- 说明:
- kublet 启动时查找配置的 --kubeletconfig 文件是否存在,如果不存在则使用 --bootstrap-kubeconfig 向 kube-apiserver 发送证书签名请求 (CSR)
- kube-apiserver 收到 CSR 请求后,对其中的 Token 进行认证(事先使用 kubeadm 创建的 token),认证通过后将请求的 user 设置为 system:bootstrap:,group 设置为 system:bootstrappers,这一过程称为 Bootstrap Token Auth
- 默认情况下,这个 user 和 group 没有创建 CSR 的权限,kubelet 启动失败,错误日志如下:
$ sudo journalctl -u kubelet -a |grep -A 2 'certificatesigningrequests' May 06 06:42:36 kube-node1 kubelet[26986]: F0506 06:42:36.314378 26986 server.go:233] failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:bootstrap:lemy40" cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope May 06 06:42:36 kube-node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a May 06 06:42:36 kube-node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
1.4.4 启动kubelet
systemctl daemon-reload systemctl enable kubelet systemctl restart kubelet systemctl status kubelet
1.4.5 检查kubelet状态
[root@k8s-master ~]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-Pa63hrEu3i0q-fXTzUbgIMq5XESsKbhdIjt9hJadsL4 1m system:bootstrap:yviv83 Pending node-csr-okE2Uika1TaEbY57V9cyIcAh6s-forX6jBlsDVm8sDY 1m system:bootstrap:mp3y9m Pending # 手动approve方式: # kubectl certificate approve node-csr-Pa63hrEu3i0q-fXTzUbgIMq5XESsKbhdIjt9hJadsL4 # kubectl describe csr node-csr-Pa63hrEu3i0q-fXTzUbgIMq5XESsKbhdIjt9hJadsL4 [root@k8s-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.10.163 Ready <none> 5m v1.10.4 192.168.10.164 Ready <none> 1m v1.10.4
提示:访问cAdvisor方法:http://192.168.10.163:4194/containers/
1.5 部署kube-proxy
1.5.1 所有node节点创建kubeconfig文件
cd /opt/kubernetes/cfg/ kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=https://192.168.10.160:8443 \ # 此处为master节点的VIP --kubeconfig=kube-proxy.kubeconfig kubectl config set-credentials kube-proxy \ --client-certificate=/opt/kubernetes/ssl/kube-proxy.pem \ --client-key=/opt/kubernetes/ssl/kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig=kube-proxy.kubeconfig kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
1.5.2 编写kube-proxy配置文件
1.5.2.1 node01节点
cat > /opt/kubernetes/cfg/kube-proxy.config.yaml <<EOF apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 192.168.10.163 clientConnection: kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig clusterCIDR: 172.30.0.0/16 healthzBindAddress: 192.168.10.163:10256 hostnameOverride: k8s-node01 kind: KubeProxyConfiguration metricsBindAddress: 192.168.10.163:10249 mode: "ipvs" EOF
1.5.2.2 node02节点
cat > /opt/kubernetes/cfg/kube-proxy.config.yaml <<EOF apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 192.168.10.164 clientConnection: kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig clusterCIDR: 172.30.0.0/16 healthzBindAddress: 192.168.10.164:10256 hostnameOverride: k8s-node02 kind: KubeProxyConfiguration metricsBindAddress: 192.168.10.164:10249 mode: "ipvs" EOF
1.5.3 编写kube-proxy启动文件
cat > /usr/lib/systemd/system/kube-proxy.service <<EOF [Unit] Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] WorkingDirectory=/var/lib/kube-proxy ExecStart=/opt/kubernetes/bin/kube-proxy \\ --config=/opt/kubernetes/cfg/kube-proxy.config.yaml \\ --alsologtostderr=true \\ --logtostderr=false \\ --log-dir=/var/log/kubernetes \\ --v=2 Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
- 说明:
- bindAddress: 监听地址
- kubeconfig: 连接 apiserver 的 kubeconfig 文件
- clusterCIDR: 必须与 kube-controller-manager 的--cluster-cidr 选项值一致;kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量,指定 --cluster-cidr 或 --masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT
- hostnameOverride: 参数值必须与 kubelet 的值一致,否则 kube-proxy 启动后会找不到该 Node,从而不会创建任何 ipvs 规则
- mode: 使用 ipvs 模式
1.5.4 启动kube-proxy
systemctl daemon-reload systemctl enable kube-proxy systemctl restart kube-proxy systemctl status kube-proxy

我的微信
如果有技术上的问题可以扫一扫我的微信