k8s二进制部署系列09-部署k8s node节点

2018年7月16日13:40:03 发表评论 1,329 views

1.1 准备环境

1.1.1 安装相关程序

yum install -y conntrack ipvsadm ipset jq iptables curl sysstat libseccomp

1.1.2 配置内核参数

cat > /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
EOF

sysctl -p /etc/sysctl.d/kubernetes.conf

1.1.3 开启内核模块

modprobe br_netfilter
modprobe ip_vs

1.1.4 设置防火墙规则

iptables -P FORWARD ACCEPT

1.1.5 创建程序工作目录

mkdir -p /var/lib/{kubelet,kube-proxy}

1.2 部署docker

1.2.1 安装docker程序

cd /server/tools/
yum localinstall -y docker-ce-18.03.1.ce-1.el7.centos.x86_64.rpm

1.2.2 创建docker配置文件

mkdir -p  /etc/docker/

cat > /etc/docker/docker-daemon.json <<EOF
{
    "registry-mirrors": ["https://hub-mirror.c.163.com", "https://docker.mirrors.ustc.edu.cn"],
    "max-concurrent-downloads": 20
}
EOF

1.2.3 配置docker启动文件

vim /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=-/run/flannel/docker
ExecStart=/usr/bin/dockerd --log-level=error $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

  • 说明：
  • flanneld 启动时将网络配置写入/run/flannel/docker 文件中，dockerd 启动前读取该文件中的环境变量 DOCKER_NETWORK_OPTIONS ，然后设置 docker0 网桥网段
  • 如果指定了多个EnvironmentFile 选项，则必须将 /run/flannel/docker 放在最后(确保 docker0 使用 flanneld 生成的 bip 参数)
  • docker 需要以 root 用于运行
  • docker 从13 版本开始，可能将iptables FORWARD chain的默认策略设置为DROP，从而导致 ping 其它 Node 上的 Pod IP 失败，遇到这种情况时，需要手动设置策略为 ACCEPT：iptables -P FORWARD ACCEPT

1.2.4 启动docker

systemctl daemon-reload
systemctl enable docker
systemctl restart docker
systemctl status docker

1.2.5 导入相关镜像

提示：由于国内网络原因无法直接从google下载镜像，所以需要手动导入相关镜像

docker load -i /tmp/coredns-1.0.6.tar
docker load -i /tmp/heapster-1.5.3.tar
docker load -i /tmp/heapster-grafana.tar
docker load -i /tmp/heapster-influxdb-amd64.tar
docker load -i /tmp/kubernetes-dashboard-amd64-v1.8.3.tar

1.3 为每个node节点创建token和kubeconfig文件

1.3.1 node01节点

kubeadm token create \
      --description kubelet-bootstrap-token \
      --groups system:bootstrappers:k8s-node01 \
      --kubeconfig ~/.kube/config
kpte71.575cug5m6gyc7rpn

cd /opt/kubernetes/cfg/

kubectl config set-cluster kubernetes \
      --certificate-authority=/opt/kubernetes/ssl/ca.pem \
      --embed-certs=true \
      --server=https://192.168.10.160:8443 \      # 此处为master节点的VIP
      --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config set-credentials kubelet-bootstrap \
      --token=kpte71.575cug5m6gyc7rpn \
      --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config set-context default \
      --cluster=kubernetes \
      --user=kubelet-bootstrap \
      --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig

1.3.2 node02节点

kubeadm token create \
      --description kubelet-bootstrap-token \
      --groups system:bootstrappers:k8s-node02 \
      --kubeconfig ~/.kube/config
ygc8ct.gpeuia26ezd467nl

kubectl config set-cluster kubernetes \
      --certificate-authority=/opt/kubernetes/ssl/ca.pem \
      --embed-certs=true \
      --server=https://192.168.10.160:8443 \
      --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config set-credentials kubelet-bootstrap \
      --token=ygc8ct.gpeuia26ezd467nl \
      --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config set-context default \
      --cluster=kubernetes \
      --user=kubelet-bootstrap \
      --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig

1.3.3 检查token状态

[[email protected] ~]# kubeadm token list --kubeconfig ~/.kube/config
TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION               EXTRA GROUPS
mp3y9m.7l0f27t3vj6wa3kq   23h       2018-07-13T10:30:15+08:00   authentication,signing   kubelet-bootstrap-token   system:bootstrappers:k8s-node02
yviv83.0iceglokz056snkl   23h       2018-07-13T10:22:17+08:00   authentication,signing   kubelet-bootstrap-token   system:bootstrappers:k8s-node01

[[email protected] ~]# kubectl get secrets  -n kube-system
NAME                                             TYPE                                  DATA      AGE
bootstrap-token-mp3y9m                           bootstrap.kubernetes.io/token         7         3m
bootstrap-token-yviv83                           bootstrap.kubernetes.io/token         7         11m

1.4 部署kubelet

1.4.1 编写kubelet配置文件

1.4.1.1 node01节点

cat > /opt/kubernetes/cfg/kubelet.config.json <<EOF
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication": {
    "x509": {
      "clientCAFile": "/opt/kubernetes/ssl/ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "address": "192.168.10.163",
  "port": 10250,
  "readOnlyPort": 0,
  "cgroupDriver": "cgroupfs",
  "hairpinMode": "promiscuous-bridge",
  "serializeImagePulls": false,
  "featureGates": {
    "RotateKubeletClientCertificate": true,
    "RotateKubeletServerCertificate": true
  },
  "clusterDomain": "cluster.local.",
  "clusterDNS": ["10.254.0.2"]
}
EOF

  • 说明：
  • address：API 监听地址，不能为0.0.1，否则 kube-apiserver、heapster 等不能调用 kubelet 的 API
  • readOnlyPort=0：关闭只读端口(默认 10255)，等效为未指定
  • anonymous.enabled：设置为 false，不允许匿名访问 10250 端口
  • x509.clientCAFile：指定签名客户端证书的 CA 证书，开启 HTTP 证书认证
  • webhook.enabled=true：开启 HTTPs bearer token 认证
  • 对于未通过 x509 证书和 webhook 认证的请求(kube-apiserver 或其他客户端)，将被拒绝，提示 Unauthorized
  • mode=Webhook：kubelet 使用 SubjectAccessReview API 查询 kube-apiserver 某 user、group 是否具有操作资源的权限(RBAC)
  • RotateKubeletClientCertificate、featureGates.RotateKubeletServerCertificate：自动 rotate 证书，证书的有效期取决于 kube-controller-manager 的 --experimental-cluster-signing-duration 参数
  • 需要 root 账户运行

1.4.1.2 node02节点

cat > /opt/kubernetes/cfg/kubelet.config.json <<EOF
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication": {
    "x509": {
      "clientCAFile": "/opt/kubernetes/ssl/ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "address": "192.168.10.164",
  "port": 10250,
  "readOnlyPort": 0,
  "cgroupDriver": "cgroupfs",
  "hairpinMode": "promiscuous-bridge",
  "serializeImagePulls": false,
  "featureGates": {
    "RotateKubeletClientCertificate": true,
    "RotateKubeletServerCertificate": true
  },
  "clusterDomain": "cluster.local.",
  "clusterDNS": ["10.254.0.2"]
}
EOF

1.4.2 编写kubelet启动文件

1.4.2.1 node01节点

cat > /usr/lib/systemd/system/kubelet.service <<EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \\
  --bootstrap-kubeconfig=/opt/kubernetes/cfg/kubelet-bootstrap.kubeconfig \\
  --cert-dir=/opt/kubernetes/ssl \\
  --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
  --config=/opt/kubernetes/cfg/kubelet.config.json \\
  --hostname-override=k8s-node01 \\
  --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest \\
  --allow-privileged=true \\
  --alsologtostderr=true \\
  --logtostderr=false \\
  --log-dir=/var/log/kubernetes \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

  • 说明：
  • 如果设置了--hostname-override 选项，则 kube-proxy 也需要设置该选项，否则会出现找不到 Node 的情况
  • --bootstrap-kubeconfig：指向 bootstrap kubeconfig 文件，kubelet 使用该文件中的用户名和 token 向 kube-apiserver 发送 TLS Bootstrapping 请求
  • K8S approve kubelet 的 csr 请求后，在--cert-dir 目录创建证书和私钥文件，然后写入 --kubeconfig 文件

1.4.2.2 node02节点

cat > /usr/lib/systemd/system/kubelet.service <<EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \\
  --bootstrap-kubeconfig=/opt/kubernetes/cfg/kubelet-bootstrap.kubeconfig \\
  --cert-dir=/opt/kubernetes/ssl \\
  --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
  --config=/opt/kubernetes/cfg/kubelet.config.json \\
  --hostname-override=k8s-node02 \\
  --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest \\
  --allow-privileged=true \\
  --alsologtostderr=true \\
  --logtostderr=false \\
  --log-dir=/var/log/kubernetes \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

1.4.3 Bootstrap Token Auth 和授予权限

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers

  • 说明：
  • kublet 启动时查找配置的 --kubeletconfig 文件是否存在，如果不存在则使用 --bootstrap-kubeconfig 向 kube-apiserver 发送证书签名请求 (CSR)
  • kube-apiserver 收到 CSR 请求后，对其中的 Token 进行认证（事先使用 kubeadm 创建的 token），认证通过后将请求的 user 设置为 system:bootstrap:，group 设置为 system:bootstrappers，这一过程称为 Bootstrap Token Auth
  • 默认情况下，这个 user 和 group 没有创建 CSR 的权限，kubelet 启动失败，错误日志如下：

$ sudo journalctl -u kubelet -a |grep -A 2 'certificatesigningrequests'
May 06 06:42:36 kube-node1 kubelet[26986]: F0506 06:42:36.314378   26986 server.go:233] failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:bootstrap:lemy40" cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope
May 06 06:42:36 kube-node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
May 06 06:42:36 kube-node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.

1.4.4 启动kubelet

systemctl daemon-reload
systemctl enable kubelet
systemctl restart kubelet
systemctl status kubelet

1.4.5 检查kubelet状态

[[email protected] ~]# kubectl get csr
NAME                                                   AGE       REQUESTOR                 CONDITION
node-csr-Pa63hrEu3i0q-fXTzUbgIMq5XESsKbhdIjt9hJadsL4   1m        system:bootstrap:yviv83   Pending
node-csr-okE2Uika1TaEbY57V9cyIcAh6s-forX6jBlsDVm8sDY   1m        system:bootstrap:mp3y9m   Pending

# 手动approve方式：
# kubectl certificate approve node-csr-Pa63hrEu3i0q-fXTzUbgIMq5XESsKbhdIjt9hJadsL4
# kubectl describe  csr node-csr-Pa63hrEu3i0q-fXTzUbgIMq5XESsKbhdIjt9hJadsL4

[[email protected] ~]# kubectl get nodes
NAME             STATUS    ROLES     AGE       VERSION
192.168.10.163   Ready     <none>    5m        v1.10.4
192.168.10.164   Ready     <none>    1m        v1.10.4

提示：访问cAdvisor方法：http://192.168.10.163:4194/containers/

1.5 部署kube-proxy

1.5.1 所有node节点创建kubeconfig文件

cd /opt/kubernetes/cfg/

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.160:8443 \       # 此处为master节点的VIP
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy \
  --client-certificate=/opt/kubernetes/ssl/kube-proxy.pem \
  --client-key=/opt/kubernetes/ssl/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

1.5.2 编写kube-proxy配置文件

1.5.2.1 node01节点

cat > /opt/kubernetes/cfg/kube-proxy.config.yaml <<EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.10.163
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
clusterCIDR: 172.30.0.0/16
healthzBindAddress: 192.168.10.163:10256
hostnameOverride: k8s-node01
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.10.163:10249
mode: "ipvs"
EOF

1.5.2.2 node02节点

cat > /opt/kubernetes/cfg/kube-proxy.config.yaml <<EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.10.164
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
clusterCIDR: 172.30.0.0/16
healthzBindAddress: 192.168.10.164:10256
hostnameOverride: k8s-node02
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.10.164:10249
mode: "ipvs"
EOF

1.5.3 编写kube-proxy启动文件

cat > /usr/lib/systemd/system/kube-proxy.service <<EOF
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \\
  --config=/opt/kubernetes/cfg/kube-proxy.config.yaml \\
  --alsologtostderr=true \\
  --logtostderr=false \\
  --log-dir=/var/log/kubernetes \\
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

  • 说明：
  • bindAddress: 监听地址
  • kubeconfig: 连接 apiserver 的 kubeconfig 文件
  • clusterCIDR: 必须与 kube-controller-manager 的--cluster-cidr 选项值一致；kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量，指定 --cluster-cidr 或 --masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT
  • hostnameOverride: 参数值必须与 kubelet 的值一致，否则 kube-proxy 启动后会找不到该 Node，从而不会创建任何 ipvs 规则
  • mode: 使用 ipvs 模式

1.5.4 启动kube-proxy

systemctl daemon-reload
systemctl enable kube-proxy
systemctl restart kube-proxy
systemctl status kube-proxy

weinxin
我的微信
如果有技术上的问题可以扫一扫我的微信
版权声明
1. 本网站名称：Leon的博客
2. 本站永久网址：https://blog.leonshadow.cn
3. 本网站的文章部分内容可能来源于网络，仅供大家学习与参考，如有侵权，请联系站长QQ632113590进行删除处理。
4. 本站一切资源不代表本站立场，并不代表本站赞同其观点和对其真实性负责。
5. 本站一律禁止以任何方式发布或转载任何违法的相关信息，访客发现请向站长举报
6. 本站资源大多存储在云盘，如发现链接失效，请联系我们我们会第一时间更新。
liyang