k8s二进制部署系列08-部署k8s master节点

2018年7月16日13:29:43 发表评论 917 views

1.1 准备master环境

1.1.1 创建密钥key

head -c 32 /dev/urandom | base64
QzItwAKKlCclTRy+XDbZh53DcTSyjmfFFCfve/MSsgE=

1.1.2 创建加密配置文件

cat > /opt/kubernetes/cfg/encryption-config.yaml <<EOF
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: QzItwAKKlCclTRy+XDbZh53DcTSyjmfFFCfve/MSsgE=
      - identity: {}
EOF

1.1.3 创建k8s程序log目录

mkdir -p /var/log/kubernetes

1.2 部署apiserver

1.2.1 配置apiserver启动文件

1.2.1.1 master节点

cat > /usr/lib/systemd/system/kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
ExecStart=/opt/kubernetes/bin/kube-apiserver \\
  --enable-admission-plugins=Initializers,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
  --anonymous-auth=false \\
  --experimental-encryption-provider-config=/opt/kubernetes/cfg/encryption-config.yaml \\
  --advertise-address=192.168.10.161 \\
  --bind-address=192.168.10.161 \\
  --insecure-port=0 \\
  --authorization-mode=Node,RBAC \\
  --runtime-config=api/all \\
  --enable-bootstrap-token-auth \\
  --service-cluster-ip-range=10.254.0.0/16 \\
  --service-node-port-range=8400-9000 \\
  --tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \\
  --tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \\
  --client-ca-file=/opt/kubernetes/ssl/ca.pem \\
  --kubelet-client-certificate=/opt/kubernetes/ssl/kubernetes.pem \\
  --kubelet-client-key=/opt/kubernetes/ssl/kubernetes-key.pem \\
  --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
  --etcd-cafile=/opt/kubernetes/ssl/ca.pem \\
  --etcd-certfile=/opt/kubernetes/ssl/kubernetes.pem \\
  --etcd-keyfile=/opt/kubernetes/ssl/kubernetes-key.pem \\
  --etcd-servers=https://192.168.10.154:2379,https://192.168.10.155:2379,https://192.168.10.156:2379 \\
  --enable-swagger-ui=true \\
  --allow-privileged=true \\
  --apiserver-count=2 \\
  --audit-log-maxage=30 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-path=/var/log/kube-apiserver-audit.log \\
  --event-ttl=1h \\
  --alsologtostderr=true \\
  --logtostderr=false \\
  --log-dir=/var/log/kubernetes \\
  --v=2
Restart=on-failure
RestartSec=5
Type=notify
#User=k8s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

  • 说明：
  • --experimental-encryption-provider-config：启用加密特性
  • --authorization-mode=Node,RBAC： 开启 Node 和 RBAC 授权模式，拒绝未授权的请求
  • --enable-admission-plugins：启用ServiceAccount 和 NodeRestriction
  • --service-account-key-file：签名 ServiceAccount Token 的公钥文件，kube-controller-manager 的--service-account-private-key-file 指定私钥文件，两者配对使用
  • --tls-*-file：指定 apiserver 使用的证书、私钥和 CA 文件。--client-ca-file 用于验证 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)请求所带的证书
  • --kubelet-client-certificate、--kubelet-client-key：如果指定，则使用 https 访问 kubelet APIs；需要为证书对应的用户(上面 kubernetes*.pem 证书的用户为 kubernetes) 用户定义 RBAC 规则，否则访问 kubelet API 时提示未授权
  • --bind-address： 不能为0.0.1，否则外界不能访问它的安全端口 6443
  • --insecure-port=0：关闭监听非安全端口(8080)
  • --service-cluster-ip-range： 指定 Service Cluster IP 地址段
  • --service-node-port-range： 指定 NodePort 的端口范围
  • --runtime-config=api/all=true： 启用所有版本的 APIs，如 autoscaling/v2alpha1
  • --enable-bootstrap-token-auth：启用 kubelet bootstrap 的 token 认证
  • --apiserver-count=3：指定集群运行模式，多台 kube-apiserver 会通过 leader 选举产生一个工作节点，其它节点处于阻塞状态
  • User=k8s：使用 k8s 账户运行

1.2.1.2 slave节点

cat > /usr/lib/systemd/system/kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
ExecStart=/opt/kubernetes/bin/kube-apiserver \\
  --enable-admission-plugins=Initializers,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
  --anonymous-auth=false \\
  --experimental-encryption-provider-config=/opt/kubernetes/cfg/encryption-config.yaml \\
  --advertise-address=192.168.10.162 \\
  --bind-address=192.168.10.162 \\
  --insecure-port=0 \\
  --authorization-mode=Node,RBAC \\
  --runtime-config=api/all \\
  --enable-bootstrap-token-auth \\
  --service-cluster-ip-range=10.254.0.0/16 \\
  --service-node-port-range=8400-9000 \\
  --tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \\
  --tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \\
  --client-ca-file=/opt/kubernetes/ssl/ca.pem \\
  --kubelet-client-certificate=/opt/kubernetes/ssl/kubernetes.pem \\
  --kubelet-client-key=/opt/kubernetes/ssl/kubernetes-key.pem \\
  --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
  --etcd-cafile=/opt/kubernetes/ssl/ca.pem \\
  --etcd-certfile=/opt/kubernetes/ssl/kubernetes.pem \\
  --etcd-keyfile=/opt/kubernetes/ssl/kubernetes-key.pem \\
  --etcd-servers=https://192.168.10.154:2379,https://192.168.10.155:2379,https://192.168.10.156:2379 \\
  --enable-swagger-ui=true \\
  --allow-privileged=true \\
  --apiserver-count=2 \\
  --audit-log-maxage=30 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-path=/var/log/kube-apiserver-audit.log \\
  --event-ttl=1h \\
  --alsologtostderr=true \\
  --logtostderr=false \\
  --log-dir=/var/log/kubernetes \\
  --v=2
Restart=on-failure
RestartSec=5
Type=notify
#User=k8s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

1.2.2 启动apiserver

systemctl daemon-reload
systemctl enable kube-apiserver
systemctl restart kube-apiserver
systemctl status kube-apiserver

1.2.3 授予 kubernetes 证书访问 kubelet API 的权限

kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

  • 说明：
  • 在执行 kubectl exec、run、logs 等命令时，apiserver 会转发到 kubelet。这里定义 RBAC 规则，授权 apiserver 调用 kubelet API

1.2.4 检查apiserver状态

[[email protected] ~]# kubectl cluster-info
Kubernetes master is running at https://192.168.10.160:8443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

1.3 部署controller-manager

1.3.1 创建和分发 kubeconfig 文件

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.160:8443 \        # 此处为master节点的VIP
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=/opt/kubernetes/ssl/kube-controller-manager.pem \
  --client-key=/opt/kubernetes/ssl/kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager \
  --cluster=kubernetes \
  --user=system:kube-controller-manager \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

scp kube-controller-manager.kubeconfig 192.168.10.161:/opt/kubernetes/cfg/
scp kube-controller-manager.kubeconfig 192.168.10.162:/opt/kubernetes/cfg/

1.3.2 配置controller-manager启动文件

cat > /usr/lib/systemd/system/kube-controller-manager.service <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/opt/kubernetes/bin/kube-controller-manager \\
  --port=0 \\
  --secure-port=10252 \\
  --bind-address=127.0.0.1 \\
  --kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\
  --service-cluster-ip-range=10.254.0.0/16 \\
  --cluster-name=kubernetes \\
  --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
  --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\
  --experimental-cluster-signing-duration=8760h \\
  --root-ca-file=/opt/kubernetes/ssl/ca.pem \\
  --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
  --leader-elect=true \\
  --feature-gates=RotateKubeletServerCertificate=true \\
  --controllers=*,bootstrapsigner,tokencleaner \\
  --horizontal-pod-autoscaler-use-rest-clients=true \\
  --horizontal-pod-autoscaler-sync-period=10s \\
  --tls-cert-file=/opt/kubernetes/ssl/kube-controller-manager.pem \\
  --tls-private-key-file=/opt/kubernetes/ssl/kube-controller-manager-key.pem \\
  --use-service-account-credentials=true \\
  --alsologtostderr=true \\
  --logtostderr=false \\
  --log-dir=/var/log/kubernetes \\
  --v=2
Restart=on
Restart=on-failure
RestartSec=5
#User=k8s

[Install]
WantedBy=multi-user.target
EOF

  • 说明：
  • --port=0：关闭监听 http /metrics 的请求，同时--address 参数无效，--bind-address 参数有效；
  • --secure-port=10252、--bind-address=0.0.0.0: 在所有网络接口监听 10252 端口的 https /metrics 请求
  • --kubeconfig：指定 kubeconfig 文件路径，kube-controller-manager 使用它连接和验证 kube-apiserver
  • --cluster-signing-*-file：签名 TLS Bootstrap 创建的证书
  • --experimental-cluster-signing-duration：指定 TLS Bootstrap 证书的有效期
  • --root-ca-file：放置到容器 ServiceAccount 中的 CA 证书，用来对 kube-apiserver 的证书进行校验
  • --service-account-private-key-file：签名 ServiceAccount 中 Token 的私钥文件，必须和 kube-apiserver 的--service-account-key-file 指定的公钥文件配对使用
  • --service-cluster-ip-range ：指定 Service Cluster IP 网段，必须和 kube-apiserver 中的同名参数一致
  • --leader-elect=true：集群运行模式，启用选举功能；被选为 leader 的节点负责处理工作，其它节点为阻塞状态
  • --feature-gates=RotateKubeletServerCertificate=true：开启 kublet server 证书的自动更新特性
  • --controllers=*,bootstrapsigner,tokencleaner：启用的控制器列表，tokencleaner 用于自动清理过期的 Bootstrap token
  • --horizontal-pod-autoscaler-*：custom metrics 相关参数，支持 autoscaling/v2alpha1
  • --tls-cert-file、--tls-private-key-file：使用 https 输出 metrics 时使用的 Server 证书和秘钥
  • --use-service-account-credentials=true:
  • User=k8s：使用 k8s 账户运行
  • kube-controller-manager 不对请求 https metrics 的 Client 证书进行校验，故不需要指定--tls-ca-file 参数，而且该参数已被淘汰

1.3.3 启动controller-manager

systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl restart kube-controller-manager
systemctl status kube-controller-manager

1.3.4 检查controller-manager状态

[[email protected] ~]# curl -s --cacert /opt/kubernetes/ssl/ca.pem https://127.0.0.1:10252/metrics |head
# HELP ClusterRoleAggregator_adds Total number of adds handled by workqueue: ClusterRoleAggregator
# TYPE ClusterRoleAggregator_adds counter
ClusterRoleAggregator_adds 3
# HELP ClusterRoleAggregator_depth Current depth of workqueue: ClusterRoleAggregator
# TYPE ClusterRoleAggregator_depth gauge
ClusterRoleAggregator_depth 0
# HELP ClusterRoleAggregator_queue_latency How long an item stays in workqueueClusterRoleAggregator before being requested.
# TYPE ClusterRoleAggregator_queue_latency summary
ClusterRoleAggregator_queue_latency{quantile="0.5"} 84683
ClusterRoleAggregator_queue_latency{quantile="0.9"} 85023

[[email protected] ~]# kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master_d78aa99e-8572-11e8-8d75-52540042d161","leaseDurationSeconds":15,"acquireTime":"2018-07-12T01:28:31Z","renewTime":"2018-07-12T01:30:26Z","leaderTransitions":1}'
  creationTimestamp: 2018-07-12T01:27:38Z
  name: kube-controller-manager
  namespace: kube-system
  resourceVersion: "308"
  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
  uid: c352457a-8572-11e8-b346-52540042d162

1.4 部署scheduler

1.4.1 创建和分发 kubeconfig 文件

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.160:8443 \     # 此处为master节点的VIP
  --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler \
  --client-certificate=/opt/kubernetes/ssl/kube-scheduler.pem \
  --client-key=/opt/kubernetes/ssl/kube-scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-context system:kube-scheduler \
  --cluster=kubernetes \
  --user=system:kube-scheduler \
  --kubeconfig=kube-scheduler.kubeconfig

kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

scp kube-scheduler.kubeconfig 192.168.10.161:/opt/kubernetes/cfg/
scp kube-scheduler.kubeconfig 192.168.10.162:/opt/kubernetes/cfg/

1.4.2 配置scheduler启动文件

cat > /usr/lib/systemd/system/kube-scheduler.service <<EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/opt/kubernetes/bin/kube-scheduler \\
  --address=127.0.0.1 \\
  --kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\
  --leader-elect=true \\
  --alsologtostderr=true \\
  --logtostderr=false \\
  --log-dir=/var/log/kubernetes \\
  --v=2
Restart=on-failure
RestartSec=5
#User=k8s

[Install]
WantedBy=multi-user.target
EOF

  • 说明：
  • --address：在0.0.1:10251 端口接收 http /metrics 请求；kube-scheduler 目前还不支持接收 https 请求
  • --kubeconfig：指定 kubeconfig 文件路径，kube-scheduler 使用它连接和验证 kube-apiserver
  • --leader-elect=true：集群运行模式，启用选举功能；被选为 leader 的节点负责处理工作，其它节点为阻塞状态
  • User=k8s：使用 k8s 账户运行

1.4.3 启动scheduler

systemctl daemon-reload
systemctl enable kube-scheduler
systemctl restart kube-scheduler
systemctl status kube-scheduler

1.4.4 检查scheduler状态

[[email protected] ~]# curl -s http://127.0.0.1:10251/metrics |head
# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 7.5091e-05
go_gc_duration_seconds{quantile="0.25"} 8.3633e-05
go_gc_duration_seconds{quantile="0.5"} 0.000114825
go_gc_duration_seconds{quantile="0.75"} 0.000262873
go_gc_duration_seconds{quantile="1"} 0.000870801

[[email protected] ~]# kubectl get endpoints kube-scheduler --namespace=kube-system  -o yaml
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master_8b241fd1-8574-11e8-8781-52540042d161","leaseDurationSeconds":15,"acquireTime":"2018-07-12T01:40:24Z","renewTime":"2018-07-12T01:41:22Z","leaderTransitions":0}'
  creationTimestamp: 2018-07-12T01:40:23Z
  name: kube-scheduler
  namespace: kube-system
  resourceVersion: "667"
  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-scheduler
  uid: 8b5bc785-8574-11e8-b346-52540042d162

1.5 配置自动approve kubelet CSR 请求

提示：仅在master节点执行一次即可。

cat > /opt/kubernetes/cfg/csr-crb.yaml <<EOF
 # Approve all CSRs for the group "system:bootstrappers"
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: auto-approve-csrs-for-group
 subjects:
 - kind: Group
   name: system:bootstrappers
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
   apiGroup: rbac.authorization.k8s.io
---
 # To let a node of the group "system:nodes" renew its own credentials
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: node-client-cert-renewal
 subjects:
 - kind: Group
   name: system:nodes
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
   apiGroup: rbac.authorization.k8s.io
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/selfnodeserver"]
  verbs: ["create"]
---
 # To let a node of the group "system:nodes" renew its own server credentials
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: node-server-cert-renewal
 subjects:
 - kind: Group
   name: system:nodes
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: approve-node-server-renewal-csr
   apiGroup: rbac.authorization.k8s.io
EOF

kubectl apply -f /opt/kubernetes/cfg/csr-crb.yaml

  • 说明：
  • kubelet 启动后使用 --bootstrap-kubeconfig 向 kube-apiserver 发送 CSR 请求，当这个 CSR 被 approve 后，kube-controller-manager 为 kubelet 创建 TLS 客户端证书、私钥和 --kubeletconfig 文件。
  • 注意：kube-controller-manager 需要配置--cluster-signing-cert-file 和 --cluster-signing-key-file参数，才会为 TLS Bootstrap 创建证书和私钥。
weinxin
我的微信
如果有技术上的问题可以扫一扫我的微信
版权声明
1. 本网站名称：Leon的博客
2. 本站永久网址：https://blog.leonshadow.cn
3. 本网站的文章部分内容可能来源于网络，仅供大家学习与参考，如有侵权，请联系站长QQ632113590进行删除处理。
4. 本站一切资源不代表本站立场，并不代表本站赞同其观点和对其真实性负责。
5. 本站一律禁止以任何方式发布或转载任何违法的相关信息，访客发现请向站长举报
6. 本站资源大多存储在云盘，如发现链接失效，请联系我们我们会第一时间更新。
liyang