k8s自动伸缩

发表于 2020-10-17 分类于 Kubernetes

讲解 Kubernetes 自动伸缩，涵盖 HPA 水平 Pod 伸缩的创建验证与 Cluster Autoscaler 节点伸缩。

HPA : Horizontal Pod Autoscaling, Pod 水平自动伸缩
KPA : Knative Pod Autoscaler, 基于请求数对 Pod 自动扩缩容，KPA 的主要限制在于它不支持基于 CPU 的自动扩缩容。
VPA : Vertical Pod Autoscaler, 垂直 Pod 自动扩缩容，VPA 会基于 Pod 的资源使用情况自动为集群设置资源占用的限制
CA : Cluster Autoscaler 是一个独立程序，是用来弹性伸缩 kubernetes 集群的, 一般云厂商都有这个功能

HPA

Pod 水平自动伸缩（Horizontal Pod Autoscaler，HPA）可以根据应用的 CPU 利用率, 内存占用率等水位信息，动态地增加或者减少 Pod 副本数量。

使用 HPA 时，要提前部署好 metrics-server. (也可以直接使用 Custom Metrics 来执行用户指定的扩展策略，这里的整个过程都是非常灵活和可定制的。)

补
Kubernetes 里的 Custom Metrics 机制，也是借助 Aggregator APIServer 扩展机制来实现的。这里的具体原理是，当你把 Custom Metrics APIServer 启动之后，Kubernetes 里就会出现一个叫作 custom.metrics.k8s.io 的 API。而当你访问这个 URL 时，Aggregator 就会把你的请求转发给 Custom Metrics APIServer 。（APIServer --enable-aggregator-routing=true 详细待补充）

wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml （需要修改）

[root@k8s01 ~]# kubectl apply -f components.yaml
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
Warning: apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.apps/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created

[root@k8s01 ~]# kubectl get deploy -n kube-system metrics-server
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   1/1     1            1           26s


# 创建 deployment
[root@k8s01 ~]#  kubectl apply -f myapp-deploy-hpa.yaml
deployment.apps/nginx-deployment created

# 暴露服务
[root@k8s01 ~]# kubectl expose deployment/myapp-deployment -n demo
service/myapp-deployment exposed

[root@k8s01 ~]# kubectl get svc -n demo
NAMESPACE     NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
demo          myapp-deployment   ClusterIP   10.104.238.121   <none>        80/TCP                   7s

[root@k8s01 ~]# kubectl get endpoints -n demo   # 获取endpoints 对象
NAME                ENDPOINTS             AGE
myapp-deployment    10.244.1.12:80,10.244.2.13:80     20s

# 创建 hpa
[root@k8s01 ~]# kubectl autoscale deploy myapp-deployment -n demo --cpu-percent=50 --min=1 --max=10
horizontalpodautoscaler.autoscaling/myapp-deployment autoscaled

myapp-deploy-hpa.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
  namespace: demo
spec:
  selector:
    matchLabels:
      app: myapp
      usage: hpa
  replicas: 1
  template:
    metadata:
      labels:
        app: myapp
        usage: hpa
    spec:
      containers:
      - name: myapp
        image: imwl/myapp:v1
        ports:
        - containerPort: 80
        resources:
          requests: # 把quota设置得小一点，方便做压力测试
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"

components.yaml

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:aggregated-metrics-reader
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: imwl/metrics-server:v0.3.7   # 修改
        imagePullPolicy: IfNotPresent
        args:
          - /metrics-server
          - --cert-dir=/tmp
          - --secure-port=4443
          - --kubelet-insecure-tls  ## 增加
          - --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname ## 增加
        ports:
        - name: main-port
          containerPort: 4443
          protocol: TCP
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
      nodeSelector:
        kubernetes.io/os: linux
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: main-port
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

创建内存 hpa

[root@k8s01 ~]# cat hpa-memory.yaml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa
spec:
  maxReplicas: 10
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx-hpa
  metrics:
  - type: Resource
    resource:
      name: memory
      targetAverageUtilization: 60

[root@k8s01 ~]# kubectl apply -f hpa-memory.yaml
horizontalpodautoscaler.autoscaling/nginx-hpa created
# 查看创建的 hpa
[root@xianchaomaster1 ~]# kubectl get hpa

验证 hpa

开三个窗口

用压测工具ApacheBench 进行压力测试

[root@k8s01 ~]#  kubectl run demo-benchmark --image httpd:2.4.46-alpine -n demo -it sh
If you don't see a command prompt, try pressing enter.
/usr/local/apache2 # ab -n 50000 -c 500 -s 60 http://10.104.238.121/
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.104.238.121 (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests

Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Completed 50000 requests
Finished 50000 requests


Server Software:        nginx/1.12.2
Server Hostname:        10.104.238.121
Server Port:            80

Document Path:          /
Document Length:        65 bytes

Concurrency Level:      500
Time taken for tests:   75.398 seconds
Complete requests:      50000
Failed requests:        0
Total transferred:      14800000 bytes
HTML transferred:       3250000 bytes
Requests per second:    663.15 [#/sec] (mean)
Time per request:       753.976 [ms] (mean)
Time per request:       1.508 [ms] (mean, across all concurrent requests)
Transfer rate:          191.69 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0  741 478.4   1003    3013
Processing:     0    7  24.2      1    1645
Waiting:        0    7  24.1      1    1644
Total:          0  748 471.2   1004    3214

Percentage of the requests served within a certain time (ms)
  50%   1004
  66%   1004
  75%   1005
  80%   1005
  90%   1006
  95%   1006
  98%   1008
  99%   1013
 100%   3214 (longest request)
/usr/local/apache2 #

hpa 的信息如下

[root@k8s01 ~]# kubectl get hpa -n demo -w
NAME               REFERENCE                     TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
myapp-deployment   Deployment/myapp-deployment   0%/50%    1         10        1          21m
myapp-deployment   Deployment/myapp-deployment   72%/50%   1         10        1          21m
myapp-deployment   Deployment/myapp-deployment   72%/50%   1         10        2          22m
myapp-deployment   Deployment/myapp-deployment   64%/50%   1         10        2          22m
myapp-deployment   Deployment/myapp-deployment   64%/50%   1         10        3          23m
myapp-deployment   Deployment/myapp-deployment   0%/50%    1         10        3          23m
myapp-deployment   Deployment/myapp-deployment   0%/50%    1         10        3          28m
myapp-deployment   Deployment/myapp-deployment   0%/50%    1         10        1          28m

deployment 信息如下

[root@k8s01 ~]# kubectl get deploy -n demo -w
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
myapp-deployment   1/1     1            1           1s
myapp-deployment   1/2     1            1           3m43s
myapp-deployment   1/2     1            1           3m43s
myapp-deployment   1/2     1            1           3m43s
myapp-deployment   1/2     2            1           3m43s
myapp-deployment   2/2     2            2           4m4s
myapp-deployment   2/3     2            2           4m44s
myapp-deployment   2/3     2            2           4m44s
myapp-deployment   2/3     2            2           4m44s
myapp-deployment   2/3     3            2           4m44s
myapp-deployment   3/3     3            3           4m46s  # 很长一段时间才降下来，防止意外
myapp-deployment   3/1     3            3           10m
myapp-deployment   3/1     3            3           10m
myapp-deployment   1/1     1            1           10m

Cluster Autoscaler

CA 主要用来监听（watch）集群中未被调度的 Pod （即 Pod 暂时由于某些调度策略、抑或资源不满足，导致无法被成功调度），然后确定是否可以通过增加节点资源来解决无法调度的问题。

如果可以的话，就会调用对应的 cloud provider 接口，向集群中增加新的节点。当然 CA 在创建新的节点资源前，也会尝试是否可以将正在运行的一部分 Pod “挤压”到某些节点上，从而让这些未被调度的 Pod 可以被调度，如果可行的话，CA 会将这些 Pod 进行驱逐

后续

除了 HPA 和 CA 以外，还有 Vertical Pod Autoscaler (VPA)可以帮我们确定 Pod 中合适的 CPU 和 Memory 区间。在实际使用的时候，注意千万不要同时使用 HPA 和 VPA，以免造成异常。

使用 HPA 的时候，也尽量对 Deployment 这类对象进行操作，避免对 ReplicaSet 操作。毕竟 ReplicaSet 由 Deployment 管理着，一旦 Deployment 更新了，旧的 ReplicaSet 会被新的 ReplicaSet 替换掉。