configMap

发表于 分类于
详解 ConfigMap 的多种创建与使用方式及热更新机制，以及 DownwardAPI 注入 Pod 元数据的方法。

configMap

ConfigMap : 许多应用程序会从 配置文件 、 命令行参数 或 环境变量 中读取配置信息。

ConfigMap API : 给我们提供了向容器中注入配置信息的机制，可以被用来保存单个属性，也可以用来保存整个配置文件或者 JSON 二进制大对象

configmap 注入方式有两种

  1. configMap 做为存储卷
  2. 将 configMap 通过 env 中 configMapKeyRef 注入到容器中。

ConfigMap 的创建

使用目录创建

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
$ mkdir configMap
$ echo "enemies=aliens
lives=3
enemies.cheat=true
enemies.cheat.level=noGoodRotten
secret.code.passphrase=UUDDLRLRBABAS
secret.code.allowed=true
secret.code.lives=30" >  configMap/game.properties

$ echo "color.good=purple
color.bad=yellow
allow.textmode=true
how.nice.to.look=fairlyNice"  > configMap/ui.properties

[root@k8s01 storage]# kubectl create configmap game-config --from-file=configMap/
configmap/game-config created
[root@k8s01 storage]# kubectl describe cm game-config
Name:         game-config
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
game.properties:
----
enemies=aliens
lives=3
enemies.cheat=true
enemies.cheat.level=noGoodRotten
secret.code.passphrase=UUDDLRLRBABAS
secret.code.allowed=true
secret.code.lives=30

ui.properties:
----
color.good=purple
color.bad=yellow
allow.textmode=true
how.nice.to.look=fairlyNice

Events:  <none>

--from-file 指定目录下的所有文件都会被用在 ConfigMap 里面创建一个键值对， 键的名字就是文件名，值就是文件的内容

使用文件创建

只要指定为一个文件就可以从单个文件中创建 ConfigMap

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
[root@k8s01 storage]# kubectl create configmap game-config-2 --from-file=configMap/game.properties  --from-file=configMap/ui.properties
configmap/game-config-2 created
[root@k8s01 storage]# kubectl get configmaps game-config-2 -o yaml
apiVersion: v1
data:
  game.properties: |
    enemies=aliens
    lives=3
    enemies.cheat=true
    enemies.cheat.level=noGoodRotten
    secret.code.passphrase=UUDDLRLRBABAS
    secret.code.allowed=true
    secret.code.lives=30
  ui.properties: |
    color.good=purple
    color.bad=yellow
    allow.textmode=true
    how.nice.to.look=fairlyNice
kind: ConfigMap
metadata:
  creationTimestamp: "2020-08-27T15:59:05Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:game.properties: {}
        f:ui.properties: {}
    manager: kubectl
    operation: Update
    time: "2020-08-27T15:59:05Z"
  name: game-config-2
  namespace: default
  resourceVersion: "235477"
  selfLink: /api/v1/namespaces/default/configmaps/game-config-2
  uid: c283cbf9-1eeb-49d8-8a1e-999099f4e97f

--from-file 这个参数可以使用多次，你可以使用两次分别指定上个实例中的那两个配置文件，效果就跟指定整个目录是一样的

使用字面值创建

使用文字值创建，利用 --from-literal 参数传递配置信息，该参数可以使用多次，格式如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
[root@k8s01 storage]# kubectl create configmap special-config --from-literal=special.how=very --from-literal=special.type=charm
configmap/special-config created
[root@k8s01 storage]# kubectl get configmaps special-config -o yaml
apiVersion: v1
data:
  special.how: very
  special.type: charm
kind: ConfigMap
metadata:
  creationTimestamp: "2020-08-27T16:00:02Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:special.how: {}
        f:special.type: {}
    manager: kubectl
    operation: Update
    time: "2020-08-27T16:00:02Z"
  name: special-config
  namespace: default
  resourceVersion: "235621"
  selfLink: /api/v1/namespaces/default/configmaps/special-config
  uid: 7ab83245-87dc-41eb-be95-584d68ed259e
[root@k8s01 storage]# kubectl describe cm special-config
Name:         special-config
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
special.how:
----
very
special.type:
----
charm
Events:  <none>
1
2
3
4
[root@k8s01 storage]# kubectl delete cm --all
configmap "game-config" deleted
configmap "game-config-2" deleted
configmap "special-config" deleted

Pod 中使用 ConfigMap

使用 ConfigMap 来替代环境变量

special-config.yaml

1
2
3
4
5
6
7
8
apiVersion: v1
kind: ConfigMap
metadata:
  name: special-config
  namespace: default
data:
  special.how: very
  special.type: charm

env-config.yaml

1
2
3
4
5
6
7
apiVersion: v1
kind: ConfigMap
metadata:
  name: env-config
  namespace: default
data:
  log_level: INFO

test-pod01.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
apiVersion: v1
kind: Pod
metadata:
  name: test-pod01
spec:
  containers:
  - name: test-container
    image: nginx:1.7.9
    command: ["/bin/sh","-c","env"]
    env:
    - name: SPECIAL_LEVEL_KEY
      valueFrom:
        configMapKeyRef:
          name: special-config
          key: special.how
    - name: SPECIAL_TYPE_KEY
      valueFrom:
        configMapKeyRef:
          name: special-config
          key: special.type
    envFrom:
    - configMapRef:
        name: env-config
  restartPolicy: Never
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[root@k8s01 storage]# kubectl apply -f special-config.yaml
configmap/special-config created
[root@k8s01 storage]# kubectl apply -f env-config.yaml
configmap/env-config created
[root@k8s01 storage]# kubectl logs test-pod  # 已读取 configMap
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT=443
NGINX_SVC_SERVICE_HOST=10.109.24.218
HOSTNAME=test-pod
HOME=/root
NGINX_SVC_PORT=tcp://10.109.24.218:80
NGINX_SVC_SERVICE_PORT=80
SPECIAL_TYPE_KEY=charm  # charm
NGINX_SVC_PORT_80_TCP_ADDR=10.109.24.218
NGINX_SVC_PORT_80_TCP_PORT=80
NGINX_SVC_PORT_80_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
NGINX_VERSION=1.7.9-1~wheezy
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
NGINX_SVC_PORT_80_TCP=tcp://10.109.24.218:80
SPECIAL_LEVEL_KEY=very  # very
log_level=INFO  # INFO
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_SERVICE_HOST=10.96.0.1
PWD=/

通过环境变量引入：使用 envfrom

用 ConfigMap 设置命令行参数

test-pod02.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: v1
kind: Pod
metadata:
  name: test-pod02
spec:
  containers:
  - name: test-container
    image: nginx:1.7.9
    command: ["/bin/sh","-c","echo $(SPECIAL_LEVEL_KEY) $(SPECIAL_TYPE_KEY)"]
    env:
    - name: SPECIAL_LEVEL_KEY
      valueFrom:
        configMapKeyRef:
          name: special-config
          key: special.how
    - name: SPECIAL_TYPE_KEY
      valueFrom:
        configMapKeyRef:
          name: special-config
          key: special.type
  restartPolicy: Never
1
2
3
4
5
6
7
8
[root@k8s01 storage]# kubectl apply -f test-pod02.yaml
pod/test-pod02 created
[root@k8s01 storage]# kubectl get pod
NAME         READY   STATUS      RESTARTS   AGE
test-pod     0/1     Completed   0          10m
test-pod02   0/1     Completed   0          13s
[root@k8s01 storage]# kubectl logs test-pod02
very charm

通过数据卷插件使用ConfigMap

在数据卷里面使用这个 ConfigMap，有不同的选项。最基本的就是将文件填入数据卷，在这个文件中，键就是文件名，键值就是文件内容
test_pod03.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: v1
kind: Pod
metadata:
  name: test-pod03
spec:
  containers:
  - name: test-container
    image: nginx:1.7.9
    command: ["/bin/sh","-c","sleep 600s"]
    volumeMounts:
    - name: config-volume
      mountPath: /etc/config
  volumes:
  - name: config-volume
    configMap:
      name: special-config
  restartPolicy: Never
1
2
3
4
5
[root@k8s01 storage]# kubectl exec test-pod03 -it -- /bin/bash
root@test-pod03:/# cat /etc/config/special.how  # 文件名
very # 文件内容
root@test-pod03:/# cat /etc/config/special.type
charm

ConfigMap 的热更新

test04.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
apiVersion: v1
kind: ConfigMap
metadata:
  name: log-config
  namespace: default
data:
  log_level: INFO
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx-test
spec:
  replicas: 1
  selector:
    matchLabels:          # 必须缩进在 selector 下面；写成同级会变成 spec 下的未知字段，
      run: my-nginx       # 而 selector 为空，apply 直接被拒：selector: Required value
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx:1.7.9
        ports:
          - containerPort: 80
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
      volumes:
      - name: config-volume
        configMap:
          name: log-config
1
2
3
4
[root@k8s01 storage]# kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2
my-nginx-test-59bf4fc6f8-lg2s2
[root@k8s01 storage]# kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` -it -- cat /etc/config/log_level
INFO  # 文件 /etc/config/log_level 的值为 INFO

修改 ConfigMap

1
2
3
4
5
6
[root@k8s01 storage]# kubectl edit configmap log-config  # 修改log_level的值为DEBUG等待大概 10 秒钟时间，再次查看环境变量的值
configmap/log-config edited
[root@k8s01 storage]# kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` -it -- cat /etc/config/log_level
INFO  # 刚开始没变
[root@k8s01 storage]# kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` -it -- cat /etc/config/log_level
DEBUG  # 后来变成 DEBUG

ConfigMap 更新后滚动更新 Pod
更新 ConfigMap 目前并不会触发相关 Pod 的滚动更新，可以通过修改 pod annotations 的方式强制触发滚动更新

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
[root@k8s01 storage]# kubectl describe deployment my-nginx-test
Name:                   my-nginx-test
Namespace:              default
CreationTimestamp:      Thu, 27 Aug 2020 13:07:56 -0400
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               run=my-nginx
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  run=my-nginx
  Containers:
   my-nginx:
    Image:        nginx:1.7.9
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:
      /etc/config from config-volume (rw)
  Volumes:
   config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      log-config
    Optional:  false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   my-nginx-test-59bf4fc6f8 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  37s   deployment-controller  Scaled up replica set my-nginx-test-59bf4fc6f8 to 1



[root@k8s01 storage]# kubectl patch deployment my-nginx-test --patch '{"spec": {"template": {"metadata": {"annotations":{"version/config": "20200826" }}}}}'
deployment.apps/my-nginx-test patched


[root@k8s01 storage]# kubectl describe deployment my-nginx-test
Name:                   my-nginx-test
Namespace:              default
CreationTimestamp:      Thu, 27 Aug 2020 12:50:33 -0400
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 2
Selector:               run=my-nginx
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:       run=my-nginx
  Annotations:  version/config: 20200826
  Containers:
   my-nginx:
    Image:        nginx:1.7.9
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:
      /etc/config from config-volume (rw)
  Volumes:
   config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      log-config
    Optional:  false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   my-nginx-test-74f5cf4498 (1/1 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  15m    deployment-controller  Scaled up replica set my-nginx-test-59bf4fc6f8 to 1
  Normal  ScalingReplicaSet  3m21s  deployment-controller  Scaled up replica set my-nginx-test-74f5cf4498 to 1
  Normal  ScalingReplicaSet  3m18s  deployment-controller  Scaled down replica set my-nginx-test-59bf4fc6f8 to 0

这个例子里我们在 .spec.template.metadata.annotations 中添加 version/config，每次通过修改 version/config 来触发滚动更新

！！！更新 ConfigMap 后：

  1. 使用该 ConfigMap 挂载的 Env 不会同步更新
  2. 使用该 ConfigMap 挂载的 Volume 中的数据需要一段时间（实测大概10秒）才能同步更新

DownwardAPI

DownwardAPI 可以获取 Pod 对象中定义的字段，比如 Pod 的标签（Labels）、Pod 的 IP 地址及 Pod 所在的命名空间（namespace）等。

Downward API 有两种使用方法，既支持环境变量注入，也支持通过 Volume 挂载。

DownwardAPI 使用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
[root@k8s01 stroage]# cat downwardapi-volume-demo.yaml
apiVersion: v1
kind: Pod
metadata:
  name: downwardapi-volume-demo
  namespace: demo
  labels:
    zone: us-east-coast
    cluster: downward-api-test-cluster1
    rack: rack-123
  annotations:
    annotation1: "345"
    annotation2: "456"
spec:
  containers:
    - name: volume-test-container
      image: busybox:1.28
      command: ["sh", "-c"]
      args:
      - while true; do
          if [[ -e /etc/podinfo/labels ]]; then
            echo -en '\n\n'; cat /etc/podinfo/labels; fi;
          if [[ -e /etc/podinfo/annotations ]]; then
            echo -en '\n\n'; cat /etc/podinfo/annotations; fi;
          sleep 5;
        done;
      volumeMounts:
        - name: podinfo
          mountPath: /etc/podinfo
  volumes:
    - name: podinfo
      downwardAPI:
        items:
          - path: "labels"
            fieldRef:
              fieldPath: metadata.labels
          - path: "annotations"
            fieldRef:
              fieldPath: metadata.annotations

查看

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
[root@k8s01 stroage]# kubectl apply -f downwardapi-volume-demo.yaml
pod/downwardapi-volume-demo created

[root@k8s01 stroage]# kubectl logs -n demo -f downwardapi-volume-demo


cluster="downward-api-test-cluster1"
rack="rack-123"
zone="us-east-coast"

annotation1="345"
annotation2="456"
kubectl.kubernetes.io/last-applied-configuration="{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{\"annotation1\":\"345\",\"annotation2\":\"456\"},\"labels\":{\"cluster\":\"downward-api-test-cluster1\",\"rack\":\"rack-123\",\"zone\":\"us-east-coast\"},\"name\":\"downwardapi-volume-demo\",\"namespace\":\"demo\"},\"spec\":{\"containers\":[{\"args\":[\"while true; do if [[ -e /etc/podinfo/labels ]]; then echo -en '\\\\n\\\\n'; cat /etc/podinfo/labels; fi; if [[ -e /etc/podinfo/annotations ]]; then echo -en '\\\\n\\\\n'; cat /etc/podinfo/annotations; fi; sleep 5; done;\"],\"command\":[\"sh\",\"-c\"],\"image\":\"busybox:1.28\",\"name\":\"volume-test-container\",\"volumeMounts\":[{\"mountPath\":\"/etc/podinfo\",\"name\":\"podinfo\"}]}],\"volumes\":[{\"downwardAPI\":{\"items\":[{\"fieldRef\":{\"fieldPath\":\"metadata.labels\"},\"path\":\"labels\"},{\"fieldRef\":{\"fieldPath\":\"metadata.annotations\"},\"path\":\"annotations\"}]},\"name\":\"podinfo\"}]}}\n"
kubernetes.io/config.seen="2021-01-30T22:50:35.136835626+08:00"
kubernetes.io/config.source="api"


[root@k8s01 stroage]# kubectl exec -n demo  downwardapi-volume-demo  -it  -- /bin/sh
/ # ls
bin   dev   etc   home  proc  root  sys   tmp   usr   var
/ # cd  /etc/podinfo
/etc/podinfo # ls
annotations  labels
/etc/podinfo # cat labels
cluster="downward-api-test-cluster1"
rack="rack-123"
/etc/podinfo # cat annotations
annotation1="345"
annotation2="456"
cni.projectcalico.org/podIP="10.244.236.130/32"
cni.projectcalico.org/podIPs="10.244.236.130/32"
kubectl.kubernetes.io/last-applied-configuration="{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{\"annotation1\":\"345\",\"annotation2\":\"456\"},\"labels\":{\"cluster\":\"downward-api-test-cluster1\",\"rack\":\"rack-123\",\"zone\":\"us-east-coast\"},\"name\":\"downwardapi-volume-demo\",\"namespace\":\"demo\"},\"spec\":{\"containers\":[{\"args\":[\"while true; do if [[ -e /etc/podinfo/labels ]]; then echo -en '\\\\n\\\\n'; cat /etc/podinfo/labels; fi; if [[ -e /etc/podinfo/annotations ]]; then echo -en '\\\\n\\\\n'; cat /etc/podinfo/annotations; fi; sleep 5; done;\"],\"command\":[\"sh\",\"-c\"],\"image\":\"busybox:1.28\",\"name\":\"volume-test-container\",\"volumeMounts\":[{\"mountPath\":\"/etc/podinfo\",\"name\":\"podinfo\"}]}],\"volumes\":[{\"downwardAPI\":{\"items\":[{\"fieldRef\":{\"fieldPath\":\"metadata.labels\"},\"path\":\"labels\"},{\"fieldRef\":{\"fieldPath\":\"metadata.annotations\"},\"path\":\"annotations\"}]},\"name\":\"podinfo\"}]}}\n"
kubernetes.io/config.seen="2021-01-30T22:50:35.136835626+08:00"
kubernetes.io/config.source="api"

数据变量更新

对于挂载目录，文件

1
2
3
4
5
6
7
8
9
10
11
/ # ls -alh /etc/podinfo/
total 0
drwxrwxrwt    3 root     root         120 Jan 30 14:50 .
drwxr-xr-x    1 root     root          21 Jan 30 14:50 ..
drwxr-xr-x    2 root     root          80 Jan 30 14:50 ..2021_01_30_14_50_36.745335770
lrwxrwxrwx    1 root     root          31 Jan 30 14:50 ..data -> ..2021_01_30_14_50_36.745335770
lrwxrwxrwx    1 root     root          18 Jan 30 14:50 annotations -> ..data/annotations
lrwxrwxrwx    1 root     root          13 Jan 30 14:50 labels -> ..data/labels


## kubelet 会定期同步检查已经挂载的 ConfigMap等 是否是最新的，如果更新了，就是创建一个新的文件夹存放最新的内容，并同步修改..data指向的软链接

对于环境变量

目前有个开源工具 Reloader，通过 watch ConfigMap 和 Secret 等，一旦发现对象更新，就自动触发对 Deployment 或 StatefulSet 等工作负载对象进行滚动升级。

可以参考 ConfigMap 的热更新

不可变 Secret 和 ConfigMap

添加字段

1
immutable: true   # 字段名是 immutable；拼错属于未知字段，会被忽略/告警，ConfigMap 依旧可改