rook-ceph 存储使用方法

发表于 分类于
汇总 rook-ceph 三种存储在 Kubernetes 中的用法，并给出直接使用 rbd 与 cephfs 的原生方式与选择建议。

集群的部署与运维见 rook-ceph 部署与运维，本文只讲集群跑起来之后怎么用：先是块存储、共享文件存储、对象存储在 Kubernetes 里的标准用法，再是不经过 PVC、直接用 rbd 与 cephfs 命令的原生方式，最后给出两类方式的对比与选择建议。PV/PVC 的基础概念可参考 PV-PVC(statefulset示例)。

⚠️ 注：下文清单基于 rook v1.5、ceph v15.2.x 记录。VolumeSnapshot 相关资源由 external-snapshotter 以 CRD 形式单独提供，版本要与 k8s、CSI 驱动匹配（本文用的是 snapshot.storage.k8s.io/v1 与 v6.2.1 控制器），旧的 v1beta1 清单在新版控制器中已不再支持。

在 Kubernetes 中使用三种存储

k8s 侧的调用链是固定的：

1
pvc --> storageClassName --> pv

这条链上真正干活的是 CSI：csi-rbdplugin-provisioner 里的 external-provisioner 监听到 PVC，调用 CSI 的 CreateVolume，由 ceph-csi 在 pool 里建出 rbd image，之后才生成 PV 对象——PV 是产物，不是执行者；rbd map、mkfs、挂载则是节点上的 csi-rbdplugin DaemonSet 做的。另外只有块存储才创建 rbd image，共享文件存储创建的是 CephFS 的 subvolume，对象存储根本不走 PVC/PV 这条路。Kubernetes 访问 ceph 集群需要配置文件和认证文件，这部分 rook-ceph 会自动创建好，直接用即可。

完整的组件与动作对照表如下，排障时靠它决定该去看哪个 Pod 的日志：

阶段 由谁执行 跑在哪 对应的 CSI 调用 在 Ceph 侧做了什么
创建卷 external-provisioner csi-*plugin-provisioner Deployment CreateVolume rbd：建 image；cephfs：建 subvolume
挂载到节点 external-attacher 同上 Deployment ControllerPublishVolume rbd 需要；cephfs 不需要这一步
节点侧准备 csi-*plugin DaemonSet（每个节点） NodeStageVolume rbd map + mkfs（首次）
挂进容器 csi-*plugin 同上 DaemonSet NodePublishVolume bind mount 到 Pod 的目录
扩容 external-resizer provisioner Deployment ControllerExpandVolume rbd resize + 文件系统扩容
快照 external-snapshotter provisioner Deployment CreateSnapshot rbd snap create / cephfs subvolume snapshot

这张表最实用的地方是把故障和日志位置对应起来：

症状 卡在哪一步 该看谁的日志
PVC 一直 Pending CreateVolume 没成功 kubectl logs -n rook-ceph deploy/csi-rbdplugin-provisioner -c csi-provisioner
PVC 已 Bound，Pod 卡在 ContainerCreating NodeStage/NodePublish 失败 kubectl logs -n rook-ceph ds/csi-rbdplugin -c csi-rbdplugin（要看Pod 所在那个节点上的那个副本）
扩容不生效 ControllerExpand 或文件系统没跟上 resizer 容器；PVC 上还会留 FileSystemResizePending 条件

看错对象是最常见的时间浪费——PVC Pending 去翻 DaemonSet 日志（那里根本没动过），或者挂载失败去看 provisioner（它早就完成了）。

顺带解释 StorageClass 里为什么要三组 secret，它们分别对应上表的不同阶段、用的 Ceph 用户权限也不同：

1
2
3
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner      # CreateVolume/Delete
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner # Expand
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node              # map/mkfs/mount

分开的意义在于最小权限：node 那组只需要读写 image 的能力，不需要建/删 pool 和 image 的权限——而 DaemonSet 跑在每个业务节点上，暴露面比 provisioner 大得多。少配或配错其中一组，表现就是上表里对应那一行的阶段失败。

三种存储的定位是：

  • 块存储：创建一个 pod 独占使用的块设备。
  • 共享文件系统：创建要在多个 pod（或多台主机）之间共享的文件系统。
  • 对象存储：创建一个在 k8s 集群内部和外部都可以访问的 S3 接口存储。

块存储

块存储是最常用的一种，一个 PVC 对应一个 rbd image，只能被单个节点挂载读写。

创建 pool 与 StorageClass

先建一个副本池，enableRBDStats: true 用于开启 rbd 的 io 统计信息：

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph # namespace:cluster
spec:
  failureDomain: host
  replicated:
    size: 3
    requireSafeReplicaSize: true
  # 开启 rbd io 统计信息
  enableRBDStats: true

再建对应的 StorageClass，后面所有块存储的 PVC 都指向它。默认走内核客户端（krbd），内核版本低、不支持部分 rbd 特性时可以加 mounter: rbd-nbd 改用 nbd。

这两者不只是”内核低就换”这么简单，取舍矩阵如下：

krbd（默认） rbd-nbd
实现位置 内核模块 用户态进程（在 plugin pod 里）
性能 好 有损耗（多一次内核↔用户态往返）
支持的 image feature 受内核版本限制 全部支持
在线扩容 支持得好 依赖版本，历史上有过问题
plugin pod 重启的后果 无影响（挂载在内核里） 连接中断，老版本 ceph-csi 不会自动 reattach，卷变成只读或 IO error

最后一行是选 nbd 之前必须知道的代价：rbd-nbd 进程跑在 csi-rbdplugin DaemonSet 的容器里，一旦这个 pod 被重启（升级 Rook、节点驱逐、OOM），走 nbd 挂载的卷就断了。所以能用 krbd 就别用 nbd——优先升内核，或者关掉内核不支持的 feature。

内核版本与 feature 的对应关系：

feature 最低内核 说明
layering 3.10 克隆/快照的基础，CentOS 7 也有
exclusive-lock 4.9 防止多客户端同时写
data-pool 4.11 数据和元数据分池
object-map / fast-diff / deep-flatten / journaling 不支持 librbd 独有，任何内核都用不了

所以 imageFeatures 的配法就一句话：老内核（3.10）只写 layering；4.9 以上可以加 exclusive-lock；那四个 librbd-only 的永远别写进 krbd 的 SC。

还有一个全文没提但很容易吃掉容量的点——fstype 与空间回收：

1
csi.storage.k8s.io/fstype: ext4      # 或 xfs

RBD 是精简置备（thin provisioning）的，image 只在实际写入时才占池容量。但文件系统删文件时默认不会告诉底层块设备”这些块不要了”，于是池里的占用只增不减——kubectl get pvc 看着用了 10G，ceph df 里这个池已经吃掉 80G。

解决靠 discard/trim：挂载时加 discard 选项（实时回收，对性能有一定影响），或者在容器里定期 fstrim -v /data（批量回收，更推荐）。

ext4 和 xfs 都支持，区别在于 xfs 不能缩小——配合前面说的”PVC 只能扩不能缩”，选 xfs 就彻底没有回头路了。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
allowVolumeExpansion: true
parameters:
    clusterID: rook-ceph
    pool: replicapool
    imageFormat: "2"
    imageFeatures: layering
    csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
    csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
    csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
    csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
    csi.storage.k8s.io/fstype: ext4
    mounter: rbd-nbd

静态供给：手动创建 PV（不建议）

手动方式需要自己先用 rbd 命令建好 image，再写一个 PV 指向它：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
apiVersion: v1
kind: PersistentVolume
metadata:
  name: ceph-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  rbd:
    monitors:
      - 10.68.86.239:6789
      - 10.68.41.247:6789
      - 10.68.19.37:6789
    pool: replicapool
    image: ceph-image
    user: admin
    secretRef:
      name: ceph-secret
    fsType: ext4
    readOnly: false
  persistentVolumeReclaimPolicy: Retain   # Recycle 只有 HostPath 和 NFS 两个 in-tree 插件实现过，整体已废弃；给 rbd 卷写 Recycle，PVC 删除后 PV 不会被回收，而是进入 Failed（事件是 no volume plugin matched），只能人工清理

PV 里直接写 rbd: 用的是 k8s 内置（in-tree）的 rbd 卷插件，需要自己维护 monitor 地址和 ceph-secret，新版 Kubernetes 已经废弃并移除了它，所以实际使用请走下面的 CSI 动态供给。

动态供给：PVC + Pod

声明 PVC 时只要指定 storageClassName，PV 由 CSI 自动创建：

pvc-demo.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-demo
  labels:
    app: pvc-demo
spec:
  storageClassName: rook-ceph-block # 指定 storageClassName
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1G

pod-with-pvc-demo.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: v1
kind: Pod
metadata:
  name: pod-demo
spec:
  containers:
  - name: demo
    image: nginx:latest
    imagePullPolicy: IfNotPresent
    ports:
    - name: www
      protocol: TCP
      containerPort: 80
    volumeMounts:
    - name: rbd
      mountPath: /usr/share/nginx/html
  volumes:
  - name: rbd
    persistentVolumeClaim:
      claimName: pvc-demo

PVC 变成 Bound、pod 里能看到 lost+found 就说明挂载成功了：

1
2
3
4
5
6
7
8
9
10
11
12
[root@imwl-01 ~]# kubectl get pv |grep pvc-demo
pvc-ed593fbd-2475-43fd-a946-2e2d5fb0c582   1Gi        RWO            Delete           Bound      default/pvc-demo                          rook-ceph-block                   61s
[root@imwl-01 ~]# kubectl get pvc
NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
pvc-demo   Bound    pvc-ed593fbd-2475-43fd-a946-2e2d5fb0c582   1Gi        RWO            rook-ceph-block   64s
[root@imwl-01 ~]# kubectl get pod
NAME       READY   STATUS    RESTARTS   AGE
pod-demo   1/1     Running   0          21m

[root@imwl-01 ~]# kubectl exec -it pod-demo -- ls -l /usr/share/nginx/html # 有挂载上
total 16
drwx------ 2 root root 16384 Mar 27 08:42 lost+found

StatefulSet 动态供给

有状态应用更常见的写法是用 volumeClaimTemplates，每个副本各自拿一块独立的 rbd：

sts-with-pvc.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet   # 使用 StatefulSet 必须先建立一个无头服务
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: "nginx"
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www     # 与下面对应
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      storageClassName: rook-ceph-block

三个副本会各自生成一个 pvc/pv：

1
2
3
4
5
6
7
8
9
10
11
12
[root@imwl-01 ceph]# kubectl get pv |grep www
pvc-59ff8d81-1426-49cf-9f79-d267a22f16cf   1Gi        RWO            Delete           Bound      default/www-web-1                         rook-ceph-block                   61s
pvc-c1afd98d-1df3-4c84-9798-2f193ca07fbc   1Gi        RWO            Delete           Bound      default/www-web-0                         rook-ceph-block                   72s
pvc-edbc09cb-3106-4e10-8d1f-33f0768fb6a8   1Gi        RWO            Delete           Bound      default/www-web-2                         rook-ceph-block                   29s
[root@imwl-01 ceph]# kubectl get pvc |grep web
www-web-0   Bound    pvc-c1afd98d-1df3-4c84-9798-2f193ca07fbc   1Gi        RWO            rook-ceph-block   82s
www-web-1   Bound    pvc-59ff8d81-1426-49cf-9f79-d267a22f16cf   1Gi        RWO            rook-ceph-block   70s
www-web-2   Bound    pvc-edbc09cb-3106-4e10-8d1f-33f0768fb6a8   1Gi        RWO            rook-ceph-block   38s
[root@imwl-01 ceph]# kubectl get pod |grep web
web-0      1/1     Running   0          92s
web-1      1/1     Running   0          80s
web-2      1/1     Running   0          48s

共享文件存储

共享文件存储基于 CephFS，可以同时挂载到多个 pod 或多台机器。

创建 CephFilesystem

filesystem.yaml，会拉起 MDS：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: myfs
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
  dataPools:
    - replicated:
        size: 3
  preserveFilesystemOnDelete: true
  metadataServer:
    activeCount: 1
    activeStandby: true

创建 StorageClass

fsName 要和上面的文件系统名字对应：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-cephfs
# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
provisioner: rook-ceph.cephfs.csi.ceph.com
allowVolumeExpansion: true
parameters:
  # clusterID is the namespace where operator is deployed.
  clusterID: rook-ceph

  # CephFS filesystem name into which the volume shall be created
  fsName: myfs   # 与上面的对应

  # Ceph pool into which the volume shall be created
  # Required for provisionVolume: "true"
  pool: myfs-data0

  # The secrets contain Ceph admin credentials. These are generated automatically by the operator
  # in the same namespace as the cluster.
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph

reclaimPolicy: Delete

确认 mds pod 起来了、StorageClass 和文件系统都在：

1
2
3
4
5
6
7
8
9
10
11
12
[root@imwl-175 ~]# kubectl -n rook-ceph get pod -l app=rook-ceph-mds
NAME                                    READY   STATUS    RESTARTS   AGE
rook-ceph-mds-myfs-a-c676cc498-nknl4    1/1     Running   0          22h
rook-ceph-mds-myfs-b-545776bddb-nxw4l   1/1     Running   0          22h

[root@imwl-175 ~]# kubectl get storageclasses.storage.k8s.io
NAME              PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-ceph-block   rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true                   20h
rook-cephfs       rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   14m

[root@imwl-175 ~]# kubectl exec -it -n rook-ceph $(kubectl get pod -n rook-ceph -l app=rook-ceph-tools -o=jsonpath='{.items[0].metadata.name}') -- ceph fs ls
name: myfs, metadata pool: myfs-metadata, data pools: [myfs-data0 ]

集群整体状态怎么看见 rook-ceph 部署与运维。

在 Pod 中使用

CephFS 的 PVC 用 ReadWriteMany，多个副本挂同一个 PVC：

fs-deploy.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: rook-cephfs
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
spec:
  selector:
    matchLabels:
      app: myapp
  replicas: 3
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: imwl/myapp:v1
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"

        volumeMounts:
          - name: cephfs-pvc
            mountPath: /tmp
      volumes:
      - name: cephfs-pvc
        persistentVolumeClaim:
          claimName: cephfs-pvc
          readOnly: false

三个副本共用一个 RWX 的 pv：

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@imwl-01 ceph]# kubectl get pod
NAME                                READY   STATUS    RESTARTS   AGE
myapp-deployment-79c9bf589c-hgkrk   1/1     Running   0          21m
myapp-deployment-79c9bf589c-hm856   1/1     Running   0          21m
myapp-deployment-79c9bf589c-lbtck   1/1     Running   0          21m

[root@imwl-01 ceph]# kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cephfs-pvc   Bound    pvc-929a3293-4bc5-46f6-be7e-782fbf822149   1Gi        RWX            rook-cephfs    22m

[root@imwl-01 ceph]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                     STORAGECLASS             REASON   AGE
pvc-929a3293-4bc5-46f6-be7e-782fbf822149   1Gi        RWX            Delete           Bound    default/cephfs-pvc                        rook-cephfs                       22m

在一个 pod 里写文件，另一个 pod 立刻能读到：

1
2
3
4
5
6
7
8
9
10
[root@imwl-01 ceph]# kubectl exec -it myapp-deployment-79c9bf589c-hgkrk -- sh
/tmp # echo 1 > /tmp/1.txt
/tmp # cat /tmp/1.txt
1
/tmp # exit

[root@imwl-01 ceph]# kubectl exec -it myapp-deployment-79c9bf589c-lbtck -- sh  # 另一个 pod 读取，内容相同
/ # cat /tmp/1.txt
1
/ # exit

要把这个文件系统挂到主机目录上，见下文「原生方式直接使用 ceph」中的 CephFS 挂载。

对象存储

对象存储不走 PVC，而是由 RGW 提供 S3 接口，通过 ObjectBucketClaim 申请 bucket，集群内外都能访问。

创建 CephObjectStore

object.yaml 会拉起 RGW：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: my-store
  namespace: rook-ceph # namespace:cluster
spec:
  # The pool spec used to create the metadata pools. Must use replication.
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
      # Disallow setting pool with replica 1, this could lead to data loss without recovery.
      # Make sure you're *ABSOLUTELY CERTAIN* that is what you want
      requireSafeReplicaSize: true
    parameters:
      # Inline compression mode for the data pool
      # Further reference: https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#inline-compression
      compression_mode: none
      # gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool
      # for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size
      #target_size_ratio: ".5"
  # The pool spec used to create the data pool. Can use replication or erasure coding.
  dataPool:
    failureDomain: host
    replicated:
      size: 3
      requireSafeReplicaSize: true
    parameters:
      compression_mode: none
      #target_size_ratio: ".5"
  # Whether to preserve metadata and data pools on object store deletion
  preservePoolsOnDelete: false
  # The gateway service configuration
  gateway:
    # A reference to the secret in the rook namespace where the ssl certificate is stored
    # sslCertificateRef:
    # A reference to the secret in the rook namespace where the ca bundle is stored
    # caBundleRef:
    # The port that RGW pods will listen on (http)
    port: 80
    # The port that RGW pods will listen on (https). An ssl certificate is required.
    # securePort: 443
    # The number of pods in the rgw deployment
    instances: 1
    # The affinity rules to apply to the rgw deployment.
    placement:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - rook-ceph-rgw
              # topologyKey: */zone can be used to spread RGW across different AZ
              # Use <topologyKey: failure-domain.beta.kubernetes.io/zone> in k8s cluster if your cluster is v1.16 or lower
              # Use <topologyKey: topology.kubernetes.io/zone>  in k8s cluster is v1.17 or upper
              topologyKey: kubernetes.io/hostname
    # A key/value list of annotations
    annotations:
    # A key/value list of labels
    labels:
    resources:
    # The requests and limits set here, allow the object store gateway Pod(s) to use half of one CPU core and 1 gigabyte of memory
    #  limits:
    #    cpu: "500m"
    #    memory: "1024Mi"
    #  requests:
    #    cpu: "500m"
    #    memory: "1024Mi"
    priorityClassName: system-cluster-critical
  # service endpoint healthcheck
  healthCheck:
    # Configure the pod probes for the rgw daemon
    startupProbe:
      disabled: false
    readinessProbe:
      disabled: false
  # security oriented settings
  # security:
  # To enable the Server Side Encryption configuration properly don't forget to uncomment the Secret at the end of the file
  #   kms: # configures RGW with AWS-SSE:KMS
  #     # name of the config map containing all the kms connection details
  #     connectionDetails:
  #        KMS_PROVIDER: "vault"
  #        VAULT_ADDR: VAULT_ADDR_CHANGE_ME # e,g: http://vault.my-domain.com:8200
  #        VAULT_BACKEND_PATH: "rook"
  #        VAULT_SECRET_ENGINE: "kv"
  #        VAULT_BACKEND: v2
  #     # name of the secret containing the kms authentication token
  #     tokenSecretName: rook-vault-token
  #   s3: # configures RGW with AWS-SSE:S3
  #     # name of the config map containing all the kms connection details
  #     connectionDetails:
  #        KMS_PROVIDER: "vault"
  #        VAULT_ADDR: VAULT_ADDR_CHANGE_ME # e,g: http://vault.my-domain.com:8200
  #        VAULT_BACKEND_PATH: "rook"
  #        VAULT_SECRET_ENGINE: "transit"
  #     # name of the secret containing the kms authentication token
  #     tokenSecretName: rook-vault-token
# # UNCOMMENT THIS TO ENABLE A KMS CONNECTION
# # Also, do not forget to replace both:
# #  * ROOK_TOKEN_CHANGE_ME: with a base64 encoded value of the token to use
# #  * VAULT_ADDR_CHANGE_ME: with the Vault address
# ---
# apiVersion: v1
# kind: Secret
# metadata:
#   name: rook-vault-token
#   namespace: rook-ceph # namespace:cluster
# data:
#   token: ROOK_TOKEN_CHANGE_ME

暴露到集群外

集群外访问需要一个 NodePort，targetPort 与上面 gateway.port 对应：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
apiVersion: v1
kind: Service
metadata:
  name: rook-ceph-rgw-my-store-external
  namespace: rook-ceph # namespace:cluster
  labels:
    app: rook-ceph-rgw
    rook_cluster: rook-ceph # namespace:cluster
    rook_object_store: my-store
spec:
  ports:
    - name: rgw
      port: 80 # service port mentioned in object store crd
      protocol: TCP
      targetPort: 80 # 与 gateway 上对应
      nodePort: 31088
  selector:
    app: rook-ceph-rgw
    rook_cluster: rook-ceph # namespace:cluster
    rook_object_store: my-store
  sessionAffinity: None
  type: NodePort

通过 ObjectBucketClaim 申请 bucket

bucket 也走 StorageClass + Claim 的模式，先建一个存储桶用的 StorageClass：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-delete-bucket
provisioner: rook-ceph.ceph.rook.io/bucket # driver:namespace:cluster
# set the reclaim policy to delete the bucket and all objects
# when its OBC is deleted.
reclaimPolicy: Delete
parameters:
   objectStoreName: my-store
   objectStoreNamespace: rook-ceph # namespace:cluster
   # To accommodate brownfield cases reference the existing bucket name here instead
   # of in the ObjectBucketClaim (OBC). In this case the provisioner will grant
   # access to the bucket by creating a new user, attaching it to the bucket, and
   # providing the credentials via a Secret in the namespace of the requesting OBC.
   #bucketName:

再用 ObjectBucketClaim 向它申请，下面会创建一个以 ceph-bkt 为前缀的 bucket：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: ceph-delete-bucket
spec:
  # To create a new bucket specify either `bucketName` or
  # `generateBucketName` here. Both cannot be used. To access
  # an existing bucket the bucket name needs to be defined in
  # the StorageClass referenced here, and both `bucketName` and
  # `generateBucketName` must be omitted in the OBC.
  #bucketName:
  generateBucketName: ceph-bkt
  storageClassName: rook-ceph-delete-bucket
  additionalConfig:
    # To set for quota for OBC
    #maxObjects: "1000"
    #maxSize: "2G"

RGW pod、两个 svc、新的 StorageClass 和 bucket 都能看到，radosgw-admin bucket list 在 toolbox 里执行：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[root@imwl-01 ceph]# kubectl -n rook-ceph get pod -l app=rook-ceph-rgw
NAME                                        READY   STATUS    RESTARTS   AGE
rook-ceph-rgw-my-store-a-74d58648b8-5r9r2   2/2     Running   0          2m27s

[root@imwl-01 ceph]# kubectl -n rook-ceph get svc -l app=rook-ceph-rgw
NAME                              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
rook-ceph-rgw-my-store            ClusterIP   10.68.25.246   <none>        80/TCP         14m
rook-ceph-rgw-my-store-external   NodePort    10.68.15.54    <none>        80:31088/TCP   4m20s

[root@imwl-01 ceph]# kubectl get storageclasses.storage.k8s.io
NAME                      PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-ceph-block           rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true                   26h
rook-ceph-block-retain    rook-ceph.rbd.csi.ceph.com      Retain          Immediate           true                   26h
rook-ceph-delete-bucket   rook-ceph.ceph.rook.io/bucket   Delete          Immediate           false                  105s
rook-cephfs               rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   26h

[root@imwl-01 ceph]# kubectl get objectbucketclaims.objectbucket.io
NAME                 AGE
ceph-delete-bucket   108s

[root@imwl-01 ceph]# radosgw-admin bucket list
[
    "ceph-bkt-1afbdd70-2a65-4aad-9e7a-8c7b3bd5acb7"
]

用 s3cmd 访问

OBC 创建成功后，访问信息分别放在同名的 configmap 与 secret 里：

1
2
3
4
5
6
7
8
9
10
11
# endpoint
kubectl get configmaps ceph-delete-bucket -o jsonpath='{.data.BUCKET_HOST}'
rook-ceph-rgw-my-store.rook-ceph.svc

# access key
kubectl get secrets ceph-delete-bucket -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d
<REDACTED_ACCESS_KEY>

# secret key
kubectl get secrets ceph-delete-bucket -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d
<REDACTED_SECRET_KEY>

拿这三项配置 s3cmd 即可。集群内直接用 rook-ceph-rgw-my-store.rook-ceph.svc:80，集群外把它换成 节点IP:31088：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
[root@rook-ceph-tools-8558bfc844-9rmjm /]# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: <REDACTED_ACCESS_KEY>
Secret Key: <REDACTED_SECRET_KEY>
Default Region [US]:

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: rook-ceph-rgw-my-store.rook-ceph.svc:80

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: rook-ceph-rgw-my-store.rook-ceph.svc:80/%(bucket)

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: no

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

上传、下载、删除都正常：

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@rook-ceph-tools-8558bfc844-9rmjm /]#  s3cmd ls
2023-03-27 09:59  s3://ceph-bkt-1afbdd70-2a65-4aad-9e7a-8c7b3bd5acb7
[root@rook-ceph-tools-8558bfc844-9rmjm /]# s3cmd put /etc/passwd* s3://ceph-bkt-1afbdd70-2a65-4aad-9e7a-8c7b3bd5acb7
upload: '/etc/passwd' -> 's3://ceph-bkt-1afbdd70-2a65-4aad-9e7a-8c7b3bd5acb7/passwd'  [1 of 2]
 1295 of 1295   100% in    0s    91.34 KB/s  done
upload: '/etc/passwd-' -> 's3://ceph-bkt-1afbdd70-2a65-4aad-9e7a-8c7b3bd5acb7/passwd-'  [2 of 2]
 1231 of 1231   100% in    0s    23.42 KB/s  done

[root@rook-ceph-tools-8558bfc844-9rmjm /]#  s3cmd get s3://ceph-bkt-1afbdd70-2a65-4aad-9e7a-8c7b3bd5acb7/passwd ./
download: 's3://ceph-bkt-1afbdd70-2a65-4aad-9e7a-8c7b3bd5acb7/passwd' -> './passwd'  [1 of 1]

[root@rook-ceph-tools-8558bfc844-9rmjm /]#  s3cmd rm s3://ceph-bkt-1afbdd70-2a65-4aad-9e7a-8c7b3bd5acb7/passwd
delete: 's3://ceph-bkt-1afbdd70-2a65-4aad-9e7a-8c7b3bd5acb7/passwd'

创建独立用户

OBC 自带的用户权限很低，连建桶都不行：

1
2
[root@rook-ceph-tools-8558bfc844-9rmjm /]# s3cmd mb s3://test
ERROR: S3 error: 400 (TooManyBuckets)

需要自己建一个对象存储用户，配额和权限都可以在这里放开：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
  name: my-user
  namespace: rook-ceph # namespace:cluster
spec:
  store: my-store
  displayName: "my display name"
  # Quotas set on the user
  # quotas:
     # maxBuckets: 100
     # maxSize: 10G
     # maxObjects: 10000
  # Additional permissions given to the user
  # capabilities:
     # user: "*"
     # bucket: "*"
     # metadata: "*"
     # usage: "*"
     # zone: "*"

用户的 AccessKey/SecretKey/Endpoint 会写进一个 secret，取出来替换掉 ~/.s3cfg 里的旧凭据：

1
2
3
[root@imwl-01 ~]# kubectl -n rook-ceph get secrets rook-ceph-object-user-my-store-my-user
NAME                                     TYPE                 DATA   AGE
rook-ceph-object-user-my-store-my-user   kubernetes.io/rook   3      42s
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# kubectl -n rook-ceph get secrets rook-ceph-object-user-my-store-my-user -o yaml
apiVersion: v1
data:
  AccessKey: <REDACTED_BASE64>
  Endpoint: aHR0cDovL3Jvb2stY2VwaC1yZ3ctbXktc3RvcmUucm9vay1jZXBoLnN2Yzo4MA==
  SecretKey: <REDACTED_BASE64>
kind: Secret
metadata:
  creationTimestamp: "2023-03-27T10:43:23Z"
  labels:
    app: rook-ceph-rgw
    rook_cluster: rook-ceph
    rook_object_store: my-store
    user: my-user
  name: rook-ceph-object-user-my-store-my-user
  namespace: rook-ceph
  ownerReferences:
  - apiVersion: ceph.rook.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: CephObjectStoreUser
    name: my-user
    uid: d940de90-3ec2-4006-a565-cb38751b8a88
  resourceVersion: "247646"
  uid: 7523f966-1e4a-4b88-9ea4-74a157e2d0c8
type: kubernetes.io/rook

换成新用户后就能建桶了：

1
2
3
4
5
[root@rook-ceph-tools-8558bfc844-9rmjm /]# s3cmd mb s3://test
Bucket 's3://test/' created

[root@rook-ceph-tools-8558bfc844-9rmjm /]# s3cmd ls
2023-03-28 02:57  s3://test

卷快照与克隆

rbd 和 cephfs 的 PVC 都支持通过 CSI 打快照、从快照恢复，或者直接克隆一份，前提是集群里装了 snapshot-controller。

部署 snapshot-controller

VolumeSnapshot 不是 k8s 内置资源，需要单独部署 external-snapshotter 的 CRD 和控制器，控制器镜像这里换成了 imwl/snapshot-controller:v6.2.1：

1
2
3
4
5
6
7
8
9
10
11
12
# kubectl apply -k client/config/crd/
customresourcedefinition.apiextensions.k8s.io/volumesnapshotclasses.snapshot.storage.k8s.io configured
customresourcedefinition.apiextensions.k8s.io/volumesnapshotcontents.snapshot.storage.k8s.io configured
customresourcedefinition.apiextensions.k8s.io/volumesnapshots.snapshot.storage.k8s.io configured

# kubectl apply -k deploy/kubernetes/snapshot-controller/
serviceaccount/snapshot-controller unchanged
role.rbac.authorization.k8s.io/snapshot-controller-leaderelection unchanged
clusterrole.rbac.authorization.k8s.io/snapshot-controller-runner unchanged
rolebinding.rbac.authorization.k8s.io/snapshot-controller-leaderelection unchanged
clusterrolebinding.rbac.authorization.k8s.io/snapshot-controller-role unchanged
deployment.apps/snapshot-controller unchanged

确认 api 版本和控制器 pod：

1
2
3
4
5
6
7
# kubectl api-versions
snapshot.storage.k8s.io/v1

[root@imwl-01 external-snapshotter-master]# kubectl get pods -n kube-system -l app=snapshot-controller
NAME                                   READY   STATUS    RESTARTS   AGE
snapshot-controller-5899869978-4xwxq   1/1     Running   0          51m
snapshot-controller-5899869978-95nth   1/1     Running   0          52m

rbd 快照与恢复

先建一个 rbd 的 VolumeSnapshotClass：

1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-rbdplugin-snapclass
driver: rook-ceph.rbd.csi.ceph.com # driver:namespace:operator
parameters:
  # Specify a string that identifies your cluster. Ceph CSI supports any
  # unique string. When Ceph CSI is deployed by Rook use the Rook namespace,
  # for example "rook-ceph".
  clusterID: rook-ceph # namespace:cluster
  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/snapshotter-secret-namespace: rook-ceph #
deletionPolicy: Delete

再对某个 PVC 打快照，下面是给 postgres-operator 命名空间下的 pg 数据盘 test-pgha1-p9qd-pgdata 做备份。注意 VolumeSnapshot 与源 PVC 必须在同一个 namespace：

1
2
3
4
5
6
7
8
9
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: rbd-pvc-snapshot
  namespace: postgres-operator
spec:
  volumeSnapshotClassName: csi-rbdplugin-snapclass
  source:
    persistentVolumeClaimName: test-pgha1-p9qd-pgdata

READYTOUSE 为 true 才算成功，为 false 就要定位原因，之前遇到的问题就是 namespace 不对：

1
2
3
[root@imwl-01 ~]# kubectl get volumesnapshots.snapshot.storage.k8s.io -A
NAMESPACE           NAME               READYTOUSE   SOURCEPVC               SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS             SNAPSHOTCONTENT                                    CREATIONTIME   AGE
postgres-operator   rbd-pvc-snapshot   true         test-pgha1-p9qd-pgdata                           20Gi          csi-rbdplugin-snapclass   snapcontent-5408b1d0-8c80-4196-b655-51d29e7d695a   7m52s          7m52s

恢复的方式是新建一个以快照为 dataSource 的 PVC，再挂到 pod 里查看：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: postgres-operator
  name: rbd-pvc-restore
spec:
  storageClassName: rook-ceph-block
  dataSource:
    name: rbd-pvc-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: csirbd-demo-pod
  namespace: postgres-operator
spec:
  containers:
   - name: web-server
     image: nginx
     volumeMounts:
       - name: mypvc
         mountPath: /mnt
  volumes:
   - name: mypvc
     persistentVolumeClaim:
       claimName: rbd-pvc-restore
       readOnly: false

rbd 克隆

不经过快照也可以直接克隆一个 PVC，把 dataSource 的 kind 换成 PersistentVolumeClaim：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: postgres-operator
  name: rbd-pvc-clone
spec:
  storageClassName: rook-ceph-block
  dataSource:
    name: test-pgha1-p9qd-pgdata
    kind: PersistentVolumeClaim ## dataSource 定义了创建克隆 PVC 的来源
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi # 克隆的大小需要等于或大于源 PVC
---
apiVersion: v1
kind: Pod
metadata:
  name: csirbd-demo-pod2
  namespace: postgres-operator
spec:
  containers:
   - name: web-server
     image: nginx
     volumeMounts:
       - name: mypvc
         mountPath: /mnt
  volumes:
   - name: mypvc
     persistentVolumeClaim:
       claimName: rbd-pvc-clone
       readOnly: false

两个 PVC 都 Bound，挂进去能看到完整的数据库目录：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[root@imwl-01 ~]# kubectl get pvc -n postgres-operator
NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS             AGE
test-pgha1-99kr-pgdata   Bound    pvc-413310df-f1b0-4c28-9c1d-3ecacad20dbe   20Gi       RWO            rook-ceph-block-retain   32m
test-pgha1-p9qd-pgdata   Bound    pvc-a1d3521a-ded6-4e2d-97b1-d4e34f94fae6   20Gi       RWO            rook-ceph-block-retain   32m
test-repo1               Bound    pvc-5670d2fd-ed04-448a-9611-c418098d132a   40Gi       RWO            rook-ceph-block-retain   32m
rbd-pvc-restore         Bound    pvc-5c0af5a1-37d3-468d-bd45-331b7aa3695f   20Gi       RWO            rook-ceph-block          22m
rbd-pvc-clone           Bound    pvc-d24d021c-a841-49a5-a4d4-f2ae67e6913e   20Gi       RWO            rook-ceph-block          20s

[root@imwl-01 ~]# kubectl exec -it -n postgres-operator csirbd-demo-pod -- bash  # 快照恢复出来的
root@csirbd-demo-pod:/# ls -l /mnt/
total 28
drwxrws---  2 root tape 16384 Mar 29 10:23 lost+found
drwx--S--- 19   26 tape  4096 Mar 29 10:23 pg13
drwx------  3   26 tape  4096 Mar 29 10:24 pg13_wal
drwxr-sr-x  3   26 tape  4096 Mar 29 10:23 pgbackrest

[root@imwl-01 ~]# kubectl exec -it -n postgres-operator csirbd-demo-pod2 -- bash  # 直接克隆出来的
root@csirbd-demo-pod2:/# ls -l mnt/
total 28
drwxrws---  2 root tape 16384 Mar 29 10:23 lost+found
drwx--S--- 19   26 tape  4096 Mar 29 16:00 pg13
drwx------  3   26 tape  4096 Mar 29 17:05 pg13_wal
drwxr-sr-x  3   26 tape  4096 Mar 29 10:23 pgbackrest

cephfs 快照与克隆

CephFS 的快照同样不太推荐用，流程和 rbd 一致，先建快照类：

1
2
3
4
5
6
7
8
9
10
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-cephfsplugin-snapclass
driver: rook-ceph.cephfs.csi.ceph.com # driver:namespace:operator
parameters:
  clusterID: rook-ceph # namespace:cluster
  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/snapshotter-secret-namespace: rook-ceph
deletionPolicy: Delete

再对前面的 cephfs-pvc 打快照：

1
2
3
4
5
6
7
8
9
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  namespace: default
  name: cephfs-pvc-snapshot
spec:
  volumeSnapshotClassName: csi-cephfsplugin-snapclass
  source:
    persistentVolumeClaimName: cephfs-pvc
1
2
3
[root@imwl-01 ~]# kubectl get volumesnapshot
NAME                  READYTOUSE   SOURCEPVC    SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS                SNAPSHOTCONTENT                                    CREATIONTIME   AGE
cephfs-pvc-snapshot   true         cephfs-pvc                           1Gi           csi-cephfsplugin-snapclass   snapcontent-dc1adc4a-aabd-4efc-9918-200b2d01dc25   55s            55s

克隆出一个新的 RWX PVC 并挂载使用：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: default # 名称空间要和源 PVC 一致
  name: cephfs-pvc-clone
spec:
  storageClassName: rook-cephfs
  dataSource:
    name: cephfs-pvc
    kind: PersistentVolumeClaim
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: csifs-clone-pod
  namespace: default
spec:
  containers:
   - name: web-server
     image: nginx
     volumeMounts:
       - name: mypvc
         mountPath: /mnt
  volumes:
   - name: mypvc
     persistentVolumeClaim:
       claimName: cephfs-pvc-clone
       readOnly: false

克隆盘里的内容和源盘一致：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@imwl-01 ~]# kubectl exec -it csifs-clone-pod -- sh  # 内容一致
# ls /mnt
1.txt
# cat /mnt/1.txt
1

[root@imwl-01 ~]# kubectl get pvc
NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cephfs-pvc         Bound    pvc-929a3293-4bc5-46f6-be7e-782fbf822149   1Gi        RWX            rook-cephfs    67m
cephfs-pvc-clone   Bound    pvc-123d95f8-414d-4cec-ae77-1671ba897917   1Gi        RWX            rook-cephfs    2m53s
[root@imwl-01 ~]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                     STORAGECLASS             REASON   AGE
pvc-123d95f8-414d-4cec-ae77-1671ba897917   1Gi        RWX            Delete           Bound    default/cephfs-pvc-clone                  rook-cephfs                       2m55s
pvc-929a3293-4bc5-46f6-be7e-782fbf822149   1Gi        RWX            Delete           Bound    default/cephfs-pvc                        rook-cephfs                       67m

这些快照在 Ceph 侧是什么

上面几节都跑通了，但没说清 CSI 的 VolumeSnapshot 落到 Ceph 上究竟变成了什么——不知道这一层，就解释不了”为什么删源快照会失败”。

RBD：rbd snap 加一次 clone。

创建 VolumeSnapshot 时 ceph-csi 执行的是 rbd snap create，仅此而已，是个轻量的元数据操作。但从快照 restore 出一个新 PVC 时做的是 rbd clone——新 image 是父快照的 COW（写时复制）克隆，没被改写过的块仍然指向父快照的数据。

这就带来一条依赖链，也是最常见的报错来源：

1
源 PVC → 源快照 → restore 出来的新 PVC（clone，仍依赖父快照）

只要 clone 还在，源快照就删不掉（Ceph 报 snapshot ... is protected from removal），源 PVC 也删不干净。想切断依赖得把 clone flatten 掉，即把父快照的数据真正拷进自己：

1
rbd flatten <pool>/<clone-image>    # 拷完就独立了，代价是占满整个 image 的空间

rbd_default_clone_format 决定行为细节：

format 1 format 2（较新版本默认）
父快照必须 protect 是 否
能否删掉有 clone 的父快照 不能 能（快照进 trash，数据留到 clone 都 flatten 完）
兼容性 老客户端可读 需要较新的 librbd/内核

这正好解释了后面「原生方式」一节里手工 clone 为什么必须先 rbd snap protect——那是 format 1 的要求；而 CSI 走 format 2，所以在 K8s 侧感觉不到 protect 这一步。

ceph-csi 还能配成 restore 时自动 flatten（rbdHardMaxCloneDepth / rbdSoftMaxCloneDepth），避免克隆链越叠越深——链太深会明显拖慢读性能，因为读未改写的块要沿链回溯。

CephFS：subvolume snapshot。

cephfs 侧走的是 ceph fs subvolume snapshot create，restore 时把快照内容拷贝到新 subvolume（不是 COW），所以恢复一个大目录会真的花时间和空间，但也没有上面那种依赖链问题。

顺带补上”CephFS 快照不推荐”的依据。 前面两处都写了不建议用却没给原因，具体是这几条：

  1. 多活 MDS 下限制多。 max_mds > 1 时快照曾长期不被支持；即使支持之后，跨多个 MDS 的目录树打快照仍需额外协调，出问题概率高于单活。
  2. 删快照的开销集中且不可控。 删一个快照要清理它引用的所有 COW 对象，这个动作由 MDS 承担，大目录树可能让 MDS 卡住数秒到数十分钟，期间整个文件系统的元数据操作都受影响。
  3. 和配额、子目录挂载相互影响。 .snap 是虚拟目录、会出现在每一级目录下；配额目录里快照如何计费、子目录挂载时 .snap 的可见性，历史上都有过行为变化。
  4. 数量有上限。 单个目录的快照数受 mds_max_snaps_per_dir 限制（默认 100），当”每小时备份”用会很快撞上去。

结论是：cephfs 快照适合”手工打一个、用完就删”，不适合做自动化的定期备份——那种需求更适合用 rbd 快照，或者在文件系统之上做应用级备份。

原生方式直接使用 ceph

不经过 PVC，直接用 ceph/rbd 命令操作集群，适合排查问题、给集群外的机器用，以及做导出导入式的备份。命令可以在 toolbox 里执行，也可以在装了 ceph-common 并写好 /etc/ceph/ceph.conf、/etc/ceph/keyring 的主机上执行，环境准备见 rook-ceph 部署与运维。

rbd 常用命令

一条完整的链路：建 pool、建 image、映射成块设备、格式化挂载、扩容、再看 object 落在哪个 pg 和 osd 上。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
[root@imwl-03 ceph]# ceph osd pool create ceph-demo 64 64  # 创建 pool 池
pool 'ceph-demo' created

[root@imwl-03 ceph]# rbd create ceph-demo/rdb-demo.img --size 1G # 在 pool 池中创建 rbd
[root@imwl-03 ceph]# rbd create -p ceph-demo --image rdb-demo.img --size 1G  # 等价写法

[root@imwl-03 ceph]# rbd -p ceph-demo ls # 查看 RBD
rdb-demo.img

[root@imwl-03 ceph]# rbd remove ceph-demo/rdb-demo1.img  # 移除 rbd
Removing image: 100% complete...done.

[root@imwl-03 ceph]# rbd info ceph-demo/rdb-demo.img  # 查看 RBD 信息
rbd image 'rdb-demo.img':
        size 1 GiB in 256 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 13cb8cac0db3
        block_name_prefix: rbd_data.13cb8cac0db3
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten # krbd 只实现了一部分特性，而且内核越老支持越少（layering 3.10+、exclusive-lock 4.9+）；object-map/fast-diff/deep-flatten 是 librbd 独有的，任何内核都不支持，所以老内核上一般只保留 layering
        op_features:
        flags:
        create_timestamp: Wed Mar 29 12:43:30 2023
        access_timestamp: Wed Mar 29 12:43:30 2023
        modify_timestamp: Wed Mar 29 12:43:30 2023

[root@imwl-03 ceph]# rbd map ceph-demo/rdb-demo.img  # 映射 RBD
/dev/rbd0

[root@imwl-03 ceph]# rbd device list  # 查看块设备列表
id  pool       namespace  image         snap  device
0   ceph-demo             rdb-demo.img  -     /dev/rbd0

[root@imwl-03 ceph]# mkfs.ext4 /dev/rbd0 # 格式化块设备
mke2fs 1.46.5 (30-Dec-2021)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 4affc63b-caad-44dc-a148-04af7a68bdcd
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

[root@imwl-03 ceph]# mount /dev/rbd0 /media/  # 挂载块设备
[root@imwl-03 ceph]# df -h /media
Filesystem               Size  Used Avail Use% Mounted on
/dev/rbd0                974M   24K  907M   1% /media
[root@imwl-03 ceph]# echo 1 > /media/1.txt
[root@imwl-03 ceph]# cat /media/1.txt
1

扩容分两步，先扩 rbd（只支持扩大），再扩文件系统，数据不受影响：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@imwl-03 ceph]# rbd resize ceph-demo/rdb-demo.img --size 2G # RBD 块存储扩容
# 缩容其实也支持，只是要显式加 --allow-shrink（不加才拒绝，是防误操作的保护）：
#   rbd resize ceph-demo/rdb-demo.img --size 1G --allow-shrink
# 但必须先在文件系统层缩小（resize2fs 得先 umount，xfs 根本不支持缩小），否则数据损坏，
# 所以实践上还是"只扩不缩"。PVC 层则是 CSI 的限制，确实只能扩容。
Resizing image: 100% complete...done.

[root@imwl-03 ceph]# rbd info ceph-demo/rdb-demo.img | grep size # 块设备扩容成功
        size 2 GiB in 512 objects

[root@imwl-03 ceph]# df -h  /media  # 文件系统未扩容
Filesystem               Size  Used Avail Use% Mounted on
/dev/rbd0                974M   28K  907M   1% /media

[root@imwl-03 ceph]# resize2fs /dev/rbd0  # 扩容文件系统
resize2fs 1.46.5 (30-Dec-2021)
Filesystem at /dev/rbd0 is mounted on /media; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/rbd0 is now 524288 (4k) blocks long.

[root@imwl-03 ceph]# df -h /media  # 文件系统扩容成功
Filesystem               Size  Used Avail Use% Mounted on
/dev/rbd0                2.0G   28K  1.9G   1% /media

[root@imwl-03 ceph]# cat /media/1.txt  # 文件未损坏
1

一个 rbd image 在底层就是一堆 object，可以顺着 object 查到 pg 和 osd：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[root@imwl-03 ceph]# rados -p ceph-demo ls |grep rbd_data.13cb8cac0db3  #  RBD 对应的 object
rbd_data.13cb8cac0db3.0000000000000120
rbd_data.13cb8cac0db3.0000000000000080
rbd_data.13cb8cac0db3.0000000000000100
rbd_data.13cb8cac0db3.0000000000000060
rbd_data.13cb8cac0db3.0000000000000020
rbd_data.13cb8cac0db3.00000000000000e0
rbd_data.13cb8cac0db3.0000000000000004
rbd_data.13cb8cac0db3.00000000000000a0
rbd_data.13cb8cac0db3.0000000000000000

[root@imwl-03 ceph]# rados -p ceph-demo stat rbd_data.13cb8cac0db3.0000000000000120 # object 信息
ceph-demo/rbd_data.13cb8cac0db3.0000000000000120 mtime 2023-03-29T13:57:36.000000+0800, size 8192

[root@imwl-03 ceph]# ceph osd map ceph-demo rbd_data.13cb8cac0db3.0000000000000120 # object 对应的 pg 和 osd
osdmap e69 pool 'ceph-demo' (4) object 'rbd_data.13cb8cac0db3.0000000000000120' -> pg 4.26c148c4 (4.4) -> up ([5,1,3], p5) acting ([5,1,3], p5)

[root@imwl-03 ceph]# ceph osd tree # osd 分布情况
ID  CLASS  WEIGHT   TYPE NAME               STATUS  REWEIGHT  PRI-AFF
-1         0.58612  root default
-5         0.19537      host 192-168-2-131
 0    hdd  0.09769          osd.0               up   1.00000  1.00000
 3    hdd  0.09769          osd.3               up   1.00000  1.00000
-7         0.19537      host 192-168-2-132
 1    hdd  0.09769          osd.1               up   1.00000  1.00000
 4    hdd  0.09769          osd.4               up   1.00000  1.00000
-3         0.19537      host 192-168-2-133
 2    hdd  0.09769          osd.2               up   1.00000  1.00000
 5    hdd  0.09769          osd.5               up   1.00000  1.00000

手工建的 pool 没有声明用途，集群会报 HEALTH_WARN，给它标上类型即可恢复：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[root@imwl-03 ceph]# ceph health detail # 集群健康详细
HEALTH_WARN 1 pool(s) do not have an application enabled; 1 mgr modules have recently crashed
[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled
    application not enabled on pool 'ceph-demo'
    use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.

[root@imwl-03 ceph]# ceph osd pool application get ceph-demo # 查看当前 pool 的 application 定义
{}

[root@imwl-03 ceph]# ceph osd pool application enable ceph-demo rbd # 设置资源池的类型，方便管理
enabled application 'rbd' on pool 'ceph-demo'

[root@imwl-03 ceph]# ceph osd pool application get ceph-demo
{
    "rbd": {}
}

# 上面 health detail 里其实是两条告警，POOL_APP_NOT_ENABLED 只是其中一条；
# 另一条 RECENT_MGR_MODULE_CRASH 不会因为给 pool 打了 application 标记而消失，得单独归档：
[root@imwl-03 ceph]# ceph crash archive-all

[root@imwl-03 ceph]# ceph -s | grep health  # 两条都处理完才会恢复健康
    health: HEALTH_OK

去除 image feature

krbd 只支持 rbd image feature 的一个子集，内核越老支持越少（CentOS 7 的 3.10 基本只有 layering），碰到 RBD image feature set mismatch 就把内核不支持的特性关掉，注意要从后往前去除（存在依赖关系），或者改用用户态的 rbd-nbd：

1
2
3
4
5
6
7
8
9
10
11
12
[root@imwl-03 ceph]# rbd feature disable ceph-demo/rdb-demo.img deep-flatten
[root@imwl-03 ceph]# rbd feature disable ceph-demo/rdb-demo.img fast-diff
[root@imwl-03 ceph]# rbd feature disable ceph-demo/rdb-demo.img object-map
rbd: failed to update image features: (22) Invalid argument
2023-03-29T14:41:39.003+0800 7f613642da00 -1 librbd::Operations: one or more requested features are already disabled
[root@imwl-03 ceph]# rbd info ceph-demo/rdb-demo.img | grep features
        features: layering, exclusive-lock
        op_features:
[root@imwl-03 ceph]# rbd feature disable ceph-demo/rdb-demo.img exclusive-lock
[root@imwl-03 ceph]# rbd info ceph-demo/rdb-demo.img | grep features
        features: layering
        op_features:

上面 object-map 报「already disabled」是因为它随 fast-diff 一起被关掉了，属正常现象。

回收站

删除 image 前可以先丢进回收站并设置过期时间，误删还能还原：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[imwl@imwl-03 ~]$ rbd create ceph-demo/ceph-trash.img --size 10G

[imwl@imwl-03 ~]$ rbd trash move ceph-demo/ceph-trash.img --expires-at 20230330 # RBD 放入回收站，并设置过期时间
rbd: image ceph-trash.img will expire at 2023-03-30T00:00:00.000000+0800

[imwl@imwl-03 ~]$ rbd -p ceph-demo ls
rdb-demo.img

[imwl@imwl-03 ~]$ rbd -p ceph-demo trash ls # 查看 RBD 回收站信息
53231e410ba1 ceph-trash.img

[imwl@imwl-03 ~]$ rbd trash restore -p ceph-demo 53231e410ba1 # 还原
[imwl@imwl-03 ~]$ rbd -p ceph-demo ls
ceph-trash.img
rdb-demo.img

快照

继续用上面挂载到 /media 的 rdb-demo.img。要注意快照回滚之后已挂载的设备不会自动生效，必须 umount + unmap 再重新 map 挂载：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[root@imwl-03 ~]# rbd snap create ceph-demo/rdb-demo.img@snap_20230329 # 创建快照
Creating snap: 100% complete...done.

[root@imwl-03 ~]# rbd snap ls ceph-demo/rdb-demo.img  # 查看快照
SNAPID  NAME           SIZE   PROTECTED  TIMESTAMP
     4  snap_20230329  2 GiB             Wed Mar 29 15:35:55 2023

[root@imwl-03 ~]# rm -rf /media/1.txt  # 模拟文件删除
[root@imwl-03 ~]# ls /media/
lost+found
[root@imwl-03 ~]# rbd snap rollback ceph-demo/rdb-demo.img@snap_20230329  # 使用快照恢复
Rolling back to snapshot: 100% complete...done.
[root@imwl-03 ~]# ls /media/  # 没有恢复，需要重新挂载
lost+found
[root@imwl-03 ~]# umount /media
[root@imwl-03 ~]# mount /dev/rbd0 /media/  # 无法挂载，需要重新 map
mount: /media: can't read superblock on /dev/rbd0

[root@imwl-03 ~]# rbd unmap ceph-demo/rdb-demo.img # unmap
[root@imwl-03 ~]# rbd map ceph-demo/rdb-demo.img  # map
/dev/rbd0
[root@imwl-03 ~]# mount /dev/rbd0 /media/
[root@imwl-03 ~]# cat /media/1.txt # 已经恢复
1

[root@imwl-03 ~]# rbd snap remove ceph-demo/rdb-demo.img@snap_20230329  # 删除快照
Removing snap: 100% complete...done.
[root@imwl-03 ~]# rbd snap purge ceph-demo/rdb-demo.img # 也可以这样删除所有快照
[root@imwl-03 ~]# rbd snap ls ceph-demo/rdb-demo.img

克隆

克隆是基于快照做的，而且快照必须先被保护，克隆出来的 image 依赖父快照：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
[root@imwl-03 ~]# rbd snap create ceph-demo/rdb-demo.img@template # 创建快照
Creating snap: 100% complete...done.

[root@imwl-03 ~]# rbd clone  ceph-demo/rdb-demo.img@template ceph-demo/vm1-clone.img # 快照需要被保护才能克隆
2023-03-29T15:55:55.341+0800 7f3c9964b640 -1 librbd::image::CloneRequest: 0x5650e49dbf90 validate_parent: parent snapshot must be protected
rbd: clone error: (22) Invalid argument

[root@imwl-03 ~]# rbd snap protect ceph-demo/rdb-demo.img@template  # 保护快照，使其无法被删除
[root@imwl-03 ~]# rbd snap rm ceph-demo/rdb-demo.img@template
Removing snap: 0% complete...failed.
2023-03-29T15:54:33.329+0800 7f749ab24a00 -1 librbd::Operations: snapshot is protected

[root@imwl-03 ~]# rbd clone  ceph-demo/rdb-demo.img@template ceph-demo/vm1-clone.img # 克隆
[root@imwl-03 ~]# rbd -p ceph-demo ls
ceph-trash.img
rdb-demo.img
vm1-clone.img

[root@imwl-03 ~]# rbd -p ceph-demo info vm1-clone.img
rbd image 'vm1-clone.img':
        size 2 GiB in 512 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 5824261366b9
        block_name_prefix: rbd_data.5824261366b9
        format: 2
        features: layering
        op_features:
        flags:
        create_timestamp: Wed Mar 29 15:56:33 2023
        access_timestamp: Wed Mar 29 15:56:33 2023
        modify_timestamp: Wed Mar 29 15:56:33 2023
        parent: ceph-demo/rdb-demo.img@template  # 依赖
        overlap: 2 GiB

# 数据信息还存在
[root@imwl-03 ~]# rbd device map ceph-demo/vm1-clone.img
/dev/rbd1
[root@imwl-03 ~]# mkdir /mnt/vm1
[root@imwl-03 ~]# mount /dev/rbd1 /mnt/vm1
[root@imwl-03 ~]# cat /mnt/vm1/1.txt
1

rbd flatten 可以解除这层依赖，让克隆盘变成独立的 image，之后删掉父快照也不影响它：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@imwl-03 ~]# rbd children ceph-demo/rdb-demo.img@template
ceph-demo/vm1-clone.img

[root@imwl-03 ~]# rbd flatten ceph-demo/vm1-clone.img
Image flatten: 100% complete...done.

[root@imwl-03 ~]# rbd info ceph-demo/vm1-clone.img | grep -E "parent|features" # 已经没有 parent 信息
        features: layering
        op_features:

[root@imwl-03 ~]# rbd snap unprotect ceph-demo/rdb-demo.img@template
[root@imwl-03 ~]# rbd snap rm ceph-demo/rdb-demo.img@template
Removing snap: 100% complete...done.
[root@imwl-03 ~]# rbd device ls
id  pool       namespace  image          snap  device
0   ceph-demo             rdb-demo.img   -     /dev/rbd0
1   ceph-demo             vm1-clone.img  -     /dev/rbd1

导出与导入

rbd export 把快照导成本地文件，rbd import 再导回集群，可以用来做冷备或跨集群迁移：

1
2
3
4
5
6
[root@imwl-03 ~]# rbd snap create  ceph-demo/vm1-clone.img@template
Creating snap: 100% complete...done.
[root@imwl-03 ~]# rbd export ceph-demo/vm1-clone.img@template vm1-clone.img
Exporting image: 100% complete...done.
[root@imwl-03 ~]# ls -hl vm1-clone.img
-rw-r--r--. 1 root root 2.0G Mar 29 16:21 vm1-clone.img

清掉原目录后导入到一个新 image，数据完整回来：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@imwl-03 ~]# rm -rf /mnt/vm1/*
[root@imwl-03 ~]# umount /mnt/vm1

[root@imwl-03 ~]# rbd import vm1-clone.img ceph-demo/vm1-clone-new.img  # 导入
Importing image: 100% complete...done.

[root@imwl-03 ~]# rbd -p ceph-demo ls
ceph-trash.img
rdb-demo.img
vm1-clone-new.img
vm1-clone.img

[root@imwl-03 ~]# rbd device map ceph-demo/vm1-clone-new.img
/dev/rbd2

[root@imwl-03 ~]# mount /dev/rbd2 /mnt/vm1/  # 数据恢复
[root@imwl-03 ~]# ls -l /mnt/vm1/
total 20
-rw-r--r--. 1 root root     2 Mar 29 13:56 1.txt
drwx------. 2 root root 16384 Mar 29 12:47 lost+found

增量备份与恢复

两次快照之间的差异可以用 export-diff/import-diff 单独搬运，恢复时先导入全量，再叠加增量：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[root@imwl-03 ~]# rbd snap create ceph-demo/vm1-clone-new.img@v1
Creating snap: 100% complete...done.
[root@imwl-03 ~]# echo 2 > /mnt/vm1/2.txt
[root@imwl-03 ~]# rbd snap create ceph-demo/vm1-clone-new.img@v2
Creating snap: 100% complete...done.

[root@imwl-03 ~]# rbd snap ls ceph-demo/vm1-clone-new.img
SNAPID  NAME  SIZE   PROTECTED  TIMESTAMP
    11  v1    2 GiB             Wed Mar 29 16:28:45 2023
    12  v2    2 GiB             Wed Mar 29 16:29:43 2023

# 导出
[root@imwl-03 ~]# rbd export ceph-demo/vm1-clone-new.img@v1 vm1-clone-new-2.img # 全量导出
Exporting image: 100% complete...done.
[root@imwl-03 ~]# rbd export-diff --from-snap v1 ceph-demo/vm1-clone-new.img@v2 vm1-clone-new-2.img@v2 # 增量导出。不带 --from-snap 导出的是"从 image 创建到 @v2 的全部差异"，约等于全量；这里之所以看着也能 import-diff 成功，只因为目标 image 刚由全量 import 出来
Exporting image: 100% complete...done.

# 导入
[root@imwl-03 ~]# rbd import vm1-clone-new-2.img ceph-demo/vm1-clone-new2.img
Importing image: 100% complete...done.
[root@imwl-03 ~]# rbd import-diff  vm1-clone-new-2.img@v2 ceph-demo/vm1-clone-new2.img
Importing image diff: 100% complete...done.

# 挂载查看，两次快照的文件都在
[root@imwl-03 ~]# rbd device map ceph-demo/vm1-clone-new2.img
/dev/rbd3
[root@imwl-03 ~]# mkdir /mnt/vm1new
[root@imwl-03 ~]# mount /dev/rbd3 /mnt/vm1new
[root@imwl-03 ~]# ls -l /mnt/vm1new
total 24
-rw-r--r--. 1 root root     2 Mar 29 13:56 1.txt
-rw-r--r--. 1 root root     2 Mar 29 16:29 2.txt
drwx------. 2 root root 16384 Mar 29 12:47 lost+found

挂载 CephFS

先确认文件系统存在，前面用 filesystem.yaml 建的 myfs 就可以直接用：

1
2
[root@imwl-03 ~]# ceph fs ls
name: myfs, metadata pool: myfs-metadata, data pools: [myfs-data0 ]

挂载需要 mon 地址和 admin 的 key，在能访问集群的机器上可以从 toolbox 里取：

1
2
3
4
5
6
7
8
9
10
11
12
mon_endpoints=$(kubectl exec -it $(kubectl get pod  -n rook-ceph -l app=rook-ceph-tools -o=jsonpath='{.items[0].metadata.name}') -n rook-ceph -- grep mon_host /etc/ceph/ceph.conf | awk '{print $3}'|sed -e 's/\r//g')

my_secret=$(kubectl exec -it $(kubectl get pod  -n rook-ceph -l app=rook-ceph-tools -o=jsonpath='{.items[0].metadata.name}') -n rook-ceph -- grep key /etc/ceph/keyring | awk '{print $3}')

mkdir /mnt/ceph
mount -t ceph ${mon_endpoints}:/ /mnt/ceph -o name=admin,secret=$my_secret

# 也可以只挂某个子目录，假设有子目录 direct_mount
# mount -t ceph ${mon_endpoints}:/direct_mount /mnt/ceph -o name=admin,secret=$my_secret

# 卸载
umount -lf /mnt/ceph

在各台主机上都执行一遍，/mnt/ceph 里的内容就是共享的。展开后的内核挂载命令形如：

1
2
3
4
[root@imwl-03 ~]# mount -t ceph 192.168.2.132:6789,192.168.2.131:6789,192.168.2.133:6789:/ /mnt/ceph-kernel -o name=admin,secret=<REDACTED_CEPH_KEY>
[root@imwl-03 ~]# df -h /mnt/ceph-kernel
Filesystem                                                  Size  Used Avail Use% Mounted on
192.168.2.132:6789,192.168.2.131:6789,192.168.2.133:6789:/  190G     0  190G   0% /mnt/ceph-kernel

内核不支持 mount -t ceph 或需要更灵活的挂载时，改用用户态的 ceph-fuse（需要先安装 ceph-fuse，并把配置文件写到 /etc/ceph）：

1
2
3
4
5
6
7
8
9
10
[root@imwl-03 ~]# ceph-fuse -o rw   -n client.admin -m  192.168.2.132:6789,192.168.2.131:6789,192.168.2.133:6789 /mnt/ceph-user
2023-03-29T14:17:34.968+0800 7f01aa0b0180 -1 init, newargv = 0x55fe75940d00 newargc=17
ceph-fuse[1102634]: starting ceph client
ceph-fuse[1102634]: starting fuse
[root@imwl-03 ~]# df -h /mnt/ceph-user
Filesystem      Size  Used Avail Use% Mounted on
ceph-fuse       190G     0  190G   0% /mnt/ceph-user

# ceph-fuse 也可以只挂子目录
# ceph-fuse -m 192.168.2.133:6789,192.168.2.132:6789,192.168.2.131:6789 -r /direct_mount /mnt/vm1new

两种方式挂的是同一个文件系统，内容互通：

1
2
3
[root@imwl-03 ~]# echo hello > /mnt/ceph-kernel/hello.txt
[root@imwl-03 ~]# cat /mnt/ceph-user/hello.txt
hello

CephFS 快照

同样不推荐使用。开启后在目录下的 .snap 里创建子目录就是打快照，恢复靠拷回来：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# ceph fs set myfs  allow_new_snaps 1
enabled new snapshots

# mkdir .snap/snap1  # 创建快照。这是 MDS 侧的元数据快照加数据 COW，瞬间完成、不复制任何数据，也不额外占空间；.snap/snap1 只是一个只读视图。只有在快照存在期间修改或删除原文件，才会因为写时复制产生额外占用（下面 cp 回来那一步才是真实拷贝）

# ls -l .snap/snap1
total 1
-rw-r--r--. 1 root root 2 Mar 29 22:24 3.txt
# rm -rf 3.txt
# ls
# cp -ra .snap/snap1/* ./  # 恢复快照
# ls -l
total 1
-rw-r--r--. 1 root root 2 Mar 29 22:24 3.txt
# rmdir .snap/snap1  # 删除快照

对象存储的原生访问

对象存储没有「原生」与「k8s」之分，RGW 暴露的就是 S3 接口，直接参考上文「对象存储」一节的 s3cmd 用法即可，只是 endpoint 在集群外要换成 节点IP:31088。

对比与选择建议

方式 访问模式 适用场景 注意点
块存储 RBD RWO，单节点读写 数据库、消息队列等有状态应用，StatefulSet 每副本一块盘 只能扩容不能缩容；同一块盘不能被多节点同时挂载
共享文件 CephFS RWX，多节点读写 多副本共享配置、日志、上传目录，也可挂到主机 依赖 MDS，元数据操作密集的场景要关注 MDS 压力
对象存储 RGW S3 接口 备份、图片、制品仓库，集群内外都能访问 不走 PVC，需要自己管理 bucket 与用户凭据

在 k8s 里跑的应用一律走 PVC + StorageClass，交给 CSI 去创建和挂载，不要手工建 PV 或手工 rbd map 给容器用，否则调度、回收、扩容都得自己兜着。

原生的 rbd/cephfs 命令主要用在三个地方：一是排查问题，确认到底是 ceph 侧还是 CSI 侧的故障；二是给集群外的机器、虚拟机提供存储；三是做 export/export-diff 这类导出式备份，以及跨集群迁移。

快照与克隆两条路都有：CSI 的 VolumeSnapshot 适合在 k8s 内做数据盘备份和快速拉起同样的数据（比如上文的 pg 数据盘），原生的 rbd snap/rbd clone 更灵活但要自己管理保护、依赖和回滚后的重新挂载。CephFS 的快照两种方式都不太推荐用于生产。