k8s 静态 PV 集成 ceph 块设备存储
静态 PV 使用步骤
- 创建存储池并启用 RBD 功能。
- 创建 Ceph 用户,提供给 k8s 使用。
- 在 k8s 上安装 ceph-common 客户端。
- 复制
ceph.conf
以及 admin 用户的 keyring 文件到 k8s 节点(master and node)。 - 创建 Secret 资源,以 keyring 的 key 为 data。
- 静态 PV 使用:
- 创建 PV
- 创建 PVC
- 在 Ceph 存储池里创建对应的 RBDImage
- 创建 POD
安装 ceph 客户端
在部署 kubernetes 之前我们就已经有了 Ceph 集群,因此我们可以直接拿来用。但是 kubernetes 的所有节点(尤其是 master 节点)上依然需要安装 ceph 客户端。
apt-get install -y ceph-common
还需要将 ceph 的配置文件 ceph.conf
放在所有节点的 /etc/ceph
目录下。
配置 Secret 资源
生成 Ceph secret
# ceph auth get-key client.admin |base64 QVFBS1RobGZEVysySXhBQXBVekozNGZ0RG1OTVVhRGlWL1NCMEE9PQ==
创建 ceph-secret.yaml
文件内容为:
apiVersion: v1 kind: Secret metadata: name: ceph-secret data: key: QVFBS1RobGZEVysySXhBQXBVekozNGZ0RG1OTVVhRGlWL1NCMEE9PQ==
创建块存储
创建 pool kcha
创建 rbd
$ rbd create --size 102400 kcha/mysqldata
移除 features
$ rbd feature disable kcha/mysqldata exclusive-lock, object-map, fast-diff, deep-flatten
查看 mon
查看 mon 状态:
# ceph mon stat e2: 2 mons at {a=[v2:172.18.32.119:3300/0,v1:172.18.32.119:6789/0],b=[v2:172.18.83.170:3300/0,v1:172.18.83.170:6789/0]}, election epoch 16, leader 0 a, quorum 0,1 a,b
创建 PV
创建 pv-mysql.yaml
文件内容为:
apiVersion: v1 kind: PersistentVolume metadata: name: mysqldata namespace: manti-infra spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce rbd: monitors: - 172.18.32.119:6789 - 172.18.83.170:6789 pool: kcha image: mysqldata user: admin secretRef: name: ceph-secret namespace: default fsType: ext4 readOnly: false persistentVolumeReclaimPolicy: Recycle storageClassName: mysqldata
创建 PVC
创建 pvc-mysql.yaml
文件内容为:
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysqldata namespace: manti-infra spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: mysqldata
创建 pod
apiVersion: apps/v1 kind: Deployment metadata: name: mysql namespace: manti-infra spec: replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: imagePullSecrets: - name: harborsecret containers: - name: mysql image: mysql:v1.0.6 imagePullPolicy: IfNotPresent ports: - containerPort: 3306 env: - name: MYSQL_ROOT_PASSWORD value: 123456 volumeMounts: - name: mysqldata subPath: mysql mountPath: /var/lib/mysql volumes: - name: mysqldata persistentVolumeClaim: claimName: mysqldata initContainers: - name: increase-fd-ulimit image: busybox command: ["sh", "-c", "ulimit -n 65536"] securityContext: privileged: true
验证服务
$ kg pod -o wide NAME READY STATUS RESTARTS AGE IP mysql-544c557fdb-nrp92 1/1 Running 0 25m 192.168.74.76 $ mysql -h192.168.74.76 -P3306 -uroot -p123456
问题解决
rbd feature disable
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable". In some cases useful info is found in syslog - try "dmesg | tail" or so. rbd: map failed: (6) No such device or address
移除 features
$ rbd feature disable kcha/mysqldata exclusive-lock, object-map, fast-diff, deep-flatten
data directory has files
部署后日志报错:
$ k logs mysql-66946b4c48-7dngn Initializing database 2020-09-01T10:34:38.008752-00:00 0 [ERROR] --initialize specified but the data directory has files in it. Aborting. 2020-09-01T10:34:38.008819-00:00 0 [ERROR] Aborting
参考资料:
- https://github.com/docker-library/mysql/issues/186
- https://dev.mysql.com/doc/refman/5.7/en/server-options.html#option_mysqld_ignore-db-dir
原因:
一个新的 ext4 磁盘分区通常不为空。有一个 lost+found
目录,已知 mysql 会阻塞该目录。您可以尝试添加 --ignore-db-dir=lost+found
到,CMD
以确保确定。
解决方案:
增加启动参数:
name: mysql-master image: mysql:5.7 args: - "--ignore-db-dir=lost+found"
增加子目录
volumeMounts: - name: mysqldata subPath: mysql mountPath: /var/lib/mysql
相关资料
相关文章
未找到相关的 Issues 进行评论
请联系 @jueee 初始化创建