docker使用volplugin对接存储-(3)启动volplugin相关服务
2017-10-12
Tip: 此文是"docker使用volplugin对接存储"的第3节，你可以在这里开始阅读整个系列。
配置docker mount-shared选项
由于volplugin需要和宿主机共享挂载点目录，也就是'/mnt'目录，而docker默认是禁止该动作的，因此需要修改下docker配置
[root@dev25 ~]# sed -i s/"MountFlags=slave"/"MountFlags=shared"/g /usr/lib/systemd/system/docker.service
[root@dev25 ~]# systemctl daemon-reload
[root@dev25 ~]# systemctl restart docker
生成volplugin镜像
基于上一篇"编译volplugin插件"所描述的，已经在容器中编译好了volplugin，此时我们基于该容器，创建出volplugin镜像
[root@dev25 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
70ec8e9120d7 docker.io/centos "/bin/bash" 38 minutes ago Up 38 minutes hopeful_pare
[root@dev25 ~]# docker commit -m "create volplugin images" -a "maming" 70ec8e9120d7 maming.org/docker/volplugin:20171012002
[root@dev25 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
maming.org/docker/volplugin 20171012002 e6ca03eb527f 7 seconds ago 798.4 MB
docker.io/centos latest 196e0ce0c9fb 4 weeks ago 196.6 MB
准备ceph环境
到上一步为止，volplugin运行环境已经准好了，接下来准备一个独立ceph存储环境，这里使用了ceph的一个示例docker镜像进行一键安装
[root@dev25 ~]# yum install ceph-common
[root@dev25 ~]# docker pull ceph/demo
[root@dev25 ~]# docker run --privileged -d --net=host -v /etc/ceph:/etc/ceph --name=ceph -e MON_IP=172.24.8.25 -e CEPH_PUBLIC_NETWORK=172.24.8.0/24 ceph/demo
# 修改ceph.conf，设置卷的默认features，在[global]节中追加rbd_default_features = 1
[root@dev25 ~]# cat /etc/ceph/ceph.conf
[global]
fsid = 2e23c20e-edb5-454f-8c25-9495e94a6fc7
mon initial members = dev25
mon host = 172.24.8.25
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd crush chooseleaf type = 0
osd journal size = 100
osd pool default pg num = 8
osd pool default pgp num = 8
osd pool default size = 1
public network = 172.24.8.0/24
cluster network = 172.24.8.0/24
rbd_default_features = 1 # 增加了这里
[client.radosgw.gateway]
rgw dns name = dev25
[client.restapi]
public addr = 0.0.0.0:5000
restapi base url = /api/v0.1
restapi log level = warning
log file = /var/log/ceph/ceph-restapi.log
启动etcd服务
[root@dev25 ~]# docker pull quay.io/coreos/etcd
[root@dev25 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
maming.org/docker/volplugin 20171012002 e6ca03eb527f 2 minutes ago 798.4 MB
quay.io/coreos/etcd latest e8325015c810 5 days ago 35.73 MB
docker.io/centos latest 196e0ce0c9fb 4 weeks ago 196.6 MB
[root@dev25 ~]# docker run --name volplugin-etcd -itd --net host quay.io/coreos/etcd etcd --advertise-client-urls http://0.0.0.0:2379 --listen-client-urls http://0.0.0.0:2379
[root@dev25 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b125c1a6d741 quay.io/coreos/etcd "etcd --advertise-cli" 30 seconds ago Up 28 seconds volplugin-etcd
70ec8e9120d7 docker.io/centos "/bin/bash" 45 minutes ago Up 45 minutes hopeful_pare
启动volplugin服务
# 启动容器
[root@dev25 ~]# docker run --net host --privileged -d -i \
-v /dev:/dev \
-v /etc/ceph:/etc/ceph \
-v /var/lib/ceph:/var/lib/ceph \
-v /lib/modules:/lib/modules:ro \
-v /mnt:/mnt:shared \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /run/docker/plugins:/run/docker/plugins \
-v /var/run/ceph:/var/run/ceph \
-v /sys/fs/cgroup:/sys/fs/cgroup \
--name volplug-all-in-one maming.org/docker/volplugin:20171012002 /bin/bash
[root@dev25 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
786fdf27faf0 maming.org/docker/volplugin:20171012002 "/bin/bash" 3 seconds ago Up 2 seconds volplug-all-in-one
b125c1a6d741 quay.io/coreos/etcd "etcd --advertise-cli" 25 hours ago Up 25 hours volplugin-etcd
# 进入容器
[root@dev25 ~]# docker exec -ti 786fdf27faf0 /bin/bash
# 由于在运行容器的时候，使用了--net host选项，因此进入容器里面的时候，容器的名称也会变成宿主机的名称，例如如下的dev25，在后面操作的时候，不要进错终端了
# 以下命令在volplugin容器里面执行，开3个容器的终端分别执行
[root@dev25 ~]# /opt/golang/bin/apiserver --listen 0.0.0.0:9005 --etcd http://172.24.8.25:2379
[root@dev25 ~]# /opt/golang/bin/volsupervisor --etcd http://172.24.8.25:2379
[root@dev25 ~]# /opt/golang/bin/volplugin --plugin-name "volcontiv" --etcd http://172.24.8.25:2379
启动服务后会报如下错误，这个是由于还没有配置volplugin导致
ERRO[0000] Error fetching global configuration: 100: Key not found (/volplugin/global-config) [11]
创建global.json/policy.json这2个配置文件
[root@dev25 ~]# cat global.json
{
"TTL": 30,
"Debug": true,
"Timeout": 20,
"MountPath": "/mnt/ceph"
}
[root@dev25 ~]# cat policy.json
{
"name": "test1",
"backends": {
"crud": "ceph",
"mount": "ceph",
"snapshot": "ceph"
},
"driver": {
"pool": "rbd"
},
"create": {
"size": "10MB"
},
"runtime": {
"snapshots": true,
"snapshot": {
"frequency": "30m",
"keep": 20
}
},
"filesystems": {
"ext4": "mkfs.ext4 -m0 %"
}
}
导入配置文件
# 在volplugin容器中执行
# 导入全局配置文件
[root@dev25 ~]# /opt/golang/bin/volcli global upload < global.json
# 导入策略配置文件
[root@dev25 ~]# /opt/golang/bin/volcli policy upload policy1 < policy.json
验证
上面3个服务都正常运行时，就可以使用volplugin插件的volcli进行操作了
# 以下命令也在容器里面执行
# 创建卷
[root@dev25 ~]# /opt/golang/bin/volcli volume create --opt size=10G policy1/xx
# 获取卷列表
[root@dev25 ~]# /opt/golang/bin/volcli volume list-all
policy1/xx
# 使用ceph原生的命令查看卷
[root@dev25 ~]# rbd list rbd
policy1.xx
# 删除卷
[root@dev25 ~]# /opt/golang/bin/volcli volume remove policy1/xx
总结
到此为止，volplugin插件已经可以愉快的运行了