基于docker搭建cephfs分布式文件系统

妙音 posted @ 2020年5月27日 12:32 in ceph with tags Ceph , 560 阅读
 

目的 

 
在一台机器上, 利用多块硬盘, 搭建一个cephfs文件系统. 具体来说就是1个mon, 1个mds, 1个mgr, 3个osd
 

注意

 
a. 使用vmware会很方便
 
b. 安装过程中会遇到很多问题,我都没有记录, 尽量安装下面步骤
 

环境准备

 
a. vmware虚拟机fedora30
 
b. 添加3块虚拟机硬盘 /dev/sdb  /dev/sdc /dev/sdd (osd最少需要3个,需要有3块磁盘)
 
c. ceph容器版本 ceph/daemon:latest-luminous
 
 

搭建步骤

 
1. 下载镜像
 
docker pull ceph/daemon:latest-luminous
 
2. 挂载硬盘
vmware虚拟机添加硬盘很方便, 直接加就可以. fdisk -l 查看硬盘
 
3. 清理硬盘
 
# 格式化
mkfs.xfs /dev/sdb -f
mkfs.xfs /dev/sdc -f
mkfs.xfs /dev/sdd -f

# 如果已经是xfs格式, 上面命令并不能清除已有数据, 需要用zap_device清理
docker run -d --net=host --name=osd0  --rm \
--privileged=true \
-v /dev/:/dev/ \
-e OSD_DEVICE=/dev/sde  \
 ceph/daemon:latest-luminous zap_device
 
4. 准备目录
 
/root/ceph
/root/ceph/etc
/root/ceph/lib
 
 
5. 启动mon (监控节点必需)
 
docker run -d --net=host  --name=mon \
-v /root/ceph/etc:/etc/ceph \
-v /root/ceph/lib/:/var/lib/ceph/ \
-e MON_IP=192.168.10.125 \
-e CEPH_PUBLIC_NETWORK=192.168.10.0/24 \
 ceph/daemon:latest-luminous mon
 
6. 启动mgr(可以不用)
 
docker run -d --net=host --name=mgr \
-v /root/ceph/etc:/etc/ceph  \
-v /root/ceph/lib/:/var/lib/ceph  \
ceph/daemon:latest-luminous  mgr
 
7. 启动osd
 
# 修改-name和OSD_DEVICE启动三个osd
docker run -d --net=host --name=osd0   \
--privileged=true \
-v /root/ceph/etc:/etc/ceph  \
-v /root/ceph/lib/:/var/lib/ceph  \
-v /dev/:/dev/ \
-e OSD_DEVICE=/dev/sdb  \
-e OSD_TYPE=disk \
 ceph/daemon:latest-luminous osd
 
8. 启动mds(cephfs系统必需)
 
#  一定要在osd之后创建启动, 因为CEPHFS_CREATE=1会创建cephfs文件系统,受osd数量影响
docker run -d --net=host --name=mds \
-v /root/ceph/etc:/etc/ceph \
-v /root/ceph/lib/:/var/lib/ceph/ \
-e CEPHFS_CREATE=1 \      # 默认创建cephfs文件系统
  ceph/daemon:latest-luminous mds
 
9. 进入mon查看ceph状态
 
# 进入容器
docker exec -it mon bash

# 查看状态
[root@localhost /]# ceph -s
  cluster:
    id:     4d74fd53-84e0-47e6-a06c-5418e4b3b653
    health: HEALTH_WARN
            1 MDSs report slow metadata IOs
            2 osds down
            34/51 objects misplaced (66.667%)
            Reduced data availability: 4 pgs inactive, 16 pgs stale
            Degraded data redundancy: 16 pgs undersized
            too few PGs per OSD (4 < min 30)

  services:
    mon: 1 daemons, quorum localhost
    mgr: localhost(active)
    mds: cephfs-1/1/1 up  {0=localhost=up:creating}
    osd: 5 osds: 2 up, 4 in

  data:
    pools:   2 pools, 16 pgs
    objects: 17 objects, 2.19KiB
    usage:   4.01GiB used, 75.6GiB / 79.6GiB avail
    pgs:     25.000% pgs not active
             34/51 objects misplaced (66.667%)
             12 stale+active+undersized+remapped
             4  stale+undersized+peered
 
10. ceph调参: too few PGs per OSD (4 < min 30)
 
存储池的pg_num, pgp_num太小了, 设置大一点
 
ceph osd pool set cephfs_data pg_num 64
ceph osd pool set cephfs_data pgp_num 64
ceph osd pool set  cephfs_metadata pg_num 32
ceph osd pool set  cephfs_metadata pgp_num 32
 
11. ceph调参: mds: cephfs-1/1/1 up  {0=localhost=up:creating}
 
mds一直处在creating状态, 因为默认I/O需要的最小副本数是2,我们需要调成1
 
ceph osd pool set cephfs_metadata min_size 1
ceph osd pool set cephfs_data min_size 1
 
12. 再看ceph状态, mds状态是active表示cephfs搭建好了
 
mds: cephfs-1/1/1 up  {0=localhost=up:active}
 
13. 挂载cephfs目录(直接mount)
 
# 获取key
cat /root/ceph/etc/ceph.client.admin.keyring
# 直接挂载
mount -t ceph 192.168.10.125:6789:/ /root/abc -o name=admin,secret=AQAvoctebqeuBRAAp+FoatmQ5CUlSlo8dmvGAg==
# 取消挂载
umount /root/abc
 
14. 挂载cephfs目录(ceph-fuse)
 
# 安装ceph-fuse
yum install ceph-fuse

# 挂载(-k指定key -c表示配置文件)
ceph-fuse -m 192.168.10.125:6789 /root/abc1 -k /root/ceph/etc/ceph.client.admin.keyring  -c /root/ceph/etc/ceph.conf

#取消挂载
umount /root/abc1
 
15. 查看结果
 
df -h

192.168.10.125:6789:/                     18G     0   18G    0% /root/abc
ceph-fuse                                 18G     0   18G    0% /root/abc1
 

 

Avatar_small
Jonny jack 说:
2020年10月20日 20:34

A very awesome blog post. We are really grateful for your blog post. You will find a lot of approaches after visiting your post. Erste Hilfe Kurs

Avatar_small
Top SEO 说:
2020年10月31日 03:39

Truly, this article is really one of the very best in the history of articles. I am a antique ’Article’ collector and I sometimes read some new articles if I find them interesting. And I found this one pretty fascinating and it should go into my collection. Very good work! ทางเข้าslotxo

Avatar_small
Top SEO 说:
2020年11月03日 18:44

Truly, this article is really one of the very best in the history of articles. I am a antique ’Article’ collector and I sometimes read some new articles if I find them interesting. And I found this one pretty fascinating and it should go into my collection. Very good work! Legit investment sites,

Avatar_small
Top SEO 说:
2020年11月04日 18:31

You there, this is really good post here. Thanks for taking the time to post such valuable information. Quality content is what always gets the visitors coming. Ottawa Web Design

Avatar_small
Digital_Zone 说:
2020年11月08日 21:59

I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article. Webdesign

Avatar_small
Digital_Zone 说:
2020年11月08日 22:08

I admit, I have not been on this web page in a long time... however it was another joy to see It is such an important topic and ignored by so many, even professionals. I thank you to help making people more aware of possible issues. Erotische massage

Avatar_small
Digital_Zone 说:
2020年11月08日 22:14

Wow! This could be one of the most useful blogs we have ever come across on thesubject. Actually excellent info! I’m also an expert in this topic so I can understand your effort. Warmtepomp installateur

Avatar_small
Digital_Zone 说:
2020年11月08日 22:23

That is the excellent mindset, nonetheless is just not help to make every sence whatsoever preaching about that mather. Virtually any method many thanks in addition to i had endeavor to promote your own article in to delicius nevertheless it is apparently a dilemma using your information sites can you please recheck the idea. thanks once more. Webdesign

Avatar_small
Digital_Zone 说:
2020年11月08日 23:02

I admire this article for the well-researched content and excellent wording. I got so involved in this material that I couldn’t stop reading. I am impressed with your work and skill. Thank you so much. Zoekmachine optimalisatie

Avatar_small
Digital_Zone 说:
2020年11月08日 23:08

Remarkable article, it is particularly useful! I quietly began in this, and I'm becoming more acquainted with it better! Delights, keep doing more and extra impressive! Warmtepomp

Avatar_small
Digital_Zone 说:
2020年11月08日 23:13

Thank you a bunch for sharing this with all of us you actually realize what you are talking about! Bookmarked. Please also seek advice from my site =). We could have a hyperlink change contract between us! SEO zoekmachine optimalisatie

Avatar_small
Digital_Zone 说:
2020年11月08日 23:19

Hey, I am so thrilled I found your blog, I am here now and could just like to say thank for a tremendous post and all round interesting website. Please do keep up the great work. I cannot be without visiting your blog again and again. Reiki ontspanningsmassage

Avatar_small
Digital_Zone 说:
2020年11月08日 23:24

You have done a great job on this article. It’s very readable and highly intelligent. You have even managed to make it understandable and easy to read. You have some real writing talent. Thank you. Webdesigner

Avatar_small
Digital_Zone 说:
2020年11月08日 23:30

It is a good site post without fail. Not too many people would actually, the way you just did. I am impressed that there is so much information about this subject that has been uncovered and you’ve defeated yourself this time, with so much quality. Good Works! SEO

Avatar_small
Digital_Zone 说:
2020年11月08日 23:35

Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I'll be subscribing to your feed and I hope you post again soon. Big thanks for the useful info. Warmtepompen

Avatar_small
Digital_Zone 说:
2020年11月08日 23:40

Thanks so much for this information. I have to let you know I concur on several of the points you make here and others may require some further review, but I can see your viewpoint. Noten

Avatar_small
Digital_Zone 说:
2020年11月08日 23:45

Wow, What an Outstanding post. I found this too much informatics. It is what I was seeking for. I would like to recommend you that please keep sharing such type of info.If possible, Thanks. Ayurvedische massage

Avatar_small
Digital_Zone 说:
2020年11月08日 23:50

I am happy to find this post very useful for me, as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. Thanks for sharing. SEO bureau

Avatar_small
Digital_Zone 说:
2020年11月08日 23:54

You have performed a great job on this article. It’s very precise and highly qualitative. You have even managed to make it readable and easy to read. You have some real writing talent. Thank you so much. Zonnepanelen kopen

Avatar_small
Digital_Zone 说:
2020年11月09日 00:00

Just admiring your work and wondering how you managed this blog so well. It’s so remarkable that I can't afford to not go through this valuable information whenever I surf the internet! Noten kopen

Avatar_small
Digital_Zone 说:
2020年11月09日 00:05

Impressive web site, Distinguished feedback that I can tackle. Im moving forward and may apply to my current job as a pet sitter, which is very enjoyable, but I need to additional expand. Regards. Noten kopen online

Avatar_small
Digital_Zone 说:
2020年11月09日 00:12

I’ve been surfing online more than three hours today, yet I never found any interesting article like yours. It’s pretty worth enough for me. In my opinion, if all webmasters and bloggers made good content as you did, the web will be a lot more useful than ever before. Herbal hot compress massage

Avatar_small
Digital_Zone 说:
2020年11月09日 00:17

Pretty good post. I have just stumbled upon your blog and enjoyed reading your blog posts very much. I am looking for new posts to get more precious info. Big thanks for the useful info. Erotische massages

Avatar_small
Digital_Zone 说:
2020年11月09日 00:21

Outstanding article! I want people to know just how good this information is in your article. Your views are much like my own concerning this subject. I will visit daily your blog because I know. It may be very beneficial for me. Webdesign bureau

Avatar_small
Digital_Zone 说:
2020年11月09日 00:26

Thanks for a very interesting blog. What else may I get that kind of info written in such a perfect approach? I’ve a undertaking that I am simply now operating on, and I have been at the look out for such info. Webdesignlab

Avatar_small
Digital_Zone 说:
2020年11月09日 01:02

I know your expertise on this. I must say we should have an online discussion on this. Writing only comments will close the discussion straight away! And will restrict the benefits from this information. Warmtepompen

Avatar_small
Digital_Zone 说:
2020年11月09日 01:06

The content is utmost interesting! I have completely enjoyed reading your points and have come to the conclusion that you are right about many of them. You are great, and your efforts are outstanding! Ramen kopen

Avatar_small
Digital_Zone 说:
2020年11月09日 01:11

Succeed! It could be one of the most useful blogs we have ever come across on the subject. Excellent info! I’m also an expert in this topic so I can understand your effort very well. Thanks for the huge help. SEO marketing

Avatar_small
Digital_Zone 说:
2020年11月09日 01:15

Thanks a lot for sharing this excellent info! I am looking forward to seeing more posts by you as soon as possible! I have judged that you do not compromise on quality. Warmtepomp verwarming


登录 *


loading captcha image...
(输入验证码)
or Ctrl+Enter