基于docker搭建cephfs分布式文件系统
目的
在一台机器上, 利用多块硬盘, 搭建一个cephfs文件系统. 具体来说就是1个mon, 1个mds, 1个mgr, 3个osd
注意
a. 使用vmware会很方便
b. 安装过程中会遇到很多问题，我都没有记录, 尽量安装下面步骤
环境准备
a. vmware虚拟机fedora30
b. 添加3块虚拟机硬盘 /dev/sdb /dev/sdc /dev/sdd (osd最少需要3个,需要有3块磁盘)
c. ceph容器版本 ceph/daemon:latest-luminous
搭建步骤
1. 下载镜像
docker pull ceph/daemon:latest-luminous
2. 挂载硬盘
vmware虚拟机添加硬盘很方便, 直接加就可以. fdisk -l 查看硬盘
3. 清理硬盘
# 格式化 mkfs.xfs /dev/sdb -f mkfs.xfs /dev/sdc -f mkfs.xfs /dev/sdd -f # 如果已经是xfs格式, 上面命令并不能清除已有数据, 需要用zap_device清理 docker run -d --net=host --name=osd0 --rm \ --privileged=true \ -v /dev/:/dev/ \ -e OSD_DEVICE=/dev/sde \ ceph/daemon:latest-luminous zap_device
4. 准备目录
/root/ceph /root/ceph/etc /root/ceph/lib
5. 启动mon (监控节点必需)
docker run -d --net=host --name=mon \ -v /root/ceph/etc:/etc/ceph \ -v /root/ceph/lib/:/var/lib/ceph/ \ -e MON_IP=192.168.10.125 \ -e CEPH_PUBLIC_NETWORK=192.168.10.0/24 \ ceph/daemon:latest-luminous mon
6. 启动mgr(可以不用)
docker run -d --net=host --name=mgr \ -v /root/ceph/etc:/etc/ceph \ -v /root/ceph/lib/:/var/lib/ceph \ ceph/daemon:latest-luminous mgr
7. 启动osd
# 修改-name和OSD_DEVICE启动三个osd docker run -d --net=host --name=osd0 \ --privileged=true \ -v /root/ceph/etc:/etc/ceph \ -v /root/ceph/lib/:/var/lib/ceph \ -v /dev/:/dev/ \ -e OSD_DEVICE=/dev/sdb \ -e OSD_TYPE=disk \ ceph/daemon:latest-luminous osd
8. 启动mds(cephfs系统必需)
# 一定要在osd之后创建启动, 因为CEPHFS_CREATE=1会创建cephfs文件系统，受osd数量影响 docker run -d --net=host --name=mds \ -v /root/ceph/etc:/etc/ceph \ -v /root/ceph/lib/:/var/lib/ceph/ \ -e CEPHFS_CREATE=1 \ # 默认创建cephfs文件系统 ceph/daemon:latest-luminous mds
9. 进入mon查看ceph状态
# 进入容器
docker exec -it mon bash
# 查看状态
[root@localhost /]# ceph -s
cluster:
id: 4d74fd53-84e0-47e6-a06c-5418e4b3b653
health: HEALTH_WARN
1 MDSs report slow metadata IOs
2 osds down
34/51 objects misplaced (66.667%)
Reduced data availability: 4 pgs inactive, 16 pgs stale
Degraded data redundancy: 16 pgs undersized
too few PGs per OSD (4 < min 30)
services:
mon: 1 daemons, quorum localhost
mgr: localhost(active)
mds: cephfs-1/1/1 up {0=localhost=up:creating}
osd: 5 osds: 2 up, 4 in
data:
pools: 2 pools, 16 pgs
objects: 17 objects, 2.19KiB
usage: 4.01GiB used, 75.6GiB / 79.6GiB avail
pgs: 25.000% pgs not active
34/51 objects misplaced (66.667%)
12 stale+active+undersized+remapped
4 stale+undersized+peered
10. ceph调参: too few PGs per OSD (4 < min 30)
存储池的pg_num, pgp_num太小了, 设置大一点
ceph osd pool set cephfs_data pg_num 64 ceph osd pool set cephfs_data pgp_num 64 ceph osd pool set cephfs_metadata pg_num 32 ceph osd pool set cephfs_metadata pgp_num 32
11. ceph调参: mds: cephfs-1/1/1 up {0=localhost=up:creating}
mds一直处在creating状态, 因为默认I/O需要的最小副本数是2，我们需要调成1
ceph osd pool set cephfs_metadata min_size 1 ceph osd pool set cephfs_data min_size 1
12. 再看ceph状态, mds状态是active表示cephfs搭建好了
mds: cephfs-1/1/1 up {0=localhost=up:active}
13. 挂载cephfs目录(直接mount)
# 获取key cat /root/ceph/etc/ceph.client.admin.keyring # 直接挂载 mount -t ceph 192.168.10.125:6789:/ /root/abc -o name=admin,secret=AQAvoctebqeuBRAAp+FoatmQ5CUlSlo8dmvGAg== # 取消挂载 umount /root/abc
14. 挂载cephfs目录(ceph-fuse)
# 安装ceph-fuse yum install ceph-fuse # 挂载(-k指定key -c表示配置文件) ceph-fuse -m 192.168.10.125:6789 /root/abc1 -k /root/ceph/etc/ceph.client.admin.keyring -c /root/ceph/etc/ceph.conf #取消挂载 umount /root/abc1
15. 查看结果
df -h 192.168.10.125:6789:/ 18G 0 18G 0% /root/abc ceph-fuse 18G 0 18G 0% /root/abc1
2020年10月20日 20:34
A very awesome blog post. We are really grateful for your blog post. You will find a lot of approaches after visiting your post. Erste Hilfe Kurs
2020年10月31日 03:39
Truly, this article is really one of the very best in the history of articles. I am a antique ’Article’ collector and I sometimes read some new articles if I find them interesting. And I found this one pretty fascinating and it should go into my collection. Very good work! ทางเข้าslotxo
2020年11月03日 18:44
Truly, this article is really one of the very best in the history of articles. I am a antique ’Article’ collector and I sometimes read some new articles if I find them interesting. And I found this one pretty fascinating and it should go into my collection. Very good work! Legit investment sites,
2020年11月04日 18:31
You there, this is really good post here. Thanks for taking the time to post such valuable information. Quality content is what always gets the visitors coming. Ottawa Web Design
2020年11月08日 21:59
I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article. Webdesign
2020年11月08日 22:08
I admit, I have not been on this web page in a long time... however it was another joy to see It is such an important topic and ignored by so many, even professionals. I thank you to help making people more aware of possible issues. Erotische massage
2020年11月08日 22:14
Wow! This could be one of the most useful blogs we have ever come across on thesubject. Actually excellent info! I’m also an expert in this topic so I can understand your effort. Warmtepomp installateur
2020年11月08日 22:23
That is the excellent mindset, nonetheless is just not help to make every sence whatsoever preaching about that mather. Virtually any method many thanks in addition to i had endeavor to promote your own article in to delicius nevertheless it is apparently a dilemma using your information sites can you please recheck the idea. thanks once more. Webdesign
2020年11月08日 23:02
I admire this article for the well-researched content and excellent wording. I got so involved in this material that I couldn’t stop reading. I am impressed with your work and skill. Thank you so much. Zoekmachine optimalisatie
2020年11月08日 23:08
Remarkable article, it is particularly useful! I quietly began in this, and I'm becoming more acquainted with it better! Delights, keep doing more and extra impressive! Warmtepomp
2020年11月08日 23:13
Thank you a bunch for sharing this with all of us you actually realize what you are talking about! Bookmarked. Please also seek advice from my site =). We could have a hyperlink change contract between us! SEO zoekmachine optimalisatie
2020年11月08日 23:19
Hey, I am so thrilled I found your blog, I am here now and could just like to say thank for a tremendous post and all round interesting website. Please do keep up the great work. I cannot be without visiting your blog again and again. Reiki ontspanningsmassage
2020年11月08日 23:24
You have done a great job on this article. It’s very readable and highly intelligent. You have even managed to make it understandable and easy to read. You have some real writing talent. Thank you. Webdesigner
2020年11月08日 23:30
It is a good site post without fail. Not too many people would actually, the way you just did. I am impressed that there is so much information about this subject that has been uncovered and you’ve defeated yourself this time, with so much quality. Good Works! SEO
2020年11月08日 23:35
Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I'll be subscribing to your feed and I hope you post again soon. Big thanks for the useful info. Warmtepompen
2020年11月08日 23:40
Thanks so much for this information. I have to let you know I concur on several of the points you make here and others may require some further review, but I can see your viewpoint. Noten
2020年11月08日 23:45
Wow, What an Outstanding post. I found this too much informatics. It is what I was seeking for. I would like to recommend you that please keep sharing such type of info.If possible, Thanks. Ayurvedische massage
2020年11月08日 23:50
I am happy to find this post very useful for me, as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. Thanks for sharing. SEO bureau
2020年11月08日 23:54
You have performed a great job on this article. It’s very precise and highly qualitative. You have even managed to make it readable and easy to read. You have some real writing talent. Thank you so much. Zonnepanelen kopen
2020年11月09日 00:00
Just admiring your work and wondering how you managed this blog so well. It’s so remarkable that I can't afford to not go through this valuable information whenever I surf the internet! Noten kopen
2020年11月09日 00:05
Impressive web site, Distinguished feedback that I can tackle. Im moving forward and may apply to my current job as a pet sitter, which is very enjoyable, but I need to additional expand. Regards. Noten kopen online
2020年11月09日 00:12
I’ve been surfing online more than three hours today, yet I never found any interesting article like yours. It’s pretty worth enough for me. In my opinion, if all webmasters and bloggers made good content as you did, the web will be a lot more useful than ever before. Herbal hot compress massage
2020年11月09日 00:17
Pretty good post. I have just stumbled upon your blog and enjoyed reading your blog posts very much. I am looking for new posts to get more precious info. Big thanks for the useful info. Erotische massages
2020年11月09日 00:21
Outstanding article! I want people to know just how good this information is in your article. Your views are much like my own concerning this subject. I will visit daily your blog because I know. It may be very beneficial for me. Webdesign bureau
2020年11月09日 00:26
Thanks for a very interesting blog. What else may I get that kind of info written in such a perfect approach? I’ve a undertaking that I am simply now operating on, and I have been at the look out for such info. Webdesignlab
2020年11月09日 01:02
I know your expertise on this. I must say we should have an online discussion on this. Writing only comments will close the discussion straight away! And will restrict the benefits from this information. Warmtepompen
2020年11月09日 01:06
The content is utmost interesting! I have completely enjoyed reading your points and have come to the conclusion that you are right about many of them. You are great, and your efforts are outstanding! Ramen kopen
2020年11月09日 01:11
Succeed! It could be one of the most useful blogs we have ever come across on the subject. Excellent info! I’m also an expert in this topic so I can understand your effort very well. Thanks for the huge help. SEO marketing
2020年11月09日 01:15
Thanks a lot for sharing this excellent info! I am looking forward to seeing more posts by you as soon as possible! I have judged that you do not compromise on quality. Warmtepomp verwarming
2020年12月14日 14:59
Great post I would like to thank you for the efforts you have made in writing this interesting and knowledgeable article.
2020年12月19日 20:02
This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post. Karabakh is Azerbaijan
2020年12月20日 16:48
I am happy to find this post very useful for me, as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. Thanks for sharing. Crawley Accountants
2020年12月20日 16:55
It is my first visit to your blog, and I am very impressed with the articles that you serve. Give adequate knowledge for me. Thank you for sharing useful material. I will be back for the more great post. East Grinstead Accountants
2020年12月20日 19:50
Cool stuff you have got and you keep update all of us. Java
2020年12月20日 22:28
This is such a great resource that you are providing and you give it away for free. I love seeing blog that understand the value of providing a quality resource for free. online mock exam
2020年12月22日 21:09
No doubt this is an excellent post I got a lot of knowledge after reading good luck. Theme of blog is excellent there is almost everything to read, Brilliant post. 私はこの更新された
2020年12月23日 01:22
Hello I am so delighted I located your blog, I really located you by mistake, while I was watching on google for something else, Anyways I am here now and could just like to say thank for a tremendous post and a all round entertaining website. Please do keep up the great work.
2020年12月23日 20:59
That is the excellent mindset, nonetheless is just not help to make every sence whatsoever preaching about that mather. Virtually any method many thanks in addition to i had endeavor to promote your own article in to delicius nevertheless it is apparently a dilemma using your information sites can you please recheck the idea. thanks once more. kasyno online blik
2020年12月25日 18:10
El Jefe del Crimen Organizado en Paraguay Horacio Cartes, es uno de los principales culpables del estado del país debido a su mala gestión y a tramas ilegales que tiene por detrás de sus miles de millones de ganancias.
2020年12月25日 18:36
Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!Thanks
2020年12月26日 13:40
I am happy to find your distinguished way of writing the post. Now you make it easy for me to understand and implement the concept. Thank you for the post. guy names girl name start with c names to call your girlfriend middle name
2020年12月26日 16:36
Positive site, where did u come up with the information on this posting?I have read a few of the articles on your website now, and I really like your style. Thanks a million and please keep up the effective work. 澳洲代写
2020年12月28日 01:26
Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!Thanks
2020年12月30日 03:15
Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!Thanks
2020年12月30日 18:08
i am for the first time here. I found this board and I in finding It truly helpful & it helped me out a lot. I hope to present something back and help others such as you helped me. Schwimmbecken
2021年1月01日 01:27
Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!Thanks!
2021年1月03日 19:14
I found that site very usefull and this survey is very cirious, I ' ve never seen a blog that demand a survey for this actions, very curious... 温哥华代写
2021年1月04日 18:46
This is a great post. I like this topic.This site has lots of advantage.I found many interesting things from this site. It helps me in many ways.Thanks for posting this again. 36th America’s Cup
2021年1月05日 22:15
Hey what a brilliant post I have come across and believe me I have been searching out for this similar kind of post for past a week and hardly came across this. Thank you very much and will look for more postings from you. Aufstellpool
2021年1月05日 22:51
I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article. Sandfilteranlage
2021年1月05日 23:13
Very informative post! There is a lot of information here that can help any business get started with a successful social networking campaign. Pool selber bauen
2021年1月06日 02:15
Great job for publishing such a beneficial web site. Your web log isn’t only useful but it is additionally really creative too.
2021年1月06日 02:43
You there, this is really good post here. Thanks for taking the time to post such valuable information. Quality content is what always gets the visitors coming..
2021年1月11日 17:30
It proved to be Very helpful to me and I am sure to all the commentators here! daftar situs judi slot online terpercaya
2023年8月04日 06:56
Nicely presented information in this post, I prefer to read this kind of stuff. The quality of content is fine and the conclusion is good. Thanks for the post. I am from Imamia Quran Academy. Online Shia Quran Madrasa is second to none in providing education to Shia women’s. So our center has selected Shia Female Quran teacher with higher education from Madrasas. Join our Online Shia Quran Academy now, and Start your free 3 days Trial.
2023年10月11日 17:25
Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It's always nice when you can not only be informed, but also entertained! Fortune Tiger
2024年1月05日 21:02
It's good information for me, thanks for your share, and I'd leave my link of here.