基于docker搭建cephfs分布式文件系统

妙音 posted @ 2020年5月27日 12:32 in ceph with tags Ceph , 881 阅读
 

目的 

 
在一台机器上, 利用多块硬盘, 搭建一个cephfs文件系统. 具体来说就是1个mon, 1个mds, 1个mgr, 3个osd
 

注意

 
a. 使用vmware会很方便
 
b. 安装过程中会遇到很多问题,我都没有记录, 尽量安装下面步骤
 

环境准备

 
a. vmware虚拟机fedora30
 
b. 添加3块虚拟机硬盘 /dev/sdb  /dev/sdc /dev/sdd (osd最少需要3个,需要有3块磁盘)
 
c. ceph容器版本 ceph/daemon:latest-luminous
 
 

搭建步骤

 
1. 下载镜像
 
docker pull ceph/daemon:latest-luminous
 
2. 挂载硬盘
vmware虚拟机添加硬盘很方便, 直接加就可以. fdisk -l 查看硬盘
 
3. 清理硬盘
 
# 格式化
mkfs.xfs /dev/sdb -f
mkfs.xfs /dev/sdc -f
mkfs.xfs /dev/sdd -f

# 如果已经是xfs格式, 上面命令并不能清除已有数据, 需要用zap_device清理
docker run -d --net=host --name=osd0  --rm \
--privileged=true \
-v /dev/:/dev/ \
-e OSD_DEVICE=/dev/sde  \
 ceph/daemon:latest-luminous zap_device
 
4. 准备目录
 
/root/ceph
/root/ceph/etc
/root/ceph/lib
 
 
5. 启动mon (监控节点必需)
 
docker run -d --net=host  --name=mon \
-v /root/ceph/etc:/etc/ceph \
-v /root/ceph/lib/:/var/lib/ceph/ \
-e MON_IP=192.168.10.125 \
-e CEPH_PUBLIC_NETWORK=192.168.10.0/24 \
 ceph/daemon:latest-luminous mon
 
6. 启动mgr(可以不用)
 
docker run -d --net=host --name=mgr \
-v /root/ceph/etc:/etc/ceph  \
-v /root/ceph/lib/:/var/lib/ceph  \
ceph/daemon:latest-luminous  mgr
 
7. 启动osd
 
# 修改-name和OSD_DEVICE启动三个osd
docker run -d --net=host --name=osd0   \
--privileged=true \
-v /root/ceph/etc:/etc/ceph  \
-v /root/ceph/lib/:/var/lib/ceph  \
-v /dev/:/dev/ \
-e OSD_DEVICE=/dev/sdb  \
-e OSD_TYPE=disk \
 ceph/daemon:latest-luminous osd
 
8. 启动mds(cephfs系统必需)
 
#  一定要在osd之后创建启动, 因为CEPHFS_CREATE=1会创建cephfs文件系统,受osd数量影响
docker run -d --net=host --name=mds \
-v /root/ceph/etc:/etc/ceph \
-v /root/ceph/lib/:/var/lib/ceph/ \
-e CEPHFS_CREATE=1 \      # 默认创建cephfs文件系统
  ceph/daemon:latest-luminous mds
 
9. 进入mon查看ceph状态
 
# 进入容器
docker exec -it mon bash

# 查看状态
[root@localhost /]# ceph -s
  cluster:
    id:     4d74fd53-84e0-47e6-a06c-5418e4b3b653
    health: HEALTH_WARN
            1 MDSs report slow metadata IOs
            2 osds down
            34/51 objects misplaced (66.667%)
            Reduced data availability: 4 pgs inactive, 16 pgs stale
            Degraded data redundancy: 16 pgs undersized
            too few PGs per OSD (4 < min 30)

  services:
    mon: 1 daemons, quorum localhost
    mgr: localhost(active)
    mds: cephfs-1/1/1 up  {0=localhost=up:creating}
    osd: 5 osds: 2 up, 4 in

  data:
    pools:   2 pools, 16 pgs
    objects: 17 objects, 2.19KiB
    usage:   4.01GiB used, 75.6GiB / 79.6GiB avail
    pgs:     25.000% pgs not active
             34/51 objects misplaced (66.667%)
             12 stale+active+undersized+remapped
             4  stale+undersized+peered
 
10. ceph调参: too few PGs per OSD (4 < min 30)
 
存储池的pg_num, pgp_num太小了, 设置大一点
 
ceph osd pool set cephfs_data pg_num 64
ceph osd pool set cephfs_data pgp_num 64
ceph osd pool set  cephfs_metadata pg_num 32
ceph osd pool set  cephfs_metadata pgp_num 32
 
11. ceph调参: mds: cephfs-1/1/1 up  {0=localhost=up:creating}
 
mds一直处在creating状态, 因为默认I/O需要的最小副本数是2,我们需要调成1
 
ceph osd pool set cephfs_metadata min_size 1
ceph osd pool set cephfs_data min_size 1
 
12. 再看ceph状态, mds状态是active表示cephfs搭建好了
 
mds: cephfs-1/1/1 up  {0=localhost=up:active}
 
13. 挂载cephfs目录(直接mount)
 
# 获取key
cat /root/ceph/etc/ceph.client.admin.keyring
# 直接挂载
mount -t ceph 192.168.10.125:6789:/ /root/abc -o name=admin,secret=AQAvoctebqeuBRAAp+FoatmQ5CUlSlo8dmvGAg==
# 取消挂载
umount /root/abc
 
14. 挂载cephfs目录(ceph-fuse)
 
# 安装ceph-fuse
yum install ceph-fuse

# 挂载(-k指定key -c表示配置文件)
ceph-fuse -m 192.168.10.125:6789 /root/abc1 -k /root/ceph/etc/ceph.client.admin.keyring  -c /root/ceph/etc/ceph.conf

#取消挂载
umount /root/abc1
 
15. 查看结果
 
df -h

192.168.10.125:6789:/                     18G     0   18G    0% /root/abc
ceph-fuse                                 18G     0   18G    0% /root/abc1
 

 

Avatar_small
Jonny jack 说:
2020年10月20日 20:34

A very awesome blog post. We are really grateful for your blog post. You will find a lot of approaches after visiting your post. Erste Hilfe Kurs

Avatar_small
Top SEO 说:
2020年10月31日 03:39

Truly, this article is really one of the very best in the history of articles. I am a antique ’Article’ collector and I sometimes read some new articles if I find them interesting. And I found this one pretty fascinating and it should go into my collection. Very good work! ทางเข้าslotxo

Avatar_small
Top SEO 说:
2020年11月03日 18:44

Truly, this article is really one of the very best in the history of articles. I am a antique ’Article’ collector and I sometimes read some new articles if I find them interesting. And I found this one pretty fascinating and it should go into my collection. Very good work! Legit investment sites,

Avatar_small
Top SEO 说:
2020年11月04日 18:31

You there, this is really good post here. Thanks for taking the time to post such valuable information. Quality content is what always gets the visitors coming. Ottawa Web Design

Avatar_small
Digital_Zone 说:
2020年11月08日 21:59

I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article. Webdesign

Avatar_small
Digital_Zone 说:
2020年11月08日 22:08

I admit, I have not been on this web page in a long time... however it was another joy to see It is such an important topic and ignored by so many, even professionals. I thank you to help making people more aware of possible issues. Erotische massage

Avatar_small
Digital_Zone 说:
2020年11月08日 22:14

Wow! This could be one of the most useful blogs we have ever come across on thesubject. Actually excellent info! I’m also an expert in this topic so I can understand your effort. Warmtepomp installateur

Avatar_small
Digital_Zone 说:
2020年11月08日 22:23

That is the excellent mindset, nonetheless is just not help to make every sence whatsoever preaching about that mather. Virtually any method many thanks in addition to i had endeavor to promote your own article in to delicius nevertheless it is apparently a dilemma using your information sites can you please recheck the idea. thanks once more. Webdesign

Avatar_small
Digital_Zone 说:
2020年11月08日 23:02

I admire this article for the well-researched content and excellent wording. I got so involved in this material that I couldn’t stop reading. I am impressed with your work and skill. Thank you so much. Zoekmachine optimalisatie

Avatar_small
Digital_Zone 说:
2020年11月08日 23:08

Remarkable article, it is particularly useful! I quietly began in this, and I'm becoming more acquainted with it better! Delights, keep doing more and extra impressive! Warmtepomp

Avatar_small
Digital_Zone 说:
2020年11月08日 23:13

Thank you a bunch for sharing this with all of us you actually realize what you are talking about! Bookmarked. Please also seek advice from my site =). We could have a hyperlink change contract between us! SEO zoekmachine optimalisatie

Avatar_small
Digital_Zone 说:
2020年11月08日 23:19

Hey, I am so thrilled I found your blog, I am here now and could just like to say thank for a tremendous post and all round interesting website. Please do keep up the great work. I cannot be without visiting your blog again and again. Reiki ontspanningsmassage

Avatar_small
Digital_Zone 说:
2020年11月08日 23:24

You have done a great job on this article. It’s very readable and highly intelligent. You have even managed to make it understandable and easy to read. You have some real writing talent. Thank you. Webdesigner

Avatar_small
Digital_Zone 说:
2020年11月08日 23:30

It is a good site post without fail. Not too many people would actually, the way you just did. I am impressed that there is so much information about this subject that has been uncovered and you’ve defeated yourself this time, with so much quality. Good Works! SEO

Avatar_small
Digital_Zone 说:
2020年11月08日 23:35

Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I'll be subscribing to your feed and I hope you post again soon. Big thanks for the useful info. Warmtepompen

Avatar_small
Digital_Zone 说:
2020年11月08日 23:40

Thanks so much for this information. I have to let you know I concur on several of the points you make here and others may require some further review, but I can see your viewpoint. Noten

Avatar_small
Digital_Zone 说:
2020年11月08日 23:45

Wow, What an Outstanding post. I found this too much informatics. It is what I was seeking for. I would like to recommend you that please keep sharing such type of info.If possible, Thanks. Ayurvedische massage

Avatar_small
Digital_Zone 说:
2020年11月08日 23:50

I am happy to find this post very useful for me, as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. Thanks for sharing. SEO bureau

Avatar_small
Digital_Zone 说:
2020年11月08日 23:54

You have performed a great job on this article. It’s very precise and highly qualitative. You have even managed to make it readable and easy to read. You have some real writing talent. Thank you so much. Zonnepanelen kopen

Avatar_small
Digital_Zone 说:
2020年11月09日 00:00

Just admiring your work and wondering how you managed this blog so well. It’s so remarkable that I can't afford to not go through this valuable information whenever I surf the internet! Noten kopen

Avatar_small
Digital_Zone 说:
2020年11月09日 00:05

Impressive web site, Distinguished feedback that I can tackle. Im moving forward and may apply to my current job as a pet sitter, which is very enjoyable, but I need to additional expand. Regards. Noten kopen online

Avatar_small
Digital_Zone 说:
2020年11月09日 00:12

I’ve been surfing online more than three hours today, yet I never found any interesting article like yours. It’s pretty worth enough for me. In my opinion, if all webmasters and bloggers made good content as you did, the web will be a lot more useful than ever before. Herbal hot compress massage

Avatar_small
Digital_Zone 说:
2020年11月09日 00:17

Pretty good post. I have just stumbled upon your blog and enjoyed reading your blog posts very much. I am looking for new posts to get more precious info. Big thanks for the useful info. Erotische massages

Avatar_small
Digital_Zone 说:
2020年11月09日 00:21

Outstanding article! I want people to know just how good this information is in your article. Your views are much like my own concerning this subject. I will visit daily your blog because I know. It may be very beneficial for me. Webdesign bureau

Avatar_small
Digital_Zone 说:
2020年11月09日 00:26

Thanks for a very interesting blog. What else may I get that kind of info written in such a perfect approach? I’ve a undertaking that I am simply now operating on, and I have been at the look out for such info. Webdesignlab

Avatar_small
Digital_Zone 说:
2020年11月09日 01:02

I know your expertise on this. I must say we should have an online discussion on this. Writing only comments will close the discussion straight away! And will restrict the benefits from this information. Warmtepompen

Avatar_small
Digital_Zone 说:
2020年11月09日 01:06

The content is utmost interesting! I have completely enjoyed reading your points and have come to the conclusion that you are right about many of them. You are great, and your efforts are outstanding! Ramen kopen

Avatar_small
Digital_Zone 说:
2020年11月09日 01:11

Succeed! It could be one of the most useful blogs we have ever come across on the subject. Excellent info! I’m also an expert in this topic so I can understand your effort very well. Thanks for the huge help. SEO marketing

Avatar_small
Digital_Zone 说:
2020年11月09日 01:15

Thanks a lot for sharing this excellent info! I am looking forward to seeing more posts by you as soon as possible! I have judged that you do not compromise on quality. Warmtepomp verwarming

Avatar_small
SaaS Security 说:
2020年12月14日 14:59

Great post I would like to thank you for the efforts you have made in writing this interesting and knowledgeable article.

Avatar_small
seoservise 说:
2020年12月19日 20:02

This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post. Karabakh is Azerbaijan

Avatar_small
Digital_Zone 说:
2020年12月20日 16:48

I am happy to find this post very useful for me, as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. Thanks for sharing. Crawley Accountants

Avatar_small
Digital_Zone 说:
2020年12月20日 16:55

It is my first visit to your blog, and I am very impressed with the articles that you serve. Give adequate knowledge for me. Thank you for sharing useful material. I will be back for the more great post. East Grinstead Accountants

Avatar_small
seoservise 说:
2020年12月20日 19:50

Cool stuff you have got and you keep update all of us. Java

Avatar_small
seoservise 说:
2020年12月20日 22:28

This is such a great resource that you are providing and you give it away for free. I love seeing blog that understand the value of providing a quality resource for free. online mock exam

Avatar_small
seoservise 说:
2020年12月22日 21:09

No doubt this is an excellent post I got a lot of knowledge after reading good luck. Theme of blog is excellent there is almost everything to read, Brilliant post. 私はこの更新された

Avatar_small
Web Design Boksburg 说:
2020年12月23日 01:22

Hello I am so delighted I located your blog, I really located you by mistake, while I was watching on google for something else, Anyways I am here now and could just like to say thank for a tremendous post and a all round entertaining website. Please do keep up the great work.

Avatar_small
seoservise 说:
2020年12月23日 20:59

That is the excellent mindset, nonetheless is just not help to make every sence whatsoever preaching about that mather. Virtually any method many thanks in addition to i had endeavor to promote your own article in to delicius nevertheless it is apparently a dilemma using your information sites can you please recheck the idea. thanks once more. kasyno online blik

Avatar_small
Horacio cartes 说:
2020年12月25日 18:10

El Jefe del Crimen Organizado en Paraguay Horacio Cartes, es uno de los principales culpables del estado del país debido a su mala gestión y a tramas ilegales que tiene por detrás de sus miles de millones de ganancias.

Avatar_small
my reviews google 说:
2020年12月25日 18:36

Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!Thanks

Avatar_small
Pak24tv 说:
2020年12月26日 13:40

I am happy to find your distinguished way of writing the post. Now you make it easy for me to understand and implement the concept. Thank you for the post. guy names girl name start with c names to call your girlfriend middle name

Avatar_small
seoservise 说:
2020年12月26日 16:36

Positive site, where did u come up with the information on this posting?I have read a few of the articles on your website now, and I really like your style. Thanks a million and please keep up the effective work. 澳洲代写

Avatar_small
website designers fr 说:
2020年12月28日 01:26

Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!Thanks

Avatar_small
website designers fr 说:
2020年12月30日 03:15

Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!Thanks

Avatar_small
jacksonseo 说:
2020年12月30日 18:08

i am for the first time here. I found this board and I in finding It truly helpful & it helped me out a lot. I hope to present something back and help others such as you helped me. Schwimmbecken

Avatar_small
jacksonseo 说:
2020年12月30日 18:56

A debt of gratitude is in order for giving late reports with respect to the worry, I anticipate read more. Stahlwandpool

Avatar_small
download Kiss 918 AP 说:
2021年1月01日 01:27

Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!Thanks!

Avatar_small
google ratings and r 说:
2021年1月01日 18:25

Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!Thanks

Avatar_small
seoservise 说:
2021年1月02日 16:26

Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!Thanks UofT代写

Avatar_small
seoservise 说:
2021年1月03日 19:14

I found that site very usefull and this survey is very cirious, I ' ve never seen a blog that demand a survey for this actions, very curious... 温哥华代写

Avatar_small
seoservise 说:
2021年1月04日 18:46

This is a great post. I like this topic.This site has lots of advantage.I found many interesting things from this site. It helps me in many ways.Thanks for posting this again. 36th America’s Cup

Avatar_small
jacksonseo 说:
2021年1月05日 22:15

Hey what a brilliant post I have come across and believe me I have been searching out for this similar kind of post for past a week and hardly came across this. Thank you very much and will look for more postings from you. Aufstellpool

Avatar_small
jacksonseo 说:
2021年1月05日 22:51

I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article. Sandfilteranlage

Avatar_small
jacksonseo 说:
2021年1月05日 23:13

Very informative post! There is a lot of information here that can help any business get started with a successful social networking campaign. Pool selber bauen

Avatar_small
methoxetamine vendor 说:
2021年1月06日 02:15

Great job for publishing such a beneficial web site. Your web log isn’t only useful but it is additionally really creative too.

Avatar_small
Google Knowledge Pan 说:
2021年1月06日 02:43

You there, this is really good post here. Thanks for taking the time to post such valuable information. Quality content is what always gets the visitors coming..

Avatar_small
https://vshareoffici 说:
2021年1月10日 00:52

You there, this is really good post here. Thanks for taking the time to post such valuable information. Quality content is what always gets the visitors coming.

Avatar_small
seoservise 说:
2021年1月11日 17:30

It proved to be Very helpful to me and I am sure to all the commentators here! daftar situs judi slot online terpercaya

Avatar_small
seoservise 说:
2021年1月12日 20:31

Hello There. I found your blog using msn. This is an extremely well written article. I will be sure to bookmark it and return to read more of your useful information. Thanks for the post. I’ll certainly comeback. kasyna online

Avatar_small
seoservise 说:
2021年1月13日 15:11

A great content material as well as great layout. Your website deserves all of the positive feedback it’s been getting. I will be back soon for further quality contents. kasyno online

Avatar_small
seoservise 说:
2021年1月13日 19:55

I am very much pleased with the contents you have mentioned. I wanted to thank you for this great article. casino online

Avatar_small
seoservise 说:
2021年1月14日 16:24 I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here! keep up the good work... online kaszinó
Avatar_small
mens bracelets 说:
2021年1月15日 01:20

You there, this is really good post here. Thanks for taking the time to post such valuable information. Quality content is what always gets the visitors coming.

Avatar_small
domains for sale 说:
2021年1月15日 01:30

I like your post. It is good to see you verbalize from the heart and clarity on this important subject can be easily observed...

Avatar_small
Free Psychic Reading 说:
2021年1月15日 01:39

Wow! Such an amazing and helpful post this is. I really really love it. It's so good and so awesome. I am just amazed. I hope that you continue to do your work like this in the future also

Avatar_small
seoservise 说:
2021年1月16日 17:47

Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with more information? It is extremely helpful for me. Webdesign Tienen

Avatar_small
seoservise 说:
2021年1月16日 20:39

This is such a great resource that you are providing and you give it away for free. I love seeing blog that understand the value of providing a quality resource for free. Chilliwack Chain Link Fencing

Avatar_small
seoservise 说:
2021年1月16日 21:12

This blog is really great. The information here will surely be of some help to me. Thanks!. Lead generation marketing

Avatar_small
seoservise 说:
2021年1月17日 15:52

Please let me know if you’re looking for a article writer for your site. You have some really great posts and I feel I would be a good asset. If you ever want to take some of the load off, I’d absolutely love to write some material for your blog in exchange for a link back to mine. Please send me an email if interested. Thank you! Webdesign Turnhout

Avatar_small
seoservise 说:
2021年1月17日 19:14

I really like your take on the issue. I now have a clear idea on what this matter is all about.. Leadgeneratie

Avatar_small
seoservise 说:
2021年1月17日 21:51

Thank you a bunch for sharing this with all of us you actually realize what you are talking about! Bookmarked. Please also seek advice from my site =). We could have a hyperlink change contract between us! Webdesign Turnhout

Avatar_small
callboy jobs in bang 说:
2021年1月18日 01:31

Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!Thanks

Avatar_small
mens rope bracelets 说:
2021年1月18日 01:46

Wow! Such an amazing and helpful post this is. I really really love it. It's so good and so awesome. I am just amazed. I hope that you continue to do your work like this in the future also..

Avatar_small
telegram 说:
2021年1月18日 01:58

I’m going to read this. I’ll be sure to come back. thanks for sharing. and also This article gives the light in which we can observe the reality. this is very nice one and gives indepth information. thanks for this nice article...

Avatar_small
seoservise 说:
2021年1月18日 15:58

Hi there! Nice stuff, do keep me posted when you post again something like this! Wat is leadgeneratie

Avatar_small
seoservise 说:
2021年1月18日 19:19

I wanted to thank you for this excellent read!! I definitely loved every little bit of it. I have you bookmarked your site to check out the new stuff you post. Webdesign Vilvoorde

Avatar_small
seoservise 说:
2021年1月18日 22:40

I know this is one of the most meaningful information for me. And I'm animated reading your article. But should remark on some general things, the website style is perfect; the articles are great. Thanks for the ton of tangible and attainable help. Lead generation website

Avatar_small
seoservise 说:
2021年1月20日 15:03

Please let me know if you’re looking for a article writer for your site. You have some really great posts and I feel I would be a good asset. If you ever want to take some of the load off, I’d absolutely love to write some material for your blog in exchange for a link back to mine. Please send me an email if interested. Thank you! Webdesign Sint-Truiden

Avatar_small
seoservise 说:
2021年1月20日 17:52

Positive site, where did u come up with the information on this posting?I have read a few of the articles on your website now, and I really like your style. Thanks a million and please keep up the effective work. Leads genereren

Avatar_small
seoservise 说:
2021年1月20日 20:18

This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here keep up the good work Webdesign Limburg

Avatar_small
seoservise 说:
2021年1月21日 16:29

I am incapable of reading articles online very often, but I’m happy I did today. It is very well written, and your points are well-expressed. I request you warmly, please, don’t ever stop writing. SEO Limburg

Avatar_small
seoservise 说:
2021年1月21日 19:15

This is a smart blog. I mean it. You have so much knowledge about this issue, and so much passion. You also know how to make people rally behind it, obviously from the responses. Website laten maken

Avatar_small
seoservise 说:
2021年1月21日 21:56

I know this is one of the most meaningful information for me. And I'm animated reading your article. But should remark on some general things, the website style is perfect; the articles are great. Thanks for the ton of tangible and attainable help. Leads kopen

Avatar_small
https://www.healthcr 说:
2021年1月23日 21:46

We have sell some products of different custom boxes.it is very useful and very low price please visits this site thanks and please share this post with your friends

Avatar_small
https://presidentsda 说:
2021年1月23日 21:59

Wow, What a Excellent post. I really found this to much informatics. It is what i was searching for.I would like to suggest you that please keep sharing such type of info.Thanks

Avatar_small
situs judi online 说:
2021年1月23日 22:20

Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!Thanks

Avatar_small
schlüsseldienst müns 说:
2021年1月24日 19:33

I’m going to read this. I’ll be sure to come back. thanks for sharing. and also This article gives the light in which we can observe the reality. this is very nice one and gives indepth information. thanks for this nice article...

Avatar_small
brick pointing compa 说:
2021年1月26日 19:27

I wanted to thank you for this excellent read!! I definitely loved every little bit of it. I have you bookmarked your site to check out the new stuff you post.

Avatar_small
รีวิวUFA365 说:
2021年1月26日 19:52

Hello I am so delighted I located your blog, I really located you by mistake, while I was watching on google for something else, Anyways I am here now and could just like to say thank for a tremendous post and a all round entertaining website. Please do keep up the great work.

Avatar_small
ایزوله پروتئین آب پن 说:
2021年1月26日 20:04

Great job for publishing such a beneficial web site. Your web log isn’t only useful but it is additionally really creative too.

Avatar_small
laundry hamper 说:
2021年1月28日 01:36

Hello I am so delighted I located your blog, I really located you by mistake, while I was watching on google for something else, Anyways I am here now and could just like to say thank for a tremendous post and a all round entertaining website. Please do keep up the great work.


登录 *


loading captcha image...
(输入验证码)
or Ctrl+Enter