MooseFS 2.0.8.X 安装和使用手册

MFS工作的主要流程

mfs_pic.png

  • MFS的读数据过程

1.client当需要一个数据时,首先向master server发起查询请求

2.master server检索自己的数据,获取到数据所在的可用数据服务器位置ip/port/chunk_id

3.master server将数据服务器(chunk server)的地址发送给客户端

4.chunk server将数据发送给客户端

  • MFS的写数据过程

1.当客户端有数据写需求时,首先向master server提供元文件元数据信息请求存储地址(元数据信息如下:文件名/大小/分数等)

2.master server根据写文件的元数据信息,到chunk server创建新的数据块

3.chunk server返回创建成功的消息

4.master server将chunk server的地址(chunk_ip/port/chunk_id)返回给客户端

5.客户端向chunk server写数据

6.chunk server返回给客户端写成功的消息

7.客户端将此次写完成结束信号和一些信息发送到master server来更新文件的长度和最后修改时间

  • MFS的删除文件过程

1.客户端有删除操作时,首先向master server发送删除信息

2.master server定位到相应元数据信息进行删除,并将chunk server上块的删除操作加入队列异步清理

3.响应客户端删除成功的信号

  • MFS修改文件内容过程

1.客户端有修改文件内容时,先向master server 发送操作信息

2.master server申请新的块给.swp文件

3.客户端关闭文件后,会向master server发送关闭信息

4.master server 会检测内容是否有更新,若有,则申请新的块存放更改后的文件,删除原有块和.swp文件块

5.若无,则直接删除.swp文件块

  • MFS重命名文件过程

1.客户端重命名文件时,向master server 发送操作信息

2.master server直接修改元数据信息中的文件名,返回重命名完成信息

  • MFS遍历文件的过程

1.遍历文件不需要访问chunk server,当有客户端遍历请求时,向master server发送操作信息

2.master server返回相应元数据信息

3.客户端接收到信息后显示

  • master 的changelog日志分析
tailf /var/lib/mfs/changelog.0.mfs

此文件在master首次启动且没有客户端操作时是不存在的,若有客户端进行了操作则生成。

复制文件cp ~/install.log /mnt/mfs_test/

121: 1457491177|CREATE(1,install.log,1,420,18,0,0,0):16

122: 1457491177|ACQUIRE(3,16)

123: 1457491177|WRITE(16,0,1):8

124: 1457491177|LENGTH(16,57855)

125: 1457491177|UNLOCK(8)

新建文件touch /mnt/mfs_test/sll_test_file

138: 1457491558|CREATE(1, sll _test_file,1,438,18,0,0,0):17

139: 1457491558|ACQUIRE(3,17)

140: 1457491558|ATTR(17,420,0,0,1457491558,1457491558)

删除文件rm -fr install.log

129: 1457491240|UNLINK(1,install.log):16

删除文件的冒号后面的数字会递增

182: 1457491998|UNLINK(1, sll _test_file):17

183: 1457491998|CHUNKDEL(13,1)

184: 1457492001|CHUNKDEL(12,1)

185: 1457492019|UNLINK(1, sll _test_file_1):18

186: 1457492026|UNLINK(1, sll _test_file_2):19

读取挂载目录文件(tab挂载目录的文件名)

130: 1457491349|ACCESS(1)

修改文件内容echo “i am zhengda” > /mnt/mfs_test/zhengda_test_file

132: 1457491440|ACQUIRE(3,15)

133: 1457491440|LENGTH(15,0)

133: 1457491440|LENGTH(15,0)

134: 1457491440|WRITE(15,0,1):9 冒号后面这个9会随着每次修改递增

135: 1457491440|LENGTH(15,13)

136: 1457491440|UNLOCK(9)

MFS源码安装

(MooseFS版本2.0.83,OS版本CentOS 6.5)

准备工作:

所有机器均配置好hostname、hosts记录,并关闭iptables,机器信息如下:

192.168.6.200 master

192.168.6.201 metalogger

192.168.6.202 chunk1

192.168.6.203 chunk2

192.168.6.204 chunk3

涉及的源码包信息如下:

moosefs-2.0.83-1.tar.gz

1、安装master

源码编译安装,因不需要chunkserver,此处configure加上参数–disable-mfschunkserver

groupadd mfs

useradd -g mfs mfs

tar xzf moosefs-2.0.83-1.tar.gz

cd moosefs-2.0.83

./configure –prefix=/usr –sysconfdir=/etc –localstatedir=/var/lib –with-default-user=mfs –with-default-group=mfs –disable-mfschunkserver

make&&make install

初始化master配置

cd /etc/mfs

cp mfsexports.cfg.dist mfsexports.cfg

cp mfsmaster.cfg.dist mfsmaster.cfg

cp mfstopology.cfg.dist mfstopology.cfg

初始化matadata文件

cd /var/lib/mfs/

cp metadata.mfs.empty metadata.mfs

启动master服务,共启动3个监听端口:9419,9420,9421。

9419:用于监听metalogger请求

9420:用于监听chunkserver请求

9421:用于web管理界面的监控端口/与mfsmount交互的端口

[root@master mfs]# mfsmaster start

open files limit has been set to: 4096

working directory: /var/lib/mfs

lockfile created and locked

initializing mfsmaster modules …

exports file has been loaded

topology file has been loaded

loading metadata …

metadata file has been loaded

no charts data file – initializing empty charts

master <-> metaloggers module: listen on *:9419

master <-> chunkservers module: listen on *:9420

main master server module: listen on *:9421

mfsmaster daemon initialized properly

mfsmaster daemon initialized properly

启动web管理界面,启动监听端口9425,访问web界面http://192.168.6.200:9425,因master的主机名不是为默认的mfsmaster,而web界面默认会尝试连接mfsmaster此主机名的服务器导致失败,手工填入实际的master主机名即可。

[root@master mfs]# mfscgiserv

lockfile created and locked

starting simple cgi server (host: any , port: 9425 , rootpath: /usr/share/mfscgi)

当前master有如下监听端口

[root@master yum.repos.d]# netstat -ntlp|grep -E “mfs|python”

tcp 0 0 0.0.0.0:9419 0.0.0.0:* LISTEN 1858/mfsmaster

tcp 0 0 0.0.0.0:9420 0.0.0.0:* LISTEN 1858/mfsmaster

tcp 0 0 0.0.0.0:9421 0.0.0.0:* LISTEN 1858/mfsmaster

tcp 0 0 0.0.0.0:9425 0.0.0.0:* LISTEN 1854/python

2、安装metalogger

源码编译安装,与master参数相同

groupadd mfs

useradd -g mfs mfs

tar xzf moosefs-2.0.83-1.tar.gz

cd moosefs-2.0.83

./configure –prefix=/usr –sysconfdir=/etc –localstatedir=/var/lib –with-default-user=mfs –with-default-group=mfs –disable-mfschunkserver

make&&make install

初始化metalogger配置文件

cd /etc/mfs

cp mfsmetalogger.cfg.dist mfsmetalogger.cfg

修改MASTER_HOST为实际master的主机名

启动metalogger

[root@metalogger /]# mfsmetalogger start

open files limit has been set to: 4096

working directory: /var/lib/mfs

lockfile created and locked

initializing mfsmetalogger modules …

mfsmetalogger daemon initialized properly

3、安装chunkserver

源码编译安装,加上参数–disable-mfsmaster –disable-mfsmetalogger,不安装master和metalogger。

groupadd mfs

useradd -g mfs mfs

tar xzf moosefs-2.0.83-1.tar.gz

cd moosefs-2.0.83

./configure –prefix=/usr –sysconfdir=/etc –localstatedir=/var/lib –with-default-user=mfs –with-default-group=mfs –disable-mfsmaster –disable-mfsmetalogger

make&&make install

初始化chunkserver配置文件

cd /etc/mfs

cp mfschunkserver.cfg.dist mfschunkserver.cfg

cp mfshdd.cfg.dist mfshdd.cfg

将mfschunkserver.cfg中的MASTER_HOST改为实际master的主机名

—新建目录/mfs_chunk_dir_1/,在mfshdd.cfg中最后一行加入以下内容:

/mfs_chunk_dir_1/

—增加mfsd对此目录的权限

chown -R mfs.mfs /mfs_chunk_dir_1/

启动chunkserver,共启动一个监听端口:9422,用于与master通信。

[root@chunk1 /]# mfschunkserver start

open files limit has been set to: 16384

working directory: /var/lib/mfs

lockfile created and locked

setting glibc malloc arena max to 8

setting glibc malloc arena test to 1

initializing mfschunkserver modules …

hdd space manager: path to scan: /mfs_chunk_dir_1/

hdd space manager: start background hdd scanning (searching for available chunks)

main server module: listen on *:9422

no charts data file – initializing empty charts

mfschunkserver daemon initialized properly

根据现网MFS的搭建经验,chunkserver所挂载的本地空间超过16T,使用mkfs.ext4格式化会出现错误,以下为解决方法:

1、建分区表和分区

parted /dev/sda

mklabel gpt

mkpart primary 0 -1

2、安装新版mkfs,支持大于16T分区

tar zxvf e2fsprogs-1.42.13.tar.gzz

cd e2fsprogs-1.42.13

mkdir build

cd build/

../configure

make

make install

查看新版是否生效

/sbin/mkfs.ext4 -V

并新增mkfs配置

vim /etc/mke2fs.conf

ext4 = {

features = has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize

inode_size = 256

auto_64-bit_support = 1 ——>加上这一句

}

格式化文件系统

mkfs.ext4 /dev/sda1

MooseFS官方推荐采用XFS文件系统,以下为操作过程:

安装xfs模块

yum install kmod-xfs xfsprogs(或者yum install xfs*)

载入xfs文件系统模块

modprobe xfs

查看是否载入了xfs模块

lsmod |grep xfs

格式化文件系统

mkfs.xfs /dev/sdb1

4、安装客户端mfsmount

源码编译安装,使用参数–enable-mfsmount,由于不需要安装master等角色,增加参数–disable-mfsmaster –disable-mfsmetalogger –disable-mfschunkserver

yum install fuse fuse-devel -y(也可源码安装)

tar xzf moosefs-2.0.83-1.tar.gz

cd moosefs-2.0.83

./configure –prefix=/usr –sysconfdir=/etc –localstatedir=/var/lib –disable-mfsmaster –disable-mfsmetalogger –disable-mfschunkserver –enable-mfsmount

make&&make install

若没有yum源,也可源码安装fuse,如下:

tar xzf fuse-2.8.4.tar.gz

cd fuse-2.8.4

./configure

make

make install

目录挂载,进行文件读写测试

mfsmount /mnt/mfs_test/ -H master

[root@chunk1 dir0]# mount

/dev/mapper/vg_centos2016-lv_root on / type ext4 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

tmpfs on /dev/shm type tmpfs (rw,rootcontext=”system_u:object_r:tmpfs_t:s0″)

/dev/sda1 on /boot type ext4 (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

192.168.6.200:9421 on /mnt/mfs_test type fuse.mfs (rw,nosuid,nodev,allow_other)

5、增加自启动配置

master

/etc/rc.local 增加如下内容:/usr/sbin/mfsmaster start

metalogger

/etc/rc.local 增加如下内容:/usr/sbin/mfsmetalogger start

chunkserver

/etc/rc.local 增加如下内容:/usr/sbin/mfschunkserver start

/etc/fstab 增加如下内容: /dev/sdb1 /mfs1 xfs defaults 1 2

MFS YUM安装

(MooseFS版本2.0.88,OS版本CentOS 6.5)

1、下载密钥和导入仓库

针对6系列:

curl “http://ppa.moosefs.com/RPM-GPG-KEY-MooseFS” > /etc/pki/rpm-gpg/RPM-GPG-KEY-MooseFS

curl “http://ppa.moosefs.com/MooseFS-stable-el6.repo” > /etc/yum.repos.d/MooseFS.repo

针对7系列

curl “http://ppa.moosefs.com/RPM-GPG-KEY-MooseFS” > /etc/pki/rpm-gpg/RPM-GPG-KEY-MooseFS

curl “http://ppa.moosefs.com/MooseFS-stable-el6.repo” > /etc/yum.repos.d/MooseFS.repo

验证MooseFS仓库可用性

2、安装master

yum install moosefs-master moosefs-cli moosefs-cgi moosefs-cgiserv

安装完成生成如下命令:

mfscgiserv mfscli mfsmaster mfsmetadump mfsmetarestore

/etc/mfs/下生成配置文件:

mfsexports.cfg mfsexports.cfg.dist mfsmaster.cfg mfsmaster.cfg.dist mfstopology.cfg mfstopology.cfg.dist

3、安装metalogger

yum install moosefs-metalogger

安装完有如下命令:

mfsmetalogger

/etc/mfs/增加的配置文件:

mfsmetalogger.cfg mfsmetalogger.cfg.dist

修改MASTER_HOST为实际master的主机名

4、安装chunkserver

mfschunkserver mfschunktool

/etc/mfs/增加的配置文件:

mfschunkserver.cfg mfschunkserver.cfg.dist mfshdd.cfg mfshdd.cfg.dist

5、安装客户端mfsmount

yum install moosefs-client

安装完有如下命令:

mfsappendchunks mfsdelquota mfsfilepaths mfsgetgoal mfsmakesnapshot mfsrgettrashtime mfsseteattr mfssettrashtime

mfscheckfile mfsdirinfo mfsfilerepair mfsgetquota mfsmount mfsrsetgoal mfssetgoal mfstools

mfsdeleattr mfsfileinfo mfsgeteattr mfsgettrashtime mfsrgetgoal mfsrsettrashtime mfssetquota

/etc/mfs/下有配置文件:

mfsmount.cfg mfsmount.cfg.dist

keepalived配置

群集环境:

角色 IP
Master 10.1.220.85
Metalogger 10.1.220.86
VIP 10.1.220.87

1、master安装keepalived

tar xzf keepalived-1.2.19.tar.gz

cd keepalived-1.2.19cd keepalived-1.2.19

./configure

make&&make install

初始化配置文件

cp /usr/local/etc/rc.d/init.d/keepalived /etc/init.d/

cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/

mkdir /etc/keepalived/

cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/

cp /usr/local/sbin/keepalived /usr/sbin/

备份默认配置

mv /etc/keepalived/keepalived.conf /etc/keepalived/bk.keepalived.conf

新建/etc/keepalived/keepalived.conf,内容如下

global_defs {

notification_email {

zhengda@aspire.cn

}

notification_email_from warning@moose.fs

smtp_server localhost

smtp_connect_timeout 30

router_id MFS2016

}

vrrp_instance VI_1 {

state MASTER

interface eth0

virtual_router_id 86

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass HelloWorld123

}

virtual_ipaddress {

10.1.220.87

}

}

修改keepalived默认日志路径

sed -i ‘/^KEEPALIVED_OPTIONS/d’ /etc/sysconfig/keepalived

echo “KEEPALIVED_OPTIONS=\”-D -d -S 0\”” >> /etc/sysconfig/keepalived

echo “# keepalived -S 0 ” >> /etc/rsyslog.conf

echo “local0.* /var/log/keepalived.log” >> /etc/rsyslog.conf

/etc/init.d/rsyslog restart

/etc/init.d/keepalived restart

tail /var/log/keepalived.log

启动keepalived,查看/var/log/keepalived.log是否正常

/etc/init.d/keepalived start

/var/log/messages关于keepalived内容如下,已新增群集地址10.1.220.87:

Mar 11 14:36:03 localhost Keepalived[22703]: Starting Keepalived v1.2.19 (03/11,2016)

Mar 11 14:36:03 localhost Keepalived[22704]: Starting Healthcheck child process, pid=22705

Mar 11 14:36:03 localhost Keepalived[22704]: Starting VRRP child process, pid=22706

Mar 11 14:36:03 localhost Keepalived_vrrp[22706]: Netlink reflector reports IP 10.1.220.85 added

Mar 11 14:36:03 localhost Keepalived_vrrp[22706]: Netlink reflector reports IP fe80::47c:1eff:fe01:185d added

Mar 11 14:36:03 localhost Keepalived_vrrp[22706]: Registering Kernel netlink reflector

Mar 11 14:36:03 localhost Keepalived_vrrp[22706]: Registering Kernel netlink command channel

Mar 11 14:36:03 localhost Keepalived_vrrp[22706]: Registering gratuitous ARP shared channel

Mar 11 14:36:03 localhost Keepalived_vrrp[22706]: Opening file ‘/etc/keepalived/keepalived.conf’.

Mar 11 14:36:03 localhost Keepalived_vrrp[22706]: Truncating auth_pass to 8 characters

Mar 11 14:36:03 localhost Keepalived_vrrp[22706]: Configuration is using : 65866 Bytes

Mar 11 14:36:03 localhost Keepalived_vrrp[22706]: Using LinkWatch kernel netlink reflector…

Mar 11 14:36:03 localhost Keepalived_vrrp[22706]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]

Mar 11 14:36:03 localhost Keepalived_healthcheckers[22705]: Netlink reflector reports IP 10.1.220.85 added

Mar 11 14:36:03 localhost Keepalived_healthcheckers[22705]: Netlink reflector reports IP fe80::47c:1eff:fe01:185d added

Mar 11 14:36:03 localhost Keepalived_healthcheckers[22705]: Registering Kernel netlink reflector

Mar 11 14:36:03 localhost Keepalived_healthcheckers[22705]: Registering Kernel netlink command channel

Mar 11 14:36:03 localhost Keepalived_healthcheckers[22705]: Opening file ‘/etc/keepalived/keepalived.conf’.

Mar 11 14:36:03 localhost Keepalived_healthcheckers[22705]: Configuration is using : 10729 Bytes

Mar 11 14:36:03 localhost Keepalived_healthcheckers[22705]: Using LinkWatch kernel netlink reflector…

Mar 11 14:36:04 localhost Keepalived_vrrp[22706]: VRRP_Instance(VI_1) Transition to MASTER STATE

Mar 11 14:36:05 localhost Keepalived_vrrp[22706]: VRRP_Instance(VI_1) Entering MASTER STATE

Mar 11 14:36:05 localhost Keepalived_vrrp[22706]: VRRP_Instance(VI_1) setting protocol VIPs.

Mar 11 14:36:05 localhost Keepalived_vrrp[22706]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.220.87

Mar 11 14:36:05 localhost Keepalived_healthcheckers[22705]: Netlink reflector reports IP 10.1.220.87 added

Mar 11 14:36:10 localhost Keepalived_vrrp[22706]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.220.87

查看master的群集IP

[root@NFJD-YPTGLQ-JKPT-006 /]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 06:7c:1e:01:18:5d brd ff:ff:ff:ff:ff:ff

inet 10.1.220.85/24 brd 10.1.220.255 scope global eth0

inet 10.1.220.87/32 scope global eth0 ———————————————>此为群集IP,已生效

inet6 fe80::47c:1eff:fe01:185d/64 scope link

valid_lft forever preferred_lft forever

2、metalogger安装keepalived

tar xzf keepalived-1.2.19.tar.gz

./configure

make&&make install

初始化配置文件

cp /usr/local/etc/rc.d/init.d/keepalived /etc/init.d/

cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/

mkdir /etc/keepalived/

cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/

cp /usr/local/sbin/keepalived /usr/sbin/

备份默认配置

mv /etc/keepalived/keepalived.conf /etc/keepalived/bk.keepalived.conf

新建/etc/keepalived/keepalived.conf,内容如下

global_defs {

notification_email {

zhengda@aspire.cn

}

notification_email_from warning@moose.fs

smtp_server localhost

smtp_connect_timeout 30

router_id MFS2016

}

vrrp_instance VI_1 {

state BACKUP

interface eth0

virtual_router_id 86

priority 99

advert_int 1

authentication {

auth_type PASS

auth_pass HelloWorld123

}

virtual_ipaddress {

10.1.220.87

}

}

修改keepalived默认日志路径

sed -i ‘/^KEEPALIVED_OPTIONS/d’ /etc/sysconfig/keepalived

echo “KEEPALIVED_OPTIONS=\”-D -d -S 0\”” >> /etc/sysconfig/keepalived

echo “# keepalived -S 0 ” >> /etc/rsyslog.conf

echo “local0.* /var/log/keepalived.log” >> /etc/rsyslog.conf

/etc/init.d/rsyslog restart

/etc/init.d/keepalived restart

tail /var/log/keepalived.log

启动keepalived,查看/var/log/keepalived.log是否正常

/etc/init.d/keepalived start

/var/log/messages关于keepalived内容如下,可见metalogger的keepalived进入了backup状态:

Mar 11 15:11:35 localhost Keepalived[8170]: Starting Keepalived v1.2.19 (03/11,2016)

Mar 11 15:11:35 localhost Keepalived[8171]: Starting Healthcheck child process, pid=8172

Mar 11 15:11:35 localhost Keepalived[8171]: Starting VRRP child process, pid=8173

Mar 11 15:11:35 localhost Keepalived_healthcheckers[8172]: Netlink reflector reports IP 10.1.220.86 added

Mar 11 15:11:35 localhost Keepalived_healthcheckers[8172]: Netlink reflector reports IP fe80::416:7eff:fe01:185e added

Mar 11 15:11:35 localhost Keepalived_healthcheckers[8172]: Registering Kernel netlink reflector

Mar 11 15:11:35 localhost Keepalived_healthcheckers[8172]: Registering Kernel netlink command channel

Mar 11 15:11:35 localhost Keepalived_healthcheckers[8172]: Opening file ‘/etc/keepalived/keepalived.conf’.

Mar 11 15:11:35 localhost Keepalived_healthcheckers[8172]: Configuration is using : 10727 Bytes

Mar 11 15:11:35 localhost Keepalived_vrrp[8173]: Netlink reflector reports IP 10.1.220.86 added

Mar 11 15:11:35 localhost Keepalived_vrrp[8173]: Netlink reflector reports IP fe80::416:7eff:fe01:185e added

Mar 11 15:11:35 localhost Keepalived_vrrp[8173]: Registering Kernel netlink reflector

Mar 11 15:11:35 localhost Keepalived_vrrp[8173]: Registering Kernel netlink command channel

Mar 11 15:11:35 localhost Keepalived_vrrp[8173]: Registering gratuitous ARP shared channel

Mar 11 15:11:35 localhost Keepalived_vrrp[8173]: Opening file ‘/etc/keepalived/keepalived.conf’.

Mar 11 15:11:35 localhost Keepalived_vrrp[8173]: Truncating auth_pass to 8 characters

Mar 11 15:11:35 localhost Keepalived_vrrp[8173]: Configuration is using : 65864 Bytes

Mar 11 15:11:35 localhost Keepalived_vrrp[8173]: Using LinkWatch kernel netlink reflector…

Mar 11 15:11:35 localhost Keepalived_healthcheckers[8172]: Using LinkWatch kernel netlink reflector…

Mar 11 15:11:35 localhost Keepalived_vrrp[8173]: VRRP_Instance(VI_1) Entering BACKUP STATE ——>进入backup状态

Mar 11 15:11:35 localhost Keepalived_vrrp[8173]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]

3、keepalived可用性切换测试

停止master上的keepalived

/etc/init.d/keepalived stop

观察metalogger的/var/log/messages变化,keepalived进入master状态,且群集地址已切换到metalogger。

Mar 11 15:11:35 localhost Keepalived_vrrp[8173]: VRRP_Instance(VI_1) Entering BACKUP STATE

Mar 11 15:11:35 localhost Keepalived_vrrp[8173]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]

Mar 11 15:15:51 localhost Keepalived_vrrp[8173]: VRRP_Instance(VI_1) Transition to MASTER STATE

Mar 11 15:15:52 localhost Keepalived_vrrp[8173]: VRRP_Instance(VI_1) Entering MASTER STATE ——->进入master状态

Mar 11 15:15:52 localhost Keepalived_vrrp[8173]: VRRP_Instance(VI_1) setting protocol VIPs.

Mar 11 15:15:52 localhost Keepalived_vrrp[8173]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.220.87

Mar 11 15:15:52 localhost Keepalived_healthcheckers[8172]: Netlink reflector reports IP 10.1.220.87 added

持续ping群集地址10.1.220.87,在切换过程中有3-5秒左右的时间是不通的

4、master和metalogger切换测试

在metalogger的keepalived中加入以下配置,意思是当keepalived检测到状态转为master时,执行目标脚本(注意脚本或者目录权限设置为755,否则会因权限问题导致脚本没有执行,并且在修改keepalived的脚本路径时务必要重启keepalived)

vrrp_sync_group VG1 {

group {

VI_1

}

notify_master “/opt/to_master.sh”

}

to_master.sh脚本内容如下,停止metalogger,启动master并通过changelog恢复metadata,再启动web界面

#!/bin/bash

/usr/local/mfs/sbin/mfsmetalogger stop

/usr/local/mfs/sbin/mfsmaster -a

/usr/local/mfs/sbin/mfscgiserv start

三台chunkserver修改hosts,将mfsmaster指向10.1.220.87,并重启chunkserver。

/usr/local/mfs/sbin/mfschunkserver restart

在客户端10.1.175.8重新挂载指向群集地址10.1.220.87;并执行脚本,持续对挂载的mfs目录的文件进行写入操作:

/usr/local/mfs/bin/mfsmount /opt/test/ -H 10.1.220.87 -S /test/

停止master的keepalived服务

/etc/init.d/keepalived stop

触发metalogger的keepalived执行to_master.sh脚本

查看写入程序结果,切换前后时间会有几十秒停滞。

MFS版本升级方法

当前融合通信MFS版本为2.0.81,而官方最新稳定版为2.0.88,以下为升级到新版2.0.88的操作过程:

1、master升级

为防止产生数据差异,停止mfsmaster,将旧版目录转移改名

mv /usr/local/mfs/ /usr/local/2.0.81.mfs

安装新版

cd ~/moosefs-2.0.88

./configure –prefix=/usr/local/mfs –with-default-user=mfs –with-default-group=mfs –disable-mfschunkserver

make&&make install

生成如下目录:

[root@NFJD-YPTGLQ-JKPT-006 /]# /usr/local/mfs/

bin/ etc/ sbin/ share/ var/

查看版本

[root@NFJD-YPTGLQ-JKPT-006 /]# /usr/local/mfs/sbin/mfsmaster -v

version: 2.0.88-1

恢复meta数据

rm -fr /usr/local/mfs/var/mfs/*

cp /usr/local/2.0.81.mfs/var/mfs/* /usr/local/mfs/var/mfs/

恢复配置文件

cd /usr/local/mfs/etc/mfs/

mkdir bk

mv *.dist bk

cp /usr/local/2.0.81.mfs/etc/mfs/* /usr/local/mfs/etc/mfs/

启动mfsmaster

[root@NFJD-YPTGLQ-JKPT-006 mfs]# /usr/local/mfs/sbin/mfsmaster start

open files limit has been set to: 4096

working directory: /usr/local/mfs/var/mfs

lockfile created and locked

initializing mfsmaster modules …

exports file has been loaded

mfstopology configuration file (/usr/local/mfs/etc/mfstopology.cfg) not found – using defaults

loading metadata …

loading sessions data … ok (0.0000)

loading objects (files,directories,etc.) … ok (0.2840)

loading names … ok (0.7050)

loading deletion timestamps … ok (0.0000)

loading quota definitions … ok (0.0000)

loading xattr data … ok (0.0000)

loading posix_acl data … ok (0.0000)

loading open files data … ok (0.0000)

loading chunkservers data … ok (0.0000)

loading chunks data … ok (0.1386)

checking filesystem consistency … ok

connecting files and chunks … ok

all inodes: 1000110

directory inodes: 104

file inodes: 1000006

chunks: 1000006

metadata file has been loaded

stats file has been loaded

master <-> metaloggers module: listen on *:9419

master <-> chunkservers module: listen on *:9420

main master server module: listen on *:9421

mfsmaster daemon initialized properly

启动web界面

[root@NFJD-YPTGLQ-JKPT-006 mfs]# /usr/local/mfs/sbin/mfscgiserv start

lockfile created and locked

starting simple cgi server (host: any , port: 9425 , rootpath: /usr/local/mfs/share/mfscgi)

版本已更新为2.0.88

[root@NFJD-YPTGLQ-JKPT-006 sbin]# /usr/local/mfs/sbin/mfsmaster -v

version: 2.0.88-1

2、metalogger升级

停止metalogger服务

mv /usr/local/mfs/ /usr/local/2.0.81.mfs

cd ~/moosefs-2.0.88

./configure –prefix=/usr/local/mfs –with-default-user=mfs –with-default-group=mfs –disable-mfschunkserver

make&&make install

查看版本

[root@NFJD-YPTGLQ-JKPT-007 moosefs-2.0.88]# /usr/local/mfs/sbin/mfsmetalogger -v

version: 2.0.88-1

恢复配置文件(meta数据无需恢复,直接从mfsmaster重新同步)

cd /usr/local/mfs/etc/mfs/

mkdir bk

mv *.dist bk

cp /usr/local/2.0.81.mfs/etc/mfs/* /usr/local/mfs/etc/mfs/

启动metalogger

[root@NFJD-YPTGLQ-JKPT-007 mfs]# /usr/local/mfs/sbin/mfsmetalogger start

open files limit has been set to: 4096

working directory: /usr/local/mfs/var/mfs

lockfile created and locked

initializing mfsmetalogger modules …

mfsmetalogger daemon initialized properly

版本已更新为2.0.88

[root@NFJD-YPTGLQ-JKPT-007 ~]# /usr/local/mfs/sbin/mfsmetalogger -v

version: 2.0.88-1

3、chunkserver升级

停止chunkserver

mv /usr/local/mfs/ /usr/local/2.0.81.mfs

cd ~/moosefs-2.0.88

./configure –prefix=/usr/local/mfs –disable-mfsmaster –disable-mfsmetalogger

make&&make install

查看版本

[root@NFJD-YPTGLQ-JKPT-001 moosefs-2.0.88]# /usr/local/mfs/sbin/mfschunkserver -v

version: 2.0.88-1

恢复配置文件

cd /usr/local/mfs/etc/mfs/

mkdir bk

mv *.dist bk

cp /usr/local/2.0.81.mfs/etc/mfs/* /usr/local/mfs/etc/mfs/

恢复var数据

rm -fr /usr/local/mfs/var/mfs/*

cp /usr/local/2.0.81.mfs/var/mfs/* /usr/local/mfs/var/mfs/

修改mfs目录权限,此步很重要,否则chunkserver无法启动,因为复制/usr/local/2.0.81.mfs/var/mfs/chunkserverid.mfs的时候,此文件的属主已变为root,且权限为640,导致mfschunkserver无法读取而报错(hdd space manager: chunkserver without meta id shouldn’t use drive with defined meta id)

chown -R mfs.mfs /usr/local/mfs/

启动chunkserver

[root@NFJD-YPTGLQ-JKPT-001 mfs]# /usr/local/mfs/sbin/mfschunkserver start

open files limit has been set to: 16384

working directory: /usr/local/mfs/var/mfs

lockfile created and locked

setting glibc malloc arena max to 8

setting glibc malloc arena test to 1

initializing mfschunkserver modules …

hdd space manager: path to scan: /mfs02/

hdd space manager: path to scan: /mfs/

hdd space manager: start background hdd scanning (searching for available chunks)

main server module: listen on *:9422

stats file has been loaded

mfschunkserver daemon initialized properly

版本已更新为2.0.88

[root@NFJD-YPTGLQ-JKPT-001 ~]# /usr/local/mfs/sbin/mfschunkserver -v

version: 2.0.88-1

以下是可批量执行的脚本,用于多台chunkserver版本升级

———————————————————————————————–

/usr/local/mfs/sbin/mfschunkserver stop

mv /usr/local/mfs/ /usr/local/2.0.81.mfs

cd /root/

tar xzf moosefs-2.0.88-1.tar.gz

cd moosefs-2.0.88

./configure –prefix=/usr/local/mfs –with-default-user=mfs –with-default-group=mfs –disable-mfsmaster –disable-mfsmetalogger

make&&make install

cd /usr/local/mfs/etc/mfs/

mkdir bk

mv *.dist bk

cp /usr/local/2.0.81.mfs/etc/mfs/* /usr/local/mfs/etc/mfs/

rm -fr /usr/local/mfs/var/mfs/*

cp /usr/local/2.0.81.mfs/var/mfs/* /usr/local/mfs/var/mfs/

chown -R mfs.mfs /usr/local/mfs/

/usr/local/mfs/sbin/mfschunkserver start

/usr/local/mfs/sbin/mfschunkserver -v

———————————————————————————————–

融合通信MFS所有角色版本已全都升级为2.0.88,如下图:

MFS配置文件说明

1、master

  • mfsmaster.cfg mfs主配置文件(定义相关参数,用户、组等设定)
#WORKING_USER和WORKING_GROUP:是运行master server的用户和组;

#SYSLOG_IDENT:是master server在syslog中的标识,也就是说明这是由master server产生的;

#LOCK_MEMORY:是否执行mlockall()以避免mfsmaster 进程溢出(默认为0,即否);

#NICE_LEVE:运行的优先级(如果可以默认是 -19; 注意: 进程必须是用root启动);

#EXPORTS_FILENAME:被挂接目录及其权限控制文件的存放位置

#DATA_PATH:metadata files and lock file存放路径,此目录下大致有以下文件:metadata,changelog,sessions,stats,lock。

#BACK_LOGS:metadata的change log文件数目(默认是 50);

#REPLICATIONS_DELAY_INIT:(initial delay in seconds before starting replications)初始延迟复制的时间(默认是300s);

#REPLICATIONS_DELAY_DISCONNECT:(replication delay in seconds after chunkserver disconnection) chunkserver断开后复制延迟(默认是3600s);

#MATOML_LISTEN_HOST:用于metalogger连接的IP地址(默认是*,代表任何IP);

#MATOML_LISTEN_PORT:监听metalogger请求的端口地址(默认是9419);

#MATOCS_LISTEN_HOST:用于chunkserver连接的IP地址(默认是*,代表任何IP);

#MATOCS_LISTEN_PORT:监听chunkserver连接的端口地址(默认是9420);

#MATOCU_LISTEN_HOST:用于客户端挂接连接的IP地址(默认是*,代表任何IP);

#MATOCU_LISTEN_PORT:监听客户端挂载连接的端口地址(默认是9421);

#CHUNKS_LOOP_TIME :(Chunks loop frequency in seconds)chunks的回环频率(默认是:300秒);

#CHUNKS_DEL_LIMIT:(Maximum number of chunks to delete in one loop)在一个loop中可以删除chunks的最大数 (默认:100)

#CHUNKS_WRITE_REP_LIMIT:(Maximum number of chunks to replicate to one chunkserver in one loop)在一个loop里复制到一个chunkserver的最大chunk数目(默认是1)

#CHUNKS_READ_REP_LIMIT:(Maximum number of chunks to replicate from one chunkserver in one loop)在一个loop里从一个chunkserver复制的最大chunk数目(默认是5)

#REJECT_OLD_CLIENTS:弹出低于1.6.0的客户端挂接(0或1,默认是0)

  • mfsexports.cfg mfs被挂接目录及其权限控制文件
该文件每一个条目分为三部分:

第一部分:客户端的ip 地址

第二部分:被挂接的目录

第三部分:客户端拥有的权限

修改完此文件后,需要执行mfsmaster reload才能生效,否则客户端挂载新建的目录权限时会报错:mfsmaster register error: Permission denied

文件内容项用tab键来隔开

(1)地址可以指定的几种表现形式:

*所有的ip 地址

n.n.n.n单个ip 地址

n.n.n.n/bIP 网络地址/位数掩码

n.n.n.n/m.m.m.mIP 网络地址/子网掩码

f.f.f.f-t.t.t.tIP 段

(2)目录部分需要注意两点:

/标识MooseFS 根;

.表示MFSMETA 文件系统

(3)权限部分:

ro只读模式共享

rw读写的方式共享

alldirs允许挂载任何指定的子目录

maproot映射为root,还是指定的用户

password指定客户端密码

md5pass指定客户端MD5密码

以下配置会让password无效,要把定义根目录的alldirs权限删掉

10.1.220.88 /zdtest1 rw,alldirs,admin,maproot=0:0,password=111111

10.1.220.88 / rw,alldirs,admin,maproot=0:0

  • mfstopology.cfg 定义 MFS 网络拓扑结构的文件

此配置用于复杂网络结构才需配置,默认留空即可。

2、metalogger

  • mfsmetalogger.cfg 主配置文件
# WORKING_USER = mfs

# WORKING_GROUP = mfs

# SYSLOG_IDENT = mfsmetalogger

# LOCK_MEMORY = 0

# NICE_LEVEL = -19

# DATA_PATH = /usr/local/mfs/var/mfs

# BACK_LOGS = 50

# META_DOWNLOAD_FREQ = 1 # metadata download frequency in hours (default is 24, at most BACK_LOGS/2) 。metadata元数据下载间隔时间(默认是24小时,单位是小时,至多是BACK_LOGS的1/2)

# MASTER_RECONNECTION_DELAY = 5 # delay in seconds before trying to reconnect to master after disconnection。在失去连接之后延迟多少秒重新连接master

# MASTER_HOST = 10.1.220.87

# MASTER_PORT = 9419

# MASTER_TIMEOUT = 60 # timeout (in seconds) for master connections。Master连接超时时间(单位是秒)

# deprecated, to be removed in MooseFS 1.7

# LOCK_FILE = /var/run/mfs/mfsmetalogger.lock

3、chunkserver

  • mfschunkserver.cfg 主配置文件
# WORKING_USER = mfs

# WORKING_GROUP = mfs

# SYSLOG_IDENT = mfschunkserver

# LOCK_MEMORY = 0

# NICE_LEVEL = -19

# DATA_PATH = /usr/local/mfs/var/mfs

# MASTER_RECONNECTION_DELAY = 5 (delay in seconds before trying to reconnect to master after disconnection)在失去连接之后延迟多少秒重新连接master

# MASTER_HOST = 10.1.220.87 元数据服务器的名称或地址,可以是主机名,也可以是ip地址。只要数据存储服务器能访问到元数据服务器就行。

# MASTER_PORT = 9420

# MASTER_TIMEOUT = 60

# CSSERV_LISTEN_HOST = * #IP address to listen on for client (mount) connections (* means any) 允许挂载的客户端连接的IP地址(*允许全部)

# CSSERV_LISTEN_PORT = 9422

# CSSERV_TIMEOUT = 5 #timeout (in seconds) for client (mount) connections 客户端挂载连接的超时时间(单位为秒)

# HDD_CONF_FILENAME = /usr/local/mfs/etc/mfshdd.cfg #分配给MFS使用的磁盘空间配置文件的位置

# HDD_TEST_FREQ = 10 #chunk test period in seconds 块的测试期(单位为秒)

# deprecated, to be removed in MooseFS 1.7

# LOCK_FILE = /var/run/mfs/mfschunkserver.lock

# BACK_LOGS = 50

  • mfshdd.cfg 本地存储挂载配置文件
建议在chunkserver上划分单独的空间给MooseFS使用,这样做的好处是便于管理剩余空间。最好是一个单独的硬盘或者一个raid卷,且单个存储不小于2G(因为chunk初始化磁盘时需占用一定空间)。

MFS主要的命令

1、master角色

mfsmaster

  • /usr/local/mfs/sbin/mfsmaster 用于master进程的启动和停止,重启

用法:

mfsmaster [-c CFGFILE] [-u] [-f] [-t LOCKTIMEOUT] [ACTION]

  • -v 打印版本号
  • -h 试用帮助
  • -c指定配置文件路径
  • -u未定义值的log
  • -f 前端运行,不进程化
  • -t等待lockfile的时间,默认60秒
  • ACTION 支持参数start, stop, restart, reload, test 和kill,不加参数默认为start;使用test会显示”mfsmaster pid: PID”或者”mfsmaster is not running”;kill动作会发送一个SIGKILL给到当前运行的master进程;SIGHUP 或者 reload用来使master重新读取所有的配置文件。
  • mfsmetarestore用来把changelog恢复metadata,已在版本1.7以后去除,现在是用mfsmaster –a来代替
  • mfsmetadump用来将metadata导出为文件
  • mfsmaster进程内存占用测试
100W个文件

[root@NFJD-YPTGLQ-JKPT-006 mfs]# ps aux –sort -rss |grep -E “mfsmaster|COMMAND”

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

mfs 20113 0.2 10.2 845952 825684 ? S< Mar02 34:47 /usr/local/mfs/sbin/mfsmaster start

root 18410 0.0 0.0 103244 816 pts/0 S+ 18:14 0:00 grep -E mfsmaster|COMMAND

200W文件

[root@NFJD-YPTGLQ-JKPT-006 mfs]# ps aux –sort -rss |grep -E “mfsmaster|COMMAND”

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

mfs 20113 0.2 13.0 1076868 1055660 ? S< Mar02 39:41 /usr/local/mfs/sbin/mfsmaster start

root 18970 0.0 0.0 103244 820 pts/0 S+ 19:25 0:00 grep -E mfsmaster|COMMAND

300W文件

[root@NFJD-YPTGLQ-JKPT-006 mfs]# ps aux –sort -rss |grep -E “mfsmaster|COMMAND”

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

mfs 20113 0.2 15.8 1300636 1279508 ? S< Mar02 44:47 /usr/local/mfs/sbin/mfsmaster start

root 19558 0.0 0.0 103244 820 pts/0 S+ 20:40 0:00 grep -E mfsmaster|COMMAND

400W文件

[root@NFJD-YPTGLQ-JKPT-006 mfs]# ps aux –sort -rss |grep -E “mfsmaster|COMMAND”

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

mfs 20113 0.3 18.7 1531252 1510364 ? S< Mar02 49:45 /usr/local/mfs/sbin/mfsmaster start

root 19968 0.0 0.0 103244 820 pts/0 S+ 21:31 0:00 grep -E mfsmaster|COMMAND

%MEM 内存使用占比

VSZ 所使用的虚拟内存大小

RSS 所使用的物理内存大小

–sort -rss 按物理内存使用大小排序

结论:增加100w文件,内存使用平均增加226912 KB

命令行接口mfscli

/usr/local/mfs/bin/mfscli 用来监控MFS状态的命令行工具

/usr/local/mfs/bin/mfscli –SIN 显示master的详细信息

CGI Server

/usr/local/mfs/sbin/mfscgiserv,mfs的web界面工具,启动后默认监听9425端口,使用浏览器来访问,用于查看mfs的运行状态,包括挂载的目录、在线客户端、配额、负载情况等等。

mfscgiserv支持的参数有start, stop, restart or test,不加参数默认为start;test动作用于返回两种信息:

ACTION is one of start, stop, restart or test. Default action is restart. The test action

”mfscgiserv pid: PID”或者 ”mfscgiserv is not running”。

2、metalogger角色

/usr/local/mfs/sbin/mfsmetalogger metalogger的主程序

mfsmetalogger支持的参数有start, stop, restart, reload, test 和 kill;不加参数默认是restart。

3、chunkserver角色

/usr/local/mfs/sbin/mfschunkserver chunkserver的主程序

mfschunkserver支持的参数有start, stop, restart, reload, test 和 kill;不加参数默认为restart;使用test会显示” mfschunkserver pid: PID”或者” mfschunkserver is not running”;kill动作会发送一个SIGKILL给到当前运行的chunkserver进程;SIGHUP 或者 reload用来使chunkserver重新读取所有的配置文件。

4、mfs客户端

mfsmount

/usr/local/mfs/bin/mfsmount

1、挂载数据分区

挂载根目录,mfsexports.cfg增加配置 10.1.220.88 / rw,admin,maproot=0:0

mfsmount -H 10.1.220.87 /opt/mfs_root_dir/

挂载子目录,mfsexports.cfg增加配置10.1.220.88 /zdtest1 rw,alldirs,admin,maproot=0:01

mfsmount -H 10.1.220.87 -S /zdtest1/ /opt/zdtest1/

mfsmount -H 10.1.220.87 -o mfssubfolder=/zdtest1/ /opt/zdtest1/

使用密码挂载,mfsexports.cfg增加配置10.1.220.88 /zdtest1 rw,alldirs,admin,maproot=0:0,password=111111

mfsmount -H 10.1.220.87 -S /zdtest1/ /opt/zdtest1/ -o mfspassword=111111

使用MD5密文挂载

mfsexports.cfg增加配置10.1.220.88 /zd_md5 ro,maproot=0:0,md5pass=e10adc3949ba59abbe56e057f20f883e

echo -n 123456 | openssl md5 获得密文e10adc3949ba59abbe56e057f20f883e

mfsmount -H 10.1.220.87 -S /zd_md5/ /opt/zd_md5/ -o mfsmd5pass=e10adc3949ba59abbe56e057f20f883e

也可以使用明文密码挂载

mfsmount -H 10.1.220.87 -o mfssubfolder=/zd_md5/ /opt/zd_md5/ -o mfspassword=123456

2、挂载MFSMETA分区,参数-m, –meta, -o mfsmeta都可以

/usr/local/mfs/bin/mfsmount -H 10.1.220.87 -m /opt/zdMFSMETA/

mount查看挂载情况,MFSMETA分区的类型为fuse.mfsmeta,而数据分区的type为fuse.mfs

[root@NFJD-YPTGLQ-JKPT-005 mnt]# mount|grep :9421

10.1.220.85:9421 on /opt/mfsdata type fuse.mfsmeta (rw,nosuid,nodev,allow_other)

10.1.220.85:9421 on /mnt type fuse.mfs (rw,nosuid,nodev,allow_other)

10.1.220.87:9421 on /opt/test type fuse.mfs (rw,nosuid,nodev,allow_other)

10.1.220.87:9421 on /opt/zdtest type fuse.mfs (rw,nosuid,nodev,allow_other)

10.1.220.87:9421 on /opt/zdMFSMETA type fuse.mfsmeta (rw,nosuid,nodev,allow_other)

mfs客户端工具

1、mfsgetgoal、mfssetgoal、mfscheckfile、mfsfileinfo,副本数量设置与状态查询

其中mfsrgetgoal和mfsrsetgoal分别相当于mfsgetgoal -r 、mfssetgoal -r,即递归执行,已不推荐使用

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfsfileinfo sll.txt

sll.txt:

chunk 0: 00000000003D3446_00000001 / (id:4011078 ver:1)

copy 1: 10.1.175.5:9422

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfscheckfile sll.txt

sll.txt:

chunks with 1 copy: 1

 

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfsgetgoal sll.txt

sll.txt: 1

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfssetgoal 3 sll.txt

sll.txt: 3

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfsgetgoal sll.txt

sll.txt: 3

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfsfileinfo sll.txt

sll.txt:

chunk 0: 00000000003D3446_00000001 / (id:4011078 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.5:9422

 

设置副本数位4时,还是只有3个副本?因为chunkserver只有3台,生效的副本数量的上限就是在线chunkserver的数量。

假设当前chunkserver只有3台,设置4副本,当后面新加一台chunkserver,再查看副本数量是否为4。

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfssetgoal 4 sll.txt

sll.txt: 4

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfsfileinfo sll.txt

sll.txt:

chunk 0: 00000000003D3446_00000001 / (id:4011078 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.5:9422

copy 3: 10.1.175.6:9422

 

生成100M的文件,再测试,因mfs会对文件数据进行分片,形成chunk,每个chunk为64MB,所以100M文件需要两个chunk。

而上述的sll.txt才十几个字节,所以只需要一个chunk

[root@NFJD-YPTGLQ-JKPT-005 test]# dd if=/dev/zero of=100m.file bs=1M count=100

100+0 records in

100+0 records out

104857600 bytes (105 MB) copied, 0.424163 s, 247 MB/s

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfsfileinfo 100m.file

100m.file:

chunk 0: 00000000003D3447_00000001 / (id:4011079 ver:1)

copy 1: 10.1.175.4:9422

chunk 1: 00000000003D3448_00000001 / (id:4011080 ver:1)

copy 1: 10.1.175.6:9422

 

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfscheckfile 100m.file

100m.file:

chunks with 1 copy: 2

 

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfsgetgoal 100m.file

100m.file: 1

设置副本数量为2

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfssetgoal 2 100m.file

100m.file: 2

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfsgetgoal 100m.file

100m.file: 2

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfscheckfile 100m.file

100m.file:

chunks with 2 copies: 2

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfsfileinfo 100m.file

100m.file:

chunk 0: 00000000003D3447_00000001 / (id:4011079 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.5:9422

chunk 1: 00000000003D3448_00000001 / (id:4011080 ver:1)

copy 1: 10.1.175.5:9422

copy 2: 10.1.175.6:9422

设置副本数量为5

因为chunkserver只有3台,生效的副本数量的上限就是在线chunkserver的数量。

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfssetgoal 5 zd.txt

zd.txt: 5

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfscheckfile zd.txt

zd.txt:

chunks with 3 copies: 1

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfssetgoal 5 100m.file

100m.file: 5

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfscheckfile 100m.file

100m.file:

chunks with 3 copies: 2

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfsfileinfo 100m.file

100m.file:

chunk 0: 00000000003D3447_00000001 / (id:4011079 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.5:9422

copy 3: 10.1.175.6:9422

chunk 1: 00000000003D3448_00000001 / (id:4011080 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.5:9422

copy 3: 10.1.175.6:9422

 

 

递归模式,包括目录和文件都设置了副本

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfssetgoal -r 2 /opt/test/tmp/

/opt/test/tmp/:

inodes with goal changed: 4

inodes with goal not changed: 0

inodes with permission denied: 0

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsgetgoal -r /opt/test/tmp/

/opt/test/tmp/:

files with goal 2 : 3

directories with goal 2 : 1

 

修改其中一个文件的副本数为3,再进行递归查询副本数

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfssetgoal 3 /opt/test/tmp/100m.file

/opt/test/tmp/100m.file: 3

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsgetgoal -r /opt/test/tmp/

/opt/test/tmp/:

files with goal 2 : 2

files with goal 3 : 1

directories with goal 2 : 1

 

当对目录设置副本时,后续在此目录新建的对象(文件和目录)都会继承副本的设置

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfssetgoal 3 /opt/test/tmp1/

/opt/test/tmp1/: 3

[root@NFJD-YPTGLQ-JKPT-005 test]# /usr/local/mfs/bin/mfsgetgoal /opt/test/tmp1/

/opt/test/tmp1/: 3

[root@NFJD-YPTGLQ-JKPT-005 test]# cd /opt/test/tmp1

[root@NFJD-YPTGLQ-JKPT-005 tmp1]# dd if=/dev/zero of=10M.file bs=1M count=10 新建文件10M.file

10+0 records in

10+0 records out

10485760 bytes (10 MB) copied, 0.0635792 s, 165 MB/s

[root@NFJD-YPTGLQ-JKPT-005 tmp1]# /usr/local/mfs/bin/mfsgetgoal 10M.file —>查看10M.file的goal设置,已经默认为3,系统全局goal是1

10M.file: 3

[root@NFJD-YPTGLQ-JKPT-005 tmp1]# /usr/local/mfs/bin/mfsfileinfo 10M.file —>查看10M.file的详情

10M.file:

chunk 0: 00000000003D3450_00000001 / (id:4011088 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.5:9422

copy 3: 10.1.175.6:9422

2、mfsrgettrashtime mfssettrashtime,文件实际从mfs彻底删除的时间设置与查询,也支持递归参数-r,默认回收时间为86400秒(24小时)

root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsgettrashtime -r /opt/test/tmp/

/opt/test/tmp/:

files with trashtime 86400 : 3

directories with trashtime 86400 : 1

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsgettrashtime /opt/test/tmp/

/opt/test/tmp/: 86400

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsgettrashtime /opt/test/tmp/100m.file

/opt/test/tmp/100m.file: 86400

设置回收时间为60秒

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfssettrashtime 60 /opt/test/tmp/100m.file

/opt/test/tmp/100m.file: 60

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsgettrashtime -r /opt/test/tmp/

/opt/test/tmp/:

files with trashtime 60 : 1

files with trashtime 86400 : 2

directories with trashtime 86400 : 1

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsgettrashtime /opt/test/tmp/100m.file

/opt/test/tmp/100m.file: 60

3、mfsdirinfo用来查看对象的摘要数量,对象可以是文件或者目录,摘要内容包括inodes、目录、文件等数量的信息。

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsdirinfo /opt/test/tmp/

/opt/test/tmp/:

inodes: 3

directories: 1

files: 2

chunks: 4

length: 209715200

size: 209735680

realsize: 419471360

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsdirinfo /opt/test/tmp/100m.file.1

/opt/test/tmp/100m.file.1:

inodes: 1

directories: 0

files: 1

chunks: 2

length: 104857600

size: 104867840

realsize: 209735680

4、mfsfilerepair

用于修复出现读取I/O错误的损坏文件,使其可读。

情况一:丢失的chunk,会将丢失的chunk全部用0来填充。

情况二:版本不匹配的chunk,会将这些chunk的版本设置为mfsmaster已知的,并且在chunkserver中能找到最高的版本。

第二种情况建议是待文件修复完成后,复制一份,然后将原数据删除。

5、mfsappendchunks 目标文件 源文件

追加chunks到一个文件,追加文件块到另一个文件.如果目标文件不存在,则会创建一个空文件,然后继续将块进行追加

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsappendchunks 100m.file.1.appendchunk 100m.file.1

[root@NFJD-YPTGLQ-JKPT-005 tmp]# ls

100m.file.1 100m.file.1.appendchunk 100m.file.2

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsfileinfo 100m.file.1.appendchunk

100m.file.1.appendchunk:

chunk 0: 00000000003D344C_00000001 / (id:4011084 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.6:9422

chunk 1: 00000000003D344D_00000001 / (id:4011085 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.5:9422

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsfileinfo 100m.file.1

100m.file.1:

chunk 0: 00000000003D344C_00000001 / (id:4011084 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.6:9422

chunk 1: 00000000003D344D_00000001 / (id:4011085 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.5:9422

6、mfsmakesnapshot

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsmakesnapshot 100m.file.1 100m.file.1.makesnapshot

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsfileinfo 100m.file.1.makesnapshot

100m.file.1.makesnapshot:

chunk 0: 00000000003D344C_00000001 / (id:4011084 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.6:9422

chunk 1: 00000000003D344D_00000001 / (id:4011085 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.5:9422

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsfileinfo 100m.file.1

100m.file.1:

chunk 0: 00000000003D344C_00000001 / (id:4011084 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.6:9422

chunk 1: 00000000003D344D_00000001 / (id:4011085 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.5:9422

7、mfsappendchunks和mfsmakesnapshot两者区别

mfsappendchunks相当于浅复制,mfsmakesnapshot相当于深复制。

[root@NFJD-YPTGLQ-JKPT-005 tmp]# !dd

dd if=/dev/zero of=10M.file bs=1M count=10

10+0 records in

10+0 records out

10485760 bytes (10 MB) copied, 0.0376699 s, 278 MB/s

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsfileinfo 10M.file ———>新建一个文件

10M.file:

chunk 0: 00000000003D3456_00000001 / (id:4011094 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.6:9422

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsappendchunks 10M.file.appendchunk 10M.file ———>使用mfsappendchunks生成文件

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsmakesnapshot 10M.file 10M.file.makesnapshot ———>使用mfsmakesnapshot生成文件

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsfileinfo * ———>查看文件信息,chunk均为一样

10M.file:

chunk 0: 00000000003D3456_00000001 / (id:4011094 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.6:9422

10M.file.appendchunk:

chunk 0: 00000000003D3456_00000001 / (id:4011094 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.6:9422

10M.file.makesnapshot:

chunk 0: 00000000003D3456_00000001 / (id:4011094 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.6:9422

[root@NFJD-YPTGLQ-JKPT-005 tmp]# echo “111” >> 10M.file ———>分别修改三个文件

[root@NFJD-YPTGLQ-JKPT-005 tmp]# echo “111” >> 10M.file.appendchunk

[root@NFJD-YPTGLQ-JKPT-005 tmp]# echo “111” >> 10M.file.makesnapshot

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfsfileinfo *

10M.file:

chunk 0: 00000000003D3457_00000001 / (id:4011095 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.6:9422

10M.file.appendchunk:

chunk 0: 00000000003D3458_00000001 / (id:4011096 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.6:9422

10M.file.makesnapshot: ———>除了mfsmakesnapshot的文件chunk不变,其他两个改变

chunk 0: 00000000003D3456_00000001 / (id:4011094 ver:1)

copy 1: 10.1.175.4:9422

copy 2: 10.1.175.6:9422

8、mfsgetquota和mfssetquota,mfs的容量配额设置工具,小写是软配额,大写是硬配额。

硬配额无法违反,软配额可以,有效期为7天(默认无法更改)

-i -I inode 目录inode数量的限额

-l -L length 文件长度的限额

-s -S size 客户端能使用的空间(不受goal影响)

-r -R realsize 实际占用的空间(文件大小*goal数量)

  • length和size的关系,一个文件的size比length多出5K(the chunk uses really 64 MiB of data + 5KiB header)
设置目录的文件大小的硬配额

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfssetquota -S 100M /opt/test/tmp/

/opt/test/tmp/: (current values | soft quota | hard quota)

inodes | 3 | – | – |

length | 31457304 | – | – |

size | 31669248 | – | 100000000 |

realsize | 63338496 | – | – |

测试写入大小为200M的文件,报错Disk quota exceeded,文件有写入,大小为64M,因为一个chunk的大小就是64M。

200M的文件需要4个chunk,当写完一个chunk后,已经占用了64M的配额,在写入第二个chunk时,实际使用量为128M?

[root@NFJD-YPTGLQ-JKPT-005 tmp]# dd if=/dev/zero of=200M.file bs=1M count=200

dd: writing `200M.file’: Disk quota exceeded

dd: closing output file `200M.file’: Disk quota exceeded

验证如下:

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfssetquota -S 10M /opt/test/tmp/ ——–>设置文件大小硬配额为10M

/opt/test/tmp/: (current values | soft quota | hard quota)

inodes | 0 | – | – |

length | 0 | – | – |

size | 0 | – | 10000000 |

realsize | 0 | – | – |

[root@NFJD-YPTGLQ-JKPT-005 tmp]# dd if=/dev/zero of=1M.file bs=1M count=200 ——–>写入1M的文件却报错,说明mfs创建一个文件就得用掉64M配额

dd: writing `1M.file’: Disk quota exceeded

dd: closing output file `1M.file’: Disk quota exceeded

硬配额100M,写入大小为70M的文件也提示超额,写入60M的文件正常

[root@NFJD-YPTGLQ-JKPT-005 tmp]# /usr/local/mfs/bin/mfssetquota -S 100M /opt/test/tmp/

/opt/test/tmp/: (current values | soft quota | hard quota)

inodes | 1 | – | – |

length | 67108864 | – | – |

size | 67113984 | – | 100000000 |

realsize | 134227968 | – | – |

[root@NFJD-YPTGLQ-JKPT-005 tmp]# dd if=/dev/zero of=1M.file bs=1M count=70

dd: closing output file `1M.file’: Disk quota exceeded

[root@NFJD-YPTGLQ-JKPT-005 tmp]# dd if=/dev/zero of=1M.file bs=1M count=60

60+0 records in

60+0 records out

62914560 bytes (63 MB) copied, 0.271172 s, 232 MB/s

设置挂载磁盘限额(-R,实际占用的空间(文件大小*goal数量))

[root@NFJD-YPTGLQ-JKPT-005 mnt]# mkdir zdtest ————->在mfs根目录新建目录zdtest

[root@NFJD-YPTGLQ-JKPT-005 mnt]# /usr/local/mfs/bin/mfssetquota -R 1G zdtest/ ————->并设置目录硬配额为1G

zdtest/: (current values | soft quota | hard quota)

inodes | 0 | – | – |

length | 0 | – | – |

size | 0 | – | – |

realsize | 0 | – | 1000000000 |

[root@NFJD-YPTGLQ-JKPT-005 mnt]# /usr/local/mfs/bin/mfsmount -H 10.1.220.87 -S /zdtest/ /opt/zdtest/ ————->客户端指定挂载子目录/zdtest/

mfsmaster accepted connection with parameters: read-write,restricted_ip,admin ; root mapped to root:root

[root@NFJD-YPTGLQ-JKPT-005 mnt]# df -h ————->查看挂载的mfs空间大小

挂载的mfs空间限制为1G

对于已挂载的目录,mfssetquota的对realsize修改是实时的,即设置realsize限额之后,df -h能看到最新的空间大小。

设置挂载磁盘限额(-S,客户端能使用的空间(不受goal影响))

单位为b(字节),设置限额为2TB

[root@NFJD-YPTGLQ-JKPT-005 MMPT]# /usr/local/mfs/bin/mfssetquota -S 2199023255552 WAPMH01/

WAPMH01/: (current values | soft quota | hard quota)

inodes | 11989041 | – | – |

length | 414931984274 | – | – |

size | 1049932998656 | – | 2199023255552 |

realsize | 2099865997312 | – | – |

查看客户端挂载

性能测试

1、dd测试

读写速度在250M左右浮动

测试写速度:

[root@mfsmaster_bk current]# dd if=/dev/zero of=/opt/zdtest1/TenGB.iso bs=1024M count=10 conv=fdatasync

10+0 records in

10+0 records out

10737418240 bytes (11 GB) copied, 45.2058 s, 238 MB/s

测试读速度:

[root@mfsmaster_bk current]# dd if=/opt/zdtest1/TenGB.iso of=/dev/zero bs=1024M count=10 iflag=direct

10+0 records in

10+0 records out

10737418240 bytes (11 GB) copied, 50.1252 s, 214 MB/s

2、iozone工具测试

下载源码包wget http://www.iozone.org/src/current/iozone3_434.tar

tar xf iozone3_434.tar

cd iozone3_434/src/current

make linux-AMD64

生成执行文件./iozone

./iozone -a -n 512m -g 10g -i 0 -i 1 -f /opt/zdtest1/10G.file -Rb ./iozone.xls -C

测试结果如下,写速度都在200多MB/S左右徘徊,读速度随着文件大小有下降的趋势。

(图表横列数字为块大小,竖列为文件大小,中间的数值为速度)

iozone测试数据

3、XFS和EXT4性能测试

创建GPT分区

parted /dev/sdb

mklabel gpt

mkpart primary 0 -1

创建XFS文件系统

yum install kmod-xfs xfsprogs(或者yum install xfs*)

modprobe xfs //载入xfs文件系统模块

lsmod |grep xfs //查看是否载入了xfs模块

mkfs.xfs /dev/sdb1

测试结果

融合通信10.1.175.7 ext4 raid0

[root@NFJD-YPTGLQ-JKPT-004 mfs1]# dd if=/dev/zero of=100GB.file bs=100M count=1000

1000+0 records in

1000+0 records out

104857600000 bytes (105 GB) copied, 139.796 s, 750 MB/s

[root@NFJD-YPTGLQ-JKPT-004 mfs1]# dd if=/dev/zero of=100GB.file bs=50M count=2000

2000+0 records in

2000+0 records out

104857600000 bytes (105 GB) copied, 144.864 s, 724 MB/s

[root@NFJD-YPTGLQ-JKPT-004 mfs1]# dd if=/dev/zero of=100GB.file bs=1000M count=100

100+0 records in

100+0 records out

104857600000 bytes (105 GB) copied, 148.059 s, 708 MB/s

[root@NFJD-YPTGLQ-JKPT-004 mfs1]# dd if=/dev/zero of=100GB.file bs=20M count=5000

5000+0 records in

5000+0 records out

104857600000 bytes (105 GB) copied, 142.09 s, 738 MB/s

融合通信10.1.175.7 xfs raid0

[root@NFJD-YPTGLQ-JKPT-004 mfs2]# dd if=/dev/zero of=100GB.file bs=50M count=2000

2000+0 records in

2000+0 records out

104857600000 bytes (105 GB) copied, 110.588 s, 948 MB/s

[root@NFJD-YPTGLQ-JKPT-004 mfs2]# dd if=/dev/zero of=100GB.file bs=100M count=1000

1000+0 records in

1000+0 records out

104857600000 bytes (105 GB) copied, 118.856 s, 882 MB/s

[root@NFJD-YPTGLQ-JKPT-004 mfs2]# dd if=/dev/zero of=100GB.file bs=20M count=5000

5000+0 records in

5000+0 records out

104857600000 bytes (105 GB) copied, 109.941 s, 954 MB/s

[root@NFJD-YPTGLQ-JKPT-004 mfs2]# dd if=/dev/zero of=100GB.file bs=1000M count=100

100+0 records in

100+0 records out

104857600000 bytes (105 GB) copied, 112.963 s, 928 MB/s

通过dd测试,XFS在读写速度上优于EXT4

inode数量对比

/dev/sdb1 大小1951193236

/dev/sdd1 大小5854355420

ext4 inode数量

/dev/sdb1 ext3 117M 27K 117M 1% /mfs

/dev/sdd1 ext4 350M 19K 350M 1% /mfs02

xfs inode数量

/dev/sdb1 xfs 373M 3 373M 1% /mfs

/dev/sdd1 xfs 1.1G 3 1.1G 1% /mfs02

XFS在同样容量情况下,inode数量是ext4的3倍。

MFS配额(quotas)配置

以融合通信MooseFS为例,部署信息如下:

角色 IP
Master 10.1.220.85
Metalogger 10.1.220.86
VIP 10.10.1.87
Chunkserver1 10.1.175.4
Chunkserver2 10.1.175.5
Chunkserver3 10.1.175.6

1、mfssetquota –R

(实际占用的空间=文件大小*goal数量)

初始状态如下:

server状态

Disks状态

Quotas状态

在路径/zdtest1中写入50G的文件,检查MFS各项指标如下:

硬限额realsize(-R)已使用98G,Disks实际占用也为98G

再写入50G文件,提示超出限额

[root@mfsmaster_bk zdtest1]# dd if=/dev/zero of=50GB.file bs=100M count=500

500+0 records in

500+0 records out

52428800000 bytes (52 GB) copied, 193.985 s, 270 MB/s

[root@mfsmaster_bk zdtest1]# dd if=/dev/zero of=50GB-2.file bs=100M count=500

dd: writing `50GB-2.file’: Disk quota exceeded

dd: closing output file `50GB-2.file’: Disk quota exceeded

  • 结论:

目录配置的硬限额realsize(-R)为186G,客户端看到的容量是187G,写入两个50G文件报错超出限额。

说明goal设置为2,配额设置为realsize(-R)时:

  • 客户端所挂载目录的实际可用容量为 容量/2,如此类推
  • 得出整个MFS实际可用容量为 总容量/2

2、mfssetquota –S

(客户端能使用的空间(不受goal影响))

size(-S)根据官方文档描述,此配置不受goal设置的影响。

删除文件,重新配置为size(-S),再次写入两个50G文件

初始状态

先写入50G文件

所生成的文件占用szie为49G,realsize为98G

再次写入50G文件,写入成功

[root@mfsmaster_bk zdtest1]# dd if=/dev/zero of=50GB.file bs=100M count=500

500+0 records in

500+0 records out

52428800000 bytes (52 GB) copied, 209.607 s, 250 MB/s

[root@mfsmaster_bk zdtest1]# dd if=/dev/zero of=50GB-2.file bs=100M count=500

500+0 records in

500+0 records out

52428800000 bytes (52 GB) copied, 217.545 s, 241 MB/s

总共成功写入两个50G文件,文件占用空间size为98G,实际占用MFS空间realsize为195G

结论:

目录限额配置为size(-S),goal设置为2时:

客户端能看到多少空间,就能用多少空间,现网中推荐使用该种方法进行设置,以防引起用户端的疑惑。

文件实际占用MFS空间为size*goal。

3、size(-S)与realsize(-R)区别

区别是:-S设置多少,客户端就能使用多少,不受goal数量的影响

-S的实际占用空间为容量*2;-R为最终占用MFS的空间大小。

举例(goal为2的情况):

  • 配置目录/zdtest1 的size(-S)为100G,客户端能够使用100G,实际占用MFS空间为200G
  • 配置目录/zdtest1 的realsize(-R)为100G,客户端只能使用50G,实际占用MFS空间为100G

4、现网操作注意点

请使用如下方法来设置配额:

单位为b(字节),如果用TB单位来设置会导致实际配额不精确,以下命令是设置限额为2TB

/usr/local/mfs/bin/mfssetquota -S 2199023255552 WAPMH01/

1 GB = 1073741824 b

1 TB = 1099511627776 b

-S 设置客户端能使用的空间(不受goal限制),比如设置1T,客户看到挂载的空间大小是1T,能使用也是1T,若此时goal设置为2,实际占用服务器空间为1*2=2T

-R 设置实际可使用的空间,比如设置1T,客户看到挂载的空间大小是1T,若此时goal设置为2,能使用的空间其实为1T/2=500G

因此在现网使用中,请使用-S来进行设置限额,业务侧也好理解。

在分配MFS给用户使用时,请考虑当前存储的总容量,最好剩余10%左右,否则会引起MFS异常。

以资源池MFS为例,当前设置goal为2,则实际可使用的空间为 7.3*3/2=10.95T,再保留10%的空间,实际可用空间为10.95*0.9=9.855T,请规划给业务使用的空间不要超过10T。

MFS小文件容量占用测试

1、基于EXT4和XFS测试

对ext4和xfs的文件容量占用测试,block size均为4096byte,信息如下:

[root@NFJD-YPTGLQ-JKPT-005 opt]# tune2fs -l /dev/sdb1|grep -i block

Block count: 24413696

Reserved block count: 1220684

Free blocks: 23982103

First block: 0

Block size: 4096

[root@NFJD-YPTGLQ-JKPT-005 opt]# xfs_info /dev/sdb3

meta-data=/dev/sdb3 isize=256 agcount=16, agsize=1525872 blks

= sectsz=4096 attr=2, projid32bit=0

data = bsize=4096 blocks=24413952, imaxpct=25

= sunit=0 swidth=0 blks

naming =version 2 bsize=4096 ascii-ci=0

log =internal bsize=4096 blocks=11920, version=2

= sectsz=4096 sunit=1 blks, lazy-count=1

realtime =none extsz=4096 blocks=0, rtextents=0

各分区分区初始信息

[root@NFJD-YPTGLQ-JKPT-005 opt]# df -T|grep -E “/mfs|Filesystem”

Filesystem Type 1K-blocks Used Available Use% Mounted on

/dev/sdb1 ext4 96120588 192160 91045692 1% /mfs1

/dev/sdb2 ext3 96122636 192292 91047504 1% /mfs2

/dev/sdb3 xfs 97608128 33504 97574624 1% /mfs3

[root@NFJD-YPTGLQ-JKPT-005 opt]# df -Ti|grep -E “/mfs|Filesystem”

Filesystem Type Inodes IUsed IFree IUse% Mounted on

/dev/sdb1 ext4 6111232 10 6111222 1% /mfs1

/dev/sdb2 ext3 6111232 10 6111222 1% /mfs2

/dev/sdb3 xfs 97655808 3 97655805 1% /mfs3

分别往/mfs1和/mfs3写入100w个大小为2byte的文件,写入后各分区状态如下:

[root@NFJD-YPTGLQ-JKPT-005 opt]# df -T|grep -E “/mfs|Filesystem”

Filesystem Type 1K-blocks Used Available Use% Mounted on

/dev/sdb1 ext4 96120588 4225760 87012092 5% /mfs1

/dev/sdb2 ext3 96122636 192292 91047504 1% /mfs2

/dev/sdb3 xfs 97608128 4329524 93278604 5% /mfs3

[root@NFJD-YPTGLQ-JKPT-005 opt]# df -Ti|grep -E “/mfs|Filesystem”

Filesystem Type Inodes IUsed IFree IUse% Mounted on

/dev/sdb1 ext4 6111232 1000110 5111122 17% /mfs1

/dev/sdb2 ext3 6111232 10 6111222 1% /mfs2

/dev/sdb3 xfs 97655808 1000103 96655705 2% /mfs3

测试结果:

ext4测试,往/mfs1写入100w个2byte大小的文件

已使用空间=4225760-192160=4033600

每个文件占用空间=4033600/100w=4.0336kb

xfs测试,往/mfs3写入100w个2byte大小的文件

已使用空间=4329524-33504=4296020

每个文件占用空间=4296020/100w=4.29602kb

结论:

  • 在文件大小小于block size大小时,占用整个block的空间
  • 上述测试每个文件占用的空间约等于block size大小

2、MFS测试

测试方法:

1、往挂载的MFS目录中写入10w的大小为2byte的文件

2、dd生成一个大小为10G的文件(bs大小设置为10M)

在MFS根目录中新建两个测试目录quotas_test、quotas_test2,均设置-S配额为100G

[root@NFJD-YPTGLQ-JKPT-005 mfs]# /usr/local/mfs/bin/mfsgetquota quotas_test quotas_test2

quotas_test: (current values | soft quota | hard quota)

inodes | 100100 | – | – |

length | 389000 | – | – |

size | 7065600000 | – | 107374182400 |

realsize | 14131200000 | – | – |

quotas_test2: (current values | soft quota | hard quota)

inodes | 1 | – | – |

length | 10485760000 | – | – |

size | 10486563840 | – | 107374182400 |

realsize | 20973127680 | – | – |

在quotas_test写入10w的大小为2byte的文件

在quotas_test2目录,通过dd生成一个大小为10G的文件(bs大小设置为10M)

quotas状态如下:

quotas_test中文件实际大小为380K,但实际占用6.6G,因goal为2,故总占用MFS空间为13G左右

qutotas_test2文件大小为9.8G,实际占用也为9.8G,总占用MFS空间为20G左右

产生上述差异的原因:

根据上一步对ext4和xfs的测试,可知文件系统是以block size为单位进行分配,在MFS中也存在相似的情况。

根据网上搜索对MFS的block size的描述,MFS的block size为64K,再加上5K的header,说明在MFS中,一个文件至少占用空间为69K。

通过上述的测试,10w的2byte大小的文件占用空间为7065600000b,每个文件所占用空间为69K。

而dd生成的文件大小与占用空间大小相等,是因为其block size设置为10M,已大于MFS的bs大小。

结论:

  • 使用MFS存放小文件,会造成容量的大量浪费
  • 建议存储文件大小在64K以上

MFS文件与chunk数量的联系

在mfs新建目录

mkdir -p MMPT/MMSJZS01

mkdir -p MMPT/WAPMH01

mkdir -p SJFX/SJCK

mkdir -p SJFX/SJCK-JK01

目录不占用chunk数量

[root@NFJD-YPTGLQ-JKPT-005 mfs]# dd if=/dev/zero of=test.file.100M bs=10M count=10

10+0 records in

10+0 records out

104857600 bytes (105 MB) copied, 1.14097 s, 91.9 MB/s

[root@NFJD-YPTGLQ-JKPT-005 mfs]# /usr/local/mfs/bin/mfsgetgoal test.file.100M

test.file.100M: 1

[root@NFJD-YPTGLQ-JKPT-005 mfs]# /usr/local/mfs/bin/mfscheckfile test.file.100M

test.file.100M:

chunks with 1 copy: 2

[root@NFJD-YPTGLQ-JKPT-005 mfs]# /usr/local/mfs/bin/mfsfileinfo test.file.100M

test.file.100M:

chunk 0: 0000000000000001_00000001 / (id:1 ver:1)

copy 1: 10.20.1.9:9422

chunk 1: 0000000000000002_00000001 / (id:2 ver:1)

copy 1: 10.20.1.8:9422

chunk分布情况

说明goal为1的时候,数据就只有一份,而且是分散到各台服务器。

设置goal为2

[root@NFJD-YPTGLQ-JKPT-005 mfs]# /usr/local/mfs/bin/mfssetgoal -r 2 /mnt/mfs/

/mnt/mfs/:

inodes with goal changed: 8

inodes with goal not changed: 0

inodes with permission denied: 0

[root@NFJD-YPTGLQ-JKPT-005 mfs]# /usr/local/mfs/bin/mfsfileinfo test.file.100M

test.file.100M:

chunk 0: 0000000000000001_00000001 / (id:1 ver:1)

copy 1: 10.20.1.8:9422

copy 2: 10.20.1.9:9422

chunk 1: 0000000000000002_00000001 / (id:2 ver:1)

copy 1: 10.20.1.8:9422

copy 2: 10.20.1.9:9422

chunk分布情况

后续新增的文件直接继承goal为2的配置

[root@NFJD-YPTGLQ-JKPT-005 mfs]# dd if=/dev/zero of=2test.file.100M bs=10M count=10

10+0 records in

10+0 records out

104857600 bytes (105 MB) copied, 0.942131 s, 111 MB/s

[root@NFJD-YPTGLQ-JKPT-005 mfs]# /usr/local/mfs/bin/mfsfileinfo 2test.file.100M

2test.file.100M:

chunk 0: 0000000000000003_00000001 / (id:3 ver:1)

copy 1: 10.20.1.7:9422

copy 2: 10.20.1.9:9422

chunk 1: 0000000000000004_00000001 / (id:4 ver:1)

copy 1: 10.20.1.7:9422

copy 2: 10.20.1.8:9422

直接在chunkserver查找chunk文件find /mfs /mfs02 -name *.mfs|xargs du -sh

chunkserver1

[root@NFJD-YPTGLQ-JKPT-001 mfs02]# find ./ -name *.mfs|xargs du -sh

37M ./02/chunk_0000000000000002_00000001.mfs

37M ./04/chunk_0000000000000004_00000001.mfs

37M ./06/chunk_0000000000000006_00000001.mfs

[root@NFJD-YPTGLQ-JKPT-001 mfs02]# find /mfs -name *.mfs|xargs du -sh

65M /mfs/03/chunk_0000000000000003_00000001.mfs

chunkserver2

[root@NFJD-YPTGLQ-JKPT-002 mfs02]# find ./ -name *.mfs|xargs du -sh

37M ./02/chunk_0000000000000002_00000001.mfs

37M ./04/chunk_0000000000000004_00000001.mfs

65M ./05/chunk_0000000000000005_00000001.mfs

[root@NFJD-YPTGLQ-JKPT-002 mfs02]# find /mfs -name *.mfs|xargs du -sh

65M /mfs/01/chunk_0000000000000001_00000001.mfs

chunkserver3

[root@NFJD-YPTGLQ-JKPT-003 mfs02]# find ./ -name *.mfs|xargs du -sh

65M ./01/chunk_0000000000000001_00000001.mfs

65M ./03/chunk_0000000000000003_00000001.mfs

65M ./05/chunk_0000000000000005_00000001.mfs

37M ./06/chunk_0000000000000006_00000001.mfs

设置trashtime为60秒,递归设置

[root@NFJD-YPTGLQ-JKPT-005 mfs]# /usr/local/mfs/bin/mfssettrashtime 60 -r ./

./:

inodes with trashtime changed: 10

inodes with trashtime not changed: 0

inodes with permission denied: 0

[root@NFJD-YPTGLQ-JKPT-005 mfs]# /usr/local/mfs/bin/mfsgettrashtime -r ./

./:

files with trashtime 60 : 3

directories with trashtime 60 : 7

删除前面新建的3个文件,查看chunk数量变化

测试至少要3分钟file才变成0, chunk数量继续要等3分钟以上才变成0

重新创建一个文件,查看chunk变化

[root@NFJD-YPTGLQ-JKPT-005 mfs]# dd if=/dev/zero of=file.100M bs=10M count=10

10+0 records in

10+0 records out

104857600 bytes (105 MB) copied, 0.961552 s, 109 MB/s

[root@NFJD-YPTGLQ-JKPT-005 mfs]# /usr/local/mfs/bin/mfsfile

mfsfileinfo mfsfilepaths mfsfilerepair

[root@NFJD-YPTGLQ-JKPT-005 mfs]# /usr/local/mfs/bin/mfsfileinfo file.100M

file.100M:

chunk 0: 0000000000000007_00000001 / (id:7 ver:1)

copy 1: 10.20.1.7:9422

copy 2: 10.20.1.8:9422

chunk 1: 0000000000000008_00000001 / (id:8 ver:1)

copy 1: 10.20.1.8:9422

copy 2: 10.20.1.9:9422

chunk总数为4个

chunkserver上的数量

[root@NFJD-YPTGLQ-JKPT-001 mfs02]# find /mfs /mfs02 -name *.mfs|xargs du -sh

65M /mfs02/07/chunk_0000000000000007_00000001.mfs

[root@NFJD-YPTGLQ-JKPT-002 mfs02]# find /mfs /mfs02 -name *.mfs|xargs du -sh

37M /mfs/08/chunk_0000000000000008_00000001.mfs

65M /mfs02/07/chunk_0000000000000007_00000001.mfs

[root@NFJD-YPTGLQ-JKPT-003 mfs02]# find /mfs /mfs02 -name *.mfs|xargs du -sh

37M /mfs/08/chunk_0000000000000008_00000001.mfs

再新建一个同样大小文件

[root@NFJD-YPTGLQ-JKPT-005 mfs]# dd if=/dev/zero of=file2.100M bs=10M count=10

10+0 records in

10+0 records out

104857600 bytes (105 MB) copied, 0.959352 s, 109 MB/s

[root@NFJD-YPTGLQ-JKPT-005 mfs]# /usr/local/mfs/bin/mfsfileinfo file2.100M

file2.100M:

chunk 0: 0000000000000009_00000001 / (id:9 ver:1)

copy 1: 10.20.1.7:9422

copy 2: 10.20.1.9:9422

chunk 1: 000000000000000A_00000001 / (id:10 ver:1)

copy 1: 10.20.1.7:9422

copy 2: 10.20.1.8:9422

chunkserver上的数量

ansible -i 5-mfs-chunkserver.txt all -m raw -a “find /mfs /mfs02 -name *.mfs|xargs du -sh” -u tyyw03 –s

10.20.1.8 | success | rc=0 >>

37M /mfs/08/chunk_0000000000000008_00000001.mfs

65M /mfs02/07/chunk_0000000000000007_00000001.mfs

37M /mfs02/0A/chunk_000000000000000A_00000001.mfs

10.20.1.7 | success | rc=0 >>

65M /mfs/09/chunk_0000000000000009_00000001.mfs

65M /mfs02/07/chunk_0000000000000007_00000001.mfs

37M /mfs02/0A/chunk_000000000000000A_00000001.mfs

10.20.1.9 | success | rc=0 >>

37M /mfs/08/chunk_0000000000000008_00000001.mfs

65M /mfs02/09/chunk_0000000000000009_00000001.mfs

继续新建,chunk数量增加4个

[root@NFJD-YPTGLQ-JKPT-005 mfs]# dd if=/dev/zero of=file3.100M bs=10M count=10

10+0 records in

10+0 records out

104857600 bytes (105 MB) copied, 0.942412 s, 111 MB/s

[root@NFJD-YPTGLQ-JKPT-005 mfs]# /usr/local/mfs/bin/mfsfileinfo file3.100M

file3.100M:

chunk 0: 000000000000000B_00000001 / (id:11 ver:1)

copy 1: 10.20.1.8:9422

copy 2: 10.20.1.9:9422

chunk 1: 000000000000000C_00000001 / (id:12 ver:1)

copy 1: 10.20.1.7:9422

copy 2: 10.20.1.9:9422

10.20.1.7 | success | rc=0 >>

65M /mfs/09/chunk_0000000000000009_00000001.mfs

65M /mfs02/07/chunk_0000000000000007_00000001.mfs

37M /mfs02/0A/chunk_000000000000000A_00000001.mfs

37M /mfs02/0C/chunk_000000000000000C_00000001.mfs

10.20.1.9 | success | rc=0 >>

37M /mfs/08/chunk_0000000000000008_00000001.mfs

65M /mfs02/09/chunk_0000000000000009_00000001.mfs

65M /mfs02/0B/chunk_000000000000000B_00000001.mfs

37M /mfs02/0C/chunk_000000000000000C_00000001.mfs

10.20.1.8 | success | rc=0 >>

37M /mfs/08/chunk_0000000000000008_00000001.mfs

65M /mfs02/07/chunk_0000000000000007_00000001.mfs

37M /mfs02/0A/chunk_000000000000000A_00000001.mfs

65M /mfs02/0B/chunk_000000000000000B_00000001.mfs

监控方案

以资源池MooseFS为例,部署信息如下:

角色 IP
Master 10.10.1.84
Metalogger 10.10.1.85
VIP 10.10.1.88
Chunkserver1 10.20.1.7
Chunkserver2 10.20.1.8
Chunkserver3 10.20.1.9

1、端口

角色 IP 需要监控的端口
Master 10.10.1.84 9419:用于监听metalogger请求

9420:用于监听chunkserver请求

9421:用于与client(mount)的连接请求

9425:mfscgiv的服务端口

Metalogger 10.10.1.85
VIP 10.10.1.88 同master
Chunkserver1 10.20.1.7 9422:用于与client(mount)的连接请求
Chunkserver2 10.20.1.8 9422:用于与client(mount)的连接请求
Chunkserver3 10.20.1.9 9422:用于与client(mount)的连接请求

查看现网运行中的mfs端口情况

[root@Ansiable-Srv-2 zd]# ansible -i 7-mfs-server.txt all -m raw -a “netstat -ntlp|grep -Ei ‘python|mfs'” -u tyyw03 -s

10.20.1.7 | success | rc=0 >>

tcp 0 0 0.0.0.0:9422 0.0.0.0:* LISTEN 183893/mfschunkserv

10.20.1.8 | success | rc=0 >>

tcp 0 0 0.0.0.0:9422 0.0.0.0:* LISTEN 61929/mfschunkserve

10.10.1.85 | success | rc=0 >>

tcp 0 0 127.0.0.1:34995 0.0.0.0:* LISTEN 31662/mfsmount

tcp 0 0 127.0.0.1:47029 0.0.0.0:* LISTEN 40667/mfsmount

10.20.1.9 | success | rc=0 >>

tcp 0 0 0.0.0.0:9422 0.0.0.0:* LISTEN 231068/mfschunkserv

10.10.1.84 | success | rc=0 >>

tcp 0 0 0.0.0.0:9419 0.0.0.0:* LISTEN 35715/mfsmaster

tcp 0 0 0.0.0.0:9420 0.0.0.0:* LISTEN 35715/mfsmaster

tcp 0 0 0.0.0.0:9421 0.0.0.0:* LISTEN 35715/mfsmaster

tcp 0 0 0.0.0.0:9425 0.0.0.0:* LISTEN 35718/python

2、lockfile

角色 IP Lockfile路径
Master 10.10.1.84 /usr/local/mfs/var/mfs/.mfsmaster.lock

/usr/local/mfs/var/mfs/.mfscgiserv.lock

Metalogger 10.10.1.85 /usr/local/mfs/var/mfs/.mfsmetalogger.lock
VIP 10.10.1.88 同master
Chunkserver1 10.20.1.7 /usr/local/mfs/var/mfs/.mfschunkserver.lock
Chunkserver2 10.20.1.8 /usr/local/mfs/var/mfs/.mfschunkserver.lock
Chunkserver3 10.20.1.9 /usr/local/mfs/var/mfs/.mfschunkserver.lock

3、mfs进程

角色 IP 进程名
Master 10.10.1.84 /usr/local/mfs/sbin/mfsmaster

/usr/bin/python /usr/local/mfs/sbin/mfscgiserv

Metalogger 10.10.1.85 /usr/local/mfs/sbin/mfsmetalogger
VIP 10.10.1.88 同master
Chunkserver1 10.20.1.7 /usr/local/mfs/sbin/mfschunkserver
Chunkserver2 10.20.1.8 /usr/local/mfs/sbin/mfschunkserver
Chunkserver3 10.20.1.9 /usr/local/mfs/sbin/mfschunkserver

4、chunkserver的inode

因现网最近发生过一起MFS的故障,现象是client新建文件出错“ENXIO (No such device or address)”,chunkserver上日志报错“ENOSPC (No space left on device)”,查看chunkserver本地数据挂载目录情况,剩余空间充足,但inode已使用100%,导致无法新建chunk文件,从而使client新建文件失败,因此inode监控比较重要。

[root@NFJD-YPTGLQ-JKPT-003 ~]# df -hTi inode的使用率为100%

Filesystem Type Inodes IUsed IFree IUse% Mounted on

/dev/mapper/vg_jkpt03-lv_root

ext4 256K 7.2K 249K 3% /

tmpfs tmpfs 4.0M 1 4.0M 1% /dev/shm

/dev/sda1 ext4 126K 39 125K 1% /boot

/dev/mapper/vg_jkpt03-lv_home

ext4 512K 2.1K 510K 1% /home

/dev/mapper/vg_jkpt03-lv_opt

ext4 3.8M 14K 3.8M 1% /opt

/dev/mapper/vg_jkpt03-lv_tmp

ext4 512K 30 512K 1% /tmp

/dev/mapper/vg_jkpt03-lv_usr

ext4 512K 84K 429K 17% /usr

/dev/mapper/vg_jkpt03-lv_var

ext4 1.0M 3.2K 1021K 1% /var

/dev/sdb1 ext3 117M 513K 116M 1% /mfs

/dev/mapper/elk-elk–disk

ext4 466M 45K 466M 1% /opt/aspire/product/elk

/dev/sdd1 ext4 1.4M 1.4M 0 100% /mfs02

[root@NFJD-YPTGLQ-JKPT-003 ~]# df -hT 已使用空间仅为2%,但此时已无法写入文件

Filesystem Type Size Used Avail Use% Mounted on

/dev/mapper/vg_jkpt03-lv_root

ext4 4.0G 370M 3.4G 10% /

tmpfs tmpfs 16G 0 16G 0% /dev/shm

/dev/sda1 ext4 485M 38M 422M 9% /boot

/dev/mapper/vg_jkpt03-lv_home

ext4 7.9G 203M 7.3G 3% /home

/dev/mapper/vg_jkpt03-lv_opt

ext4 60G 20G 38G 34% /opt

/dev/mapper/vg_jkpt03-lv_tmp

ext4 7.9G 147M 7.4G 2% /tmp

/dev/mapper/vg_jkpt03-lv_usr

ext4 7.9G 2.5G 5.1G 33% /usr

/dev/mapper/vg_jkpt03-lv_var

ext4 16G 453M 15G 3% /var

/dev/sdb1 ext3 1.8T 34G 1.7T 2% /mfs

/dev/mapper/elk-elk–disk

ext4 7.2T 1.5T 5.4T 22% /opt/aspire/product/elk

/dev/sdd1 ext4 5.5T 68G 5.2T 2% /mfs02

5、chunkserver容量

[root@NFJD-YPTGLQ-JKPT-003 mfs02]# df -hT|grep xfs

/dev/sdb1 xfs 1.9T 151G 1.7T 9% /mfs

/dev/sdd1 xfs 5.5T 451G 5.1T 9% /mfs02

6、meta数据容量

默认的数据路径为/usr/local/mfs/var/,因此需要针对/根目录进行监控。

[root@NFJD-YPTGLQ-JKPT-004 mfs]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/VolGroup-lv_root

47G 13G 32G 28% /

tmpfs 3.9G 0 3.9G 0% /dev/shm

/dev/sda1 485M 37M 423M 9% /boot

/dev/mapper/VolGroup-lv_home

4.6G 138M 4.2G 4% /home

备份策略

1、数据文件

角色 IP 需备份的文件名(路径均在数据目录) 备份优先级
Master 10.10.1.84 metadata.mfs 1 Meta元数据,存放的是文件名和路径的对应信息;

这时Master进程还未启动时的meta元数据的文件名。

metadata.mfs.back 1 与metadata.mfs是一样,Meta元数据,存放的是文件名和路径的对应信息;

在master进程运行时的meta元数据的文件名

changelog.*.mfs 1 metadata的changelog日志,每一个小时合并到metadata.mfs中一次
stats.mfs 3 存放的是Master图表状态
Metalogger 10.10.1.85 changelog_ml.*.mfs 2 Master的changelog日志备份
metadata.ml.mfs.back 2 Master的metadata.mfs.back的最新备份
Chunkserver 10.20.1.7

10.20.1.8

10.20.1.9

chunkserverid.mfs 3 Chunkserver的id标识
csstats.mfs 3 Chunkserver的图表状态

2、配置文件

角色 IP 需备份的文件名(路径均在数据目录)
Master 10.10.1.84 mfsmaster.cfg
mfsexports.cfg
mfstopology.cfg
Metalogger 10.10.1.85 mfsmetalogger.cfg
Chunkserver 10.20.1.7

10.20.1.8

10.20.1.9

mfschunkserver.cfg
mfshdd.cfg