存储性能测试和5种工具选择

对某品牌的分布式存储进行了如下工具测试,对比普通FC存储的性能数据。

1、dd测试(一般用于本地硬盘的简单测试)
方法:测试块大小4k,8k,64k和1M,测试数据量为100G,执行dd if=/dev/zero of=/data/helloworld bs=1M count=102400
结果:平均速率约为750 MB/s
注:普通FC存储约为670 MB/s

2、hdparm测试(一般用于本地硬盘的简单测试)
方法:执行hdparm -Tt /dev/sdxxx
结果:平均速率为70-100 MB/s 左右浮动
注:普通FC存储约为360 MB/s

3、orion测试(用于测试存储的oltp性能,模拟oracle)
方法:同时测试5个块设备,执行./orion_linux_x86-64 -run simple -testname iotest123 -num_disks 5
结果:
Maximum Large MBPS=595.05 @ Small=0 and Large=10
Maximum Small IOPS=3099 @ Small=25 and Large=0
Minimum Small Latency=8.04 @ Small=24 and Large=0
详细数据:

ORION VERSION 11.1.0.7.0

Commandline:
-run simple -testname iotest123 -num_disks 5

This maps to this test:
Test: iotest123
Small IO size: 8 KB
Large IO size: 1024 KB
IO Types: Small Random IOs, Large Random IOs
Simulated Array Type: CONCAT
Write: 0%
Cache Size: Not Entered
Duration for each Data Point: 60 seconds
Small Columns:, 0
Large Columns:, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
Total Data Points: 36

Name: /dev/sdah Size: 858993459200
Name: /dev/sdai Size: 858993459200
Name: /dev/sdaj Size: 858993459200
Name: /dev/sdak Size: 858993459200
Name: /dev/sdal Size: 858993459200
5 FILEs found.

Maximum Large MBPS=595.05 @ Small=0 and Large=10
Maximum Small IOPS=3099 @ Small=25 and Large=0
Minimum Small Latency=8.04 @ Small=24 and Large=0

注:普通FC存储测试结果参考如下
Maximum Large MBPS=476.68 @ Small=0 and Large=2
Maximum Small IOPS=1784 @ Small=5 and Large=0
Minimum Small Latency=2.58 @ Small=3 and Large=0

4、iozone测试(本地硬盘或者NFS等传统存储测试)
方法:测试块大小4k,8k,64k、1M、2M、4M、8M和16M,测试数据量为200G,执行iozone -a -n 512m -g 16g -i 0 -i 1 -f /langchao/200G.file -Rb ./iozone0525.xls -C
结果:
写平均速率约为:90-120 MB/S 之间浮动
读平均速率约为:300-490 MB/S 之间浮动
详细数据:

iozone -a -n 512m -g 16g -i 0 -i 1 -f /langchao/200G.file -Rb ./iozone0525.xls -C
The top row is records sizes, the left column is file sizes
Writer Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
524288 0 0 0 0 85.98355 85.15566 84.20773 87.70587 87.61054 85.44886 85.87258 84.78848 80.15957
1048576 0 0 0 0 90.35346 90.3719 88.94122 100.7986 90.09031 91.30338 94.62239 91.12668 91.89865
2097152 0 0 0 0 105.667 103.1136 103.629 105.0496 103.6155 104.1899 103.844 118.8821 96.84508
4194304 0 0 0 0 113.024 115.282 113.8014 111.1989 112.347 112.3828 112.5376 108.3152 103.4564
8388608 0 0 0 0 116.3378 122.0652 118.3721 117.2589 121.3806 117.513 117.4191 120.5094 108.6423
16777216 0 0 0 0 117.1983 117.5814 114.9045 111.0679 114.9424 114.5299 113.3163 113.8328 107.7349
Re-writer Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
524288 0 0 0 0 145.0467 143.687 131.3047 131.1593 133.9671 134.2651 130.5808 135.3056 121.8748
1048576 0 0 0 0 139.3085 149.0021 151.1622 146.2659 151.6189 148.0571 146.0191 153.9341 127.6513
2097152 0 0 0 0 148.881 154.5301 150.4974 154.5596 187.3824 151.7502 160.7372 152.8687 137.2404
4194304 0 0 0 0 167.4625 175.5084 165.5225 172.9143 168.3728 169.8384 192.7342 168.0536 152.1222
8388608 0 0 0 0 179.4276 184.4657 183.5961 167.1096 187.8342 182.7594 184.7261 182.4028 160.9307
16777216 0 0 0 0 134.5504 133.4313 141.6385 137.6593 133.0304 133.6514 141.2211 136.83 146.6709
Reader Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
524288 0 0 0 0 312.5405 338.0374 278.0573 338.1924 337.1106 326.4768 279.5672 328.6695 221.2658
1048576 0 0 0 0 344.1645 392.1548 327.3961 333.8125 366.3674 323.4534 331.5795 313.6616 219.9376
2097152 0 0 0 0 402.7131 391.1469 360.0767 374.9855 351.3713 424.5437 376.5379 350.3262 230.4536
4194304 0 0 0 0 406.9491 385.7098 373.7435 371.4637 400.3046 369.7195 432.7948 358.2688 243.9996
8388608 0 0 0 0 439.4873 438.3206 393.1168 421.8903 427.9911 416.3369 419.459 465.4355 268.0901
16777216 0 0 0 0 459.2699 452.7642 425.5148 436.0775 434.3376 438.5376 432.2877 434.4132 273.9593
Re-reader Report
4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384
524288 0 0 0 0 310.9554 303.4453 362.6903 384.105 387.1162 349.9365 288.8842 289.5473 234.0503
1048576 0 0 0 0 415.1548 361.0204 320.0508 330.6645 388.5284 354.1505 345.9186 326.4076 221.377
2097152 0 0 0 0 384.03 369.9677 373.2726 356.6666 366.7205 408.1187 431.6969 361.5258 229.9918
4194304 0 0 0 0 399.6686 417.772 412.1506 386.4536 386.367 394.3037 381.2992 373.7435 247.9446
8388608 0 0 0 0 450.4276 441.4695 476.3091 491.2972 499.7049 498.2206 440.2827 425.4965 291.3664
16777216 0 0 0 0 470.4384 493.471 460.0883 439.3071 482.2512 479.0271 479.4607 441.2343 284.8717

注:普通FC存储测试结果参考如下
写平均速率约为:60-80 MB/S 之间浮动
读平均速率约为:140-350 MB/S 之间浮动

5、vdbench(常用于测试分布式存储)
./vdbench -t 测试vdbench是否正常使用
./vdbench -f parmfile ,parmfile指的是参数文件

—创建写的参数文件testfile.write
fsd=fsd1601,anchor=/opt/nfs_dir/1601,depth=1,width=10,files=100,size=100M
fwd=fwd46,fsd=fsd1601,operation=write,xfersize=1M,fileio=sequential,fileselect=sequential,threads=5
rd=rd1,fwd=fwd46,format=only,fwdrate=max,elapsed=500,interval=2

./vdbench -f testfile.write,开始创建文件

完成测试后会在vdbench的目录下output中生成文件,在anchor所定义的目录下生成名为1601的目录

—创建多个写
fsd=zd001,anchor=/opt/nfs_dir/vdtest1,depth=1,width=10,files=100,size=100M
fsd=zd002,anchor=/opt/nfs_dir/vdtest2,depth=1,width=10,files=100,size=100M
fsd=zd003,anchor=/opt/nfs_dir/vdtest3,depth=1,width=10,files=100,size=100M
fsd=zd004,anchor=/opt/nfs_dir/vdtest4,depth=1,width=10,files=100,size=100M
fsd=zd005,anchor=/opt/nfs_dir/vdtest5,depth=1,width=10,files=100,size=100M

fwd=zzdd001,fsd=zd001,operation=write,xfersize=1M,fileio=sequential,fileselect=sequential,threads=5
fwd=zzdd002,fsd=zd002,operation=write,xfersize=1M,fileio=sequential,fileselect=sequential,threads=5
fwd=zzdd003,fsd=zd003,operation=write,xfersize=1M,fileio=sequential,fileselect=sequential,threads=5
fwd=zzdd004,fsd=zd004,operation=write,xfersize=1M,fileio=sequential,fileselect=sequential,threads=5
fwd=zzdd005,fsd=zd005,operation=write,xfersize=1M,fileio=sequential,fileselect=sequential,threads=5

rd=rd1,fwd=(zzdd001,zzdd002,zzdd003,zzdd004,zzdd005,),format=only,fwdrate=max,elapsed=500,interval=2
部分测试结果: