I am using dd and ioping to determine read/write latency on NFS and iSCSI.
This is a test lab so the latencies will be very low compared to production environments.
For the purpose of this test I will use 2 Arch Linux VMs. One will act as an NFS and iSCSI server, the other one will be the client. server1 and client1.
dd will help me find out the read / write speed on the disk / nfs export from the client. This is useful because the read / write actually takes place on the remote server, over TCP.
ioping will show me latency. Latency is different from read/write speed, as it is the read or write response time over the network.
1. iSCSI
I mounted a LUN from server1 to client1 in /media/lun
[root@client1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdl 50G 33M 50G 1% /media/lun
[root@client1 ~]#
1.1 Use dd to perform a test write and read.
[root@client1 ~]# time dd if=/dev/zero of=/media/lun/testfile bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 2.01766 s, 133 MB/s
real 0m2.054s
user 0m0.007s
sys 0m1.677s
[root@client1 ~]#
Write performance in this case is 133 MB/s.
[root@client1 ~]# time dd if=/media/lun/testfile of=/dev/null bs=16k
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.0614949 s, 4.4 GB/s
real 0m0.064s
user 0m0.007s
sys 0m0.057s
[root@client1 ~]#
Read performance 4.4 GB/s.
1.2 Use ioping to test disk latency.
Measure disk seek rate (iops, avg)
[root@client1 ~]# ioping -R /dev/sdl
--- /dev/sdl (device 50.0 GiB) ioping statistics ---
4.9 k requests completed in 3.0 s, 1.6 k iops, 6.4 MiB/s
min/avg/max/mdev = 347 us / 609 us / 10.6 ms / 733 us
[root@client1 ~]#
Measure disk sequential speed (MiB/s)
[root@client1 ~]# ioping -R /dev/sdl
--- /dev/sdl (device 50.0 GiB) ioping statistics ---
4.9 k requests completed in 3.0 s, 1.6 k iops, 6.4 MiB/s
min/avg/max/mdev = 347 us / 609 us / 10.6 ms / 733 us
[root@client1 ~]# ioping -RL /dev/sda
--- /dev/sda (device 8.0 GiB) ioping statistics ---
918 requests completed in 3.0 s, 306 iops, 76.6 MiB/s
min/avg/max/mdev = 1.0 ms / 3.3 ms / 74.6 ms / 4.6 ms
[root@client1 ~]#
Measure read latency:
[root@client1 ~]# ioping /dev/sdl
4.0 KiB from /dev/sdl (device 50.0 GiB): request=1 time=3.3 ms
4.0 KiB from /dev/sdl (device 50.0 GiB): request=2 time=2.3 ms
4.0 KiB from /dev/sdl (device 50.0 GiB): request=3 time=2.0 ms
4.0 KiB from /dev/sdl (device 50.0 GiB): request=4 time=2.5 ms
4.0 KiB from /dev/sdl (device 50.0 GiB): request=5 time=2.1 ms
4.0 KiB from /dev/sdl (device 50.0 GiB): request=6 time=2.9 ms
4.0 KiB from /dev/sdl (device 50.0 GiB): request=7 time=2.6 ms
^C
--- /dev/sdl (device 50.0 GiB) ioping statistics ---
7 requests completed in 6.7 s, 395 iops, 1.5 MiB/s
min/avg/max/mdev = 2.0 ms / 2.5 ms / 3.3 ms / 414 us
In troubleshooting cases use one of the options
-s <size> request size (4k)
-S <wsize> working set size (1m)
to get values closer to the real life workload.
2. NFS
Mounted an NFS export from server1 into /mnt
[root@client1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
172.16.41.132:/san/vol1 2.0G 256K 2.0G 1% /mnt
[root@client1 ~]#
2.1 Test using dd. (same as for iSCSI)
[root@client1 ~]# time dd if=/dev/zero of=/mnt/testfile bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 4.73394 s, 56.7 MB/s
real 0m4.751s
user 0m0.000s
sys 0m0.993s
[root@client1 ~]#
As expected, I am getting about half the write speed as compared to iSCSI performance.
The same is valid for read performance:
[root@client1 ~]# time dd if=/mnt/testfile of=/dev/null bs=16k
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.109984 s, 2.4 GB/s
real 0m0.131s
user 0m0.007s
sys 0m0.087s
[root@client1 ~]#
2.2 Using ioping to test NFS performance:
Use the same commands, except replace /dev/sdl with /mnt (mount point). Ioping will recognize network filesystems.
[root@client1 ~]# ioping -RL /mnt
--- /mnt (nfs4 172.16.41.132:/san/vol1) ioping statistics ---
512 requests completed in 3.0 s, 171 iops, 42.8 MiB/s
min/avg/max/mdev = 3.2 ms / 5.8 ms / 15.5 ms / 2.0 ms
[root@client1 ~]#
[root@client1 ~]# ioping -RL -s 512 /mnt
--- /mnt (nfs4 172.16.41.132:/san/vol1) ioping statistics ---
22.2 k requests completed in 3.0 s, 7.6 k iops, 3.7 MiB/s
min/avg/max/mdev = 0 us / 132 us / 111.0 ms / 1.9 ms
[root@client1 ~]#
Ioping options:
[root@client1 ~]# ioping
Usage: ioping [-LABCDWRq] [-c count] [-w deadline] [-pP period] [-i interval]
[-s size] [-S wsize] [-o offset] directory|file|device
ioping -h | -v
-c <count> stop after <count> requests
-w <deadline> stop after <deadline>
-p <period> print raw statistics for every <period> requests
-P <period> print raw statistics for every <period> in time
-i <interval> interval between requests (1s)
-s <size> request size (4k)
-S <wsize> working set size (1m)
-o <offset> working set offset (0)
-L use sequential operations (includes -s 256k)
-A use asynchronous I/O
-C use cached I/O
-D use direct I/O
-W use write I/O *DANGEROUS*
-R seek rate test (same as -q -i 0 -w 3 -S 64m)
-B print final statistics in raw format
-q suppress human-readable output
-h display this message and exit
-v display version and exit
[root@client1 ~]#