If your NFS file system is mounted across a high-speed network, such as 10/40Gigabit Ethernet using JUMBO frame, then larger read and write packet sizes might enhance NFS file system performance.
The chunk size that works best for your NFS configuration depends primarily on the network configuration (MTU) and the applications performing I/O. If your applications are doing lots of small I/O, then a large chunk size might negatively affect the performance.
If the applications are using a mixture of I/O payload sizes, then you must do some experimentation with variable block sizes to select the 'sweet spot size'. However, once you have determined that the 'larger block size' is best for your environment, then MTU size can be very important because it determines 'packet fragments' on the network. If your chunk size is 8KB and the MTU is 1500, it will take six Ethernet frames to transmit the 8KB. If you increase the MTU to 9000 (9,000 bytes), the number of Ethernet frames drops to one.
Chunk vs MTU fragments [1500 bytes] | JUMBO [9000 bytes]
8KB = 6 1
32KB = 22 4
64KB = 43 8
In general factors that affect NFS performance are:
1. OS buffer size
2. NFS wsize/rsize
3. RPC slots
4. TCP Window (Static or Dynamic)
5. Ethernet MTU (Standard / Jumbo)
6. NFS threads
To avoid potentially causing denial of service on a cluster node, modify clients running RHEL 6.3 and later to use, at most, 128 RPC slots. This number corresponds with the current maximum number of RPCs available per TCP connection in ONTAP.
To do this, run the following on the NFS client (alternatively, edit the /etc/modprobe.d/sunrpc.conf file manually to use these values):
# echo "options sunrpc udp_slot_table_entries=64 tcp_slot_table_entries=128 tcp_max_slot_table_entries=128" >> /etc/modprobe.d/sunrpc.conf
Centos7 machine in my lab:
[root@centos7 modprobe.d]# cd /etc/modprobe.d/
[root@centos7 modprobe.d]# ls
lockd.conf mlx4.conf truescale.conf tuned.conf
Note: Looks like my client does not contain this file, let's create one.
[root@centos7 modprobe.d]# touch sunrpc.conf
[root@centos7 modprobe.d]# ls
lockd.conf mlx4.conf sunrpc.conf truescale.conf tuned.conf
[root@centos7 modprobe.d]# echo "options sunrpc udp_slot_table_entries=64 tcp_slot_table_entries=128 tcp_max_slot_table_entries=128" >> /etc/modprobe.d/sunrpc.conf
[root@centos7 modprobe.d]# cat sunrpc.conf
options sunrpc udp_slot_table_entries=64 tcp_slot_table_entries=128 tcp_max_slot_table_entries=128
[root@centos7 modprobe.d]# reboot
In NetApp ONTAP® 9 and later, '-v3-tcp-max-read-size' and '-v3-tcp-max-write-size' have been deprecated.
The recommendation is to leverage the option '-tcp-max-xfer-size' instead. This change also allows TCP transfer sizes of 1MB for both reads and writes. ONTAP versions prior to ONTAP 9 only allowed 1MB for reads.
Following example shows - How to change the NFS 'wsize' & 'rsize' to 1MB. This is just an example showing the commands needed to change read/write size, and it's not a 'recommendation', as every environment (Infrastructure + Application IO requirement) is different.
To change the read/write size:
1) Check the current xfer size on NetApp first:
ONTAP94::*> nfs show -fields tcp-max-xfer-size
vserver tcp-max-xfer-size
------- -----------------
SVM_NFS 65536
2) Check the read/write size on the NFS client:
[root@centos7 /]# mount
192.168.0.215:/vol_NFS on /mnt/netapp-mnt type nfs
(rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.215,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=192.168.0.215)
3) Change the xfer size on NetApp first:
ONTAP94::*> nfs modify -tcp-max-xfer-size 1048576 -vserver SVM_NFS
Warning: Setting "-tcp-max-xfer-size" to a value greater than 65536 may cause performance degradation for existing connections using smaller values. Contact technical support for guidance. Do you want to continue? {y|n}: y
Confirm it again:
ONTAP94::*> nfs show -fields tcp-max-xfer-size
vserver tcp-max-xfer-size
------- -----------------
SVM_NFS 1048576
4) unmount the existing connection(s) on NFS clients:
[root@centos7 /]# umount -f /mnt/netapp-mnt
5) mount it back:
[root@centos7 /]# mount -t nfs 192.168.0.215:/vol_NFS /mnt/netapp-mnt/
6) Confirm the new size:
[root@centos7 /]# mount
192.168.0.215:/vol_NFS on /mnt/netapp-mnt type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.215,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=192.168.0.215)
[root@centos7 /]#
According to TR-4067: In some circumstances, the number of ports used to mount or for NFS operations might be exhausted, which then causes subsequent mount and NFS operations to hang until a port is made available. For example, one scenario seen was with ESX using NFS datastores, because some legacy best practices would call for a data LIF/IP address per datastore. In environments with many volumes/datastores, this created a situation where NFS ports were overrun. The remediation for this situation would be to disable the mount-rootonly and/or the nfs-rootonly options on the NFS server. This removes the 1 to 1,024 port range limit and allows up to 65,534 ports to be used in a NFS server.
Courtesy: TR-4067 (Must read article)
No comments:
Post a Comment