When discussing NFS performance in HPC and AI environments, one Linux mount option deserves particular attention:
nconnect
At first glance, the idea is simple.
nconnect allows a Linux NFS client to establish multiple TCP connections to the same NFS server IP address for a single mount.
For example:
mount -t nfs -o vers=3,nconnect=8 10.10.10.11:/ai_data /mnt/ai_dataInstead of relying on one TCP connection between the Linux client and the selected NFS LIF, the client can establish eight.
Conceptually:
Linux HPC Client
|
|-- TCP connection 1 --\
|-- TCP connection 2 ---\
|-- TCP connection 3 ----> NetApp NFS LIF
|-- TCP connection 4 ---/
|-- TCP connection 5 --/
|-- TCP connection 6
|-- TCP connection 7
|-- TCP connection 8The NFS client can distribute NFS operations across those TCP connections, increasing transport-level concurrency and potentially making better use of high-speed networks.
But in an HPC environment, that immediately raises a more important question:
If I increase TCP parallelism, what else has to scale with it?
That is where the topic becomes interesting.
nconnect and RPC Slots Are Not the Same Thing
It is easy to confuse nconnect with NFS RPC slot tables because both influence concurrency.
They solve different problems.
Think of RPC slots as answering this question:
How many NFS operations can the client have outstanding at one time?
The Linux NFS client can maintain multiple outstanding RPC requests. In high-performance environments, the RPC slot table behaves somewhat like queue depth in storage.
If there are too few available RPC slots, the client may not generate enough outstanding work to keep a fast storage system busy.
nconnect, however, answers a different question:
Across how many TCP connections can that work travel?
A useful mental model is:
RPC Slots
=
How much NFS work can be outstanding?
nconnect
=
How many TCP connections can carry that work?So increasing RPC concurrency and increasing TCP concurrency are related, but they are not the same thing.
If the client can generate many outstanding NFS operations but they are all being pushed through a single busy TCP connection, the transport layer may become the bottleneck.
Conversely, configuring many TCP connections does not help much if the application and NFS client are generating very little concurrent work.
Where the ONTAP 128 Limit Fits
There is another layer to understand.
ONTAP associates an NFS connection with a Connection ID, or CID, and a connection can service a finite number of concurrent in-flight operations.
The often-mentioned 128 value belongs to this concurrency discussion. It should not be confused with the number of nconnect connections.
nconnect itself supports multiple TCP connections per mount, while the 128 figure relates to the amount of concurrent work associated with an NFS connection or with Linux RPC slot tuning, depending on the context.
Conceptually, one connection might look like this:
Client
|
| TCP connection / CID
|
| concurrent NFS operations
|
ONTAPWith multiple connections:
Client
|
|-- TCP / CID 1 --> concurrent operations
|-- TCP / CID 2 --> concurrent operations
|-- TCP / CID 3 --> concurrent operations
|-- TCP / CID 4 --> concurrent operations
|
ONTAPThis is why nconnect can increase effective transport parallelism.
Instead of placing all the client's NFS traffic onto one TCP flow, the Linux NFS client can distribute requests across several connections.
Now Imagine 500 HPC Clients
Suppose I have:
500 Linux compute nodes
10 NetApp NFS data LIFs
1 large shared dataset
100/200 GbE storage networkIf all 500 clients were to mount through the same LIF using:
nconnect=8that could theoretically create:
500 clients × 8 TCP connections
=
4,000 TCP connectionstoward that single endpoint.
That may increase concurrency, but it can also simply move the bottleneck somewhere else.
A better HPC design would normally distribute clients across multiple NFS LIFs.
For example:
500 HPC Clients
|
Client / DNS Distribution
|
-----------------------------------------
| | | | |
LIF1 LIF2 LIF3 ... LIF10
| | |
~50 clients ~50 ~50
|
nconnect=8
|
multiple TCP flows per clientNow there are two different scaling mechanisms at work.
The multiple LIFs distribute clients across storage network endpoints.
nconnect then provides multiple TCP connections between each individual client and the LIF it selected.
These mechanisms should not be confused.
nconnect does not automatically spread its connections across several different LIFs.
It normally creates multiple TCP connections to the same server IP address.
That distinction matters a great deal in large HPC environments.
Why This Matters for GPU Training
Consider an AI training node with eight GPUs.
The storage path may look something like this:
GPU Training Framework
|
PyTorch / TensorFlow DataLoader Workers
|
Linux Filesystem + Page Cache
|
NFS Client
|
RPC Layer
|
nconnect TCP Streams
|
NIC / Network
|
NetApp NFS LIF
|
ONTAP
|
FlexGroup / Volumes
|
Aggregates / SSD / NVMeThe GPUs themselves are not normally reading directly from NFS.
The training framework has data-loader processes or threads that read the training data, prepare it and feed the GPUs.
If dozens of data-loader workers are issuing reads in parallel, the storage stack has to be capable of sustaining that concurrency.
Otherwise, expensive GPUs can sit idle waiting for data.
This is where nconnect can help.
Instead of all NFS operations travelling through one TCP connection, several TCP flows can carry requests simultaneously.
But simply adding:
nconnect=16does not guarantee higher GPU utilisation.
It only improves one layer of the path.
Everything Else Has to Be on the Same Page
This is the most important point from an HPC perspective.
Performance is determined by the slowest component in the end-to-end path.
Improving one layer often just exposes the next bottleneck.
For nconnect to deliver meaningful benefit, several areas need to be aligned.
Application concurrency
The application must generate enough parallel I/O.
If an application reads one file synchronously using a single thread, multiple TCP connections will not magically create parallel work.
AI training frameworks often use multiple data-loader workers specifically to avoid this.
RPC concurrency
The Linux NFS client needs enough outstanding RPC operations to keep the transport busy.
If RPC concurrency is too low, having eight or sixteen TCP connections may provide little benefit.
NFS read and write sizes
rsize and wsize influence how much data is transferred per NFS operation.
Large sequential workloads and small-file workloads behave very differently.
Client CPU
High-speed NFS can become CPU intensive.
TCP processing, interrupt handling, NFS processing and context switching all consume CPU cycles.
NIC queues and RSS
Modern NICs use multiple receive and transmit queues.
RSS, IRQ placement and CPU affinity can matter greatly on high-speed networks.
Multiple TCP flows can make it easier for network processing to spread across CPU cores.
Network bandwidth
Multiple TCP connections do not help if the physical NIC or Ethernet link is already saturated.
Packet loss and retransmissions
TCP reacts strongly to packet loss.
Even a small amount of congestion or packet loss can materially affect throughput in a high-bandwidth environment.
NFS LIF distribution
Hundreds or thousands of clients should not unintentionally converge on one LIF while other interfaces remain underutilised.
The client distribution strategy matters just as much as nconnect.
ONTAP CPU and NFS processing
More concurrency on the client eventually means more work on the storage system.
The controller must have enough CPU and protocol-processing capacity to keep up.
Backend storage performance
Eventually, every NFS read or write has to be serviced by WAFL, cache, SSD, NVMe and the underlying storage layout.
If the backend cannot deliver the required IOPS or throughput, increasing TCP parallelism does not solve the problem.
Metadata performance
AI and HPC workloads are not always large sequential reads.
Datasets consisting of millions of small files can generate enormous numbers of operations such as:
LOOKUP
GETATTR
OPEN
READDIR
CREATE
REMOVEIn such environments, metadata operations may become the bottleneck long before network bandwidth is exhausted.
Dataset layout
Sometimes the biggest performance improvement does not come from NFS tuning at all.
Converting millions of tiny files into larger dataset containers, using caching, or staging frequently used data to local NVMe can make a much larger difference.
More Is Not Always Better
It is tempting to think:
nconnect=16must always be better than:
nconnect=4That is not necessarily true.
Imagine:
1,000 clients × nconnect=8
=
8,000 TCP connectionsor:
10,000 clients × nconnect=16
=
160,000 TCP connectionsAt that point, TCP connection counts, mount storms, server resources and connection-management overhead become important design considerations.
HPC tuning is not about increasing every value to the maximum.
It is about finding a balanced level of concurrency across the whole stack.
The Way I Think About nconnect
The easiest way for me to understand it is as part of a pipeline:
Application Threads
↓
RPC Concurrency
↓
nconnect TCP Parallelism
↓
NIC Queues / CPU
↓
Network Fabric
↓
Multiple NFS LIFs
↓
ONTAP NFS Processing
↓
WAFL / FlexGroup
↓
Physical StorageEvery layer must be capable of feeding the next.
If one layer can theoretically deliver 20 GB/s but the next can sustain only 5 GB/s, the application will still see something close to 5 GB/s.
That is why I would not describe nconnect as an NFS performance switch.
It is better understood as one concurrency mechanism within a much larger HPC I/O architecture.
And that, to me, is the more important lesson.
In HPC storage, performance rarely comes from a single setting.
It comes from ensuring that the application, Linux NFS client, RPC layer, TCP stack, NICs, network, storage LIFs, ONTAP processing and backend storage are all capable of moving work at approximately the same pace.
Otherwise, we do not really remove the bottleneck.
We simply move it somewhere else.
References
For further reading, useful areas to review include:
NetApp ONTAP NFS documentation — nconnect, NFS client tuning, NFS session behaviour, connection limits and NFS performance guidance.
Linux NFS client documentation — mount options, RPC slot behaviour, rsize, wsize and TCP-based NFS transport.
NetApp ONTAP performance documentation — NFS LIF design, FlexGroup, networking, protocol performance and backend storage behaviour.
Comments
Post a Comment