Gigabit Ethernet

[MaegRil]

My server has a 3Com 3c996B-T Gigabit NIC installed, and currently I'm only getting max. 25 MB/sec

Here are some more details:
Server MoBo - Supermicro P3TDL3 (http://www.supermicro.com/products/motherboard/P3/LE/P3TDL3.cfm)
Sever CPUs - 2x Intel Pentium III-S 1400 MHz Tualatins (http://processorfinder.intel.com/scripts/details.asp?sSpec=SL5XL&ProcFam=25&PkgType=ALL&SysBusSpd=ALL&CorSpd=ALL)
Server NIC - (as above) 3Com 3c996B-T (http://www.3com.com/products/en_US/detail.jsp?tab=features&sku=3C996B-T&pathtype=purchase)
Server HDDs - 2x Seagate Cheetah 10K.6 36.7 GB (http://www.seagate.com/cda/products/discsales/marketing/detail/0,1081,541,00.html) & 2x Western Digital SE 120GB (http://www.wdc.com/en/products/Products.asp?DriveID=27)

The server is running FC3 and is using the tg3 driver for the NIC.
The NIC is installed in the first 64-bit/66MHz PCI slot which I have set to run at 66 MHz.

This is just for testing purposes - but I copied the FC3 DVD ISO to all four disks in the server.
I then copied the file over NFS using four different clients simultaneously, copying from a different disk in the server for each client.
And the server never supplies more than 25 MB/sec

I really thought it would have been more.

All clients are connected to 10/100Mbps ports, and have no problems achieving full bandwidth individually for their respective ports.

The logs file on the server are reporting the link is up at 1000Mbps
The NIC LEDs say the link is up at 1000Mbps
The switch says the link is up at 1000Mbps
And I can't find any problems on the server nor the clients in the logs.

The only thing I could question is the switch I'm using:
3Com OfficeConnect Switch 8 with Gigabit Uplink (http://www.3com.com/products/en_US/detail.jsp?tab=features&sku=3C1670108&pathtype=purchase)

Any thoughts or comments on this switch?
Anything else I should look at that could improve bandwidth from the server?

I do have access to a 3Com Baseline Switch 2226 (http://www.3com.com/products/en_US/detail.jsp?tab=features&sku=3C16475A&pathtype=purchase), could I expect higher throughput from the server is I used this switch?

[AndyGreen]

Can it not be that NFS is the bottleneck here?

[MaegRil]

Or that. Maybe. I don't know.

What are the limitations of NFS?

[AndyGreen]

Google can be more help than I can.

http://www.google.com/search?q=nfs+performance&ie=UTF-8&oe=UTF-8

If you're seeing a real 25M*Bytes*ps you are definitely above 100M*bits*ps speeds (~10MBytes/sec max). Therefore you need to be looking at what damages performance at 1Gbps not at if it is at 1Gbps.

[MaegRil]

Yep, already did that search, and a search for NFS limitations.
And I already new about setting the block size.

But ..... I had not set the block size, even though I knew about it.

Why not?
(Maybe I'm a little ignorant) Transferring files over NFS on client individually gives the full 100Mbps (10MB/sec) to that client.
And transferring files over NFS on two clients simultaneously still gives 100Mbps to each client.

So, I automatically conclude that because the clients are recieving the max. throughput already then there is no need to specify the block size.

DOH! How wrong was I?

Anyways, I searched for some benchmarks for network cards and came up with this article Testing Gigabit Network Adapters on platform TYAN Trinity GC-SL (http://www.digit-life.com/articles2/gig-eth-64bit/gig-eth-64bit-apr2004-p2-3.html) at http://www.digit-life.com/

I'm not sure exactly what the default block size is for NFS on FC3, but I have an inkly it may only be 1024.
Which coresponds roughly with the performance the graphs show at digit-life.

So, anyways, after studying the charts I immediately tried block sizes of 8192.
And hey presto - all clients are receiving 100Mbps each - simultaneously.

Unfortunately I can't set the MTU any higher on the server as the clients 10/100 NICs don't support jumboframes.

So, .... I've learnt a lesson here! :)

Though, out of interest, does anyone one know what the default block size is for NFS?