Gigabit running at 100Mb speeds

[Flu]

Hi Folks,
Having an issue with the new gigabit network I just installed here. I'm using 2 types of DLink cards on the machines: DGE-550T and DGE-530T. I'm using the dl2k kernel module with the 550 cards and the sk98lin module with the 530 cards. All are recognized by Fedora Core 3 and work fine. However, they only transfer data at 100Mb speeds and not gigabit.

The Netgear 16 port gigabit switch I'm using correctly shows all cards connected at Gigabit speeds and using ethtool on the 550 cards (530 doesn't support ethtool) I correctly see: Speed 1000Mb/s listed for the gigabit card. Additionally, when the 550 cards are initialized, I can see "speed: 1000" listed inside of the dmesg output. I would assume that these cards are all set for gigabit speeds from the status messages.

However, if I transfer between two computers that each have the 530T cards, I get 11.5MB/sec via scp, which is good..but not near the speed I would expect from a gigabit network. When I use either of the machines with the 550T cards, I only get about 7.5 MB/sec. The harddrives on all these machines are easily capable of 35MB/sec or better according to hdparm tests.

Any thoughts or ideas as to why this is happening?

[macemoneta]

Several thoughts.

Make sure you're using a large enough transfer to get a reasonable measurement (a couple of GB).

Check the output of netstat -s for errors (invalid headers, segments retransmited, bad segments received, buffer overrun, Retrans). Your wiring may not be up to the task.

What is the cpu utilization during the transfer? Scp will add encryption/decryption overhead. Once the CPU hits 100%, your data transfer will be throttled.

Remember to account for TCP/IP packet overhead. More data is being sent over the interface than is actually involved in the transfer. Scp knows how much data is being moved at the application level only. Check the RX/TX data on the interface before and after your test with ifconfig eth0, for example.

Remember that sending/receiving data requires traversing the PCI bus twice (HD -> PCI -> CPU -> PCI -> NIC). The PCI bus can become a bottleneck, depending on the other devices active on it:

Bus capacity:
PCI32 33MHz = 133MB/s
PCI32 66MHz = 266MB/s
PCI64 33MHz = 266MB/s
PCI64 66MHz = 533MB/s
PCI-X 133MHz = 1066MB/s

[Flu]

First off, thanks for the quick response, and good ideas. Community help is one the main things I really love about Linux.

I checked my processor utilization, and it was rather high (93%) when using ssh. So I setup an NFS directory instead and tried using rsync to copy the data to the remote server via the NFS mount. The transfer rate was just a shade better: 12.13MB/sec instead of 11.5MB/sec.

I also verfied the netstat information and didn't see anything unsual in there. Not a single bad segment or a segment retransmitted.

I am testing the send with a 203MB file and the TX bytes before and after are right in line with that file size: 1858421050 bytes before (1.7 GiB) and 2082608813 bytes after (1.9GiB). Which indicates no other traffic is being sent across the network at the time of the transfer.

/me ponders

[macemoneta]

Well, to isolate the problem a little further, could you replace the switch with a crossover cable? I haven't seen any test/reviews on this particular model, but I've seen lot's of networking gear that doesn't live up to manufacturer's specs.

If you get the same throughput with the switch bypassed, and you are not maxed on CPU, then the problem may lie in the drivers themselves. Some drivers use microsecond delays or other performance limiting code to work around idiosyncrasies in a given device. At high data rates, those small issues can limit performance. Make sure you're running the latest kernel, as both these drivers have had performance issues in the past.

If I remember correctly, gigabit drivers also allow you to set jumbo frames. To do this, set the mtu on the interface to 9000:

ifconfig eth0 mtu 9000

This can dramatically improve performance, if your drivers support it.

[duncan]

I wasn't able to get speeds significantly higher that T100 until I bought a cat 6 crossover cable (no switch). To test, you can use ttcp (I think this is included with FC3) or get iperf. There is an option on ttcp to discard the data, thus eliminating the disk speed from the test.
Duncan

[Flu]

More great ideas, thanks much. I'll try these things out and post back with the results. I don't think I have any Cat 6 cable in house (we've been using Cat 5e), but I'll order some this afternoon.

[daschu]

Flu
I also have a DGE530T installed in a Fedora core 3 system. I'm having trouble setting up the driver. When I do a make all I get several error's. How did you install the sk98lin module?

thanks

[gin]

If I remember correctly, gigabit drivers also allow you to set jumbo frames. To do this, set the mtu on the interface to 9000:

ifconfig eth0 mtu 9000

This can dramatically improve performance, if your drivers support it.

Just a note that you have to ensure that your switches support jumbo frames. In my research I have discovered that most do not and you should look for that feature when purchasing your switch if it matters to you. I haven't seen it on any SOHO DLINK Gigabit switches yet that's for sure. Also note that often windows drivers (or was it linux drivrs, can't completely remember now) often only support upto 3000.

[Flu]

Just an FYI to everyone that the Category 6 patch cables I purchased did NOT fix this problem for me. I've yet to connect two of the gigabit machines together (bypassing the Gigabit switch), and that's my next course of action. Also the switch and/or NICs I bought don't appear to be very happy when I attempt to make use of jumbo frames.


daschu .. I think I just used the built in sk98lin driver included with FC3, I recall downloading the sk98lin driver seperatately, but I don't think I ended up compiling it.

[macemoneta]

There are several low cost SOHO GigE switches that now offer jumbo frames (thanks to a relatively recent chipset from Broadcom). The lowest cost unit I've found is made by SMC (less than $55 US) (http://shopper.cnet.com/SMC_EZ_Switch_SMC8505T_switch_5_ports/4014-6432_9-30526519.html?q=). A good review of the features is here (http://www.dealtime.com/xPR-SMC_EZ_Switch_SMC8508T~RD-156390821508).

[Flu]

That looks like a pretty good deal, thanks for the great info macemoneta.

[daschu]

Flu
Could you point me in the right direction for how to use the sk98lin driver included with FC3?

thanks

[Flu]

I have the following inside of my /etc/modprobe.conf file, and that seems to do the trick for me:
alias eth0 sk98lin
options sk98lin Speed_A=1000,DupCap_A=Full

Are you getting an error when you run modprobe sk98lin?

[daschu]

Thanks Flu!
When I modprobe sk98lin I get no errors. I still had an older Dlink 530 card installed which was causing problems. For some reason Kudzu does not detect the new card the DGE-530T. Using lspci does not detect the card either. I had to remove the old dlink card reboot and delete the old configuration in Kudzu. Then I added your comments,
alias eth0 sk98lin
options sk98lin Speed_A=1000,DupCap_A=Full
to /etc/modprobe.conf
and the new card started working at 1gigabyte speeds.

Thanks again