[Fedora 15] Slow network after upgrade

[Skiriki]

Hi; I run two servers at home, and I recently ran into a trickier sort of problem.

We'll call them ns1 and ns2. ns1 is the primary server where serious bizniz happens, and ns2 is a testbed for stuff.

Both servers are reasonably identical: both are AMD + Asus MB, with embedded Realtek chipset for ethernet, and with a second ethernet card (100% identical, bought at the same time etc).

Both servers have similar functions: they run bind, httpd, sendmail, mysql, php, several wikis, forums and such; ns1 also runs dovecot and squirrelmail. Both were assembled and installed by me.

On both servers, eth0 is the internal network's NIC, and it is trusted. eth1 is IP-masquerading, and not trusted; iptables pokes enough prerequisite holes to it, to let it work as web and mail server etc. Both servers have identical iptables setup. On both systems, eth1 routes traffic outside. On both systems, eth0 has a static IP (192.168.0.1 and .2, respectively), and eth1 receives its static IP address from my ISP (with MAC-based identification).

Both servers are connected to the same DSL box, and both servers are connected to the same switch in internal network, using same type of cable. Yay for standardization.

Originally, both servers ran Fedora 13.

I upgraded ns2 around January 2012 to Fedora 15, via DVD, using "upgrade" option, and everything went swimmingly, no problems. I have continued to update the server via yum, and everything is still just fine.

I upgraded ns1 on 1st of May, 2012, to Fedora 15, via DVD, using "upgrade" option, and performance is drastically different. Immediately after upgrading, I ran yum to retrieve update packets, and I thought that was that.

Well, no.

ns2 continues behaving splendidly.

ns1, on the other hand, slows down and then messes something between eth0 and eth1. This is detectable with a simple test: I try to download stuff when my desktop is using ns1 for routing; speed starts at 35 KB/s, then keeps dropping until it is at zero.

If I swap to ns2 for routing, and try to download again, I will get the 2 Mbit/s as I'm supposed to.

However, if I log in to ns1, and start a download off the command line (like, say, with wget), I will get 2Mbit/s again.

This stunted downloading includes downloads from the server itself, while in internal network: prior to update, it worked fast, and content was quick to load.

ethtool tells me that all NICs are running at full speed.

I have already tried to disable ipv6, up to the point of removing ip6tables from loadable services, but ns2 runs just fine with ip6tables lurking about. Both systems do not use ipv6 for any of their NICs.

I have added biosdevname=0 to kernel line in grub.conf, and it still makes no difference (and ns2 runs just fine without it).

I have tried to reinstall kernel and various other things, and none of them make a difference.

My kernel line looks like this:

kernel /vmlinuz-2.6.43.2-6.fc15.x86_64 root=UUID=9b7277da-bf2f-4ef1-be90-2f7226935c9a ro rd_MD_UUID=84894a3e:0e0dff17:4dc693b9:2565f0cc rd_MD_UUID=a4851993:0ba36d09:736988a9:b06084d4 rd_NO_LUKS rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=fi-latin1 iommu=noaperture biosdevname=0

I do not spot any obvious problems in /var/log/messages, /var/log/boot.log or dmesg.

I am starting to reach my wits' end, and I can't figure it out.

Please help me. Or at least toss some ideas, so I can try to google and figure it out on my own; I'm out of ideas.

---------- Post added at 03:26 AM ---------- Previous post was at 01:00 AM ----------

Additional information and a tiny correction:

This is eth0:
Bridge: nVidia Corporation MCP61 Ethernet (rev a2)
Subsystem: ASUSTeK Computer Inc. Device 83a4
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 43
Memory at dedf9000 (32-bit, non-prefetchable) [size=4K]
I/O ports at c480 [size=8]
Capabilities: [44] Power Management version 2
Capabilities: [50] MSI: Enable+ Count=1/8 Maskable+ 64bit+
Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
Kernel driver in use: forcedeth
Kernel modules: forcedeth

This is eth1:
Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8169/8110 Family PCI Gigabit Ethernet NIC
Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
I/O ports at d800 [size=256]
Memory at deeffc00 (32-bit, non-prefetchable) [size=256]
Expansion ROM at deec0000 [disabled] [size=128K]
Capabilities: [dc] Power Management version 2
Kernel driver in use: r8169
Kernel modules: r8169

Now, a quick google shows that some people have had problems with forcedeth drivers. Any idea if this would be it?

Would this get fixed by feeding Fedora 16 to it? Or should I try something else?

[stevea]

MPC61 and a RealTek 8169 Yikes - you didn't exactly get the pick of the litter there. Two of the worst interfaces around IMO - tho' nearly every end-user mobo uses a RealTek cheapie these days.

Yes - i was aware of forcedeth problems a year+ ago - no idea where that stands.

I can't answer your question abt the driver difference on F16, but if I were you I'd try some network performance tests using the F16 and F17beta live CDs. You can boot, then yum install 'nuttcp' or other tests.


You really need to characterize the problem better.

Doing Inet downloads might indicate a problem, but it's not easily reproducible and includes other issues like disk speed. Instead test between boxen using synthetic data. I suggest you try nuttcp and run both tcp and udp tests and the problematic system (ns2?) - on EACH interface.

Not entirely clear from your description but I assume your problem is on the realtek interface only ? Test both.
Use wireshark and examine the inter-packet gaps and look for bad packets or re-tries.

[Skiriki]

Thanks -- at this stage this has stopped being a Fedora 15 problem, and turned into Fedora 16 problem.

After (even more painful) update (GRUB2 decided not to play nice with software RAID and meant that I suddenly had to put my geek learning into use -- deceptively simple in the end, but it took some serious prodding), I squished Fedora 16 into the server, and ran full update to everything.

The problem persists; let me try to illustrate it better.

This is my normal setup:

Lots of computers at home -> [->eth0->NS1 services->eth1->] -> internet

(ns2 is a similar setup; basically, both route traffic in and out and act as a firewall.)

This used to work just fine (Fedora 13). Now, if I try to use this, my connection speeds become pitiful: Samba, sendmail, httpd -- anything accessed through eth0 of NS1 dies a pitiful byte-death, if it is on the server or in the internet.

If I log in to ns1, and tell it to access the same internet resource (which effectively bypasses eth0), speeds are just fine! eth1 works perfectly. eth0 -- the one with forcedeth driver -- apparently does not.

Further, people who access only eth1, in form of using www pages from the internet side, have not reported any slowdown effects.

So wouldn't logic dictate that the problem lurks either between eth0 and eth1, or in eth0?

Here's what lshw, for example, reports to me:

eth1 (the one which is working OK):
capabilities: pm bus_master cap_list rom ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=2.3LK-NAPI duplex=full ip=[my IP address] latency=64 link=yes maxlatency=64 mingnt=32 multicast=yes port=MII speed=100Mbit/s
resources: irq:18 ioport:d800(size=256) memory:deeffc00-deeffcff memory:deec0000-deedffff

eth0 (which seems to be the culprit):
capabilities: bridge pm msi ht bus_master cap_list ethernet physical mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=forcedeth driverversion=0.64 duplex=full ip=192.168.0.1 latency=0 link=yes maxlatency=20 mingnt=1 multicast=yes port=MII speed=1Gbit/s
resources: irq:43 memory:dedf9000-dedf9fff ioport:c480(size=8)

Check the parts in italics. Compare them to the one working OK. Do those numbers seem suspect? I confess that I have never tweaked 'em before -- what's the command I have to pound in to terminal to change 'em?

On the other hand, this is starting to get pretty academic at this point -- I'm tired, and I don't think I am going to do yet another all-nighter with this thing, so I'm just going to solve my problem by throwing money at it and go get a new NIC tomorrow (Realtek chipset, since it works), unless someone has some creative ideas in three or four hours (because then I'll be off to bed, possibly much sooner, on account of not having slept more than a handful of hours).

In any case, I appreciate all help I can get.

[stevea]

Those latency, mingnt numbers are the PCI bus setup I believe and they should not cause the problem you re seeing.

So wouldn't logic dictate that the problem lurks either between eth0 and eth1, or in eth0?

Yes - that's certainly the place to investigate first - most probably sources of the problem.

If you want to get a differnt NIC I can't blame you - those are both loser-interfaces in my book.

You'd be smart to invest in a Intel or Marvelle chipset (on a PCIx interface if you want full GigE performance).
RealTek is cheap ubiquitous Taiwanese junk. I shutoff the ReallTek on my SOHO server and replaced it with one of these.
http://www.newegg.com/Product/Produc...82E16833106033

[Skiriki]

Originally Posted by stevea

Those latency, mingnt numbers are the PCI bus setup I believe and they should not cause the problem you re seeing.

Okay. Thing is, I have never had this sort of problem before (I have had problems with nVidia's display drivers, though), and I'm solidly stumped.

Yes - that's certainly the place to investigate first - most probably sources of the problem.

Some googling suggested that forcedeth is still a problem in Fedora 16 (I found a bugtrack report from late April this year), and I'm not exactly coding-talented, so...

If you want to get a differnt NIC I can't blame you - thse are both loser-interfaces in my book.

In my case, losers can't be choosers, as I have very limited budget due to unemployment (I recycle old desktops for my servers, been doing that for over a decade); as said, the existing Realtek-based card works just fine for my modest needs (and similar cards in ns2 as well), so I'm inclined to go with that. I mostly want this damn thing working, and reduce my stress over it.

However, thanks for the tip; the local computer store may have something nice in their stash and since I'm a longtime customer, I usually get little oldtimer discounts here and there. So I'll check 'em out and see what they got.

[Skiriki]

Welp, I could barely afford the Realtek-based A-Link card, so that was it.

However, now I have functional network once again.

So clearly there's something wrong with forcedeth driver.