VLAN not recognizing incoming traffic

afberendsen · #1 29th May 2011, 12:15 AM

Hi all
I'm working for the last 10 dyas trying to solve this problem and now I have to ask for help

Before explaining the problem, I have to draw the environment. Because of post size limitations, I'll have to break this post in more than one entry.

Physical view

Code:

+--------+     +---+
|storage1|-----|   |
+--------+     | s |
               | w |
+--------+     | i |     +-------+
|router2 |-----| t |-----|router1|--->Internet
+--------+     | c |     +-------+
               | h | 
+--------+     | 8 |
|router3 |-----|   |
+--------+     +---+

Device details

switch8

Hardware: cisco WS-C3548-XL
IOS: C3500XL Software (C3500XL-C3H2S-M), Version 12.0(5)WC16, RELEASE SOFTWARE (fc1). Processor is running Enterprise Edition Software. Cluster command switch capable. Cluster member switch capable.

Interfaces: 48 FastEthernet/IEEE 802.3 interface(s). 2 Gigabit Ethernet/IEEE 802.3 interface(s)

interfaces fastEthernet 0/1 connects switch8 to router2

Code:

interface FastEthernet0/1
 description Link router2 (all networks)
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1-10,1002-1005
 switchport mode trunk
 spanning-tree portfast

interfaces fastEthernet 0/4 connects switch8 to router3

Code:

interface FastEthernet0/4
 description router3/on-board (echo router)
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1-10,1002-1005
 switchport mode trunk
 spanning-tree portfast

interfaces fastEthernet 0/5 connects switch8 to storage1

Code:

interface FastEthernet0/5
 description storage1/left ob-board/eth2 (temp connection until is set-up)
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1-10,1002-1005
 switchport mode trunk
 spanning-tree portfast

interfaces fastEthernet 0/6 connects switch8 to router1

Code:

interface FastEthernet0/6
 description Link to personal router @ office

router1
- Hardware: Cisco WAG325N Wireless-N ADSL2+ Gateway
- Firmware: Firmware Version:V1.00.12
- Software configuration:
  - Local IP address: 192.168.1.1/24
  - RIP disabled.UPnP disabled. IGMP Proxy disabled. SNMP disabled.
  - Static route: 192.168.2.0/24 192.168.1.49

router2

Hardware: Dell Optiplex GX270 Small desktop. Intel(R) Pentium(R) 4 CPU 2.66GHz. 1024MB
OS: Linux router2.berendsen.local 2.6.34.8-68.fc13.i686 #1 SMP Thu Feb 17 15:00:46 UTC 2011 i686 i686 i386 GNU/Linux
Network: eth3=>1:0c.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)

Output for netstat -r -n

Code:

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.7.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan7
192.168.6.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan6
192.168.5.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan5
192.168.4.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan4
192.168.3.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan3
192.168.2.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan2
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth3
192.168.10.0    0.0.0.0         255.255.255.0   U         0 0          0 vlan10
192.168.9.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan9
192.168.8.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan8
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth3
... (previous line repeated for all interfaces)
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth3

Output for ifconfig -a

Code:

eth3      Link encap:Ethernet  HWaddr 00:0B:DB:6F:6A:7C
          inet addr:192.168.1.49  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20b:dbff:fe6f:6a7c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3826795 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1780137 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:675516650 (644.2 MiB)  TX bytes:160326043 (152.8 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:2946 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2946 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:443934 (433.5 KiB)  TX bytes:443934 (433.5 KiB)

vlan2     Link encap:Ethernet  HWaddr 00:0B:DB:6F:6A:7C
          inet addr:192.168.2.1  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::20b:dbff:fe6f:6a7c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2438960 errors:0 dropped:0 overruns:0 frame:0
          TX packets:942825 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3407412446 (3.1 GiB)  TX bytes:67145386 (64.0 MiB)

vlan3     Link encap:Ethernet  HWaddr 00:0B:DB:6F:6A:7C
          inet addr:192.168.3.1  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::20b:dbff:fe6f:6a7c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:44921 errors:0 dropped:0 overruns:0 frame:0
          TX packets:44687 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2928473 (2.7 MiB)  TX bytes:3133249 (2.9 MiB)
(from vlan4 to vlan10, the IP address only changes the third byte [4...10])

Configuration files

Code:

==== File /etc/sysconfig/network-scripts/ifcfg-eth3 ====
DEVICE=eth3
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
NM_CONTROLLED=no
USERCTL=yes
IPV6INIT=no
IPADDR=192.168.1.49
DNS1=202.180.64.10
DNS2=202.180.64.11
GATEWAY=192.168.1.1
PREFIX=24
DOMAIN=berendsen.local
NAME="My personal network"
==== File /etc/sysconfig/network-scripts/ifcfg-vlan2 ====
VLAN=yes
VLAN_NAME_TYPE=VLAN_PLUS_VID_NO_PAD
DEVICE=vlan2
PHYSDEV=eth3
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
NM_CONTROLLED=no
USERCTL=yes
IPV6INIT=no
IPADDR=192.168.2.1
PREFIX=24
==== File /etc/sysconfig/network-scripts/ifcfg-vlan3 ====
VLAN=yes
VLAN_NAME_TYPE=VLAN_PLUS_VID_NO_PAD
DEVICE=vlan3
PHYSDEV=eth3
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
NM_CONTROLLED=no
USERCTL=yes
IPV6INIT=no
IPADDR=192.168.3.1
PREFIX=24

Continues on the next post

afberendsen · #2 29th May 2011, 12:17 AM

Continuing from previous post

router3

Hardware: Dell Optiplex GX270 Small desktop. Intel(R) Pentium(R) 4 CPU 2.66GHz. 512MB
OS: Linux router3.berendsen.local 2.6.35.13-91.fc14.i686 #1 SMP Tue May 3 13:36:36 UTC 2011 i686 i686 i386 GNU/Linux
Network: eth12=>01:0c.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)

Output for netstat -r -n

Code:

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.7.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan7
192.168.6.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan6
192.168.5.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan5
192.168.4.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan4
192.168.3.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan3
192.168.2.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan2
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth12
192.168.10.0    0.0.0.0         255.255.255.0   U         0 0          0 vlan10
192.168.9.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan9
192.168.8.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan8
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth12
... (previous line repeated for all interfaces)
0.0.0.0         192.168.2.1     0.0.0.0         UG        0 0          0 vlan2

Output for ifconfig -a

Code:

eth12     Link encap:Ethernet  HWaddr 00:0B:DB:65:ED:41
          inet addr:192.168.1.254  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20b:dbff:fe65:ed41/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:111524 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39100 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:7687856 (7.3 MiB)  TX bytes:8064875 (7.6 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:19504 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19504 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2185332 (2.0 MiB)  TX bytes:2185332 (2.0 MiB)

vlan2     Link encap:Ethernet  HWaddr 00:0B:DB:65:ED:41
          inet addr:192.168.2.254  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::20b:dbff:fe65:ed41/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:13880 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15862 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:799915 (781.1 KiB)  TX bytes:6931265 (6.6 MiB)

vlan3     Link encap:Ethernet  HWaddr 00:0B:DB:65:ED:41
          inet addr:192.168.3.254  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::20b:dbff:fe65:ed41/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2408 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19422 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:120400 (117.5 KiB)  TX bytes:816036 (796.9 KiB)
(from vlan4 to vlan10, the IP address only change the third byte [4...10])

Configuration files

Code:

==== File /etc/sysconfig/network-scripts/ifcfg-eth12 ====
DEVICE="eth12"
NM_CONTROLLED="no"
ONBOOT=yes
BOOTPROTO=none
TYPE=Ethernet
==== File /etc/sysconfig/network-scripts/ifcfg-vlan2 ====
VLAN=yes
VLAN_NAME_TYPE=VLAN_PLUS_VID_NO_PAD
DEVICE=vlan2
PHYSDEV=eth12
NM_CONTROLLED="no"
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.2.254
PREFIX=24
IPV6INIT="no"
TYPE=Ethernet
DNS1="202.180.64.10"
DNS2="202.180.64.11"
GATEWAY=192.168.2.1
==== File /etc/sysconfig/network-scripts/ifcfg-vlan3 ====
VLAN=yes
VLAN_NAME_TYPE=VLAN_PLUS_VID_NO_PAD
DEVICE=vlan3
PHYSDEV=eth12
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
NM_CONTROLLED=no
USERCTL=yes
IPV6INIT=no
IPADDR=192.168.3.254
PREFIX=24

Continues on the next post

afberendsen · #3 29th May 2011, 12:18 AM

Continuing from previous post

storage1

Hardware: ASUS M2N-SLI. AMD Athlon(tm) 64 X2 Dual Core Processor 4400+. 4096MB
OS: Linux storage1.berendsen.local 2.6.35.6-45.fc14.x86_64 #1 SMP Mon Oct 18 23:57:44 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
Network: eth2=>00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)

Output for netstat -r -n

Code:

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.7.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan7
192.168.6.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan6
192.168.5.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan5
192.168.4.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan4
192.168.3.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan3
192.168.2.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan2
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth2
192.168.10.0    0.0.0.0         255.255.255.0   U         0 0          0 vlan10
192.168.9.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan9
192.168.8.0     0.0.0.0         255.255.255.0   U         0 0          0 vlan8
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth2
... (previous line repeated for all interfaces)
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth2

Output for ifconfig -a

Code:

eth1      Link encap:Ethernet  HWaddr 54:E6:FC:85:04:5D
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8109 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:355575 (347.2 KiB)
          Interrupt:16 Base address:0x4000

eth2      Link encap:Ethernet  HWaddr 00:1E:8C:8E:D1:EA
          inet addr:192.168.1.50  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::21e:8cff:fe8e:d1ea/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:80583511 errors:0 dropped:0 overruns:0 frame:0
          TX packets:70052126 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:45146631341 (42.0 GiB)  TX bytes:37979314156 (35.3 GiB)
          Interrupt:42 Base address:0x2000

eth3      Link encap:Ethernet  HWaddr 00:1E:8C:8E:D6:82
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:21 Base address:0xc000

eth4      Link encap:Ethernet  HWaddr 54:E6:FC:84:F3:F1
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:17 Base address:0xe000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:8923 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8923 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:992371 (969.1 KiB)  TX bytes:992371 (969.1 KiB)

sit0      Link encap:IPv6-in-IPv4
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

vlan2     Link encap:Ethernet  HWaddr 00:1E:8C:8E:D1:EA
          inet addr:192.168.2.50  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::21e:8cff:fe8e:d1ea/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:888408 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2352695 errors:0 dropped:23 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:54029282 (51.5 MiB)  TX bytes:3433884793 (3.1 GiB)

vlan3     Link encap:Ethernet  HWaddr 00:1E:8C:8E:D1:EA
          inet addr:192.168.3.50  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::21e:8cff:fe8e:d1ea/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:45270 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23241 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2941992 (2.8 MiB)  TX bytes:2242976 (2.1 MiB)
(from vlan4 to vlan10, the IP address only changes the third byte [4...10])

Configuration files

Code:

==== File /etc/sysconfig/network-scripts/ifcfg-eth2 ====
DEVICE="eth2"
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV6INIT=no
NAME="Base interface for work around"
IPADDR=192.168.1.50
PREFIX=24
GATEWAY=192.168.1.1
DNS1=202.180.64.10
DNS2=202.180.64.11
DOMAIN=berendsen.local
==== File /etc/sysconfig/network-scripts/ifcfg-vlan2 ====
VLAN=yes
VLAN_NAME_TYPE=VLAN_PLUS_VID_NO_PAD
DEVICE=vlan2
PHYSDEV=eth2
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
NM_CONTROLLED=no
USERCTL=yes
IPV6INIT=no
IPADDR=192.168.2.50
PREFIX=24
NAME="Interface for infrastructure access. Should not be used for data"
==== File /etc/sysconfig/network-scripts/ifcfg-vlan3 ====
VLAN=yes
VLAN_NAME_TYPE=VLAN_PLUS_VID_NO_PAD
DEVICE=vlan3
PHYSDEV=eth2
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
NM_CONTROLLED=no
USERCTL=yes
IPV6INIT=no
IPADDR=192.168.3.50
PREFIX=24
NAME="Interface for infrastructure access. Should not be used for data"

afberendsen · #4 29th May 2011, 12:28 AM

Problem description

From storage1 I'm able to ping any interface in router2, using ping:

Code:

[root@storage1 network-scripts]# ping -n -I vlan3 192.168.3.1
PING 192.168.3.1 (192.168.3.1) from 192.168.3.50 vlan3: 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_req=1 ttl=64 time=0.289 ms
64 bytes from 192.168.3.1: icmp_req=2 ttl=64 time=0.241 ms
64 bytes from 192.168.3.1: icmp_req=3 ttl=64 time=0.249 ms
^C
--- 192.168.3.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.241/0.259/0.289/0.028 ms

From router2 I'm able to ping any interface in storage1, using ping:

Code:

# ping -n -I vlan4 192.168.4.50
PING 192.168.4.50 (192.168.4.50) from 192.168.4.1 vlan4: 56(84) bytes of data.
64 bytes from 192.168.4.50: icmp_seq=1 ttl=64 time=1.07 ms
64 bytes from 192.168.4.50: icmp_seq=2 ttl=64 time=0.154 ms
64 bytes from 192.168.4.50: icmp_seq=3 ttl=64 time=0.244 ms
64 bytes from 192.168.4.50: icmp_seq=4 ttl=64 time=0.152 ms
^C
--- 192.168.4.50 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3254ms
rtt min/avg/max/mdev = 0.152/0.406/1.077/0.389 ms

But, from/to router3, I'm only able to ping 192.168.1.254 and 192.168.2.254. And then comes the strange thing. Running ping an dtcpdump at the smae time, tracing the base interface, I can see router2 sending repplies back to router3, but those replies are never received.

router2

Code:

# tcpdump -e -n -i eth3|grep "vlan 3"|grep -i "ed:41"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes
11:26:45.275538 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 46
11:26:45.275555 00:0b:db:6f:6a:7c > 00:0b:db:65:ed:41, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Reply 192.168.3.1 is-at 00:0b:db:6f:6a:7c, length 28
11:26:46.278442 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 46
11:26:46.278453 00:0b:db:6f:6a:7c > 00:0b:db:65:ed:41, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Reply 192.168.3.1 is-at 00:0b:db:6f:6a:7c, length 28
11:26:47.280347 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 46
11:26:47.280357 00:0b:db:6f:6a:7c > 00:0b:db:65:ed:41, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Reply 192.168.3.1 is-at 00:0b:db:6f:6a:7c, length 28
11:26:49.276414 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 46
11:26:49.276426 00:0b:db:6f:6a:7c > 00:0b:db:65:ed:41, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Reply 192.168.3.1 is-at 00:0b:db:6f:6a:7c, length 28
11:26:50.278319 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 46
11:26:50.278328 00:0b:db:6f:6a:7c > 00:0b:db:65:ed:41, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Reply 192.168.3.1 is-at 00:0b:db:6f:6a:7c, length 28
11:26:51.280477 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 46
11:26:51.280490 00:0b:db:6f:6a:7c > 00:0b:db:65:ed:41, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Reply 192.168.3.1 is-at 00:0b:db:6f:6a:7c, length 28
11:26:53.277541 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 46
11:26:53.277551 00:0b:db:6f:6a:7c > 00:0b:db:65:ed:41, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Reply 192.168.3.1 is-at 00:0b:db:6f:6a:7c, length 28
11:26:54.278444 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 46
11:26:54.278454 00:0b:db:6f:6a:7c > 00:0b:db:65:ed:41, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Reply 192.168.3.1 is-at 00:0b:db:6f:6a:7c, length 28
^C147 packets captured
147 packets received by filter
0 packets dropped by kernel

router3

Code:

# ping  -n -q -I vlan3 192.168.3.1 &
# tcpdump -e -n -i eth12|grep "vlan 3"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth12, link-type EN10MB (Ethernet), capture size 65535 bytes
11:26:44.094295 00:04:27:0a:34:44 > 01:00:0c:cc:cc:cd, ethertype 802.1Q (0x8100), length 68: vlan 3, p 0, LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8000.00:04:27:0a:34:42.8010, length 42
11:26:45.263791 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 28
11:26:46.094390 00:04:27:0a:34:44 > 01:00:0c:cc:cc:cd, ethertype 802.1Q (0x8100), length 68: vlan 3, p 0, LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8000.00:04:27:0a:34:42.8010, length 42
11:26:46.266789 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 28
11:26:47.268787 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 28
11:26:48.094489 00:04:27:0a:34:44 > 01:00:0c:cc:cc:cd, ethertype 802.1Q (0x8100), length 68: vlan 3, p 0, LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8000.00:04:27:0a:34:42.8010, length 42
11:26:49.264793 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 28
11:26:50.096083 00:04:27:0a:34:44 > 01:00:0c:cc:cc:cd, ethertype 802.1Q (0x8100), length 68: vlan 3, p 0, LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8000.00:04:27:0a:34:42.8010, length 42
11:26:50.266786 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 28
11:26:51.268790 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 28
11:26:52.096682 00:04:27:0a:34:44 > 01:00:0c:cc:cc:cd, ethertype 802.1Q (0x8100), length 68: vlan 3, p 0, LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8000.00:04:27:0a:34:42.8010, length 42
11:26:53.265791 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 28
11:26:54.095027 00:04:27:0a:34:44 > 01:00:0c:cc:cc:cd, ethertype 802.1Q (0x8100), length 68: vlan 3, p 0, LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8000.00:04:27:0a:34:42.8010, length 42
11:26:54.266790 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 28
11:26:55.268787 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 28
11:26:56.095124 00:04:27:0a:34:44 > 01:00:0c:cc:cc:cd, ethertype 802.1Q (0x8100), length 68: vlan 3, p 0, LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8000.00:04:27:0a:34:42.8010, length 42
11:26:57.266789 00:0b:db:65:ed:41 > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 3, p 0, ethertype ARP, Request who-has 192.168.3.1 tell 192.168.3.254, length 28
^C132 packets captured
132 packets received by filter
0 packets dropped by kernel

Any ideas about what I'm doing wrong?

David Becker · #5 29th May 2011, 11:12 AM

You might want to try to use a unique mac address per vlan. Start by using a unique mac address on the vlan interfaces which aren't behaving as expected.

David

afberendsen · #6 29th May 2011, 09:35 PM

Quote:

Originally Posted by David Becker

You might want to try to use a unique mac address per vlan. Start by using a unique mac address on the vlan interfaces which aren't behaving as expected.

David

Thanks for the idea.

I considered that but this will be totally against the idea of having VLANs. Depending on the number of VLANs, if I have to use different MAC addresses, then will be more sensible to try to use a quad-port NIC or event a MoBo with many slots.

Since I wanto to use only one NIC and have those VLNAs working as expected, I wonder if you can help me to point what could be problem, which seems to be with the router3 server.

David Becker · #7 29th May 2011, 10:09 PM

Quote:

Originally Posted by afberendsen

Thanks for the idea.

I considered that but this will be totally against the idea of having VLANs.

Why?

Quote:

Originally Posted by afberendsen

Depending on the number of VLANs, if I have to use different MAC addresses, then will be more sensible to try to use a quad-port NIC or event a MoBo with many slots.

That would give you more throughput which is a separate consideration of logical partitioning and minimizing broadcasts.

Quote:

Originally Posted by afberendsen

Since I wanto to use only one NIC and have those VLNAs working as expected, I wonder if you can help me to point what could be problem, which seems to be with the router3 server.

Try using a different mac address for the problematic vlans in questions. You don't need a separate nic per mac address, it requires little effort and it'll rule out host or switch (config and operation) idiosyncrasies.

David

afberendsen · #8 29th May 2011, 11:01 PM

Hello David

For my understanding, VLANs are defined by 802.1q, and the extra tagging for the Ethernet frame is to allow logical groups using the same MAC for Level 1. Also, on my case, I have now, for testing purposes, three servers. But, at the end deployment, I'll have many servers and will be very hard to track and configure server by server in regards of MAC addresses.

From Wikipedia; "IEEE 802.1Q, or VLAN tagging, is a networking standard promulgated by the IEEE 802.1 work group for the sharing of a physical Ethernet network link by multiple independent logical network. IEEE 802.1Q defines the meaning of a virtual LAN (VLAN) with respect to the specific conceptual model underpinning bridging at the Media Access Control layer and to the IEEE 802.1D Spanning Tree Protocol. This protocol allows nodes on different VLANs to communicate with one another through a network switch with Network Layer (OSI layer 3) capabilities, or a router." (source http://en.wikipedia.org/wiki/IEEE_802.1Q)

Also, if in fact I need to set-up different MAC addresses for my VLANs, this implies that the Linux kernel is not ready to support VLANs. In our production environment, we have a mix of operating systems and hardware, sharing VLANs, using only Cisco switches and routers, and there are no need to assign MAC addresses for the extra VLANs.

Do you think then that my decision to use Linux for this task is not the best option? Should I use Solaris or any other U*X brand instead of Linux?

---------- Post added at 10:01 AM ---------- Previous post was at 09:48 AM ----------

Reading again that same page, maybe I found the problem. I'll make some changes on my network configuration to check if this will fix the problem.

Quote:

Clause 9 of the 1998 802.1Q standard defines the encapsulation protocol used to multiplex VLANs over a single link, by adding VLAN tags. However, it is possible to send frames either tagged or untagged, so to help explain which frames will be sent with or without tags, some vendors (most notably Cisco) use the concepts of a) trunk ports and b) the native VLAN for that trunk.
The concept of a trunk port is that once a port is designated as a trunk port, it will forward and receive tagged frames.
Frames belonging to the native VLAN do NOT carry VLAN tags when sent over the trunk. Conversely, if an untagged frame is received on a trunk port, the frame is associated with the Native VLAN for this port.

David Becker · #9 29th May 2011, 11:18 PM

Quote:

Originally Posted by afberendsen

Hello David

For my understanding, VLANs are defined by 802.1q, and the extra tagging for the Ethernet frame is to allow logical groups using the same MAC for Level 1.

Also, on my case, I have now, for testing purposes, three servers. But, at the end deployment, I'll have many servers and will be very hard to track and configure server by server in regards of MAC addresses.

From Wikipedia; "IEEE 802.1Q, or VLAN tagging, is a networking standard promulgated by the IEEE 802.1 work group for the sharing of a physical Ethernet network link by multiple independent logical network. IEEE 802.1Q defines the meaning of a virtual LAN (VLAN) with respect to the specific conceptual model underpinning bridging at the Media Access Control layer and to the IEEE 802.1D Spanning Tree Protocol. This protocol allows nodes on different VLANs to communicate with one another through a network switch with Network Layer (OSI layer 3) capabilities, or a router." (source http://en.wikipedia.org/wiki/IEEE_802.1Q)

Also, if in fact I need to set-up different MAC addresses for my VLANs, this implies that the Linux kernel is not ready to support VLANs. In our production environment, we have a mix of operating systems and hardware, sharing VLANs, using only Cisco switches and routers, and there are no need to assign MAC addresses for the extra VLANs.

Do you think then that my decision to use Linux for this task is not the best option? Should I use Solaris or any other U*X brand instead of Linux?

---------- Post added at 10:01 AM ---------- Previous post was at 09:48 AM ----------

Reading again that same page, maybe I found the problem. I'll make some changes on my network configuration to check if this will fix the problem.

You're not getting an ARP reply on router 3, at least not coming over vlan3, and the switch in question supports many different setups. ARP is not encapsulated in IP and the switch may be making all kinds of strange or unexpected decisions based on mac addresses being reused.

David

afberendsen · #10 29th May 2011, 11:34 PM

Using VLAns implies in re-suing the MAC address for all frames anyway. That's one the ideas behind VLANs. To have VLAns working, all the devices on the frame path (NIC, switch, router, operating system etc) need to understand the VLAN tagging. From that same Wikipedia page, you can see that VLAN is an extra field inside the Level 1 frame.

David Becker · #11 29th May 2011, 11:57 PM

Quote:

Originally Posted by afberendsen

Using VLAns implies in re-suing the MAC address for all frames anyway. That's one the ideas behind VLANs.

Whatever dude. VLANs are specified independent of l3 application and you're not limiting yourself to l2 switching so take the interoperation of l2 and l3 into consideration.

Quote:

Originally Posted by afberendsen

To have VLAns working, all the devices on the frame path (NIC, switch, router, operating system etc) need to understand the VLAN tagging. From that same Wikipedia page, you can see that VLAN is an extra field inside the Level 1 frame.

I'd argue that's a level 2 frame.

So why aren't you receiving an ARP reply on router 3?

David

afberendsen · #12 30th May 2011, 12:25 AM

I have to test tonight, but seems that the level 2 output frame is incorrectly forwarded, because of a bad Level 3 routing table, to a non-tagged switch port, which is dropping the vlan tag.

David Becker · #13 30th May 2011, 12:39 AM

Quote:

Originally Posted by afberendsen

I have to test tonight, but seems that the level 2 output frame is incorrectly forwarded, because of a bad Level 3 routing table, to a non-tagged switch port, which is dropping the vlan tag.

Which would be contrary to the configuration you showed us in your initial postings. One way of troubleshooting would be to use a unique mac address. You don't have to mandate the use of unique mac addresses in your corporate by-laws, just try it out for the problematic vlans while changing the scope of your tcpdump to capture based on mac addresses rather than vlans. FYI, I have linux hosts running with way more vlans, all using the same mac address connected to cisco switches but for troubleshooting's sake, I may want to change these mac addresses, if only temporarily.

Honestly, there's much more involved with switching than what you're thinking about, especially when you're using a managed switch like the one you're using. There's the switch, mac address learning, the host and all kinds of different modes of operation for each of these devices. A cisco device may function 'properly' with a cisco switch while other devices may default to a different modus operandi resulting in idiosyncrasies, which is why I suggested back in my first reply to try it out with a different mac address and the only thing I'd want to add to that is tcpdumping more than just vlan 3 (which is ostensibly easier when using a unique mac address).

David

afberendsen · #14 30th May 2011, 12:53 AM

In fact, all the details about the environment are described.

Quote:

...
interfaces fastEthernet 0/6 connects switch8 to router1
...
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth3
...
Local IP address: 192.168.1.1/24

But, for sure, changing the MAC address is out of question. If Linux cannot work with the current environment, then it will be used only for low level servers. We do not have this kind of problem with Solaris, AIX etc.

But, on the other hand, if is a mistake made by me, then I have to rectify it and test again.

David Becker · #15 30th May 2011, 03:45 AM

Quote:

Originally Posted by afberendsen

In fact, all the details about the environment are described.

No they're not.

Quote:

Originally Posted by afberendsen

But, for sure, changing the MAC address is out of question. If Linux cannot work with the current environment, then it will be used only for low level servers. We do not have this kind of problem with Solaris, AIX etc.

I know. Machines, just like employees, are replaceable.

David