Wednesday, July 15, 2009

Monitoring 6500 switch fabric, trying to understand switching mode

Hi all, last week I had an informal meeting (also called a beer) with Nicola Modena CCIE #19119 (thanks buddy)!

During the long and interesting conversation, he asked me what was the utilization of my 6500s...
I answered someting like "Cpu is under 5%", and he said "well, but traffic isn't process switched, what's about the fabric?"

Good point Nic!

So today I've read the "Configuring a Supervisor Engine 720" section of the Catalyst 6500 Release 12.2SXF and Rebuilds Software Configuration Guide...

...and tryed to understand the various show fabric outputs.

Here is my show module and show fabric:
6509# sh module 
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 24 CEF720 24 port 1000mb SFP WS-X6724-SFP Serials omitted
2 10 WiSM WLAN Service Module WS-SVC-WISM-1-K9 S..
3 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX S..
5 2 Supervisor Engine 720 (Active) WS-SUP720-3B S..
6 2 Supervisor Engine 720 (Hot) WS-SUP720-3B S..

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 001b.d4ec.7860 to 001b.d4ec.7877 2.6 12.2(14r)S5 12.2(18)SXF9 Ok
2 001c.5843.7cb0 to 001c.5843.7cbf 2.0 12.2(14r)S5 12.2(18)SXF9 Ok
3 001c.587b.20a0 to 001c.587b.20cf 2.6 12.2(14r)S5 12.2(18)SXF9 Ok
5 0019.e7d3.a2ac to 0019.e7d3.a2af 5.4 8.4(2) 12.2(18)SXF9 Ok
6 001a.2f3b.f80c to 001a.2f3b.f80f 5.4 8.4(2) 12.2(18)SXF9 Ok

Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
1 Centralized Forwarding Card WS-F6700-CFC S.. 3.1 Ok
2 Centralized Forwarding Card WS-SVC-WISM-1-K9-D S.. 2.0 Ok
3 Centralized Forwarding Card WS-F6700-CFC S.. 3.1 Ok
5 Policy Feature Card 3 WS-F6K-PFC3B S.. 2.3 Ok
5 MSFC3 Daughterboard WS-SUP720 S.. 3.0 Ok
6 Policy Feature Card 3 WS-F6K-PFC3B S.. 2.3 Ok
6 MSFC3 Daughterboard WS-SUP720 S.. 3.0 Ok

Mod Online Diag Status
---- -------------------
1 Pass
2 Pass
3 Pass
5 Pass
6 Pass
6509#
6509#sh fabric
show fabric active:
Active fabric card in slot 5
Backup fabric card in slot 6

show fabric mode:
Global switching mode is Compact
dCEF mode is not enforced for system to operate
Fabric module is not required for system to operate
Modules are allowed to operate in bus mode
Truncated mode is allowed, due to presence of CEF720, Standby supervisor module

Module Slot Switching Mode
1 Crossbar
2 Crossbar
3 Crossbar
5 dCEF
6 Crossbar

show fabric congestion management
:
Fabric clear-block is off.

show fabric status all:
slot channel speed module fabric
status status
1 0 20G OK OK
2 1 N/A OK OK
3 0 20G OK OK
3 1 20G OK OK
5 0 20G OK OK
6 0 20G OK OK

show fabric utilization all:
slot channel speed Ingress % Egress %
1 0 20G 1 1
2 1 8G 0 0
3 0 20G 0 0
3 1 20G 0 1
5 0 20G 0 1
6 0 20G 0 0

show fabric errors all:
Module errors:
slot channel crc hbeat sync DDR sync
1 0 0 0 0 0
2 1 0 0 0 0
3 0 0 0 0 0
3 1 0 0 0 0
5 0 0 0 0 0
6 0 0 0 0 0

Fabric errors:
slot channel sync buffer timeout
1 0 0 0 0
2 1 0 0 0
3 0 0 0 0
3 1 0 0 0
5 0 0 0 0
6 0 0 0 0



This is our core L3, it has two Sup720-3B installed and configured as hot-standby, two switching modules (24 sfp and 48 Rj45) and a (double) Wism installed.
As you can see from show fabric utilization, we're using 1% of the 20Gbps bus capacity for modules 1,3,5....

For a better understanding of the 6500 architecture, read the interesting white paper: Cisco Catalyst 6500 Architecture White Paper
Another interesting resource is: Notes on Cisco Catalyst 6500 Architecture

If I don't have misunderstood, here the "global switching mode" is "Compact", due to the presence of CEF720 switching modules that have a "Crossbar" connection and due to SUP720-3B, so according with the whitepaper: "In this mode of operation, the switch can achieve centralized performance of up to 30Mpps independent of packet size.".

Not bad! We are under-utilizing a lot this platform!

Let's take a look to a distribution L3 switch:

6506#sh module
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 24 CEF720 24 port 1000mb SFP WS-X6724-SFP Serials omitted
2 24 CEF720 24 port 1000mb SFP WS-X6724-SFP S...
3 48 48 port 10/100/1000mb EtherModule WS-X6148-GE-TX S...
5 2 Supervisor Engine 720 (Active) WS-SUP720-3B S...

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 0021.a0b4.00b0 to 0021.a0b4.00c7 3.3 12.2(18r)S1 12.2(33)SXH4 Ok
2 0021.a07e.fbd8 to 0021.a07e.fbef 3.3 12.2(18r)S1 12.2(33)SXH4 Ok
3 0021.a08c.aa10 to 0021.a08c.aa3f 7.2 7.2(1) 8.7(0.22)BUB Ok
5 0021.1bff.09dc to 0021.1bff.09df 5.7 8.5(2) 12.2(33)SXH4 Ok

Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
1 Centralized Forwarding Card WS-F6700-CFC S.. 4.1 Ok
2 Centralized Forwarding Card WS-F6700-CFC S.. 4.1 Ok
5 Policy Feature Card 3 WS-F6K-PFC3B S.. 2.4 Ok
5 MSFC3 Daughterboard WS-SUP720 S.. 3.2 Ok

Mod Online Diag Status
---- -------------------
1 Pass
2 Pass
3 Pass
5 Pass

6506#sh fabric
show fabric active:
Active fabric card in slot 5
No backup fabric card in the system

show fabric mode:
Global switching mode is Truncated
dCEF mode is not enforced for system to operate
Fabric module is not required for system to operate
Modules are allowed to operate in bus mode
Truncated mode is allowed, due to presence of CEF720 module

Module Slot Switching Mode
1 Crossbar
2 Crossbar
3 Bus
5 Bus

show fabric congestion management:
Fabric clear-block is off (operational).

show fabric status all:
slot channel speed module fabric hotStandby Standby Standby
status status support module fabric
1 0 20G OK OK Y(not-hot)
2 0 20G OK OK Y(not-hot)
5 0 20G OK OK Y(not-hot)

show fabric utilization all:
slot channel speed Ingress % Egress %
1 0 20G 0 0
2 0 20G 0 0
5 0 20G 0 0

show fabric errors all:
Module errors:
slot channel crc hbeat sync DDR sync
1 0 0 0 0 0
2 0 0 0 0 0
5 0 0 0 0 0

Fabric errors:
slot channel sync buffer timeout
1 0 0 0 0
2 0 0 0 0
5 0 0 0 0


with a quick look to "Ethernet and Gigabit Ethernet Switching modules" I realized that module 3 in this 6506E, 48 port 10/100/1000mb EtherModule (WS-X6148-GE-TX) has only a "BUS" connection, it's a so called "classic line card", that's why the switching mode is "Truncated".

Let's take a look to wikipedia: http://en.wikipedia.org/wiki/Catalyst_6500 simple, nice photo..
6509 from wikipedia

No comments: