These links are all made using Virtual Port Channels, which are cross-chassis aggregated links - these are abbreviated "vPC" in Cisco parlance; some other vendors call this MLAG (Multi-chassis Link AGgregation).
vPC offers benefits over Spanning Tree in that it doesn't block any links and avoids the use of Spanning Tree as a redundancy protocol, leaving it just to prevent loops (which is what it does best).
Design
Our existing data centre network uses Extreme Summit X450s and X460s as ToR units, connected back by two links: one to each upstream switch, with one blocked by Spanning Tree. The basic equivalent replacement (in absence of anyone requesting anything more fancy) is a Nexus 2248TP (48x 100/1000 copper switch) with four FEX fabric links: 2 to each EoR 56128P.This requires two ports on each EoR unit to be aggregated as part of a 4 port vPC group across the two chassis. Once that link is set up, it will be configured as a FEX fabric link to attach the ToR FEX unit.
Configuring vPC peering
To configure vPC, the first thing to do is turn it on as a feature, along with LACP, which will be needed to set up the peer link:n5k-bottom(config)# feature vpc
n5k-bottom(config)# feature lacp
- the peer link carries the bulk of the traffic (keeping the MAC tables in step and forwarding traffic between the peer switches, in the event of some downstream connectivity failing - we're using a pair of diversely-routed 40GE links for this
- the keepalive link is used to exchange status and keepalive messages and just determine if the peer switch is still there and operating; not much traffic is exchanged across this link and it isn't critical, as long as the other link does not fail - we're using a single 1GE link for this
vPC keepalive link
The keepalive link requires an IP address for the two switches to communicate which defaults to the management interface mgmt0 (in the management VRF). This interface is a copper connection on the rear of the 56128P and we don't typically run copper between racks, so we're going to use a fibre link on the front panel.As this is a single physical link and there is no reason to extend it elsewhere, we'll configure it a non-switchport interface with an IP address directly in the default VRF but using IP addresses we don't use on the University network:
n5k-bottom(config)# int e1/48
n5k-bottom(config-if)# desc vpc-keepalive
n5k-bottom(config-if)# no switchport
n5k-bottom(config-if)# ip address 192.168.100.1/30
... and then 192.168.100.2/30 on the peer switch.
vPC domain
Once the keepalive link is set up, the vPC peering can be set up by creating a vpc domain. A switch can participate in only a single vPC domain and it must agree on both peers.n5k-bottom(config)# vpc domain 1
n5k-bottom(config-vpc-domain)# peer-keepalive destination 192.168.100.2 vrf default source 192.168.100.1
n5k-bottom(config-vpc-domain)# system-priority 8192
n5k-bottom(config-vpc-domain)# role priority 8192
The source IP address must be specified when the default VRF is used. The two priorities are as follows:
- the system-priority MUST MATCH on both switches and is used as the LACP system priority for a link; it should be lowered (= higher priority, as per LACP/STP) on the 'upstream' end of a link - I've used 8192
- the role priority (note there is no hyphen in this option name) SHOULD DIFFER on each switch and should be lowered (= higher priority) on the switch you wish to be the primary/active one in the event of a failure of the peer link - I use 8192 on the primary and 16384 on the secondary
vPC peer link
The peer link is a special type of switched link that must itself be a port channel: the interfaces to be used for it must first be grouped and then that group assigned as a peer link. A link cannot be configured as a peer link until the vpc domain has been created, so this step must be done last:n5k-bottom(config)# int e1/49-50
n5k-bottom(config-if-range)# channel-group 1 mode active
n5k-bottom(config)# int e1/49
n5k-bottom(config-if)# desc vpc-peer/1
n5k-bottom(config)# int e1/50
n5k-bottom(config-if)# desc vpc-peer/2
n5k-bottom(config)# int po1
n5k-bottom(config-if)# desc vpc-peerlink
n5k-bottom(config-if)# switchport mode trunk
n5k-bottom(config-if)# switchport trunk allowed vlan remove 1
n5k-bottom(config-if)# vpc peer-link
n5k-bottom(config-if)# switchport mode trunk
n5k-bottom(config-if)# switchport trunk allowed vlan remove 1
n5k-bottom(config-if)# vpc peer-link
The VLANs to be supported across vPC must be present on both switches and added to the peer link - in the event of a link somewhere failing, it is possible traffic will need to pass across the peer link. Any VLANs which are not to be used across vPC should be removed explicitly - we never use the default VLAN (= 1) so I've removed that; if any uplink VLANs are present, they probably should also be removed, but it's probably best to keep the list generally open to avoid mistakes caused by missing individual VLANs out (which isn't what we normally do with our trunks, as I don't like VLANs going places I didn't explicitly want).
Adjacency formed
n5k-bottom# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 0
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled (timeout = 240 seconds)
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po1 up -
We're now ready to set up ports in a vPC.
FEXs
FEXs are controlled entirely from their parent switch (in our case, the EoR 56128Ps) and have no local configuration, including their firmware (which will be upgraded as required). Their identity and configuration is governed by which ports they are connected to, allowing a ToR unit to be replaced or swapped about and all setup will automatically be applied.Before FEXs can be used, the feature must be enabled:
n5k-bottom(config)# feature fex
When a FEX is attached to a parent switch, it will be discovered but not action will be taken with it (including updating its firmware) until it is brought online:
n5k-bottom# show fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
--- -------- Discovered N2K-C2248TP-E-1GE FOX1848GCMB
Configuring and associating a FEX via vPC
To configure a vPC group of ports to be used by FEXs across switches into a vPC group, they must first be put into a local [static] port channel group; note that LACP is not supported. The port channel interface can then be assigned a vPC number and set associated with a FEX and then the port channel interface itself assigned a vPC number to match it against a group on the peer switch:
n5k-bottom(config-if)# int e1/1-2
n5k-bottom(config-if-range)# desc b1
n5k-bottom(config-if)# int e1/1
n5k-bottom(config-if-range)# desc b1/1
n5k-bottom(config-if)# int e1/2
n5k-bottom(config-if-range)# desc b1/2
n5k-bottom(config-if)# desc b1
n5k-bottom(config-if)# fex associate 101
n5k-bottom(config-if)# vpc 101
n5k-bottom(config)# fex 101
n5k-bottom(config-fex)# desc b1
n5k-bottom(config-fex)# desc b1
FEXs are numbered 100-199 - for consistency and the avoidance of mistakes, it is common to use the same local port channel and vPC numbers as the FEX is assigned. The FEX number controls the ethernet interface numbers - for example, FEX 101 with 48 ports will have interfaces on the controlling switch called Ethernet101/1/1-48.
Once a FEX has been associated, it will download an updated image, if required:
n5k-bottom# show fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101 b1 Image Download N2K-C2248TP-E-1GE FOX1848GCMB
When that's complete, it will reboot and come online properly, making its interfaces available for configuration:
n5k-top# show fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101 b1 Online N2K-C2248TP-E-1GE FOX1848GCMB
n5k-top# show int status fex 101
--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Eth101/1/1 -- notconnec 1 auto auto --
Eth101/1/2 -- notconnec 1 auto auto --
Eth101/1/3 -- notconnec 1 auto auto --
...
