Wednesday, 18 February 2015

FEXs (Fabric Extenders) and vPCs (Virtual Port Channels)

The new data centre design makes extensive uses of Nexus 2k-series FEXs (Fabric EXtenders) as ToR (Top of Rack) units connected back to pairs of Nexus 56128Ps as the EoR (End of Row) units.

These links are all made using Virtual Port Channels, which are cross-chassis aggregated links - these are abbreviated "vPC" in Cisco parlance; some other vendors call this MLAG (Multi-chassis Link AGgregation).

vPC offers benefits over Spanning Tree in that it doesn't block any links and avoids the use of Spanning Tree as a redundancy protocol, leaving it just to prevent loops (which is what it does best).


Our existing data centre network uses Extreme Summit X450s and X460s as ToR units, connected back by two links: one to each upstream switch, with one blocked by Spanning Tree.  The basic equivalent replacement (in absence of anyone requesting anything more fancy) is a Nexus 2248TP (48x 100/1000 copper switch) with four FEX fabric links: 2 to each EoR 56128P.

This requires two ports on each EoR unit to be aggregated as part of a 4 port vPC group across the two chassis.  Once that link is set up, it will be configured as a FEX fabric link to attach the ToR FEX unit.

Configuring vPC peering

To configure vPC, the first thing to do is turn it on as a feature, along with LACP, which will be needed to set up the peer link:

n5k-bottom(config)# feature vpc
n5k-bottom(config)# feature lacp

vPC requires the pair of switches involved are linked together with two types of links:
  • the peer link carries the bulk of the traffic (keeping the MAC tables in step and forwarding traffic between the peer switches, in the event of some downstream connectivity failing - we're using a pair of diversely-routed 40GE links for this
  • the keepalive link is used to exchange status and keepalive messages and just determine if the peer switch is still there and operating; not much traffic is exchanged across this link and it isn't critical, as long as the other link does not fail - we're using a single 1GE link for this

vPC keepalive link

The keepalive link requires an IP address for the two switches to communicate which defaults to the management interface mgmt0 (in the management VRF).  This interface is a copper connection on the rear of the 56128P and we don't typically run copper between racks, so we're going to use a fibre link on the front panel.

As this is a single physical link and there is no reason to extend it elsewhere, we'll configure it a non-switchport interface with an IP address directly in the default VRF but using IP addresses we don't use on the University network:

n5k-bottom(config)# int e1/48
n5k-bottom(config-if)# desc vpc-keepalive
n5k-bottom(config-if)# no switchport
n5k-bottom(config-if)# ip address

... and then on the peer switch.

vPC domain

Once the keepalive link is set up, the vPC peering can be set up by creating a vpc domain.  A switch can participate in only a single vPC domain and it must agree on both peers.

n5k-bottom(config)# vpc domain 1
n5k-bottom(config-vpc-domain)# peer-keepalive destination vrf default source
n5k-bottom(config-vpc-domain)# system-priority 8192
n5k-bottom(config-vpc-domain)# role priority 8192

The source IP address must be specified when the default VRF is used.  The two priorities are as follows:
  • the system-priority MUST MATCH on both switches and is used as the LACP system priority for a link; it should be lowered (= higher priority, as per LACP/STP) on the 'upstream' end of a link - I've used 8192
  • the role priority (note there is no hyphen in this option name) SHOULD DIFFER on each switch and should be lowered (= higher priority) on the switch you wish to be the primary/active one in the event of a failure of the peer link - I use 8192 on the primary and 16384 on the secondary

vPC peer link

The peer link is a special type of switched link that must itself be a port channel: the interfaces to be used for it must first be grouped and then that group assigned as a peer link.  A link cannot be configured as a peer link until the vpc domain has been created, so this step must be done last:

n5k-bottom(config)# int e1/49-50
n5k-bottom(config-if-range)# channel-group 1 mode active

n5k-bottom(config)# int e1/49
n5k-bottom(config-if)# desc vpc-peer/1

n5k-bottom(config)# int e1/50
n5k-bottom(config-if)# desc vpc-peer/2

n5k-bottom(config)# int po1
n5k-bottom(config-if)# desc vpc-peerlink
n5k-bottom(config-if)# switchport mode trunk
n5k-bottom(config-if)# switchport trunk allowed vlan remove 1
n5k-bottom(config-if)# vpc peer-link

The VLANs to be supported across vPC must be present on both switches and added to the peer link - in the event of a link somewhere failing, it is possible traffic will need to pass across the peer link.  Any VLANs which are not to be used across vPC should be removed explicitly - we never use the default VLAN (= 1) so I've removed that; if any uplink VLANs are present, they probably should also be removed, but it's probably best to keep the list generally open to avoid mistakes caused by missing individual VLANs out (which isn't what we normally do with our trunks, as I don't like VLANs going places I didn't explicitly want).

Adjacency formed

Once this is done, the peering should be formed:

n5k-bottom# show vpc
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success
Per-vlan consistency status       : success
Type-2 consistency status         : success
vPC role                          : primary
Number of vPCs configured         : 0
Peer Gateway                      : Disabled
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Enabled (timeout = 240 seconds)

vPC Peer-link status
id   Port   Status Active vlans
--   ----   ------ --------------------------------------------------
1    Po1    up     -

We're now ready to set up ports in a vPC.


FEXs are controlled entirely from their parent switch (in our case, the EoR 56128Ps) and have no local configuration, including their firmware (which will be upgraded as required).  Their identity and configuration is governed by which ports they are connected to, allowing a ToR unit to be replaced or swapped about and all setup will automatically be applied.

Before FEXs can be used, the feature must be enabled:

n5k-bottom(config)# feature fex

When a FEX is attached to a parent switch, it will be discovered but not action will be taken with it (including updating its firmware) until it is brought online:

n5k-bottom# show fex
  FEX         FEX           FEX              FEX              Fex
Number    Description      State            Model            Serial
---       --------            Discovered   N2K-C2248TP-E-1GE   FOX1848GCMB

Configuring and associating a FEX via vPC

To configure a vPC group of ports to be used by FEXs across switches into a vPC group, they must first be put into a local [static] port channel group; note that LACP is not supported.  The port channel interface can then be assigned a vPC number and set associated with a FEX and then the port channel interface itself assigned a vPC number to match it against a group on the peer switch:

n5k-bottom(config-if)# int e1/1-2
n5k-bottom(config-if-range)# desc b1
n5k-bottom(config-if-range)# channel-group 101

n5k-bottom(config-if)# int e1/1
n5k-bottom(config-if-range)# desc b1/1

n5k-bottom(config-if)# int e1/2
n5k-bottom(config-if-range)# desc b1/2

n5k-bottom(config)# int po101
n5k-bottom(config-if)# desc b1
n5k-bottom(config-if)# switchport mode fex-fabric
n5k-bottom(config-if)# fex associate 101
n5k-bottom(config-if)# vpc 101

n5k-bottom(config)# fex 101
n5k-bottom(config-fex)# desc b1

FEXs are numbered 100-199 - for consistency and the avoidance of mistakes, it is common to use the same local port channel and vPC numbers as the FEX is assigned.  The FEX number controls the ethernet interface numbers - for example, FEX 101 with 48 ports will have interfaces on the controlling switch called Ethernet101/1/1-48.

Once a FEX has been associated, it will download an updated image, if required:

n5k-bottom# show fex
  FEX         FEX           FEX              FEX              Fex
Number    Description      State            Model            Serial
101        b1             Image Download   N2K-C2248TP-E-1GE   FOX1848GCMB

When that's complete, it will reboot and come online properly, making its interfaces available for configuration:

n5k-top# show fex
  FEX         FEX           FEX              FEX              Fex
Number    Description      State            Model            Serial
101        b1                     Online   N2K-C2248TP-E-1GE   FOX1848GCMB

n5k-top# show int status fex 101

Port          Name               Status    Vlan      Duplex  Speed   Type
Eth101/1/1    --                 notconnec 1         auto    auto    --
Eth101/1/2    --                 notconnec 1         auto    auto    --
Eth101/1/3    --                 notconnec 1         auto    auto    --

No comments:

Post a Comment