Mincebert's blog

Monday, 23 February 2015

Problems with NX-OS configuration synchronization (switch-profiles)

Now that configuration synchronization (through a switch-profile in configure sync mode) has been set up, the relevant parts of the configuration across both upstream Nexus 5ks should be kept in step (in particular, the ports on the FEXs).

However, during testing, a number of issues were encountered, making this feature look very useful but need a lot of care and attention, as if they arise it is often necessary to do some fiddling about to recover things. However, so far none of these processes would appear to result in a service outage.

Running configuration getting out step

Sometimes, the running configuration can get out of step with the local configuration and synchronised switch-profile configuration - everything looks correct when you look at the files, but you get an error when you try to commit changes.

To detect this situation, compare the relevant portions of show running-config ... and show running-config switch-profile and see if they reconcile. If they don't, you can compare these with what the internal configuration of the switch says with two internal commands:

n5k-top# show run int e1/3
...
interface Ethernet1/3

n5k-top# show system internal csm info global-db cmd-tbl | sec Ethernet1/3
parent_seq_no= 0, seq_no= 34, clone_seq_no= 0, cmd= 'interface Ethernet1/3'
parent_seq_no= 34, seq_no= 35, clone_seq_no= 0, cmd= 'description b2'

n5k-top# show system internal csm info switch-profile cfgd-db cmd-tbl | sec Ethernet1/3
parent_seq_no= 0, seq_no= 146, clone_seq_no= 0, cmd= 'interface Ethernet1/3'
parent_seq_no= 146, seq_no= 150, clone_seq_no= 0, cmd= 'channel-group 102'
parent_seq_no= 146, seq_no= 147, clone_seq_no= 0, cmd= 'description b2'
parent_seq_no= 146, seq_no= 149, clone_seq_no= 0, cmd= 'fex associate 102'
parent_seq_no= 146, seq_no= 148, clone_seq_no= 0, cmd= 'switchport mode fex-fabric'

The first shows what is in the running configuration of Ethernet1/3, the second shows the local configuration and the third the one from the switch-profile (here for a block pertaining to Ethernet1/3). Clearly these don't tally up.

Somehow things have gone awry, but it's easy to fix by resyncing the database which, in this case, caused the local configuration to clear itself (but sometimes it can have other effects):

n5k-top# conf sync

Enter configuration commands, one per line. End with CNTL/Z.

n5k-top(config-sync)# resync-database

Re-synchronization of switch-profile db takes a few minutes...

Re-synchronize switch-profile db completed successfully.

n5k-top(config-sync)# end

n5k-top# show system internal csm info global-db cmd-tbl | sec Ethernet1/3

Over the next few seconds, things will sync up and sort themselves out again and you'll have your interface configuration back:

n5k-top# show run int e1/3

...

interface Ethernet1/3

description b2

switchport mode fex-fabric

fex associate 102

channel-group 102

I found an explanation of this here: http://nexp.com.ua/technologies/nx-os/troubleshooting-nx-os-config-sync/ - that's a good place to start before trying the second (more drastic) method.

Stub configuration stanza removal

If it is impossible to remove stub configuration stanzas (the opening line of a block, starting a subcontext in the configuration file - e.g. interface ... or fex ...), from either configuration, if it has been entered by mistake.

For example, if you add a FEX to the switch profile:

n5k-bottom# conf sync
n5k-bottom(config-sync)# switch-profile wcdc-b
Switch-Profile started, Profile ID is 1
n5k-bottom(config-sync-sp)# fex 103
n5k-bottom(config-sync-sp-fex)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
Please avoid other configuration changes during this time.
Commit Successful

Then you mistakenly go to configure the FEX in local (configure terminal) mode but realise your mistake, even enter no subcommands, you cannot remove the fex stanza because it would clash (= be mutually exclusive, in the parlance of configuration synchronization) with the declaration in the switch profile:

n5k-bottom# conf t
n5k-bottom(config)# fex 103
n5k-bottom(config-fex)# no fex 103
Error: Command is not mutually exclusive

You also cannot remove the declaration in the switch profile as that would clash with the one in the local configuration!

n5k-bottom(config)# conf sync
n5k-bottom(config-sync)# switch-profile wcdc-b
Switch-Profile started, Profile ID is 1
n5k-bottom(config-sync-sp)# no fex 103
n5k-bottom(config-sync-sp)# commit

Failed: Verify Failed

n5k-bottom(config-sync-sp)# show switch-profile status

...

Local information:

----------------

Status: Verify Failure

Error(s):

Following commands failed mutual-exclusion checks:

no fex 103

You can see these two commands with this handy internal command:

n5k-bottom# show system internal csm info global-db cmd-tbl | sec "fex 103"

parent_seq_no= 0, seq_no= 238, clone_seq_no= 0, cmd= 'fex 103'

n5k-bottom# show system internal csm info switch-profile cfgd-db cmd-tbl | sec "fex 103"

parent_seq_no= 0, seq_no= 237, clone_seq_no= 0, cmd= 'fex 103'

The only solution to this problem that I'm aware of is to, step-by-step on both switches (except for the first command, which will do both at the same time):

Use the no switch-profile ... profile-only all in configure sync mode to remove the profile. This will break the synchronisation and move all the commands in the profile to the local configuration on both switches.
Remove the unwanted parts in configure terminal mode.
Create a new switch-profile.
Use import running-config to read in the current configuration into the profile.
commit the new profile, which will move the commands from the local configuration into the switch-profile.
Re-establish the synchronisation peer.

This appears to be non-destructive but is a little worrying!

With an example - first break the synchronisation and copy the commands in the switch-profile to the local configuration:

n5k-bottom# conf sync

Enter configuration commands, one per line. End with CNTL/Z.

n5k-bottom(config-sync)# no switch-profile wcdc-b profile-only all

WARNING: Deleting switch-profile will not remove all commands configured under switch-profile. This will only remove the switch profile. Are you sure you want to delete the switch-profile from the system ?

Are you sure? (y/n) [n] y

Verification successful...

Proceeding to delete switch-profile. This might take a while depending on amount of configuration under a switch-profile.

Please avoid other configuration changes during this time.

Delete Successful

At this point, the commands you don't want will still be there, but only in the local configuration, where they can be removed directly on both switches, without issue:

n5k-bottom# conf t

Enter configuration commands, one per line. End with CNTL/Z.

n5k-bottom(config)# no fex 103

Now re-create the switch-profile and import the (now clean) running configuration, then commit it - I've added some show system ... commands to show the commands moving from the local configuration to the switch-profile when the commit is issued:

n5k-bottom# conf sync

Enter configuration commands, one per line. End with CNTL/Z.

n5k-bottom(config-sync)# switch-profile wcdc-b

Switch-Profile started, Profile ID is 1

n5k-bottom(config-sync-sp)# import running-config

n5k-bottom(config-sync-sp-import)# show system internal csm info global-db cmd-tbl | sec "fex 103"

parent_seq_no= 0, seq_no= 8, clone_seq_no= 0, cmd= 'fex 103'

n5k-bottom(config-sync-sp-import)# commit

Verification successful...

Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.

Please avoid other configuration changes during this time.

Commit Successful

n5k-bottom(config-sync)# show system internal csm info global-db cmd-tbl | sec "fex 103"

n5k-bottom(config-sync)# show system internal csm info switch-profile cfgd-db cmd-tbl | sec "fex 103"

parent_seq_no= 0, seq_no= 8, clone_seq_no= 0, cmd= 'fex 103'

Once this has been done on both switches, the synchronisation peering can be re-established and hopefully everything will be OK:

n5k-bottom# conf sync

Enter configuration commands, one per line. End with CNTL/Z.

n5k-bottom(config-sync)# switch-profile wcdc-b

Switch-Profile started, Profile ID is 1

n5k-bottom(config-sync-sp)# sync-peers dest 192.168.200.2

n5k-bottom(config-sync-sp)# end

Wait a few seconds:

n5k-bottom# show switch-profile status

...

Local information:

----------------

Status: Commit Success

Error(s):

Peer information:

----------------

IP-address: 192.168.200.2

Sync-status: In sync

Status: Commit Success

Error(s):

All rather messy! What NX-OS needs is some sort of "delete fex ..." command to remove the block from the configuration you're editing, but not try and remove the block from the running-config itself.

Config-revision getting out of step

I'm not sure how I got in this state, but I was testing FEX preprovisioning and what happened when the configured model mismatches the connected one, then fixing this. This messed things up when I tried to change the model - the command verified OK but failed on the peer when I tried committing, claiming there was a mismatch.

After trying to undo this, I got in a weird situation where the peer had lost the configuration of some port channel interfaces in the switch profile. On closer inspection, in show switch-profile the peer had a Config-revision listed one less than the switch I committed the change on.

I couldn't find a command to force the two switches to synchronise (including making a change and trying to commit it). Breaking the peering (by removing the sync-peers destination ... entry and re-adding it on one switch) didn't fix it, with the switch grumbling the commands mismatched:

n5k-bottom# show switch-profile status

switch-profile : wcdc-b
----------------------------------------------------------

Start-time: 221438 usecs after Mon Feb 12 02:59:18 2001
End-time: 885937 usecs after Mon Feb 12 02:59:19 2001

Profile-Revision: 11
Session-type: Import-Verify
Session-subtype: -
Peer-triggered: No
Profile-status: Verify Failed

Local information:
----------------
Status: Verify Success
Error(s):

Peer information:
----------------
IP-address: 192.168.200.2
Sync-status: Not yet merged
Merge Flags: pending_merge:1 rcv_merge:1 pending_validate:0
Status: Verify Failure
Error(s):
Validation Failed: Config validation failed as found changes on both sides. rcvd
_rev: 0, expected_rev: 0
interface Ethernet1/3
switchport mode fex-fabric
interface Ethernet1/3
fex associate 102
interface Ethernet1/3
channel-group 102
interface Ethernet1/4
switchport mode fex-fabric
interface Ethernet1/4
fex associate 102
interface Ethernet1/4
channel-group 102

Indeed, those commands listed were the ones missing on the peer but present on the one I was making the change on.

Fixing this was fairly straightforward but tedious - the missing commands just needed adding back into the switch profile and committing:

n5k-top# conf sync
n5k-top(config-sync)# switch-profile wcdc-b
Switch-Profile started, Profile ID is 1
n5k-top(config-sync-sp)# int e1/3
n5k-top(config-sync-sp-if)# switchport mode fex-fabric
n5k-top(config-sync-sp-if)# fex associate 102
n5k-top(config-sync-sp-if)# channel-group 102
n5k-top(config-sync-sp-if)# int e1/4
n5k-top(config-sync-sp-if)# switchport mode fex-fabric
n5k-top(config-sync-sp-if)# fex associate 102
n5k-top(config-sync-sp-if)# channel-group 102
n5k-top(config-sync-sp-if)# verify
Verification Successful
n5k-top(config-sync-sp)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
Please avoid other configuration changes during this time.
Commit Successful

A few seconds after this was done, the two switches automatically recovered and synced up:

n5k-top# show switch-profile status
...
Peer information:
----------------
IP-address: 192.168.200.1
Sync-status: In sync
Status: Commit Success
Error(s):

Invented/implicit and non-canonical commands

You can get in a muddle if you rely on NX-OS to insert implicit commands, or use the non-canonical forms of some commands. What do I mean by this? Well - consider the following configuration for a vPC link port channel interface:

rk-wcdc-b-b(config)# interface Port-Channel1
rk-wcdc-b-b(config-if)# switchport mode trunk
rk-wcdc-b-b(config-if)# switchport trunk allowed vlan remove 1
rk-wcdc-b-b(config-if)# vpc peer-link

Entering the last of these commands will display the following warning:

Please note that spanning tree port type is changed to "network" port type on vPC peer-link.
This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP Bridge Assurance

(which is enabled by default) is not disabled.

When we look at the configuration, we can see two changes from what we entered (ignoring the speed command, which seems not to matter):

rk-wcdc-b-b# show run int po12
...
interface port-channel12
description vpc-peer
switchport mode trunk
switchport trunk allowed vlan 2-4094
spanning-tree port type network
speed 40000
vpc peer-link

The allowed vlan list just shows what's included (not what we removed), thus storing a different command from what we entered. Also, as per the warning above, the spanning-tree port type network command has been automatically inserted.

This is all good stuff but, when you try to create a switch-profile and import the running-config and verify it, it fails on those two lines:

rk-wcdc-b-b# conf sync

rk-wcdc-b-b(config-sync)# switch-profile wcdc-b

Switch-Profile started, Profile ID is 1

rk-wcdc-b-b(config-sync-sp)# import running-config

rk-wcdc-b-b(config-sync-sp-import)# verify

Failed: Verify Failed

rk-wcdc-b-b(config-sync-sp-import)# show switch-profile status

...

Local information:

----------------

Status: Verify Failure

Error(s):

Following commands are not configured under config-terminal mode and hence cannot be imported:

interface port-channel12

switchport trunk allowed vlan 2-4094

interface port-channel12

spanning-tree port type network

This is because these commands didn't match what was exactly entered, but something that was either "invented" (inserted automatically, as per the last line) or different (as per the first line).

In this case, the issue can be fixed with the conf sync / resync-database command, but I've seen situations - usually after the switch-profile has been created and the pair of switches synced up - where things get messy and it's necessary to break the synchronisation to fix it.

It is best to try and avoid commands being implicitly inserted and enter them explicitly.

NX-OS configuration synchronisation (switch-profiles)

When using vPC with dual-attached FEXs, a large chunk of the configuration across the two parent Nexus 5k (or 7k) switches must be kept in step. For example:

VLANs must be created and deleted on both switches.
Fabric interfaces must have the same configuration with regards channel group assignments, FEX associations, etc.
Edge (host) ports must have the same switchport configuration - mode and access/trunk/native VLANs.

This can be done manually but is a bit tedious and error-prone to maintain. To help, NX-OS has a mechanism whereby parts of the configuration can be synchronised across the switches.

However, this facility is little confusing and can be problematic to set up - it's one of those things where the Cisco documentation makes sense after you understand the basics!

Key concepts

Each switch maintains a local configuration which contains things which are NOT synchronised across the switches. This is configured in the normal way, using configure terminal and has all the usual commands in it and will obviously hold things which differ between the switches.
In addition, there is a separate configuration which is synchronised between switches and holds the common elements. This is configured in a special mode, entered using configure sync in a block called a switch-profile.
The running configuration is the result of merging the two configurations. When using show running-config (including its options to look at specific parts of the configuration), the merged result is shown.
Changes to the switch profile are made on one of the switches, verifyed and commited as a single transaction (rather like a database) on both switches automatically; if they fail, things should rollback to how they were before they started.
Commands can be imported from the running configuration into the switch profile: this will remove them from the running configuration.
Synchronisation can only work across the out-of-band management VRF (which, on a Nexus 5k, limits you to the copper Management0 port on the rear of the unit); it cannot be done in-band.

Setting up synchronisation

There are three main steps to this:

Set up management interface communication and enable Cisco Fabric Services (CFS) over it
Set up a switch profile and import the running configuration into it
Establish synchronisation between the peers

You have to take care to do these in the correct order otherwise things can get in a muddle. Cisco describe this rather briefly on this page.

Set up management interface communication

Once the profile The first thing to do is allow communication between switches using the management interface and enable Cisco Fabric Services (CFS) over IPv4. CFS is used to communicate the synchronisation and only operates over the management VRF:

n5k-bottom# conf t

n5k-bottom(config)# int mgmt0

n5k-bottom(config-if)# ip addr 192.168.200.1/24

n5k-bottom(config-if)# exit

n5k-top(config)# cfs ipv4 distribute

n5k-top(config)# end

Set up the switch profile and import the running configuration

The switch profile is created in the special configure sync mode.

The profile must be given a name - this is used to identify the configuration which must be synchronised and must match across the synchronisation peers - I recommend using something to identify the area the switches and their FEXs will serve (e.g. the room and rack row).

Once this is done, the running configuration can be imported, verified and committed (the verification stage can be omitted, if required, but it's a good idea to check things first). This will move the synchronisable elements of the configuration from the running configuration into the switch profile.

n5k-bottom# conf sync

Enter configuration commands, one per line. End with CNTL/Z.

n5k-bottom(config-sync)# switch-profile wcdc-b

Switch-Profile started, Profile ID is 1

n5k-bottom(config-sync-sp)# import running-config

n5k-bottom(config-sync-sp-import)# verify

Verification Successful

n5k-bottom(config-sync-sp-import)# commit

Verification successful...

Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.

Please avoid other configuration changes during this time.

Commit Successful

All this must be done on BOTH switches independently.

Establish synchronisation between the peers

The synchronisation can now be enabled between the peers in the switch profile:

n5k-bottom# conf sync
Enter configuration commands, one per line. End with CNTL/Z.
n5k-bottom(config-sync)# switch-profile wcdc-b
Switch-Profile started, Profile ID is 1

n5k-bottom(config-sync-sp)# sync-peers destination 192.168.200.2

Once this is entered on both switches, they should find each other, exchange information and sync up, comparing the imported running configuration with each other and ensuring they agree (or, if they differ, they can be merged without conflict). This will typically take 10-20 seconds and can be checked with show switch-profile status; the Peer information / Status will change from Peer not reachable to Verify Success to Commit Success and then finally the Sync-status will say In Sync, when this is complete:

n5k-bottom# show switch-profile status

switch-profile : wcdc-b

----------------------------------------------------------

Start-time: 6070 usecs after Sun Feb 11 23:42:16 2001

End-time: 571814 usecs after Sun Feb 11 23:42:17 2001

Profile-Revision: 1

Session-type: Initial-Exchange

Session-subtype: Init-Exchange-All

Peer-triggered: Yes

Profile-status: Sync Success

Local information:

----------------

Status: Commit Success

Error(s):

Peer information:

----------------

IP-address: 192.168.200.1

Sync-status: In sync

Status: Commit Success

Error(s):

Once this has completed, and the two switches are in sync, it's ready for use.

Using synchronisation

From this point onwards, any changes to made across both switches should be made in configure sync mode, in the switch-profile then comited to take effect, rather than in configure terminal mode. For example, to set up a new FEX, you might do:

n5k-bottom# conf sync

Enter configuration commands, one per line. End with CNTL/Z.

n5k-bottom(config-sync)# switch-profile wcdc-b

Switch-Profile started, Profile ID is 1

n5k-bottom(config-sync-sp)# int e1/3-4

n5k-bottom(config-sync-sp-if-range)# desc b2

n5k-bottom(config-sync-sp-if-range)# channel-group 102

n5k-bottom(config-sync-sp-if-range)# exit

n5k-bottom(config-sync-sp)# int po102

n5k-bottom(config-sync-sp-if)# desc b2

n5k-bottom(config-sync-sp-if)# switchport mode fex-fabric

n5k-bottom(config-sync-sp-if)# fex associate 102

n5k-bottom(config-sync-sp-if)# vpc 102

n5k-bottom(config-sync-sp-if)# verify

Verification Successful

n5k-bottom(config-sync-sp)# commit

Verification successful...

Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.

Please avoid other configuration changes during this time.

Commit Successful

If something is accidentally configured in configure terminal mode, sometimes it can just be removed (with some no ... commands), but it may take some work if mutual exclusion errors are encountered - I'll cover this on a future entry.

Friday, 20 February 2015

FEX testing

Now I've got the FEXs up and running, there are various things I wanted to test and confirm so I can better understand how they work and, more importantly, how to find things that have gone wrong.

Port-VLAN inconsistency across upstream switches

When a FEX is dual-homed to a pair of upstream switches, each of those switches will have a configuration for the ports on the FEX. By default, these are NOT automatically kept in step (although something called configuration synchronization can do this, which I'll go over later as I've been looking at that too).

If the ports are configured identically on both switches, everything is hunky dory:

n5k-top(config)# int e101/1/3
n5k-top(config-if)# switchport access vlan 808

n5k-bottom(config)# int e101/1/3

n5k-bottom(config-if)# switchport access vlan 808

n5k-top# show int e101/1/3 status

--------------------------------------------------------------------------------

Port Name Status Vlan Duplex Speed Type

--------------------------------------------------------------------------------

Eth101/1/3 -- connected 808 full a-1000 --

However, if one of the switches has a different configuration, the port becomes inactive:

n5k-bottom(config)# int e101/1/3

n5k-bottom(config-if)# switchport access vlan 812

n5k-bottom# show int e101/1/3 status

--------------------------------------------------------------------------------

Port Name Status Vlan Duplex Speed Type

--------------------------------------------------------------------------------

Eth101/1/3 -- inactive 812 full auto --

This error can be explained with the "show int ... status err-vlans" command:

n5k-bottom# show int e101/1/3 status err-vlans

--------------------------------------------------------------------------------

Port Name Err-Vlans Status

--------------------------------------------------------------------------------

Eth101/1/3 -- 812 Vlan is not

configured on

remote vPC

interface

It doesn't show up in the verbose show int command output, which seems a bit of an omission.

Once corrected, the port immediately springs back into life.

Wednesday, 18 February 2015

FEXs (Fabric Extenders) and vPCs (Virtual Port Channels)

The new data centre design makes extensive uses of Nexus 2k-series FEXs (Fabric EXtenders) as ToR (Top of Rack) units connected back to pairs of Nexus 56128Ps as the EoR (End of Row) units.

These links are all made using Virtual Port Channels, which are cross-chassis aggregated links - these are abbreviated "vPC" in Cisco parlance; some other vendors call this MLAG (Multi-chassis Link AGgregation).

vPC offers benefits over Spanning Tree in that it doesn't block any links and avoids the use of Spanning Tree as a redundancy protocol, leaving it just to prevent loops (which is what it does best).

Design

Our existing data centre network uses Extreme Summit X450s and X460s as ToR units, connected back by two links: one to each upstream switch, with one blocked by Spanning Tree. The basic equivalent replacement (in absence of anyone requesting anything more fancy) is a Nexus 2248TP (48x 100/1000 copper switch) with four FEX fabric links: 2 to each EoR 56128P.

This requires two ports on each EoR unit to be aggregated as part of a 4 port vPC group across the two chassis. Once that link is set up, it will be configured as a FEX fabric link to attach the ToR FEX unit.

Configuring vPC peering

To configure vPC, the first thing to do is turn it on as a feature, along with LACP, which will be needed to set up the peer link:

n5k-bottom(config)# feature vpc
n5k-bottom(config)# feature lacp

vPC requires the pair of switches involved are linked together with two types of links:

the peer link carries the bulk of the traffic (keeping the MAC tables in step and forwarding traffic between the peer switches, in the event of some downstream connectivity failing - we're using a pair of diversely-routed 40GE links for this
the keepalive link is used to exchange status and keepalive messages and just determine if the peer switch is still there and operating; not much traffic is exchanged across this link and it isn't critical, as long as the other link does not fail - we're using a single 1GE link for this

vPC keepalive link

The keepalive link requires an IP address for the two switches to communicate which defaults to the management interface mgmt0 (in the management VRF). This interface is a copper connection on the rear of the 56128P and we don't typically run copper between racks, so we're going to use a fibre link on the front panel.

As this is a single physical link and there is no reason to extend it elsewhere, we'll configure it a non-switchport interface with an IP address directly in the default VRF but using IP addresses we don't use on the University network:

n5k-bottom(config)# int e1/48
n5k-bottom(config-if)# desc vpc-keepalive
n5k-bottom(config-if)# no switchport
n5k-bottom(config-if)# ip address 192.168.100.1/30

... and then 192.168.100.2/30 on the peer switch.

vPC domain

Once the keepalive link is set up, the vPC peering can be set up by creating a vpc domain. A switch can participate in only a single vPC domain and it must agree on both peers.

n5k-bottom(config)# vpc domain 1
n5k-bottom(config-vpc-domain)# peer-keepalive destination 192.168.100.2 vrf default source 192.168.100.1
n5k-bottom(config-vpc-domain)# system-priority 8192
n5k-bottom(config-vpc-domain)# role priority 8192

The source IP address must be specified when the default VRF is used. The two priorities are as follows:

the system-priority MUST MATCH on both switches and is used as the LACP system priority for a link; it should be lowered (= higher priority, as per LACP/STP) on the 'upstream' end of a link - I've used 8192
the role priority (note there is no hyphen in this option name) SHOULD DIFFER on each switch and should be lowered (= higher priority) on the switch you wish to be the primary/active one in the event of a failure of the peer link - I use 8192 on the primary and 16384 on the secondary

vPC peer link

The peer link is a special type of switched link that must itself be a port channel: the interfaces to be used for it must first be grouped and then that group assigned as a peer link. A link cannot be configured as a peer link until the vpc domain has been created, so this step must be done last:

n5k-bottom(config)# int e1/49-50
n5k-bottom(config-if-range)# channel-group 1 mode active

n5k-bottom(config)# int e1/49
n5k-bottom(config-if)# desc vpc-peer/1

n5k-bottom(config)# int e1/50
n5k-bottom(config-if)# desc vpc-peer/2

n5k-bottom(config)# int po1

n5k-bottom(config-if)# desc vpc-peerlink
n5k-bottom(config-if)# switchport mode trunk
n5k-bottom(config-if)# switchport trunk allowed vlan remove 1
n5k-bottom(config-if)# vpc peer-link

The VLANs to be supported across vPC must be present on both switches and added to the peer link - in the event of a link somewhere failing, it is possible traffic will need to pass across the peer link. Any VLANs which are not to be used across vPC should be removed explicitly - we never use the default VLAN (= 1) so I've removed that; if any uplink VLANs are present, they probably should also be removed, but it's probably best to keep the list generally open to avoid mistakes caused by missing individual VLANs out (which isn't what we normally do with our trunks, as I don't like VLANs going places I didn't explicitly want).

Adjacency formed

Once this is done, the peering should be formed:

n5k-bottom# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 0
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled (timeout = 240 seconds)

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po1 up -

We're now ready to set up ports in a vPC.

FEXs

FEXs are controlled entirely from their parent switch (in our case, the EoR 56128Ps) and have no local configuration, including their firmware (which will be upgraded as required). Their identity and configuration is governed by which ports they are connected to, allowing a ToR unit to be replaced or swapped about and all setup will automatically be applied.

Before FEXs can be used, the feature must be enabled:

n5k-bottom(config)# feature fex

When a FEX is attached to a parent switch, it will be discovered but not action will be taken with it (including updating its firmware) until it is brought online:

n5k-bottom# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

--- -------- Discovered N2K-C2248TP-E-1GE FOX1848GCMB

Configuring and associating a FEX via vPC

To configure a vPC group of ports to be used by FEXs across switches into a vPC group, they must first be put into a local [static] port channel group; note that LACP is not supported. The port channel interface can then be assigned a vPC number and set associated with a FEX and then the port channel interface itself assigned a vPC number to match it against a group on the peer switch:

n5k-bottom(config-if)# int e1/1-2

n5k-bottom(config-if-range)# desc b1

n5k-bottom(config-if-range)# channel-group 101

n5k-bottom(config-if)# int e1/1

n5k-bottom(config-if-range)# desc b1/1

n5k-bottom(config-if)# int e1/2

n5k-bottom(config-if-range)# desc b1/2

n5k-bottom(config)# int po101

n5k-bottom(config-if)# desc b1

n5k-bottom(config-if)# switchport mode fex-fabric

n5k-bottom(config-if)# fex associate 101

n5k-bottom(config-if)# vpc 101

n5k-bottom(config)# fex 101
n5k-bottom(config-fex)# desc b1

FEXs are numbered 100-199 - for consistency and the avoidance of mistakes, it is common to use the same local port channel and vPC numbers as the FEX is assigned. The FEX number controls the ethernet interface numbers - for example, FEX 101 with 48 ports will have interfaces on the controlling switch called Ethernet101/1/1-48.

Once a FEX has been associated, it will download an updated image, if required:

n5k-bottom# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

101 b1 Image Download N2K-C2248TP-E-1GE FOX1848GCMB

When that's complete, it will reboot and come online properly, making its interfaces available for configuration:

n5k-top# show fex

FEX FEX FEX FEX Fex

Number Description State Model Serial

------------------------------------------------------------------------

101 b1 Online N2K-C2248TP-E-1GE FOX1848GCMB

n5k-top# show int status fex 101

--------------------------------------------------------------------------------

Port Name Status Vlan Duplex Speed Type

--------------------------------------------------------------------------------

Eth101/1/1 -- notconnec 1 auto auto --

Eth101/1/2 -- notconnec 1 auto auto --

Eth101/1/3 -- notconnec 1 auto auto --

...

Tuesday, 17 February 2015

Cisco Nexus data centre network and enabling MPLS

We're just completed a tender for a new data centre network and select a design based on one of Cisco's standard models consisting of core Nexus 7010s in two physically diverse data centre locations, with each row of racks consisting of a pair of Nexus 56128Ps as the EoR switches and various Nexus 2k models as the ToR units.

Despite having plenty of experience with IOS (and, to a lesser extent, IOS XE), we've never had Nexus equipment before, in particular not used NX-OS. As such, there's a bit of a learning curve to get things up and running.

This post (and probably some following ones) is what we've encountered in the setup of the Nexus equipment and will likely cover VDCs on the 7010s, vPC (Virtual Port-Channel) on the 7010s and 56128Ps and the FEXs, as well as some NX-OS differences.

MPLS

Despite getting some licence sheets with PAKs (Product Activation Keys) for the MPLS feature set, these appear to have already been installed. However, the feature isn't activated by default and requires some work to do so.

First, in the admin VDC, the feature set must be installed:

n7k(config)# install feature-set mpls

Then, in the service VDCs (our backbone routing-only one is "cudn", the Cambridge University Data Network), the feature-set must be activated, along with the desired individual features:

n7k-cudn(config)# feature-set mpls
n7k-cudn(config)# feature mpls ldp
n7k-cudn(config)# feature mpls l3vpn

The MPLS features then become available in that VDC.

Thursday, 7 August 2014

IPv4 multicast over an MPLS VPN using mGRE

I've wanted to sort out multicast forwarding over an MPLS VPN for some time — while most institutions at the University of Cambridge don't have multicast-enabled networks, it seems a loose end that might prevent take up of the service.

After some poking about, trying to find free stuff online and failing, it seemed like MPLS and VPN Architectures vol. II from CiscoPress would explain it: in particular Chapter 7 ("Multicast VPN"). The good news is that it does.

However, the book dates from 2003 and I found an update document on Cisco's website about developments in MBGP to handle multicast over MPLS VPNs from about 2008 ("Multicast VPN - IP Multicast Support for MPLS VPNs"); there was a reprint of the book in paperback a month or so ago (late June 2014) and I thought it might have been updated to cover that. Unfortunately, the book hasn't been updated, although the changes are minimal so it doesn't hurt to start with the book and then read the update.

Note: the solution described here uses mGRE (multipoint GRE); replacing much of this is MLDP (Multicast Label Distribution Protocol) The latter looks like the right solution in the longer term but is fairly recent (dating from 2010) and needs IOS 15 on the Cisco platform, I believe. I'll look at that later.

Anyway — onto the technical bits...

The key facts

There are several key things to know in advance, which I think helps understand multicast over an MPLS VPN:

Firstly, it doesn't actually involve MPLS: multicast traffic is forwarded completely differently from unicast traffic and doesn't involve any MPLS labelled frames.
Instead, multicast traffic inside the VPN is tunnelled, by encapsulating it using GRE, inside multicast traffic in the global space. This is commonly called 'mGRE' (multipoint GRE).
In the global space, one or more MDTs (Multicast Distribution Trees) are constructed to carry this tunnelled traffic between the PE routers in the VPN. When traffic is received on one of these trees, it is de-encapsulated and forwarded to the VRF (Virtual Routing and Forwarding) instance for the VPN. Inside the VRF, traffic is forwarded into the MDT, when it needs to reach other routers in the VPN.
MDTs essentially put all the participating routers on a single layer 2 segment: when querying the PIM neighbours on the tunnel interface created by an MDT, all of the other routers will be seen.
The first MDT is known as the Default MDT — all PE routers in the VPN will join this tree and it will, by default, be used to forward all multicast traffic on the VPN.
If there is a lot of multicast traffic and it is not typically required on all the PE routers (because only some have members) then this is inefficient. To handle this situation, Data MDTs can be configured to set up separate groups in the global space — Cisco let you do this for groups where the bandwidth exceeds a certain threshold and/or with an address matched by an access list.
The Data MDTs use a pool of group addresses which is automatically recycled as groups come and go, much like a DHCP address pool. If the pool is exhausted, group addresses will be reused on a least-used (by mapping of group addresses) basis.
The indication that a particular group has been switched from the Default MDT to the Data MDT is done by a PIM message that is forwarded across the Default MDT.
Note that the decision to switch traffic to a Data MDT is done by the ingress router (nearest the source), rather than the final hop router (as would be the case with SPT switchover under regular PIM).
If the groups for the Default MDTs are in a PIM-SSM range (232.0.0.0/8, by default), the BGP address-family ipv4 mdt can be used to locate other PE routers in the MVPN. This is not necessary if the groups use PIM-SM with an RP.
The VPN must be configured to provide a BSR and RP in the normal manner: these work as normal across the MDTs (so a single BSR and RP can be set up on a PE for the VPN, or even inside the customer network).
The interfaces to CE routers work just as normal and also support BSR and RP discovery.

This arrangement is designed to be scalable and limit control plane resources in the provider network: the Data MDT group address pool allows the provider to prevent a large number of customer groups in the VPN from creating a similar number on the P routers, limiting the effect to just the PE routers.

All this business requires a good understanding of how multicast works and requires careful tuning of PIM modes (SM, SSM, Bidir) as well as the MDT group addresses, to get the best performance.

Keeping traffic separate

The main alarm bell I had with all this was about keeping traffic inside the VPN from escaping onto the global network, where it could be intercepted, or false traffic inserted.

This problem is actually straightforward: Cisco provide the ip multicast boundary command which is used to filter multicast traffic on an interface — if the group addresses used by the MDTs are in a range not permitted by the list specified with this, they will not pass across the interface, keeping the traffic safe.

In our case, we filter 239.255.0.0/16 as being 'institution private' so it doesn't ever pass across an interface between an edge subnet or institution and our backbone. If the MDTs are created in here, they should be safe.

Testing multicast boundaries

To check this all worked, I did some tests with this using GNS3 and Wireshark sniffing the virtual link between a host in the global space, which had joined the Default MDT group. By switching the filtering on and off, I could watch the effect...

Installing a boundary had an immediate affect, stopping the MDT traffic from reaching the host. The interface connecting the host disappeared from the output interface list of show ip mroute.
However, the group still showed up as being joined in the output of show ip igmp group, even though traffic did not pass. This would remain there until the IGMP reporting timer expired; other groups would refresh, but not this one
When the boundary was removed, traffic would not immediately flow, even if the interface remained in the IGMP list. However, when the IGMP membership report was retransmitted by the host, it caused the traffic to start flowing to the host again.

Example

(Update 2015-050-24 - I'm about to convert my GNS3 simulation over to MLDP, now I've upgraded to IOS 15, so I thought I'd do some tests using mGRE before dismantling it.)

A quick demo of this all in action and the output of some show commands.

Configuration

The configuration below builds on a working [non-VRF] PIM-SM on the backbone, as well as an MPLS IPv4 VPN. I'm doing this in IOS 15.

The MDTs are configured in the VRF, identically on each router, in the address-family block - the extra commands highlighted:

vrf definition eng_vrf

rd 192.84.5.238:129

route-target export 192.84.5.0:129

route-target import 192.84.5.0:129

address-family ipv4

mdt default 239.255.32.1

mdt data 239.255.36.0 0.0.0.31

mdt data threshold 10

exit-address-family

ip multicast-routing vrf eng_vrf

I've deliberately set the Data MDT switchover threshold high to avoid it being triggered, initially.

Note that IOS 15 replaces the mdt data ... threshold ... option with a separate mdt data threshold command that operates per-VRF. If you use the old format, you will receive a warning that it's deprecated.

Once enabled on the router, a VRF interface is enabled the same as a non-VRF one:

interface Ethernet1/2.210

description eng-vpn-nms

encapsulation dot1Q 210

vrf forwarding eng_vrf

ip address 129.169.10.253 255.255.255.0

ip pim bsr-border

ip pim sparse-mode

ip igmp version 3

Checking the Default MDT

Once this has been configured on all routers, the Default MDT should be formed across the backbone (using 239.255.32.1, in the above example) and they should all discover each other as if on a single network segment using PIM:

DIST-NMS#show ip pim vrf eng_vrf neighbor

PIM Neighbor Table

Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,

P - Proxy Capable, S - State Refresh Capable, G - GenID Capable

Neighbor Interface Uptime/Expires Ver DR

Address Prio/Mode

192.84.5.236 Tunnel3 00:00:15/00:01:35 v2 1 / S P G

192.84.5.237 Tunnel3 00:00:39/00:01:29 v2 1 / S P G

The above router - DIST-NMS - has found the other routers across the Default MDT on the backbone using the Tunnel3 interface; their loopback addresses in 192.84.5.x are shown. No DR is shown as that is us (our loopback is 192.84.5.238 - the highest address):

DIST-NMS#show ip pim vrf eng_vrf int tu3

Address Interface Ver/ Nbr Query DR DR

Mode Count Intvl Prior

192.84.5.238 Tunnel3 v2/S 2 30 1 192.84.5.238

You can check the MDTs and their tunnel interfaces as follows:

DIST-NMS#show ip pim vrf eng_vrf mdt

* implies mdt is the default MDT

MDT Group/Num Interface Source VRF

* 239.255.32.1 Tunnel3 Loopback0 eng_vrf

You can see the Default MDT group is visible in the global IP multicast forwarding table - note the Z flag, showing the group is used as an mGRE multicast tunnel:

DIST-NMS#show ip mroute 239.255.32.1

IP Multicast Routing Table

Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,

L - Local, P - Pruned, R - RP-bit set, F - Register flag,

T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,

X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,

U - URD, I - Received Source Specific Host Report,

Z - Multicast Tunnel, z - MDT-data group sender,

Y - Joined MDT-data group, y - Sending to MDT-data group,

G - Received BGP C-Mroute, g - Sent BGP C-Mroute,

Q - Received BGP S-A Route, q - Sent BGP S-A Route,

V - RD & Vector, v - Vector

Outgoing interface flags: H - Hardware switched, A - Assert winner

Timers: Uptime/Expires

Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 239.255.32.1), 00:06:39/stopped, RP 192.84.5.240, flags: SJCFZ

Incoming interface: Ethernet1/0, RPF nbr 192.84.5.33

Outgoing interface list:

MVRF eng_vrf, Forward/Sparse, 00:06:39/00:02:20

(192.84.5.236, 239.255.32.1), 00:05:40/00:01:10, flags: JTZ

Incoming interface: Ethernet1/0, RPF nbr 192.84.5.33

Outgoing interface list:

MVRF eng_vrf, Forward/Sparse, 00:05:40/00:00:19

(192.84.5.237, 239.255.32.1), 00:06:04/00:00:59, flags: JTZ

Incoming interface: Ethernet1/0, RPF nbr 192.84.5.33

Outgoing interface list:

MVRF eng_vrf, Forward/Sparse, 00:06:04/00:02:56

(192.84.5.238, 239.255.32.1), 00:06:39/00:02:54, flags: FT

Incoming interface: Loopback0, RPF nbr 0.0.0.0

Outgoing interface list:

Ethernet1/0, Forward/Sparse, 00:06:38/00:02:50

The PIM domain inside the VRF

The multicast forwarding table for the VRF is initially empty:

DIST-NMS# show ip mroute vrf eng_vrf

IP Multicast Routing Table

...

(*, 224.0.1.40), 00:59:21/00:02:14, RP 0.0.0.0, flags: DCL

Incoming interface: Null, RPF nbr 0.0.0.0

Outgoing interface list:

Ethernet1/2.210, Forward/Sparse, 00:59:20/00:02:14

In my simulation, the PIM-SM RP is on a router in an institutional (departmental) network, not part of the MPLS backbone. This is advertised as normal using PIM BSR and detected over the MDT:

DIST-NMS#show ip pim vrf eng_vrf rp mapping

PIM Group-to-RP Mappings

Group(s) 224.0.0.0/4

RP 129.169.252.1 (TRUMP.ENG), v2

Info source: 129.169.252.1 (TRUMP.ENG), via bootstrap, priority 64, holdtime 150

Uptime: 00:17:01, expires: 00:02:12

Adding group members

I'm using 234.131.111.10 as a test group. First, I set up two group members on institutional networks using ip igmp join-group .... Here's one:

interface FastEthernet0/0

description eng-vpn-medschl

ip address 129.169.86.86 255.255.255.0

ip igmp join-group 234.131.111.10

These show up in the multicast forwarding table on the RP, ENG-TRUMP:

ENG-TRUMP#show ip mroute 234.131.111.10

IP Multicast Routing Table

...

(*, 234.131.111.10), 00:24:05/00:03:14, RP 129.169.252.1, flags: SJC

Incoming interface: Null, RPF nbr 0.0.0.0

Outgoing interface list:

Ethernet1/0.252, Forward/Sparse, 00:21:03/00:03:14

Ethernet1/2, Forward/Sparse, 00:24:05/00:01:58

On that router, Ethernet1/2 is the link to the backbone router which is inside the VRF (and has the above configured member) and Ethernet1/0.252 has another, local member.

Adding a source

Now I can start a source by setting a ping going on an end host (with IP 129.169.10.10) to the group address and see the responses from the two members:

HOST-ENG-NMS#ping 234.131.111.10 repeat 1000

Type escape sequence to abort.

Sending 1000, 100-byte ICMP Echos to 234.131.111.10, timeout is 2 seconds:

Reply to request 0 from HOST-TRUMP.ENG (129.169.254.1), 444 ms

Reply to request 0 from HOST-MEDSCHL.ENG (129.169.86.86), 572 ms

Reply to request 1 from HOST-MEDSCHL.ENG (129.169.86.86), 208 ms

Reply to request 1 from HOST-TRUMP.ENG (129.169.254.1), 404 ms

...

The multicast forwarding table on the first hop router will show traffic being forwarded over the tunnel:

DIST-NMS#show ip mroute vrf eng_vrf 234.131.111.10

IP Multicast Routing Table

...

(*, 234.131.111.10), 00:00:10/stopped, RP 129.169.252.1, flags: SPF

Incoming interface: Tunnel3, RPF nbr 192.84.5.236

Outgoing interface list: Null

(129.169.10.10, 234.131.111.10), 00:00:10/00:03:22, flags: FT

Incoming interface: Ethernet1/2.210, RPF nbr 0.0.0.0

Outgoing interface list:

Tunnel3, Forward/Sparse, 00:00:09/00:03:20

Because there's not a sufficient level of traffic (above the configured 10kbit/s), no Data MDT has been created - we can check on the first hop router:

DIST-NMS#show ip mroute vrf eng_vrf 234.131.111.10 active

Use "show ip mfib active" to get better response time for a large number of mroutes.

Active IP Multicast Sources - sending >= 4 kbps

DIST-NMS#show ip pim vrf eng_vrf mdt send

DIST-NMS#

And on a receiving router:

DIST-HOSP#show ip pim vrf eng_vrf mdt receive detail

DIST-HOSP#

Increasing the data rate

Now everything's working, let's increase the rate of traffic by making the ping send packets without the 1s delay between them:

HOST-ENG-NMS#ping 234.131.111.10 rep 100000 timeout 0

Type escape sequence to abort.

Sending 100000, 100-byte ICMP Echos to 234.131.111.10, timeout is 0 seconds:

......................................................................

...

And check the traffic data rate:

DIST-NMS#show ip mroute vrf eng_vrf 234.131.111.10 active

Use "show ip mfib active" to get better response time for a large number of mroutes.

Active IP Multicast Sources - sending >= 4 kbps

Group: 234.131.111.10, (MCAST-131-111-10.CAM)

Source: 129.169.10.10 (HOST-NMS.ENG)

Rate: 141 pps/113 kbps(1sec), 113 kbps(last 40 secs), 3 kbps(life avg)

Now the data rate has gone above the configured 10kbps (kbit/s) threshold, a Data MDT should have been created and traffic switched across to it. This can be confirmed on the first hop router, along with the address of the new group:

DIST-NMS#show ip pim vrf eng_vrf mdt send

MDT-data send list for VRF: eng_vrf

(source, group) MDT-data group/num ref_count

(129.169.10.10, 234.131.111.10) 239.255.36.0 1

On the egress (from the mGRE cloud) router, this can be confirmed:

DIST-HOSP#show ip pim vrf eng_vrf mdt receive detail

Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,

L - Local, P - Pruned, R - RP-bit set, F - Register flag,

T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,

X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,

U - URD, I - Received Source Specific Host Report,

Z - Multicast Tunnel, z - MDT-data group sender,

Y - Joined MDT-data group, y - Sending to MDT-data group,

G - Received BGP C-Mroute, g - Sent BGP C-Mroute,

Q - Received BGP S-A Route, q - Sent BGP S-A Route,

V - RD & Vector, v - Vector

Joined MDT-data [group/mdt number : source] uptime/expires for VRF: eng_vrf

[239.255.36.0 : 192.84.5.238] 00:04:56/00:00:36

(129.169.10.10, 234.131.111.10), 00:10:39/00:01:44/00:00:36, OIF count: 1, flags: JTY

Once the source stops and the MDT times out (about 3 minutes), the Data MDT will then be destroyed.