NX-OS configuration synchronisation (switch-profiles)

When using vPC with dual-attached FEXs, a large chunk of the configuration across the two parent Nexus 5k (or 7k) switches must be kept in step.  For example:

• VLANs must be created and deleted on both switches.
• Fabric interfaces must have the same configuration with regards channel group assignments, FEX associations, etc.
• Edge (host) ports must have the same switchport configuration - mode and access/trunk/native VLANs.

This can be done manually but is a bit tedious and error-prone to maintain.  To help, NX-OS has a mechanism whereby parts of the configuration can be synchronised across the switches.

However, this facility is little confusing and can be problematic to set up - it's one of those things where the Cisco documentation makes sense after you understand the basics!

Key concepts

• Each switch maintains a local configuration which contains things which are NOT synchronised across the switches.  This is configured in the normal way, using configure terminal and has all the usual commands in it and will obviously hold things which differ between the switches.
• In addition, there is a separate configuration which is synchronised between switches and holds the common elements.  This is configured in a special mode, entered using configure sync in a block called a switch-profile.
• The running configuration is the result of merging the two configurations.  When using show running-config (including its options to look at specific parts of the configuration), the merged result is shown.
• Changes to the switch profile are made on one of the switches, verifyed and commited as a single transaction (rather like a database) on both switches automatically; if they fail, things should rollback to how they were before they started.
• Commands can be imported from the running configuration into the switch profile: this will remove them from the running configuration.
• Synchronisation can only work across the out-of-band management VRF (which, on a Nexus 5k, limits you to the copper Management0 port on the rear of the unit); it cannot be done in-band.

Setting up synchronisation

There are three main steps to this:

1. Set up management interface communication and enable Cisco Fabric Services (CFS) over it
2. Set up a switch profile and import the running configuration into it
3. Establish synchronisation between the peers

You have to take care to do these in the correct order otherwise things can get in a muddle.  Cisco describe this rather briefly on this page.

Set up management interface communication

Once the profile The first thing to do is allow communication between switches using the management interface and enable Cisco Fabric Services (CFS) over IPv4.  CFS is used to communicate the synchronisation and only operates over the management VRF:

n5k-bottom# conf t

n5k-bottom(config)# int mgmt0

n5k-bottom(config-if)# ip addr 192.168.200.1/24

n5k-bottom(config-if)# exit

n5k-top(config)# cfs ipv4 distribute

n5k-top(config)# end

Set up the switch profile and import the running configuration

The switch profile is created in the special configure sync mode.

The profile must be given a name - this is used to identify the configuration which must be synchronised and must match across the synchronisation peers - I recommend using something to identify the area the switches and their FEXs will serve (e.g. the room and rack row).

Once this is done, the running configuration can be imported, verified and committed (the verification stage can be omitted, if required, but it's a good idea to check things first).  This will move the synchronisable elements of the configuration from the running configuration into the switch profile.

n5k-bottom# conf sync

Enter configuration commands, one per line.  End with CNTL/Z.

n5k-bottom(config-sync)# switch-profile wcdc-b

Switch-Profile started, Profile ID is 1

n5k-bottom(config-sync-sp)# import running-config

n5k-bottom(config-sync-sp-import)# verify

Verification Successful

n5k-bottom(config-sync-sp-import)# commit

Verification successful...

Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.

Please avoid other configuration changes during this time.

Commit Successful

All this must be done on BOTH switches independently.

Establish synchronisation between the peers

The synchronisation can now be enabled between the peers in the switch profile:

n5k-bottom# conf sync
Enter configuration commands, one per line.  End with CNTL/Z.
n5k-bottom(config-sync)# switch-profile wcdc-b
Switch-Profile started, Profile ID is 1

n5k-bottom(config-sync-sp)# sync-peers destination 192.168.200.2

Once this is entered on both switches, they should find each other, exchange information and sync up, comparing the imported running configuration with each other and ensuring they agree (or, if they differ, they can be merged without conflict).  This will typically take 10-20 seconds and can be checked with show switch-profile status; the Peer information / Status will change from Peer not reachable to Verify Success to Commit Success and then finally the Sync-status will say In Sync, when this is complete:

n5k-bottom# show switch-profile status

switch-profile  : wcdc-b

----------------------------------------------------------

Start-time:   6070 usecs after Sun Feb 11 23:42:16 2001

End-time: 571814 usecs after Sun Feb 11 23:42:17 2001

Profile-Revision: 1

Session-type: Initial-Exchange

Session-subtype: Init-Exchange-All

Peer-triggered: Yes

Profile-status: Sync Success

Local information:

----------------

Status: Commit Success

Error(s):

Peer information:

----------------

IP-address: 192.168.200.1

Sync-status: In sync

Status: Commit Success

Error(s):

Once this has completed, and the two switches are in sync, it's ready for use.

Using synchronisation

From this point onwards, any changes to made across both switches should be made in configure sync mode, in the switch-profile then comited to take effect, rather than in configure terminal mode.  For example, to set up a new FEX, you might do:

n5k-bottom# conf sync

Enter configuration commands, one per line.  End with CNTL/Z.

n5k-bottom(config-sync)# switch-profile wcdc-b

Switch-Profile started, Profile ID is 1

n5k-bottom(config-sync-sp)# int e1/3-4

n5k-bottom(config-sync-sp-if-range)# desc b2

n5k-bottom(config-sync-sp-if-range)# channel-group 102

n5k-bottom(config-sync-sp-if-range)# exit

n5k-bottom(config-sync-sp)# int po102

n5k-bottom(config-sync-sp-if)# desc b2

n5k-bottom(config-sync-sp-if)# switchport mode fex-fabric

n5k-bottom(config-sync-sp-if)# fex associate 102

n5k-bottom(config-sync-sp-if)# vpc 102

n5k-bottom(config-sync-sp-if)# verify

Verification Successful

n5k-bottom(config-sync-sp)# commit

Verification successful...

Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.

Please avoid other configuration changes during this time.

Commit Successful

If something is accidentally configured in configure terminal mode, sometimes it can just be removed (with some no ... commands), but it may take some work if mutual exclusion errors are encountered - I'll cover this on a future entry.