Spanning Tree interoperability: PVST+ and IEEE RSTP/MSTP

First post for a while — Happy 2018!

We're planning on reconfiguring the way we do Spanning Tree on our network at the moment: we don't use it on a large scale, having banished VLANs from our core, in favour of a fully-routed network with VLANs constrained to a single building/department (although there's a smidgeon of EoMPLS for the odd layer 2 point-to-point virtual circuit).

We do, however, want to use Spanning Tree on our PoP (Point-of-Presence) switches: these are our CPE (Customer Premises Equipment) layer 2 switches that we put on the border of institutional (department, college, etc.) networks.  They act as the demarcation point between our backbone and the institutional network and try to protect the next layer of the network (the distribution routers) from layer 2 issues within institutions (since as loops creating ARP or DHCP storms)

When we replaced the PoP switches back in 2008, I did some interoperability tests between the Cisco PoP equipment we were installing (Catalyst 3560G-24PS) and various other vendors that institutions may use (since they choose their own kit and pick a variety of different vendors and models).  Aside from Spanning Tree not really being geared up to work across administrative boundaries, there were questions over how (say) HP's implementation of IEEE-standard RSTP (Rapid Spanning Tree Protocol) would interoperate with Cisco's Rapid-PVST+ (that we use, and used to relied on, since we still had cross-site VLANs back then).

I did various tests and found it all didn't play well together and it all misbehaved in difficult/unpredictable ways, so ended up blocking it (via "spanning-tree bpdufilter", with "spanning-tree portfast trunk" to avoid long delays on ports going into the forwarding state) on the ports feeding institutions and having to say it was unsupported, leaving LACP or FlexLink ("switchport backup ...") as the only mechanisms for redundant links.  However, we do get the occasional loop in institutional networks (sometimes between two ports on the PoP) and I want to try and do a better job of detecting these and stopping them impacting the network further up.

Given all that, I got a new Cisco PoP switch (we selected various models from the Catalyst 3850 range) and tried hooking it up to various other bits of kit (Cisco and non-Cisco) in odd ways to see how it behaves together.

Cisco Rapid PVST+ to Rapid PVST+

OK - obvious one first: Cisco Rapid PVST+ to Cisco Rapid PVST+.  This works fine, as you'd expect: the spanning trees on each of the VLANs find each other and interconnect, ports come up quickly when bridges are linked together, redundancy/loops are detected and it all works well.

[Also, note here, when I say "Rapid PVST+", the probably all applies to (non-Rapid) PVST+, but I haven't done any testing with that and I don't think it's worth spending time on nowadays.  That said, although no-one should be using it, it is still the default in Cisco IOS!]

The PVST+ BPDUs differ from IEEE (STP/RSTP/MSTP) BPDUs in that they are of frame type SNAP (Subnetwork Access Protocol) vs a dedicated STP BPDU type (aside from a different destination multicast MAC address).  The SNAP type is then organisation "Cisco" and vendor private protocol ID of "PVST+").  Once you get into the actual frame, it looks much the same, except that it has an "Originating VLAN (PVID)" field, identifying the VLAN ID on which it's operating:

Cisco Rapid PVST+ BPDU - VLAN 1 trunk native

IEEE RSTP BPDU IEEE RSTP BPDU

If the VLAN is tagged on the port (i.e. trunk, non-native) then the PVST+ BPDU frame will also have an 802.1Q VLAN header for the appropriate VLAN:

Cisco Rapid PVST+ BPDU - VLAN 100 tagged

(There is nothing special about this being VLAN 100 here: if the VLAN is not the native/untagged VLAN, the 802.1Q header will still be included, even if VLAN 1.)

I suppose, if you do weird things like transport one VLAN over another, you might get the PVST+ PVID differing from the one in the 802.1Q header, but I don't have that situation and I can't be bothered to trying and force it.

There are a couple of gotchas, though...

PVID Inconsistent with Rapid PVST+

The PVID field in the PVST+ BPDU is what allows two switches with mismatched native VLANs configured on a port to detect that they've been incorrectly linked together and block the VLAN on that port with "PVID_Inc"[onsistent], and the VLAN on the port to block in a "BKN" (broken) state:

cat3850#show spanning-tree vlan 100

VLAN0100
  Spanning tree enabled protocol rstp
  Root ID    Priority    16484
             Address     c4b9.cd48.1980
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    16484  (priority 16384 sys-id-ext 100)
             Address     c4b9.cd48.1980
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1             Desg BKN*4         128.1    P2p *PVID_Inc

This presents us with a problem, since some institutions have never configured their main data VLAN with the "correct" 802.1Q tag (i.e. the one we use on the backbone) and just use VLAN 1 (this goes back to when only we, on the backbone, needed VLANs — institutions didn't typically have more than one, so just used the default VLAN; now they have voice, wireless, Building Management, etc.).

I don't think there's a way to override this checking, on Cisco switches.  HP ProCurve switches running Rapid PVST+ have a "spanning-tree ignore-pvid-inconsistency" command but that only fixes the HP end.

If we want to interconnect these two VLANs, I think the only solution is to filter out the BPDUs on the link between the switches with bpdufilter:

interface GigabitEthernet1/0/1
 spanning-tree bpdufilter enable

... this only needs doing at one end (since it stops BPDUs being both sent and received), although this prevents Spanning Tree from doing anything useful on the port, as well, so we can't detect loops, etc.: the port will always forward (but will still go a listening/learning phase, taking 30s to bring up the VLAN on the port, even though it's effectively deaf to other switches).

Other vendors' Rapid PVST+

Some switch vendors support Cisco Rapid PVST+.  I've done some brief testing with HP ProCurve switches in Rapid PVST+ mode ("spanning-tree mode rapid-pvst") and they've worked fine, negotiating separate spanning trees for each of the VLANs working exactly as you'd expect a Cisco to.  The HP ProCurves also handle VLAN 1 in the same way as a Cisco, including sending the additional IEEE RSTP BPDUs, as well as the Rapid PVST+ BPDUs (see below).

When I last looked at Extreme XOS-based switches (e.g. Summit X450/X460), a few years ago, these also worked perfectly fine.  However, there was a note in the documentation about VLAN 1 not being handled correctly — I'm unsure exactly what that was, but it's probably related to the VLAN 1 dual-BPDU situation that Cisco support (see below).  We don't use them enough now (and don't use VLAN 1, either), so I haven't researched this further.

Cisco Rapid PVST+ to RSTP/MSTP

Ignoring VLAN 1 (see below!), mixing Cisco Rapid PVST+ and IEEE standard RSTP/MSTP results in the switches sending BPDUs that each other ignore — there are essentially two completely separate protocols operating independently.  However, the Rapid PVST+ BPDUs appear to just flow through the non-Rapid PVST+-aware bridge like any normal frame (since they're just sent as tagged frames on their VLANs) and will end up returning to the originating switch (or another Rapid PVST+-aware bridge) for processing (a bit like having an old-fashioned repeater!).

Obviously, the BPDUs from the IEEE bridge won't get through the Rapid PVST+ bridge network, unless the ports interconnecting them back to the IEEE bridge(s) have the VLAN presented untagged/native.

If there are multiple connections from the same Rapid PVST+ switch into a segment without a Rapid PVST+-aware bridge, the redundant connections will be treated as "Backup" ports in the Spanning Tree protocol, and blocked (similar to Alternate ports).

For example, if I take a Cisco Catalyst 3850 running Rapid PVST+ and an HP ProCurve 2920 running IEEE MSTP with three ports configured: port 1 has VLANs 100 and 200 tagged/trunk, 2 has 100 tagged/trunk and 3 has 200 tagged/trunk.  I then linked the ports to their equivalent number: 1 on the Cisco to 1 on the HP, 2-2 and 3-3.  I also set the spanning tree priority on the Cisco to 16384 for both VLANs 100 and 200.  I now get this on the Cisco:

cat3850#show spanning-tree vlan 100

VLAN0100
  Spanning tree enabled protocol rstp
  Root ID    Priority    16484
             Address     c4b9.cd48.1980
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    16484  (priority 16384 sys-id-ext 100)
             Address     c4b9.cd48.1980
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1             Desg FWD 4         128.1    P2p 
Gi1/0/2             Back BLK 4         128.2    P2p 

cat3850#show spanning-tree vlan 200

VLAN0200
  Spanning tree enabled protocol rstp
  Root ID    Priority    16584
             Address     c4b9.cd48.1980
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    16584  (priority 16384 sys-id-ext 200)
             Address     c4b9.cd48.1980
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1             Desg FWD 4         128.1    P2p 
Gi1/0/3             Back BLK 4         128.3    P2p 

This situation should work fine, although if the VLAN doesn't actually go into the same switch on the IEEE side but, instead, through a number of other switches, then you get a failure between the two halves connecting back to the Rapid PVST+ bridge, it will likely take several seconds to reconverge, since this will only be detected when the BPDUs fail to arrive on the backup port.

I did see some problems with this, however, with HP Comware switches (5500-EI, running version 3 of Comware in MSTP mode): sometimes the Rapid PVST+ BPDUs wouldn't get through and both ports would stay forwarding.  This makes me think that we shouldn't rely on this as a way to build redundant topologies but more as a way to try and detect loops.

Rapid PVST+ VLAN 1 special handling

VLAN 1 gets some special treatment with Rapid PVST+: if it is present on a port, not only do you get PVST+ BPDUs, with the PVID field and sent with a 802.1Q header, if presented tagged/non-native, you also get a regular IEEE RSTP BPDU.  The to packet captures shown first, above, were actually sent consecutively by the same switch, on the same port:

IEEE RSTP BPDU and Cisco Rapid PVST+ BPDU - VLAN 1, sent consecutively

The IEEE RSTP BPDU is always sent untagged/trunk-native (without an 802.1Q header), regardless of whether the VLAN itself is untagged/trunk-native or tagged/trunk on that port.

This BPDU allows interoperation with IEEE-standard, non-PVST+ switches, such as an HP ProCurve running MSTP (Multi Spanning Tree) or regular RSTP (non-PVST+ IEEE Rapid Spanning Tree) and for a redundant topology to be built.

However, this only works for VLAN 1 — the IEEE BPDUs will never be sent if VLAN 1 is not present, regardless of whether one is untagged/native.  In addition, if other VLANs are also present, the outcome of negotiation with a partner IEEE-standard bridge will only affect the forwarding status and spanning tree of VLAN 1.  For example, the following port on a Cisco, connected to a similarly configured HP ProCurve (in terms of VLANs), with the HP running MSTP and a priority of 8192, only reports a root bridge for VLAN 1:

cat3850#show running-config interface g1/0/1
interface GigabitEthernet1/0/1
 ! (default: switchport trunk native vlan 1)
 switchport trunk allowed vlan 1,100
 switchport mode trunk

cat3850#show spanning-tree interface g1/0/1

Vlan                Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
VLAN0001            Root FWD 4         128.1    P2p 
VLAN0100            Desg FWD 4         128.1    P2p 

... VLAN 100 will also take 30s to go through listening/learning and drop into forwarding, as if there is no partner bridge.

In our case, we don't actually use VLAN 1 anywhere on the network: we treat it as a kind of dumping ground where things end up if we haven't configured them (such as an access port with a voice VLAN but no data VLAN — there's no useful service on it, it's just used because often there has to be a VLAN of some sort specified).

[Extreme switches are nice in this area as they're able to have ports which have no VLANs present on them, and not have any untagged/trunk-native VLAN.  We had a case recently where we wanted VLAN to be tagged/non-native on an HPE Aruba wireless access point, but could only do that by making an arbitrary other, unused VLAN the untagged/native one.]

Second connection between Rapid PVST+ and IEEE RSTP/MSTP bridges with VLAN 1

If we plug in a second connection between the same two switches, using two ports configured in the same way as described above (VLAN 1 untagged/native, VLAN 100 tagged/trunk, Spanning Tree priority on IEEE bridge of 8192), VLAN 1 forms a spanning tree, but VLAN 100 just flows the BPDUs through the HP ProCurve and the Cisco sees it as a link into the same multiaccess segment and blocks it as a Backup role port:

cat3850#show spanning-tree vlan 1  

VLAN0001
  Spanning tree enabled protocol rstp
  Root ID    Priority    8192
             Address     d4c9.efb6.a680
             Cost        4
             Port        1 (GigabitEthernet1/0/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     c4b9.cd48.1980
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1             Root FWD 4         128.1    P2p 
Gi1/0/2             Altn BLK 4         128.2    P2p 

cat3850#show spanning-tree vlan 100

VLAN0100
  Spanning tree enabled protocol rstp
  Root ID    Priority    32868
             Address     c4b9.cd48.1980
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    24676  (priority 24576 sys-id-ext 100)
             Address     c4b9.cd48.1980
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1             Desg FWD 4         128.1    P2p 
Gi1/0/2             Back BLK 4         128.2    P2p

The HP forwards on all ports (since it's the root):

hp2920# show spanning-tree 

 Multiple Spanning Tree (MST) Information

  STP Enabled   : Yes

  Force Version : MSTP-operation

  IST Mapped VLANs : 1-4094

  Switch MAC Address : d4c9ef-b6a680

  Switch Priority    : 8192 

  Max Age  : 20

  Max Hops : 20   

  Forward Delay : 15

  Topology Change Count  : 6           

  Time Since Last Change : 11 secs     

  CST Root MAC Address : d4c9ef-b6a680

  CST Root Priority    : 8192        

  CST Root Path Cost   : 0           

  CST Root Port        : This switch is root

  IST Regional Root MAC Address : d4c9ef-b6a680

  IST Regional Root Priority    : 8192        

  IST Regional Root Path Cost   : 0           

  IST Remaining Hops            : 20          

  Root Guard Ports     : 

  Loop Guard Ports     : 

  TCN Guard Ports      : 

  BPDU Protected Ports :                                         

  BPDU Filtered Ports  :                                         

  PVST Protected Ports :                                         

  PVST Filtered Ports  :                                         

  Root Inconsistent Ports  :             

  Loop Inconsistent Ports  :             

                  |           Prio              | Designated    Hello         

  Port  Type      | Cost      rity State        | Bridge        Time PtP Edge

  ----- --------- + --------- ---- ------------ + ------------- ---- --- ----

  1     100/1000T | 20000     128  Forwarding   | d4c9ef-b6a680 2    Yes No  

  2     100/1000T | 20000     128  Forwarding   | d4c9ef-b6a680 2    Yes No  

This situation should work fine, I think, albeit a little confusing.

Now, if we set the VLAN 1 priority on the Cisco Rapid PVST+ switch to 4096 (lower/better than the HP ProCurve running IEEE MSTP), the root bridge moves over to the Cisco:

cat3850#show spanning-tree vlan 1 

VLAN0001

  Spanning tree enabled protocol rstp

  Root ID    Priority    4097

             Address     c4b9.cd48.1980

             This bridge is the root

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    4097   (priority 4096 sys-id-ext 1)

             Address     c4b9.cd48.1980

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi1/0/1             Desg FWD 4         128.1    P2p 

Gi1/0/2             Desg FWD 4         128.2    P2p 

cat3850#show spanning-tree vlan 100

VLAN0100

  Spanning tree enabled protocol rstp

  Root ID    Priority    32868

             Address     c4b9.cd48.1980

             This bridge is the root

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32868  (priority 32768 sys-id-ext 100)

             Address     c4b9.cd48.1980

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi1/0/1             Desg FWD 4         128.1    P2p 

Gi1/0/2             Desg FWD 4         128.2    P2p 

However, the HP treats both ports without regards to VLANs and so blocks one of them:

hp2920# show spanning-tree

 Multiple Spanning Tree (MST) Information

  STP Enabled   : Yes
  Force Version : MSTP-operation
  IST Mapped VLANs : 1-4094
  Switch MAC Address : d4c9ef-b6a680
  Switch Priority    : 8192 
  Max Age  : 20
  Max Hops : 20   
  Forward Delay : 15

  Topology Change Count  : 4           
  Time Since Last Change : 10 mins     

  CST Root MAC Address : c4b9cd-481980
  CST Root Priority    : 4097        
  CST Root Path Cost   : 20000       
  CST Root Port        : 1                  

  ...

                  |           Prio              | Designated    Hello         
  Port  Type      | Cost      rity State        | Bridge        Time PtP Edge
  ----- --------- + --------- ---- ------------ + ------------- ---- --- ----
  1     100/1000T | 20000     128  Forwarding   | c4b9cd-481980 2    Yes No  
  2     100/1000T | 20000     128  Blocking     | c4b9cd-481980 2    Yes No  

This situation could create some confusion, if there were different VLANs configured on each of the ports, but with VLAN 1 present: only one port would forward and the others would block, potentially breaking some of the other VLANs.

Joining two PVST+ VLANs in the IEEE STP bridge

This one allows you to get in a bit of a muddle!  Here's an example: a Rapid PVST+ switch with port 1 having VLAN 100 untagged/native and port 2 having VLAN 200 untagged/native is connected to an IEEE Spanning Tree bridge with both ports in the same VLAN untagged/native (some VLAN ID as one of the ones on the PVST+ bridge, or different - it doesn't matter), e.g.:

Cisco Rapid PVST+ bridge (assuming the VLANs are already created):

interface Gi1/0/1
 switchport mode trunk
 switchport trunk allowed vlan 100
 switchport trunk native vlan 100
!
interface Gi1/0/2
 switchport mode trunk
 switchport trunk allowed vlan 200
 switchport trunk native vlan 200

HP ProCurve (isn't this side easy!):

vlan 50 untag 1,2

The corresponding ports are then connected together (1-1, 2-2).

Once connected, the Rapid PVST+ BPDUs for the two different VLANs are interconnected through VLAN 50 in the IEEE STP bridge and find their way back to the Cisco switch, where you get a "PVID_Inc" (VLAN ID inconsistent) and both ports go into the "BKN" (broken) state and block, as described above, breaking both VLANs!

It makes no difference whether the IEEE STP bridge is running Spanning Tree or not, since the IEEE STP BPDUs pass by the Rapid PVST+ BPDUs and don't interact.  The only change in behaviour is that the broken state won't be detected until the ports on the IEEE bridge move into the Forwarding state (which will take 30s normally, or 4s if the HPs "auto-edge" mode is enabled), meaning the problem will initially not be noticed (unless there are other effects) and probably things will be OK for 30s (on the port that was connected first, at least), after which both ports will move into the Broken state and block, when the Rapid PVST+ BPDUs get bridged between the ports/VLANs on the Cisco.

This situation could confusing because both VLANs will break, when this situation occurs — perhaps by someone accidentally connecting a data VLAN to a voice VLAN.  Without spanning tree, both VLANs may continue functioning, to a certain extent.  However, this is probably a good thing as the problem is detected immediately, rather than weird things going on until the root cause is determined.

(One thing we do is have different HSRP group numbers for each SVI, meaning that the MAC addresses and other messages don't clash, when two VLANs are connected together by accident.  This also helps us spot what's happened, because the virtual MAC address of [say] the voice VLAN will appear on the data VLAN.)

Cisco Rapid PVST+ to No Spanning Tree

Last but not least, what about if there is no Spanning Tree running on the partner bridge?

There is obviously a distinction between it gobbling up the BPDUs (perhaps by running bpdufilter) and one which just lets the BPDUs flow through without processing with them, treating them like any other traffic.

HP ProCurve switches have spanning tree turned off by default and require it to be turned on with "spanning-tree" (that's it — no arguments!) command.  When turned on, it defaults of IEEE MSTP (at least on a ProCurve 2920; other switches may vary, especially the older ones, but HP have been pushing MSTP for a long while, as they're fairly keen on IEEE standards, even when they're not particularly great!).

Without Spanning Tree enabled, the situation, at least with HP ProCurve, is that the BPDUs flow through like normal traffic, creating a situation the same as the "Rapid PVST+ and RSTP/MSTP" interaction, above.

For Cisco Catalyst switches: if spanning tree is explicitly disabled on a VLAN with "no spanning-tree vlan ...", the switch will not gobble PVST+ BPDUs but allow them to flow through and the bridges on ports into that VLAN will see each other, in terms of Spanning Tree.  However, this doesn't appear to be true for IEEE STP BPDUs: they do seem to be filtered, even if a VLAN is presented untagged/native.

ERSPAN on a Nexus 7010 (updated 2017-10-19)

We needed to determine the cause of some faults on our server network which is based on Nexus 7010 equipment.  The ERSPAN ("Encapsulated Remote Source Port ANalyzer") feature proved excellent for this for a number of reasons:

• We didn't need to physically go to the data centre to do the monitoring.  The monitoring is directed across an IP tunnel to the capture device, which can be anywhere else on an IP network.
• ERSPAN (and SPAN in general) on the Nexus platform has very good filtering capability so we can select just the traffic we want and monitor it on a machine with significantly less bandwidth than the capture device (in my case, my office machine monitoring 8x 10Gbit/s links looking for odd packets!).
• Related to the above, actually just physically connecting the 10Gbit/s ports to my laptop would be challenge, ignoring whether it can keep up with the traffic on them!

Setting up ERSPAN

In the Admin VDC (regardless of which VDC traffic is to be monitored in), the following global command needs to be set to define the source IP address for the packets.  This address does not have to available to the particular VDC where the monitoring is done:

  monitor erspan origin ip-address 192.84.5.248 global

The actual mirroring can then be set up as follows:

Command	Meaning
monitor session 1 type erspan-source	Create an ERSPAN session number 1
erspan-id 18	ID number transmitted in the ERSPAN packet to identify this particular SPAN
vrf default	VRF into which ERSPAN traffic is to be transmitted to the destination (not the VRF being monitored)
destination ip 192.0.2.1	IP address of destination (collector of ERSPAN traffic)
source interface port-channel789 both	Interface to be monitored
filter vlan 61,62	VLANs to be selected from the above interface (to monitor a whole VLAN, use 'source vlan ...' instead).  Filters can also reference access lists and other criteria.
filter access-group FILTER-VLANACCMAP	VLAN access-map to use to filter traffic (see below).
no shut	Enable the monitor

Filtering using an ACL

If you want to filter traffic matching things like source IP addresses, port numbers, etc. you can use an access-list to match it.  The Cisco documentation isn't clear on this, and contradicts how it's done, but a good reference is here.

In short, you must create an access-list, then a vlan access-map and apply that to the ERSPAN session.  For example:

  ip access-list ERSPAN-TRAFFIC-ACL4

   permit udp host 192.0.2.6 any eq 53

  !

  vlan access-map ERSPAN-TRAFFIC-VLAM 10

   match ip address ERSPAN-TRAFFIC-ACL4

  !

  monitor session 1 type erspan-source

   filter access-group ERSPAN-TRAFFIC-VLAM

Note that, despite the command "filter access-group ..." you MUST use a VLAN access-map (at least on 7.3.1 and above).

Capturing the traffic

Once enabled, the router will send the monitored traffic to the target host.  This will arrive as GRE traffic with protocol type 0x88be (ERSPAN).  Following the GRE header will be the ERSPAN header, which contains things like the VLAN ID and ERSPAN ID, then the entire L2 frame.

Wireshark recognises the packets as ERSPAN packets without any configuration, and will correctly decode them to show the encapsulated packet directly, allowing you to watch things in real time from the comfort of your desk!

VRF Source Select in NX-OS (Cisco Nexus)

We've never used IOS's VRF Source Select feature before, so I've never had to look into it.  It's useful when you have clients in multiple VRFs on a single interface and wish to select between them based on their IP address.  However, we have a use for it on our VPN service...

We offer a standard "University VPN Service" which gives all users an address from a single pool, but institutions can pay to have a "Managed VPN Service" which is limited to a subset of users of their own choice (typically the ones in their institution) and has a dedicated pool of client addresses.  The institution can then permit this range access through firewalls and into servers.  The addresses are all routed to the VPN across a single routed link in our server network.

We provide some institutions with a private internal network using MPLS L3 VPN.  However, the VPN server itself doesn't have VRFs (and we don't really want to configure and would like to be able to use the source select feature to put their pool of addresses into the VPN.

The server router is a Nexus 7010 with NX-OS 7.2(1)D1(1).  We're running 7.2 to get use MPLS Inter-AS Option B routing working, but I don't think this is needed for the source select feature.

A bit of Googling and searching Cisco's website didn't show up a VRF source select equivalent directly, but you can roll your own very simply with inter-VRF routes and some Policy Based Routing (PBR).  Cisco's website documents this but doesn't give a complete example.

The VPN server

In real life, our VPN server is a Linux box running StrongSWAN and acting as a router (with a link subnet and the client addresses routed to it over that).  However, I'm simulating it using another VDC on the same Nexus 7010.

Here's the uplink subnet (to router R1) and the default route:

interface Ethernet2/5

  description to-r1

  ip address 1.19.0.9/24

  no shutdown

!

ip route 0.0.0.0/0 Ethernet2/5 1.19.0.1

We simulate client addresses in the global and customer VRFs with a pair of loopback interfaces:

interface loopback19

  description global

  ip address 1.0.9.1/24

!

interface loopback109

  description cust

  ip address 100.0.9.1/24

Link and default VRF on the router

The upstream router has a link to the VPN server with the client address range in the default VRF routed across it:

interface Ethernet2/6

  description to-v1

  ip address 1.19.0.1/24

  no shutdown

!

ip route 1.0.9.0/24 Ethernet2/6 1.19.0.9

Clients in the the default VRF are now reachable across the network (assuming static routes are redistributed appropriately).

The VPN client address in the default VRF can now be pinged:

route-dcr-r1# ping 1.0.9.1

PING 1.0.9.1 (1.0.9.1): 56 data bytes

64 bytes from 1.0.9.1: icmp_seq=0 ttl=254 time=1.506 ms

64 bytes from 1.0.9.1: icmp_seq=1 ttl=254 time=1.36 ms

64 bytes from 1.0.9.1: icmp_seq=2 ttl=254 time=1.339 ms

64 bytes from 1.0.9.1: icmp_seq=3 ttl=254 time=1.325 ms

64 bytes from 1.0.9.1: icmp_seq=4 ttl=254 time=1.371 ms

--- 1.0.9.1 ping statistics ---

5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 1.325/1.38/1.506 ms

Routing traffic out from the VRF

To route traffic from inside the VRF to the VPN server in the default VRF, an inter-VRF static route can easily be created:

vrf context cust

  ip route 100.0.9.0/24 Ethernet2/6 1.19.0.9 vrf default

... this says that 100.0.9.0/24 is to be routed via 1.19.0.9 (the VPN server) in VRF default on Ethernet2/6.

The route needs to be redistributed as per any normal route in the VRF.  In our case, this is redistributed as a static route (not as part of an aggregate), along with the direct (NX-OS parlance for "connected") route used on the link subnet:

route-map permit_rtmap permit 10

!

router bgp 1

  vrf cust

    address-family ipv4 unicast

      redistribute direct route-map permit_rtmap

      redistribute static route-map permit_rtmap

Selecting VRF based on source IP address

Before we can use Policy Based Routing (PBR), we need to enable it as a feature:

feature pbr

First, we create an access list to match the traffic to jump into a different VRF:

ip access-list vpn-cust-addrs

  10 permit ip 100.0.9.0/24 any 

Then we create a route-map to change the VRF:

route-map vpn-in_rtmap permit 10

  match ip address vpn-cust-addrs 

  set vrf cust

... the set statement changes the VRF of the received traffic: the next hop and output interface are derived by looking at the routing table in the cust VRF.

Next, we apply the policy routing to the interface linking to the VPN server:

interface Ethernet2/6

  ip policy route-map vpn-in_rtmap

A ping to the client addresses from inside the VRF now works from R1:

route-dcr-r1# ping 100.0.9.1 vrf cust

PING 100.0.9.1 (100.0.9.1): 56 data bytes

64 bytes from 100.0.9.1: icmp_seq=0 ttl=254 time=1.552 ms

64 bytes from 100.0.9.1: icmp_seq=1 ttl=254 time=1.291 ms

64 bytes from 100.0.9.1: icmp_seq=2 ttl=254 time=1.3 ms

64 bytes from 100.0.9.1: icmp_seq=3 ttl=254 time=1.444 ms

64 bytes from 100.0.9.1: icmp_seq=4 ttl=254 time=1.307 ms

--- 100.0.9.1 ping statistics ---

5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 1.291/1.378/1.552 ms

Overriding a video mode for a specific monitor with a Raspberry Pi

I have a Raspberry Pi I use for fiddling about.  It has an odd little 10.4" Lilliput monitor with an 800x600 native resolution and a variety of inputs, including both HDMI and DVI:

• If I use the DVI input, it reports a DMT (Display Monitor Timings - computer monitor type) with 800x600
• If I use the HDMI input, it reports a CEA (Consumer Electronics Association - TV type) with a preferred resolution of 1280x720

I prefer to use the HDMI input as the cable is thinner and easier to connect (especially important with something as light as the Pi).  However, when I do, the Pi picks a rather blurry 1280x720.

I can force this in /boot/config.txt with the following lines:

# set DMT mode
hdmi_group=2
# set 800x600 @ 60Hz
hdmi_mode=9

However, this is annoying if I plug the Pi into a different screen.  It is possible to force a specific mode only when a particular monitor is connected.

First, find out the display ID:

pi@mincepi ~ $ tvservice -n
device_name=LLP-32V3H-H6A

Then edit config.txt to add a section in square brackets to override it:

[EDID=LLP-32V3H-H6A]
hdmi_group=2
hdmi_mode=9 

[all]

The trailing [all] is to specify that the following lines apply to all monitor types and is just a safe thing to put in case some additional directives are applied below it.

Frama franking machine ethernet issues

The University has been refurbishing the old Arup Building on the New Museums Site for the Cambridge Conservation Initiative and renamed it as the David Attenborough Building.

In the new building, we had problems with a Frama franking machine which wouldn't connect to the network:

• When the machine boots up, the ethernet link goes up and the device DHCPs to get an address but then disconnects.
• When trying a connection test or trying to frank something, the link never goes live (the interface doesn't go up from a physical/line level - not getting as far as IP) and it reports an error.

Oddly, if we disconnect the franking machine and connect something else to the wallport, that device works fine, even on 1Gbit/s connections.  If we connect the franking machine directly to a laptop (in my case, a Thunderbird adapter on a MacBook Air), it also works fine and can be pinged.  We've tried other wallports, cables and switchports without resolving the issue.

Some notes:

• The machine does, by design, disconnect from the network when it's not using it as a security measure (to minimise the chance of being hacked remotely).
• The connection is via a wall socket and probably 30-40m cable.
• The franking machine only has a 10Mbit/s half-duplex ethernet interface.
• The switch the machine was connecting to is a Cisco 2960X-48LPS-L (48-port PoE+ with 10G uplink).

Fixing the port speed to 100Mbit/s half-duplex did not resolve the issue.

I ended up solving it by putting an unmanaged Netgear 8-port 10/100 switch between it and the wallport and it works fine.

However, I also wondered if PoE was upsetting the controller in the franking machine so I disabled that and removed the Netgear.  That was working when I left, so that may have fixed up, but I'll have to check in and see how they're getting on, a couple of days later.

The Double Wireshark

I've been trying to diagnose a problem with multicast forwarding over MPLS with MLDP — packets are going missing somewhere between the PE router, including the PIM Hellos, resulting in PIM neighbours occasionally timing out, even when no traffic is flowing across the MDT).

This morning, I did some packet captures over a quiet MDT (i.e. one where there was no traffic being forwarded, other than the PIM Hellos).  The backbone links are 10G and busy most of the time, making looking for a single missing packet tricky, so I changed OSPF costs and HSRP tracking object statuses to try and move as much traffic as possible away from them and making sniffing the links possible with a reasonable degree of confidence there was no packet drop.

I started at the PE router to check it was actually sending the PIM Hellos on its uplink, which it was (at the time another PE router didn't receive them).  I then moved on to one of our core (P) routers, which is also the MDT root: I needed to see if the packets were ingressing and egressing correctly and see if they were going missing.

To do this, I attached two Ethernet interfaces to my MacBook Air (one Thunderbolt and one USB) —although traffic levels were only a few megabits, I used the USB one for ingress and Thunderbolt one for egress (as the latter is arguably more critical) and ran two Wiresharks:

The core router was then set to mirror the ingress and egress ports to different monitoring destinations - Gi5/2 and Gi6/2 (the copper ports on the Supervisor cards) and attached to the MacBook Air:

The missing packets were evident by comparing the captures:

These corresponded to the destination PE router showing it had missed a PIM Hello and the timeout not resetting (and a debug ip pim vrf ... hello / terminal monitor would show it had gone missing).

I've emailed the captures and report to the support partner / Cisco.

ECMP with OSPF, BGP and MPLS

As reported earlier, we've been gradually enabling ECMP across the University backbone to make better use of links and increase effective bandwidth.  For the most part, this is pretty straightforward, but there were a few gotchas — I thought I'd document both these: the enabling and the gotchas.

Note that ECMP typically enables load-sharing and NOT load-balancing:

• Load-sharing is about distributing the traffic across active paths, probably by hashing the source and/or destination addresses of the packet, potentially resulting in an uneven distribution (especially if the traffic is between a small number of addresses).
• Load-balancing is attempts to distribute the traffic such that it is evenly split across the available paths.

Note that ECMP using routing protocols will usually only distribute traffic where routers are both the ingress and egress point: it does nothing for inbound traffic from a simple host with a single subnet gateway or static default route: in the absence of a dynamic routing protocol to do this, solutions such as Cisco's GLBP (Gateway Load Balancing Protocol) can help here (which I'm not going to cover here).

OSPFv2 and OSPFv3

OSPF is easy to do - you simply increase the number of paths used:

router ospf 1
 maximum-paths 2
!
ipv6 router ospf 1
 maximum-paths 2

... the parameter to maximum-paths specifies how many of the available [lowest and best] equal-cost paths calculated in OSPF are loaded in the router's active forwarding table.  IOS supports a value up to 6.

This command must be entered on all routers in the network and applies at each hop.  For example, in a traditional two-layer core and distribution model:

• On the ingress distribution router, it will distribute traffic across uplinks to the core routers.
• On the core routers, it will distribute traffic across the downlinks to the egress distribution routers which serve the destination address.

Once entered, the show ip route ... command can be used to confirm multipath is in operation — here on a core router:

CORE-CENT#show ip route 131.111.10.10

Routing entry for 131.111.10.0/24

  Known via "ospf 1", distance 110, metric 27, type extern 1

  Last update from 192.84.5.18 on Ethernet1/4, 00:01:07 ago

  Routing Descriptor Blocks:

  * 192.84.5.34, from 192.84.5.238, 00:01:07 ago, via Ethernet1/2

      Route metric is 27, traffic share count is 1

    192.84.5.18, from 192.84.5.234, 00:01:07 ago, via Ethernet1/4

      Route metric is 27, traffic share count is 1

Personally, I find the show ip cef ... detail command a little clearer (and explains MPLS better, when we get round to that):

CORE-CENT#show ip cef 131.111.10.10 detail 

131.111.10.0/24, epoch 0, per-destination sharing

  local label info: global/33

  nexthop 192.84.5.18 Ethernet1/4

  nexthop 192.84.5.34 Ethernet1/2

That's all there is to it, although if you use DHCP relaying, first hop redundancy (VRRP, HSRP or GLBP) and address spoofing protection, then note the messy problem with those I covered in an earlier article!

BGP

Multipath using BGP route is similar to OSPF:

router bgp 64602
 address-family ipv4 unicast
  maximum-paths 2
  maximum-paths ibgp 2
 exit-address-family
 !
 address-family ipv6 unicast
  maximum-paths 2
  maximum-paths ibgp 2
 exit-address-family

The obvious difference is the two separate commands

• maximum-paths ... applies to routes learnt over eBGP peerings
• maximum-paths ibgp ... applies to routes learnt over iBGP peerings, even if they are external in origin (i.e. learnt from an eBGP ASBR in the same AS)

Because of this, usually only the latter (the iBGP version) is required on core routers and BGP route reflectors (as they don't usually have eBGP peers).

Route Reflectors (RRs) also present an additional wrinkle — an RR will only reflect a single route to its clients: that which it itself considers the best, based on the normal BGP selection methods (which can include the IGP cost).  This does not usually cause a problem because, as long as the RR client sends the traffic to the RR, the RR can then multipath traffic from itself according to the available paths, when it forwards it on.  However, this did cause a problem when trying to share traffic across the core to our internet gateways (more later).

Note that you do not (and cannot) enable multipath for the multicast address families.  The multicast routes are for RPF checking; multipath for multicast traffic forwarding is done separately (and I've not yet looked into it, so there's nothing in this article about it).

eBGP example

Let's look at an example - 129.169.0.0/16 is a block of addresses used by the Department of Engineering — their network is AS65106,  separate from the University backbone (which is AS64602).  They connect to the backbone via a series of /30 link subnets in 193.60.93.16/28 across which operate the eBGP peerings.

(Note that we're using a contrived simulation here, so the details are not necessarily accurate to reality.)

Looking in the routing table for 129.169.80.10 on one of the core routers (which, in my example is advertised equally across two of these links):

CORE-CENT#show ip route 129.169.80.10

Routing entry for 129.169.0.0/16

  Known via "bgp 64602", distance 200, metric 0

  Tag 65106, type internal

  Last update from 193.60.93.26 00:02:11 ago

  Routing Descriptor Blocks:

    193.60.93.26, from 192.84.5.234, 00:02:11 ago

      Route metric is 0, traffic share count is 1

      AS Hops 1

      Route tag 65106

      MPLS label: none

  * 193.60.93.18, from 192.84.5.236, 00:02:11 ago

      Route metric is 0, traffic share count is 1

      AS Hops 1

      Route tag 65106

      MPLS label: none

This shows the two links via eBGP:

• one via 193.60.93.26 which was learnt from iBGP peer 192.84.5.234
• the other via 193.60.93.18 which was learnt from iBGP peer 192.84.5.236

Checking the actual forwarding on a core router with show ip cef:

CORE-CENT#show ip cef 129.169.80.10 detail   

129.169.0.0/16, epoch 0, flags rib only nolabel, rib defined all labels, per-destination sharing

  recursive via 193.60.93.18

    recursive via 193.60.93.16/30

      nexthop 192.84.5.26 Ethernet1/6

  recursive via 193.60.93.26

    recursive via 193.60.93.24/30

      nexthop 192.84.5.18 Ethernet1/4

Note the two levels of recursion:

• 193.60.93.18 and 193.60.93.26 are the addresses of the eBGP border routers from which the routes were learnt
• these match the link subnet routes 193.60.93.16/30 and 193.60.93.26/30, respectively (from the IGP)
• each of these was learnt from the IGP neighbours 192.84.5.26 (on Eth1/6) and 192.84.5.18 (on Eth1/4)

Just for completeness, let's delve into the BGP database:

CORE-CENT#show bgp ipv4 unicast 129.169.80.10

BGP routing table entry for 129.169.0.0/16, version 31

Paths: (2 available, best #2, table default)

Multipath: iBGP

  Advertised to update-groups:

     1         

  Refresh Epoch 1

  65106, (aggregated by 65106 129.169.252.1), (Received from a RR-client), (received & used)

    193.60.93.18 (metric 27) from 192.84.5.236 (192.84.5.236)

      Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, multipath(oldest)

      rx pathid: 0, tx pathid: 0

  Refresh Epoch 1

  65106, (aggregated by 65106 129.169.252.2), (Received from a RR-client), (received & used)

    193.60.93.26 (metric 27) from 192.84.5.234 (192.84.5.234)

      Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, multipath, best

      rx pathid: 0, tx pathid: 0x0

MPLS L3 VPNs

On the surface, MPLS L3 VPNs look straightforward and similar to BGP — you just need to use maximum-paths in the corresponding BGP VRF stanzas:

router bgp 64602
 address-family ipv4 unicast vrf ucs-staff_vrf
  maximum-paths 2
  maximum-paths ibgp 2
 exit-address-family
 !
 address-family ipv6 unicast vrf ucs-staff_vrf
  maximum-paths 2
  maximum-paths ibgp 2
 exit-address-family

(In the above case, the VRFs have eBGP peerings to non-MPLS peers — ones using a regular ipv4 unicast address family with the peering inside a VRF, as opposed to a vpnv4 peering with non-VRF addresses; providing the so-called "carrier's carrier" service.  Hence the need for the maximum-paths 2 line.  If the VPN was only using connected and static routes, inside the AS, only the maximum-paths ibgp 2 line would be needed as the routes would all be internal.)

Close, but no cigar: only a single path was being used — looking on the ingress PE router:

DIST-HOSP#show ip cef vrf eng_vrf 129.169.10.10 detail 

129.169.10.0/24, epoch 0

  recursive via 192.84.5.234 label 52

    nexthop 192.84.5.29 Ethernet1/0 label 18

And BGP confirms only a single route is available:

DIST-HOSP#show bgp vpnv4 unicast vrf eng_vrf 129.169.10.10

BGP routing table entry for 64602:129:129.169.10.0/24, version 6

Paths: (1 available, best #1, table eng_vrf)

Flag: 0x820

  Not advertised to any peer

  Local

    192.84.5.234 (metric 15) from 192.84.5.240 (192.84.5.240)

      Origin incomplete, metric 0, localpref 100, valid, internal, best

      Extended Community: RT:64602:129

      Originator: 192.84.5.234, Cluster list: 192.84.5.240

      Connector Attribute: count=1

       type 1 len 12 value 64602:129:192.84.5.234

      mpls labels in/out nolabel/52

Odd - let's have a look at a core RR P router (note that we have to look up the route using the RD itself (64602:129 - a combination of our AS plus a local ID) as there is no VRF configured on a P router):

CORE-CENT#show bgp vpnv4 uni rd 64602:129 129.169.10.10

BGP routing table entry for 64602:129:129.169.10.0/24, version 5

Paths: (2 available, best #1, no table)

Flag: 0x820

  Advertised to update-groups:

        2

  Local, (Received from a RR-client)

    192.84.5.234 (metric 8) from 192.84.5.234 (192.84.5.234)

      Origin incomplete, metric 0, localpref 100, valid, internal, best

      Extended Community: RT:64602:129

      Connector Attribute: count=1

       type 1 len 12 value 64602:129:192.84.5.234

      mpls labels in/out nolabel/52

  Local, (Received from a RR-client)

    192.84.5.238 (metric 8) from 192.84.5.238 (192.84.5.238)

      Origin incomplete, metric 0, localpref 100, valid, internal

      Extended Community: RT:64602:129

      Connector Attribute: count=1

       type 1 len 12 value 64602:129:192.84.5.238

      mpls labels in/out nolabel/17

Both routes are present here, but only one is being selected: the one from 192.84.5.234 (marked as "best"), based on it having a lower IP address (given no other method of preference); the one from 192.84.5.238 is being discarded.

Distinguishing Router Distinguishers

This was a bit of a mystery until I found this post by Ivan Pepelnjak which described how RRs behave with multiple VPN routes and finally made the distinction between Route Distinguishers and Route Targets clear to me (and he admits it's not clearly explained in the Cisco textbooks):

• The Route Distinguisher (RD) is used, along with the IPv4 or IPv6 addresses of the router to build a complete route ID you can imagine being in the form "RD:route", e.g. "64602:129:129.169.10.0/24" (you can see this in the first line of the output from show bgp vpnv4 ..., above).  The purpose of the RD is to distinguish a route belonging to one VPN from another in the provider network.
• The Route Target (RT) specifies which which VPNs a particular route should be imported from or exported into, when a VRF is configured on a particular router.

The important point is that a RR will only reflect a single route with a particular ID: if the PE routers are all using the same RD, the RR will only use a single one of these routes.  Changes to this involve extensions to the BGP protocol which only appeared in IOS 15.2, with the router bgp ... / bgp additional-paths ... command.

Without this new capability, the solution to this is to use a different RD on each PE router, resulting in distinct routes in the provider network.  This made us revisit how we reassign RDs and RTs:

• We now set the RD to be the public IPv4 loopback address of the PE router, instead of using the local [private] ASN; RDs support this as a standard format and leave 16 bits for the administratively-assigned ID.  This changes our RDs from 64602:id to 192.84.5.x:id.
• On the other hand, the import and export RTs typically need to be the same for all VRFs across the VPN (unless partial imports are to be used).  For consistency with the RT, we've changed those from the same ASN:id format (64602:id, same as the RD) to 192.84.5.0:id (192.84.5.0/24 being the block we use for our router backbone and loopback addresses).

(Note that the address-family vpvn4 stanza does have a maximum-paths statement, but that doesn't appear to do anything useful: I'm not sure what the point of it is!)

Once changed, both routes now show up as best on the core router as completely separate (as far as it is concerned, they're completely separate routes, albeit with the same RT), searching by the new RD (it's only on the PE router where the two routes get brought together, when they're imported by the common RT into the VRF):

CORE-CENT#show bgp vpnv4 unicast rd 192.84.5.234:129 129.169.10.10

BGP routing table entry for 192.84.5.234:129:129.169.10.0/24, version 3

Paths: (1 available, best #1, no table)

  Advertised to update-groups:

     1         

  Refresh Epoch 1

  Local, (Received from a RR-client)

    192.84.5.234 (metric 8) from 192.84.5.234 (192.84.5.234)

      Origin incomplete, metric 0, localpref 100, valid, internal, best

      Extended Community: RT:192.84.5.0:129

      mpls labels in/out nolabel/54

      rx pathid: 0, tx pathid: 0x0

CORE-CENT#show bgp vpnv4 unicast rd 192.84.5.238:129 129.169.10.10

BGP routing table entry for 192.84.5.238:129:129.169.10.0/24, version 2

Paths: (1 available, best #1, no table)

  Advertised to update-groups:

     1         

  Refresh Epoch 1

  Local, (Received from a RR-client)

    192.84.5.238 (metric 8) from 192.84.5.238 (192.84.5.238)

      Origin incomplete, metric 0, localpref 100, valid, internal, best

      Extended Community: RT:192.84.5.0:129

      mpls labels in/out nolabel/56

      rx pathid: 0, tx pathid: 0x0

Looking on the PE router, both routes now appear and are marked as "multipath".  Note that the unique router ID contains the local RD for this VRF (192.84.5.237:129):

DIST-HOSP#show bgp vpnv4 unicast vrf eng_vrf 129.169.10.10

BGP routing table entry for 192.84.5.237:129:129.169.10.0/24, version 12

Paths: (2 available, best #2, table eng_vrf)

Multipath: iBGP

  Not advertised to any peer

  Refresh Epoch 1

  Local, imported path from 192.84.5.238:129:129.169.10.0/24 (global)

    192.84.5.238 (metric 15) from 192.84.5.240 (192.84.5.240)

      Origin incomplete, metric 0, localpref 100, valid, internal, multipath(oldest)

      Extended Community: RT:192.84.5.0:129

      Originator: 192.84.5.238, Cluster list: 192.84.5.240

      mpls labels in/out nolabel/56

      rx pathid: 0, tx pathid: 0

  Refresh Epoch 1

  Local, imported path from 192.84.5.234:129:129.169.10.0/24 (global)

    192.84.5.234 (metric 15) from 192.84.5.240 (192.84.5.240)

      Origin incomplete, metric 0, localpref 100, valid, internal, multipath, best

      Extended Community: RT:192.84.5.0:129

      Originator: 192.84.5.234, Cluster list: 192.84.5.240

      mpls labels in/out nolabel/54

      rx pathid: 0, tx pathid: 0x0

Finally, show ip cef ... will confirm that multipath is in use:

DIST-HOSP#show ip cef vrf eng_vrf 129.169.10.10 detail    

129.169.10.0/24, epoch 0, flags rib defined all labels, per-destination sharing

  recursive via 192.84.5.234 label 54

    nexthop 192.84.5.29 Ethernet1/0 label 33

  recursive via 192.84.5.238 label 56

    nexthop 192.84.5.29 Ethernet1/0 label 17

Final note about ECMP with MPLS

The other thing to note about MPLS is that it is the ingress PE router which determines the egress PE router, rather than the P routers.  The reason for this is that the outer label the ingress PE router places on the traffic is that of the egress PE router and thus it selects it (rather than the P routers, on a per-hop basis): the P routers simply forward traffic according to that label.

As such, multipathing using MPLS does not require any configuration on the P routers.