"mpls ip ttl-expiration pop 1"

I encountered an odd problem when tracerouting from outside the University network to a host in the data centre network in my GNS3 simulation:

HOST-JANETW#traceroute 131.111.8.31

Type escape sequence to abort.

Tracing the route to RAVEN.CAM (131.111.8.31)

  1 GW-JANETW.JANETW.JA (128.4.0.254) 76 msec 76 msec 72 msec

  2 JANETW.ENETW.NET (146.97.41.246) 196 msec 96 msec 112 msec

  3  *  *  * 

  4  *  *  * 

  5 DCW-CUDN.DCW-SRV.NET (193.60.88.2) 176 msec 180 msec 144 msec

  6 RAVEN.CAM (131.111.8.31) 188 msec 180 msec 184 msec

[For reference, these hostnames are not the same as the real ones, but the naming system works by using OTHER-ROUTER.THIS-ROUTER.DOMAIN - so JANETW.ENETW.NET is the IP address of ENETW on the link to/from JANETW.]

Hops 3 and 4 went missing, which are those to MILL from ENETW and then to DCW[-CUDN] from MILL; once we reach DCW-SRV from DCW[-CUDN] we get responses again.  (The two DCW names are because they're different VDCs on the same Nexus 7010.)  This is shown below, with the missing hops at the tips of the red arrows:

This puzzled me for about an hour and a half - I was checking routes and trying pings from the affected hosts back to the source, which were working OK.  I then starting poking about in more details to check what exactly happened to the packets...

Plain-IPv4-over-MPLS when you don't expect it

The first thing was to see what happened to the packet on ENETW by consulting CEF to see what it would do with a packet destined for 131.111.8.31:

BDR-ENETW#show ip cef 131.111.8.31 detail 

131.111.8.0/23, epoch 0

  recursive via 193.60.88.2

    nexthop 192.84.5.133 Ethernet1/1 label 33

This shows the packet was matching the 131.111.8.0/23 route which should be sent via 193.60.88.2 (the interface on DCW-SRV coming from DCW[-CUDN]), which is reached recursively by using a next hop of 192.84.5.133 (the interface on MILL at hop 3, coming in from ENETW).  The important bit is to note that the packet will be forwarded out of interface Ethernet1/1 with MPLS label 30.

Dissecting this (as an aside - just to check there wasn't anything completely screwy) - 131.111.8.0/23 is an eBGP route to AS65110 (the server network) learnt from 193.60.88.2:

BDR-ENETW#show ip route 131.111.8.31

Routing entry for 131.111.8.0/23

  Known via "bgp 64602", distance 200, metric 0

  Tag 65110, type internal

  Last update from 193.60.88.2 01:41:18 ago

  Routing Descriptor Blocks:

  * 193.60.88.2, from 192.84.5.240, 01:41:18 ago

      Route metric is 0, traffic share count is 1

      AS Hops 1

      Route tag 65110

193.60.88.2 is, in turn, reached via OSPF with next hop 192.84.5.133, as CEF showed:

BDR-ENETW#show ip route 193.60.88.2

Routing entry for 193.60.88.2/32

  Known via "ospf 1", distance 110, metric 55, type extern 1

  Last update from 192.84.5.133 on Ethernet1/1, 01:45:33 ago

  Routing Descriptor Blocks:

  * 192.84.5.133, from 192.84.5.248, 01:45:33 ago, via Ethernet1/1

      Route metric is 55, traffic share count is 1

So, back to the label - let's see what that's for:

BDR-ENETW#show mpls ip binding remote-label 33 neighbor 192.84.5.133

  193.60.88.2/32

        out label:    33        lsr: 192.84.5.250:0   inuse

Here, I didn't expect MPLS to be in use as the destination address of the packet isn't one that we advertise labels for via LDP: we limit those to the backbone router loopback addresses and the outside addresses of eBGP peers to which we have inter-AS MPLS VPNs.

However, as our server network does operates as a separate AS from the backbone and we use MPLS forwarding between it and the backbone, we do advertise labels for its outside addresses - including 193.60.88.2/32: ENETW has labelled the packet for forwarding across the backbone to DCW-SRV because it has a BGP next hop with a label available.

Next step is to work out what happens to this MPLS frame...

Traceroute over an MPLS network

So, assuming that ENETW does the correct thing and sends the packet on via MPLS to MILL, lets see what that does by debugging ICMP and trying the traceroute again:

CORE-MILL#debug ip icmp

ICMP packet debugging is on

CORE-MILL#

Apr  8 19:53:01.591: Adding 4 bytes of label stack

Apr  8 19:53:01.595: MPLS: ICMP: time exceeded (time to live) sent to 128.4.0.1 (dest was 131.111.8.31)

This shows the MPLS frame is received and the TTL expired (by default, the PE router which turns the packet from a plain IPv4 packet into a labelled MPLS frame will copy the increasing TTL [as generated by traceroute]).  The P router (MILL) unwraps the packet inside and generates the 'TTL exceeded' ICMP message.

The critical bit, however, is that the P router will then label the packet up with its original MPLS label and forward it on to the destination (with a TTL reset to 255); it does NOT return it directly to the sender.  It has to do this as the IP addresses using the MPLS frame may not be ones which exist in the same routing space as the P router.

I verified all this by running Wireshark on the virtual interfaces inside GNS3 (incredibly useful that!) and checking it all looked the way I expected.

This I knew and understood and had investigated in the past (although not in an inter-AS situation) - what I hadn't thought of was what happens next...

The egress PE router

When the egress PE router receives the labelled ICMP 'TTL exceeded' packet, it unwraps it and tries to forward it on to the destination.  In this case, the PE router is DCW-SRV (in the server network) trying to return a packet with a source address of 192.84.5.133 to 128.4.0.1 (HOST-JANETW; an imaginary host, the one my traceroute was initiated from).  This is where things fail...

Because of the recent IP unicast RPF verification testing I've been doing, that source address is blocked from entering the backbone from the server network by the distribution routers to prevent address spoofing: the server network is not allowed to originate addresses it is not responsible for routing.

This can be confirmed by looking on the distribution router, DCW[-CUDN], part of the backbone network:

DIST-DCW#show ip int e1/2 | begin verify

  IP verify source reachable-via RX, ACL 10

   108 verification drops

   0 suppressed verification drops

   0 verification drop-rate

Some debugging shows the naughty source address:

DIST-DCW#debug ip cef drops rpf         

IP CEF drops for RPF debugging is on

DIST-DCW#

*Apr  8 20:08:38.171: CEF-Drop: Packet from 192.84.5.133 via Ethernet1/2 -- via-rx

DIST-DCW#

In the words of Colonel Hans Landa: that's a bingo!  The whole thing is illustrated below, with the rejected ICMP message from the DCW-SRV PE router being marked with a X in red:

MPLS Traceroute showing ICMP message from PE router being rejected due to RPF

It also explains why I'd never seen this problem with the Computer Laboratory in my simulation before, as I'd never bothered blocking traffic from other ASs to prevent address spoofing: I'd only ever tested filtering prefixes over eBGP; traffic filtering is something we've always done on the real network through access lists and not of interest in the simulation.

Solving the [non] problem

Once I'd worked out what the actual problem was, solving it turned out to be trivial...

In reality, this problem turns out to be academic: on our real network, we run with "no mpls ip propagate-ttl forwarded" configured on our PE routers, which prevents the TTL of the incoming packet from being copied into the MPLS frame.  This means that the TTL should never expire when crossing the backbone and those hops simply won't show up in a traceroute (so successful hops 5 and 6 will become hops 3 and 4).

However, in my GNS3 simulation, I like to expose the intermediate hops to verify complete paths from end station hosts for diagnostic and investigatory purposes.  Indeed, the reason I spotted the problem was whilst testing the paths for traffic when various failures occur, by shutting down partso the simulation.

The solution to this is to use the "mpls ip ttl-expiration pop 1" command - this causes MPLS frames with only a single label depth (which will be plain IPv4-over-MPLS frames) to be returned directly to the sender by the P router, rather than forwarding it on to the PE router to be bounced back.  As such, there is no problem with address spoofing as the IP packet starts its life inside the backbone itself.

After doing the later, my traceroute works:

HOST-JANETW#traceroute 131.111.8.31

Type escape sequence to abort.

Tracing the route to RAVEN.CAM (131.111.8.31)

  1 GW-JANETW.JANETW.JA (128.4.0.254) 16 msec 16 msec 8 msec

  2 JANETW.ENETW.NET (146.97.41.246) 20 msec 52 msec 20 msec

  3 ENETW.MILL.NET (192.84.5.133) [MPLS: Label 33 Exp 0] 68 msec 52 msec 24 msec

  4 MILL.DCW.NET (192.84.5.154) [MPLS: Label 16 Exp 0] 48 msec 68 msec 48 msec

  5 DCW-CUDN.DCW-SRV.NET (193.60.88.2) 80 msec 88 msec 68 msec

  6 RAVEN.CAM (131.111.8.31) 104 msec 56 msec 112 msec

You can even see the MPLS labels used in the previously-missing hops being returned in the ICMP extension block.

One thing that this has shown up is that, once the new data centre network is installed, traceroutes into our server network will not expose the backbone network router hops.  That wasn't something I was expecting and could be confusing for others.

Cisco Nexus data centre network and enabling MPLS

We're just completed a tender for a new data centre network and select a design based on one of Cisco's standard models consisting of core Nexus 7010s in two physically diverse data centre locations, with each row of racks consisting of a pair of Nexus 56128Ps as the EoR switches and various Nexus 2k models as the ToR units.

Despite having plenty of experience with IOS (and, to a lesser extent, IOS XE), we've never had Nexus equipment before, in particular not used NX-OS.  As such, there's a bit of a learning curve to get things up and running.

This post (and probably some following ones) is what we've encountered in the setup of the Nexus equipment and will likely cover VDCs on the 7010s, vPC (Virtual Port-Channel) on the 7010s and 56128Ps and the FEXs, as well as some NX-OS differences.

MPLS

Despite getting some licence sheets with PAKs (Product Activation Keys) for the MPLS feature set, these appear to have already been installed.  However, the feature isn't activated by default and requires some work to do so.

First, in the admin VDC, the feature set must be installed:

n7k(config)# install feature-set mpls

Then, in the service VDCs (our backbone routing-only one is "cudn", the Cambridge University Data Network), the feature-set must be activated, along with the desired individual features:

n7k-cudn(config)# feature-set mpls
n7k-cudn(config)# feature mpls ldp
n7k-cudn(config)# feature mpls l3vpn

The MPLS features then become available in that VDC.

IPv4 multicast over an MPLS VPN using mGRE

I've wanted to sort out multicast forwarding over an MPLS VPN for some time — while most institutions at the University of Cambridge don't have multicast-enabled networks, it seems a loose end that might prevent take up of the service.

After some poking about, trying to find free stuff online and failing, it seemed like MPLS and VPN Architectures vol. II from CiscoPress would explain it: in particular Chapter 7 ("Multicast VPN").  The good news is that it does.

However, the book dates from 2003 and I found an update document on Cisco's website about developments in MBGP to handle multicast over MPLS VPNs from about 2008 ("Multicast VPN - IP Multicast Support for MPLS VPNs"); there was a reprint of the book in paperback a month or so ago (late June 2014) and I thought it might have been updated to cover that.  Unfortunately, the book hasn't been updated, although the changes are minimal so it doesn't hurt to start with the book and then read the update.

Note: the solution described here uses mGRE (multipoint GRE); replacing much of this is MLDP (Multicast Label Distribution Protocol)   The latter looks like the right solution in the longer term but is fairly recent (dating from 2010) and needs IOS 15 on the Cisco platform, I believe.  I'll look at that later.

Anyway — onto the technical bits...

The key facts

There are several key things to know in advance, which I think helps understand multicast over an MPLS VPN:

• Firstly, it doesn't actually involve MPLS: multicast traffic is forwarded completely differently from unicast traffic and doesn't involve any MPLS labelled frames.
• Instead, multicast traffic inside the VPN is tunnelled, by encapsulating it using GRE, inside multicast traffic in the global space.  This is commonly called 'mGRE' (multipoint GRE).
• In the global space, one or more MDTs (Multicast Distribution Trees) are constructed to carry this tunnelled traffic between the PE routers in the VPN.  When traffic is received on one of these trees, it is de-encapsulated and forwarded to the VRF (Virtual Routing and Forwarding) instance for the VPN.  Inside the VRF, traffic is forwarded into the MDT, when it needs to reach other routers in the VPN.
• MDTs essentially put all the participating routers on a single layer 2 segment: when querying the PIM neighbours on the tunnel interface created by an MDT, all of the other routers will be seen.
• The first MDT is known as the Default MDT — all PE routers in the VPN will join this tree and it will, by default, be used to forward all multicast traffic on the VPN.
• If there is a lot of multicast traffic and it is not typically required on all the PE routers (because only some have members) then this is inefficient.  To handle this situation, Data MDTs can be configured to set up separate groups in the global space — Cisco let you do this for groups where the bandwidth exceeds a certain threshold and/or with an address matched by an access list.
• The Data MDTs use a pool of group addresses which is automatically recycled as groups come and go, much like a DHCP address pool.  If the pool is exhausted, group addresses will be reused on a least-used (by mapping of group addresses) basis.
• The indication that a particular group has been switched from the Default MDT to the Data MDT is done by a PIM message that is forwarded across the Default MDT.
• Note that the decision to switch traffic to a Data MDT is done by the ingress router (nearest the source), rather than the final hop router (as would be the case with SPT switchover under regular PIM).
• If the groups for the Default MDTs are in a PIM-SSM range (232.0.0.0/8, by default), the BGP address-family ipv4 mdt can be used to locate other PE routers in the MVPN.  This is not necessary if the groups use PIM-SM with an RP.
• The VPN must be configured to provide a BSR and RP in the normal manner: these work as normal across the MDTs (so a single BSR and RP can be set up on a PE for the VPN, or even inside the customer network).
• The interfaces to CE routers work just as normal and also support BSR and RP discovery.

This arrangement is designed to be scalable and limit control plane resources in the provider network: the Data MDT group address pool allows the provider to prevent a large number of customer groups in the VPN from creating a similar number on the P routers, limiting the effect to just the PE routers.

All this business requires a good understanding of how multicast works and requires careful tuning of PIM modes (SM, SSM, Bidir) as well as the MDT group addresses, to get the best performance.

Keeping traffic separate

The main alarm bell I had with all this was about keeping traffic inside the VPN from escaping onto the global network, where it could be intercepted, or false traffic inserted.

This problem is actually straightforward: Cisco provide the ip multicast boundary command which is used to filter multicast traffic on an interface — if the group addresses used by the MDTs are in a range not permitted by the list specified with this, they will not pass across the interface, keeping the traffic safe.

In our case, we filter 239.255.0.0/16 as being 'institution private' so it doesn't ever pass across an interface between an edge subnet or institution and our backbone.  If the MDTs are created in here, they should be safe.

Testing multicast boundaries

To check this all worked, I did some tests with this using GNS3 and Wireshark sniffing the virtual link between a host in the global space, which had joined the Default MDT group.  By switching the filtering on and off, I could watch the effect...

• Installing a boundary had an immediate affect, stopping the MDT traffic from reaching the host.  The interface connecting the host disappeared from the output interface list of show ip mroute.
• However, the group still showed up as being joined in the output of show ip igmp group, even though traffic did not pass.  This would remain there until the IGMP reporting timer expired; other groups would refresh, but not this one
• When the boundary was removed, traffic would not immediately flow, even if the interface remained in the IGMP list.  However, when the IGMP membership report was retransmitted by the host, it caused the traffic to start flowing to the host again.

Example

(Update 2015-050-24 - I'm about to convert my GNS3 simulation over to MLDP, now I've upgraded to IOS 15, so I thought I'd do some tests using mGRE before dismantling it.)

A quick demo of this all in action and the output of some show commands.

Configuration

The configuration below builds on a working [non-VRF] PIM-SM on the backbone, as well as an MPLS IPv4 VPN.  I'm doing this in IOS 15.

The MDTs are configured in the VRF, identically on each router, in the address-family block - the extra commands highlighted:

vrf definition eng_vrf

 rd 192.84.5.238:129

 route-target export 192.84.5.0:129

 route-target import 192.84.5.0:129

 !

 address-family ipv4

  mdt default 239.255.32.1

  mdt data 239.255.36.0 0.0.0.31

  mdt data threshold 10

 exit-address-family

!

ip multicast-routing vrf eng_vrf

I've deliberately set the Data MDT switchover threshold high to avoid it being triggered, initially.

Note that IOS 15 replaces the mdt data ... threshold ... option with a separate mdt data threshold command that operates per-VRF.  If you use the old format, you will receive a warning that it's deprecated.

Once enabled on the router, a VRF interface is enabled the same as a non-VRF one:

interface Ethernet1/2.210

 description eng-vpn-nms

 encapsulation dot1Q 210

 vrf forwarding eng_vrf

 ip address 129.169.10.253 255.255.255.0

 !

 ip pim bsr-border

 ip pim sparse-mode

 ip igmp version 3

Checking the Default MDT

Once this has been configured on all routers, the Default MDT should be formed across the backbone (using 239.255.32.1, in the above example) and they should all discover each other as if on a single network segment using PIM:

DIST-NMS#show ip pim vrf eng_vrf neighbor 

PIM Neighbor Table

Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,

      P - Proxy Capable, S - State Refresh Capable, G - GenID Capable

Neighbor          Interface                Uptime/Expires    Ver   DR

Address                                                            Prio/Mode

192.84.5.236      Tunnel3                  00:00:15/00:01:35 v2    1 / S P G

192.84.5.237      Tunnel3                  00:00:39/00:01:29 v2    1 / S P G

The above router - DIST-NMS - has found the other routers across the Default MDT on the backbone using the Tunnel3 interface; their loopback addresses in 192.84.5.x are shown.  No DR is shown as that is us (our loopback is 192.84.5.238 - the highest address):

DIST-NMS#show ip pim vrf eng_vrf int tu3        

Address          Interface                Ver/   Nbr    Query  DR     DR

                                          Mode   Count  Intvl  Prior

192.84.5.238     Tunnel3                  v2/S   2      30     1      192.84.5.238

You can check the MDTs and their tunnel interfaces as follows:

DIST-NMS#show ip pim vrf eng_vrf mdt

  * implies mdt is the default MDT

  MDT Group/Num   Interface   Source                   VRF

* 239.255.32.1    Tunnel3     Loopback0                eng_vrf

You can see the Default MDT group is visible in the global IP multicast forwarding table - note the Z flag, showing the group is used as an mGRE multicast tunnel:

DIST-NMS#show ip mroute 239.255.32.1

IP Multicast Routing Table

Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,

       L - Local, P - Pruned, R - RP-bit set, F - Register flag,

       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,

       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,

       U - URD, I - Received Source Specific Host Report, 

       Z - Multicast Tunnel, z - MDT-data group sender, 

       Y - Joined MDT-data group, y - Sending to MDT-data group, 

       G - Received BGP C-Mroute, g - Sent BGP C-Mroute, 

       Q - Received BGP S-A Route, q - Sent BGP S-A Route, 

       V - RD & Vector, v - Vector

Outgoing interface flags: H - Hardware switched, A - Assert winner

 Timers: Uptime/Expires

 Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 239.255.32.1), 00:06:39/stopped, RP 192.84.5.240, flags: SJCFZ

  Incoming interface: Ethernet1/0, RPF nbr 192.84.5.33

  Outgoing interface list:

    MVRF eng_vrf, Forward/Sparse, 00:06:39/00:02:20

(192.84.5.236, 239.255.32.1), 00:05:40/00:01:10, flags: JTZ

  Incoming interface: Ethernet1/0, RPF nbr 192.84.5.33

  Outgoing interface list:

    MVRF eng_vrf, Forward/Sparse, 00:05:40/00:00:19

(192.84.5.237, 239.255.32.1), 00:06:04/00:00:59, flags: JTZ

  Incoming interface: Ethernet1/0, RPF nbr 192.84.5.33

  Outgoing interface list:

    MVRF eng_vrf, Forward/Sparse, 00:06:04/00:02:56

(192.84.5.238, 239.255.32.1), 00:06:39/00:02:54, flags: FT

  Incoming interface: Loopback0, RPF nbr 0.0.0.0

  Outgoing interface list:

    Ethernet1/0, Forward/Sparse, 00:06:38/00:02:50

The PIM domain inside the VRF

The multicast forwarding table for the VRF is initially empty:

DIST-NMS# show ip mroute vrf eng_vrf

IP Multicast Routing Table

...

(*, 224.0.1.40), 00:59:21/00:02:14, RP 0.0.0.0, flags: DCL

  Incoming interface: Null, RPF nbr 0.0.0.0

  Outgoing interface list:

    Ethernet1/2.210, Forward/Sparse, 00:59:20/00:02:14

In my simulation, the PIM-SM RP is on a router in an institutional (departmental) network, not part of the MPLS backbone.  This is advertised as normal using PIM BSR and detected over the MDT:

DIST-NMS#show ip pim vrf eng_vrf rp mapping 

PIM Group-to-RP Mappings

Group(s) 224.0.0.0/4

  RP 129.169.252.1 (TRUMP.ENG), v2

    Info source: 129.169.252.1 (TRUMP.ENG), via bootstrap, priority 64, holdtime 150

         Uptime: 00:17:01, expires: 00:02:12

Adding group members

I'm using 234.131.111.10 as a test group.  First, I set up two group members on institutional networks using ip igmp join-group ....  Here's one:

interface FastEthernet0/0

 description eng-vpn-medschl

 ip address 129.169.86.86 255.255.255.0

 ip igmp join-group 234.131.111.10

These show up in the multicast forwarding table on the RP, ENG-TRUMP:

ENG-TRUMP#show ip mroute 234.131.111.10

IP Multicast Routing Table

...

(*, 234.131.111.10), 00:24:05/00:03:14, RP 129.169.252.1, flags: SJC

  Incoming interface: Null, RPF nbr 0.0.0.0

  Outgoing interface list:

    Ethernet1/0.252, Forward/Sparse, 00:21:03/00:03:14

    Ethernet1/2, Forward/Sparse, 00:24:05/00:01:58

On that router, Ethernet1/2 is the link to the backbone router which is inside the VRF (and has the above configured member) and Ethernet1/0.252 has another, local member.

Adding a source

Now I can start a source by setting a ping going on an end host (with IP 129.169.10.10) to the group address and see the responses from the two members:

HOST-ENG-NMS#ping 234.131.111.10 repeat 1000

Type escape sequence to abort.

Sending 1000, 100-byte ICMP Echos to 234.131.111.10, timeout is 2 seconds:

Reply to request 0 from HOST-TRUMP.ENG (129.169.254.1), 444 ms

Reply to request 0 from HOST-MEDSCHL.ENG (129.169.86.86), 572 ms

Reply to request 1 from HOST-MEDSCHL.ENG (129.169.86.86), 208 ms

Reply to request 1 from HOST-TRUMP.ENG (129.169.254.1), 404 ms

...

The multicast forwarding table on the first hop router will show traffic being forwarded over the tunnel:

DIST-NMS#show ip mroute vrf eng_vrf 234.131.111.10

IP Multicast Routing Table

...

(*, 234.131.111.10), 00:00:10/stopped, RP 129.169.252.1, flags: SPF

  Incoming interface: Tunnel3, RPF nbr 192.84.5.236

  Outgoing interface list: Null

(129.169.10.10, 234.131.111.10), 00:00:10/00:03:22, flags: FT

  Incoming interface: Ethernet1/2.210, RPF nbr 0.0.0.0

  Outgoing interface list:

    Tunnel3, Forward/Sparse, 00:00:09/00:03:20

Because there's not a sufficient level of traffic (above the configured 10kbit/s), no Data MDT has been created - we can check on the first hop router:

DIST-NMS#show ip mroute vrf eng_vrf 234.131.111.10 active  

Use "show ip mfib active" to get better response time for a large number of mroutes.

Active IP Multicast Sources - sending >= 4 kbps

DIST-NMS#show ip pim vrf eng_vrf mdt send                  

DIST-NMS#

And on a receiving router:

DIST-HOSP#show ip pim vrf eng_vrf mdt receive detail 

DIST-HOSP#

Increasing the data rate

Now everything's working, let's increase the rate of traffic by making the ping send packets without the 1s delay between them:

HOST-ENG-NMS#ping 234.131.111.10 rep 100000 timeout 0

Type escape sequence to abort.

Sending 100000, 100-byte ICMP Echos to 234.131.111.10, timeout is 0 seconds:

......................................................................

......................................................................

...

And check the traffic data rate:

DIST-NMS#show ip mroute vrf eng_vrf 234.131.111.10 active 

Use "show ip mfib active" to get better response time for a large number of mroutes.

Active IP Multicast Sources - sending >= 4 kbps

Group: 234.131.111.10, (MCAST-131-111-10.CAM)

   Source: 129.169.10.10 (HOST-NMS.ENG)

     Rate: 141 pps/113 kbps(1sec), 113 kbps(last 40 secs), 3 kbps(life avg)

Now the data rate has gone above the configured 10kbps (kbit/s) threshold, a Data MDT should have been created and traffic switched across to it.  This can be confirmed on the first hop router, along with the address of the new group:

DIST-NMS#show ip pim vrf eng_vrf mdt send                 

MDT-data send list for VRF: eng_vrf

  (source, group)                     MDT-data group/num   ref_count

  (129.169.10.10, 234.131.111.10)     239.255.36.0         1

On the egress (from the mGRE cloud) router, this can be confirmed:

DIST-HOSP#show ip pim vrf eng_vrf mdt receive detail 

Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,

       L - Local, P - Pruned, R - RP-bit set, F - Register flag,

       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,

       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,

       U - URD, I - Received Source Specific Host Report, 

       Z - Multicast Tunnel, z - MDT-data group sender, 

       Y - Joined MDT-data group, y - Sending to MDT-data group, 

       G - Received BGP C-Mroute, g - Sent BGP C-Mroute, 

       Q - Received BGP S-A Route, q - Sent BGP S-A Route, 

       V - RD & Vector, v - Vector

Joined MDT-data [group/mdt number : source]  uptime/expires for VRF: eng_vrf

 [239.255.36.0 : 192.84.5.238]  00:04:56/00:00:36

  (129.169.10.10, 234.131.111.10), 00:10:39/00:01:44/00:00:36, OIF count: 1, flags: JTY

Once the source stops and the MDT times out (about 3 minutes), the Data MDT will then be destroyed.