We needed to determine the cause of some faults on our server network which is based on Nexus 7010 equipment. The ERSPAN ("Encapsulated Remote Source Port ANalyzer") feature proved excellent for this for a number of reasons:
monitor erspan origin ip-address 192.84.5.248 global
- We didn't need to physically go to the data centre to do the monitoring. The monitoring is directed across an IP tunnel to the capture device, which can be anywhere else on an IP network.
- ERSPAN (and SPAN in general) on the Nexus platform has very good filtering capability so we can select just the traffic we want and monitor it on a machine with significantly less bandwidth than the capture device (in my case, my office machine monitoring 8x 10Gbit/s links looking for odd packets!).
- Related to the above, actually just physically connecting the 10Gbit/s ports to my laptop would be challenge, ignoring whether it can keep up with the traffic on them!
Setting up ERSPAN
In the Admin VDC (regardless of which VDC traffic is to be monitored in), the following global command needs to be set to define the source IP address for the packets. This address does not have to available to the particular VDC where the monitoring is done:monitor erspan origin ip-address 192.84.5.248 global
The actual mirroring can then be set up as follows:
| Command | Meaning |
|---|---|
| monitor session 1 type erspan-source | Create an ERSPAN session number 1 |
| erspan-id 18 | ID number transmitted in the ERSPAN packet to identify this particular SPAN |
| vrf default | VRF into which ERSPAN traffic is to be transmitted to the destination (not the VRF being monitored) |
| destination ip 192.0.2.1 | IP address of destination (collector of ERSPAN traffic) |
| source interface port-channel789 both | Interface to be monitored |
| filter vlan 61,62 | VLANs to be selected from the above interface (to monitor a whole VLAN, use 'source vlan ...' instead). Filters can also reference access lists and other criteria. |
| filter access-group FILTER-VLANACCMAP | VLAN access-map to use to filter traffic (see below). |
| no shut | Enable the monitor |
Filtering using an ACL
If you want to filter traffic matching things like source IP addresses, port numbers, etc. you can use an access-list to match it. The Cisco documentation isn't clear on this, and contradicts how it's done, but a good reference is here.
In short, you must create an access-list, then a vlan access-map and apply that to the ERSPAN session. For example:
ip access-list ERSPAN-TRAFFIC-ACL4
permit udp host 192.0.2.6 any eq 53
!
vlan access-map ERSPAN-TRAFFIC-VLAM 10
match ip address ERSPAN-TRAFFIC-ACL4
!
monitor session 1 type erspan-source
filter access-group ERSPAN-TRAFFIC-VLAM
Note that, despite the command "filter access-group ..." you MUST use a VLAN access-map (at least on 7.3.1 and above).
Capturing the traffic
Once enabled, the router will send the monitored traffic to the target host. This will arrive as GRE traffic with protocol type 0x88be (ERSPAN). Following the GRE header will be the ERSPAN header, which contains things like the VLAN ID and ERSPAN ID, then the entire L2 frame.
Wireshark recognises the packets as ERSPAN packets without any configuration, and will correctly decode them to show the encapsulated packet directly, allowing you to watch things in real time from the comfort of your desk!
