Thursday 1 February 2018

DHCP, RPF verify, FHRP and ECMP - the final solution

A while back, I wrote about a problem with DHCP relaying, when using an FHRP (with multiple routers) and ECMP, whilst also using RPF source verification.

My solution at the time was to use an access control list to do the RPF source verification, with some exceptions to allow the DHCP packets to be relayed across the client subnet, from the "other" router.  The RPF source verification feature ("ip verify unicast source ..." command) caused problems on the Supervisor 720s we had at the time), when used with an exception.

Since then, we've upgraded our backbone network to Catalyst 6807-XLs, all with Supervisor 6Ts and IOS 15.4.  These updated routers resolve the issue with the "ip verify unicast source ..." command, when used with an "exception" ACL (which allows packets matching the ACL to violate the RPF source verification check) — it used to punt the majority of packets up to the CPU and cause high load through the IP Input process (only the "exception" packets were handled on the ASIC).  However, the new Supervisor 6Ts (and the Supervisor 2Ts, which we've never used) do all of this in the ASICs and avoid the CPU issue.

As part of the upgrade, we redid all of the router configurations from scratch, rather than just adapting the existing configuration files with a bit of editing plus search and replace.  This gave me an opportunity to revisit this.

Refresher — how RPF source verification works


Just as a reminder, putting RPF source verification is enabled by adding a command under the SVI definition, e.g.:

interface Vlan789
 description BOTOLPHS
 ip address 172.27.89.253 255.255.255.0
 standby version 2
 standby 81 ip 172.27.89.254
 ip helper-address 172.31.1.1
 ip verify unicast source reachable-via rx

This stops any packets being admitted via this interface, unless they would normally be sent out via that interface.  This will include directly-routed/connection routes and static routes via the interface.

However, this breaks DHCP, when relayed through the "other" router (e.g. one also serving VLAN 789, on 172.27.89.252 and running HSRP to protect address 172.27.89.254) because the packet will come from the DHCP server (172.31.1.1), in via the VLAN 789 SVI on the other router, to 172.27.89.253 (to address this router relayed the packet to the DHCP server from) and so will have a source address of 172.31.1.1, which isn't reached via the interface.

To work around problems like this, the "ip verify unicast source ..." command allows you to specify an access list to exempt packets from this check, so we can use to to allow the DHCP packets.  Unfortunately, it must be a NUMBERED access list, so you're limited in the number you can create (100-199 and 2000-2699 for extended access lists), plus it's difficult to pick sensible numbers for them, given the constraints on these in IOS.

The number scalability problem


That the ACL is limited to being a numbered one creates a bit of a problem: what we really wanted to do was create an ACL per interface, to permit the packets for DHCP on that interface, e.g. for the above:

ip access-list extended 2089
 permit udp host 172.31.1.1 eq bootps 172.27.89.0 0.0.0.255 eq bootps

(Note that DHCP packets going from the server to relay agents go from port 67 ["bootps" — BOOTP server] to the same port; packets going from relay agents or servers to clients go from 67 to 68 ["bootpc" — BOOTP client].)

The problem with this is that we'd need one access list per interface and we only have a relatively few numbers to play with.  (In practice, there are probably enough numbers per router, but managing the set of them would be difficult, as we need to know which numbers are free, since we can't just pick names/numbers based on the VLAN ID, due to the range limitations.)

The solution


The workaround to this problem we came up with was to combine the RPF source verification feature with an access list.  We start by creating numbered access lists per set of DHCP servers and don't limit the destination address to this VLAN but, instead, put "any":

ip access-list extended 2031
 permit udp host 172.31.1.1 eq bootps any eq bootps

This exemption would obviously leave the SVI open to a host spoofing replies from the DHCP server to other router interfaces (on this router, or elsewhere on the network).  To protect against this, we also use an inbound ACL:

ip access-list extended VL789-IN-ACL4
 permit udp host 172.31.1.1 eq bootps 172.27.89.0 0.0.0.255 eq bootps
 deny udp host 172.31.1.1 eq bootps any eq bootps
 ...

... the access list needs to:
  • permit the relayed packets for the primary IPv4 range on this VLAN (it doesn't need to cope with secondary ranges, since those will always be relayed via the primary address on the router), then
  • deny all other packets from the DHCP server's address
We can then apply these to the interface:

interface Vlan789
 ip address 172.27.89.253 255.255.255.0
 ...
 ip helper-address 172.31.1.1
 ...
 ip verify unicast source reachable-via rx 2031
 ip access-group VL789-IN-ACL4 in
 ...

Because the RPF access list (2031) doesn't specifically mention the addresses used on this VLAN, it can be reused across multiple VLANs/subnets.  As such, we only need one access list per set of DHCP servers, which is manageable.

[In fact, in our case, we actually limit the per-VLAN inbound access list to just accepting packets to this particular router's primary IPv4 address on the VLAN, since the configuration build mechanism we have knows what that is.  As such, the access list on each router is slightly different, for the same VLAN.  However, this is probably unnecessary — it's just something we can do easily.]

NX-OS strikes again


The Cisco Nexus equipment we have (which runs NX-OS 7.3) also have the RPF source verification command but DOESN'T have the exception access list feature.  This means we can't use it, where DHCP relaying is in use — instead, we need to used regular access lists, as we did on the Supervisor 720s.

[If vPC (Virtual Port-Channel — Cisco's version of MLAG) NX-OS is supposed to process traffic addressed for the addresses of the partner bridge, for VLANs which are members of vPC.  However, this didn't seem to work for DHCP relaying — either it only applies to traffic arriving via the vPC member port (which Cisco said it didn't, when dealing with them about something else), or DHCP is an exception.  I haven't had time to look at this, yet.]

Sunday 21 January 2018

Controlling the CBR from a Texas Instruments TI-89 / TI-92 / Voyage 200

OK.  Slightly odd one this, but I'm writing it up here for three reasons:
  1. I've got a load of bits of paper on my desk at home with scribbles about it and I want to chuck them away, and
  2. I couldn't find this information anywhere on the web already, and
  3. I can't think of anywhere better to put it
I don't know if anyone else will be interested, but someone may get some use out of it.

Background


I have some old Texas Instruments calculators and a CBR 2 ultrasonic "Calculator-Based Ranger", which is designed for school classroom use to do experiments which measure and undertand distance/velocity/acceleration data at rapid intervals and then allow students to plot graphs and analyse the data on their calculators.

TI supply some software for these devices - EasyData on the TI-83/84 and a program called Ranger for the TI-89/92/Voyage 200 series.  The former is fine: easy to use and powerful.  However, the latter software I found particularly awkward to use and, seeing as it was just a program written in the TI BASIC on the calculator, which you can view and edit, I thought I might be able to do a better job of it, if only I could understand how it worked.

What's here is the result of me decoding the Ranger program and reverse engineering the protocol which the calculator uses to talk to the CBR 2.  This allowed me to write a much better program (in my opinion!) which I called "CBR Logger" — I'm quite happy to supply a copy of that to anyone that wants it.

To give you an idea about what that program, and the CBR can do, here are some screen shots from a TI-92 Plus showing the data captured from me bouncing a (soccer) football on the wood floor of my living room (processed in "Ball bounce" mode):

Logging setup dialog
Time/distance graph being traced
Time/velocity graph
Time/distance+velocity combined graph with menu

I must say, the hardest part about this wasn't reverse engineering the protocol so much as understanding the Ranger program: it's either been written to be as compact as possible and includes lots of weird hacks, or someone deliberately obscured it to stop people trying to understand it!

Note that I've got a CBR 2: I haven't got a CBR (original model), so I don't know if this applies to that too.  However,  I think it probably should, as the Ranger software claims to support both.  For brevity in these instructions, I'm just going to refer to both units as the "CBR", even though all my testing is with a CBR 2.

Basic operation


The CBR 2 connects to the calculator in one of two ways:
  • Via the standard TI calculator link cable — the 2.5mm stereo jack cable, which you use to connect two calculators together and transfer data and programs, or
  • On the TI-89 Titanium, you can use the USB B (CBR end) to USB C (calculator end) cable.
The end result is the same as the calculator hides these differences from programs written in TI BASIC.

To send data to the CBR, the "Send" command is used, which takes a list of numbers as a TI BASIC list (e.g. "Send {1,11,2,2}"}.  In the case of the CBR, the first one or two numbers specify the command number and the numbers following the parameters for that command.

To retrieve data from the CBR (either logged data, or status information about the CBR itself), the "Get" command is used.  This takes an argument which is the name of a variable to retrieve a list of numbers into.  What these numbers mean depends on the command that has been sent first.

The CBR can operate in two main modes:
  • Logging mode — the parameters for logging are set up by the calculator (interval between samples and number of samples required) and the CBR primed to sample.  When started, the CBR will log the sampled data points into its internal memory (up to 512 points' worth) and then stop.  The calculator can then retrieve the data from the internal memory of the CBR.  This method allows the CBR to be disconnected from the calculator during the experiment and reconnected, once the data has been logged, so it can be retrieved.
  • Real-Time mode — the CBR starts sampling and the calculator pulls down the data from it, as it is sampled.  The CBR doesn't record any of this information internally and so can sample data for as long as its batteries last.  The calculator can also process this data as it's captured, perhaps to draw a graph immediately, whilst the experiment is in progress.  However, the calculator cannot capture data particularly quickly, making it unsuitable for high speed events.  In addition, the calculator must obviously be connected to the CBR throughout the experiment.

Commands


The following table gives an overview of the commands that I've found (with my names for them):

NumberCommand
0Reset And Clear
1Stop or Set Up Sampling
3Start Sampling
5Select Logged Data
6Shutdown or Smooth Logged Data
7Get CBR Status

The exact meaning of these commands and the arguments they take are described below.

Of course, there may be more commands or options to what I've found, but I am, of course, reverse engineering what's in the Ranger program, and that may not use all the features.

Command 0 — Reset And Clear


The first command is {0} - it takes no parameters and resets the CBR, clearing any logged data from the internal memory of the CBR.

If invalid commands or parameters to commands are sent, sometimes the CBR gets in an error state and refuses to accept any commands until command 0 is sent to reset it.  When it's in this state, it usually flashes the LED on the front red, whenever a command is sent to it.


Command 1,0 — Stop


This command takes no parameters and just stops the CBR if it is logging in real-time mode (following command 3).


Command 1,11 — Set Up Sampling


This command is tightly integrated with command 3 (Sample).  It's unclear why the parameters specified here are separated from those with command 3, but that's the way it is!

The command is in the form:

{1,11,mode,sets}

The mode parameter is a bitwise field and contains and takes a value which is calculated as:

2 + (0 = log in CBR, or 4 = real time) + (0 = metres, 1 = ft)

The difference between logging and real-time modes is described above.

Note that the CBR will actually log in either meters or feet: it's not the software on the calculator which handles conversion between the different units (although it obviously could).

The sets parameter controls which data sets are returned in the cycle of retrieving with Get (see Getting Data, below) and is one of three values: 0 (distance only), 1 (distance and velocity) or 2 (distance, velocity and acceleration).  Choosing a mode which logs less information (e.g. 0 - distance only) doesn't free up any storage for more data points, so there isn't any particular benefit in limiting this, that I've found.  Also, even if a set is not selected here, command 5 can still be used to select the velocity and acceleration information (see below).

The sets parameter is ignored when logging in real time mode: it always returns all three measurements and time and can be omitted.

Command 3 — Start Sampling


This command follows on from 1,11 (Set Up Sampling - which you must send first, although you can send subsequent command 3s, to take another sample).  It takes quite a few parameters:

{3,interval,#samples,trigger,0,0,0,0,timing,smoothing}

The interval is the time in seconds between samples - e.g. 0.1 for a tenth of a second.  The minimum value appears to be 0.005 (1/200th of a second).

The #samples is the total number of samples to take.  The total length of the experiment will, thus, be interval * #samples seconds.  This value cannot exceed 512, which is presumably limited by the internal memory for logged data in the CBR.

To sample continuously, in real time mode, the value -1 (minus one) is specified.

Note that the value here must fit with that specified to command 1,11 — i.e. if real time mode is set there (mode +4), this value must be -1; if logging mode is set there, this value must be a positive integer in the range 1-512.

Trigger controls how sampling begins: 0 = immediately, 1 = when the TRIGGER button on the CBR itself is pressed and 7 = after 10 seconds.  The 10 second countdown timer is handled by the CBR itself: the LED on the CBR will flash green, each second in countdown, blinking rapidly in the last second before sampling begins.

The next four parameters must be zero.  I don't know if they could do anything useful!

The timing parameter controls how timings are returned in the logged data: 0 = do not return timing data, 1 = return absolute times (since the start of sampling), 2 = return relative times (time between a data point and the last, essentially the same as interval).  When running in real time mode, this parameter must be 0 and all times will be returned relative to the last sample (see the example).

The final parameter is smoothing.  This specifies the level of smoothing the CBR will perform on the data being returned, to reduce noise.  It ranges from 0 (= none), through 1 (= low), 2 (= medium) and 3 (= high) — these are the terms used in the Ranger program.  It's unclear exactly what the algorithm is for smoothing: it may refer to the number of adjacent points the value is smoothed over.  When in real time mode, this parameter must be 7.

Following this command, data is obtained with calls to Get (see below).

Command 5 — Select Logged Data


This command is used to select what part of the data logged in the CBR from a previous sampling run is to be retrieved with subsequent Gets to retrieve the data.  It takes the form:

{5,11,set,start,end}

The set parameter chooses the set of data to be retrieved next.  Data retrieved cycles between {distance, velocity, acceleration, time} (as enabled by commands 1 and 3) and this shifts that cycle to the specified data set: 0 (= distance), 1 (= velocity) or 2 (= acceleration).  See "Getting Data" below.

This command can be used to retrieve data which was not included in command 1 (e..g if command 1 specified only distance and velocity to be logged, command 5 can still be used to select acceleration data).

The start and end parameters allow a range of data points to be specified to retrieve partial data.  The start value is 1 for the first point and can range up to the total number of logged points; the end value specifies the last point to be retrieved and can be 0 to include all values from the start value to the last, so start=1, end=0 will include all logged points.

Once used, subsequently retrieved data will be restricted to the range specified, until it is reset.

This command obviously makes no sense following a real time experiment, since nothing is logged by the CBR, in that mode.

Command 6,2 — Shutdown


The Ranger program sends command {6,2} when it exits.  I assume this is some sort of "shutdown" command that puts the CBR into a low power state (which it may well do automatically, if you don't send anything to it for a few minutes).  The CBR seems to take longer to respond to the next command, after sending this, but that could be anecdotal!

Regardless, this command does not erase any logged data held in the CBR (which I assume will stay there until the batteries run out, or it's explicitly cleared with command 0).

Command 6,6 — Smooth Logged Data


This command smoothes the logged data already held in the CBR (following a non-real time sampling with command 3).  It takes a single parameter in the form:

{6,6,smoothing}

Smoothing is the level of smoothing to apply to the data and values match the smoothing parameter in command 3 (0 to 3: none-low-medium-high).

After applying this command, the data retrieved from the CBR will be appropriately smoothed (however, that algorithm works!).

The command can be called again with a different value and the smoothing reapplied (or cancelled) from the originally-logged data: smoothing only affects the data returned to the calculator, not what is stored in the CBR.

Command 7 — Get Status


This command takes no parameters but a subsequent call to Get will return a 17-element list, describing the current status of the CBR:

ElementDescription
1Always 11.210
2Last error code: 0 = no error
3Battery status: 0 = OK, 1 = low, 2 = replace
4
5Interval for sampling in seconds (as specified in command 3)
6
7Mode (as specified in command 1)
8Sets (as specified in command 1)
9Smoothing (as specified in command 3)
10#samples (as specified in command 3); -1 if sampling in real time; rather oddly returns 99 if the CBR is cleared with command 0
11Timing (as specified in command 3)
12
13
14Status: 1 = not set up / initialised, 2 = armed (waiting for TRIGGER button or in 10s countdown), 3 = sampling in progress, 4 = done (logged data available)
15Start (as specified in command 5)
16End (as specified in command 5)
17

The rows with no description I don't know what they mean!

The 11 in element 1 and its recurrence in various commands make me think the CBR may be something like "device type 11" and possibly different values would be used to select different parts of the CBL (Calculator-Based Laboratory) equipment set, which I believe the CBR can be connected to, as well as directly to the calculator.

After using this command, subsequent Gets will revert to returning data from the CBR.

Error codes


There are probably lots of these, but I did start making a list of some, by deliberately sending invalid parameters to the CBR and then checking the "last error code" in the status list.  Here are some I found:

CodeMeaning/cause
12Invalid second parameter to command 1 (not 0 or 11)
32Invalid interval (in command 3)
33#samples out of range (more than 512)
34Invalid mode (in command 1) or trigger (in command 3)
39Timing parameter invalid (in command 3)

Getting Data


Once data has been logged, or real time sampling started, no specified command is required to retrieve data from the CBR, although some commands (such command 5) can be used to affect what is returned.  A call to Get will just return a list of the data.

The way data is returned is different based on whether the CBR is operating in logging mode or real time mode.

Logging Mode Data


In logging mode, calls to Get will cycle between the different types of data set to be returned in command 1, followed by the timing of the data points, if enabled in command 3:

# in cycleData
1Distance
2Velocity (if command 1, set >= 1)
3Acceleration (if command 1, set >= 2
4Timing (if command 3, timing > 0): absolute (timing = 1) or relative (timing = 2)

Get will return a list with the length of the number of data points logged: first to last.

There is no way to tell at which point you are in the cycle, although command 5 can be used to select a particular set of data, although not the timing information.

Real Time Mode Data


In real time mode, Get always returns a 4 element list, containing the distance, velocity, acceleration and relative time (since the last data point retrieved).  Since the CBR returns the relative time, the calculator doesn't have to handle any delays or timing on its part: if the CBR samples multiple points in that time, it does some sort of averaging over the period between Gets.

If no data is available because the calculator is trying to retrieve data before the CBR has sampled any, the Get call will wait until some is available and the relative time will be that of the timing interval.

Obviously, how quickly you can sample the CBR depends on your program, but I managed to pull data from it and plot points on the screen in real time at about 0.1 to 0.15s intervals on a TI-92 Plus (which is slightly faster than an original TI-92 and TI-89).  For shorter intervals, you're best using logging mode.

Examples


You can easily control the CBR by manually entering commands on the Home screen of the calculator.  I'll give you a couple of examples here, just to show how everything fits together.

Logging mode example


Start by initialising the CBR:

Send {0}

Then set it up for logging mode (+ 0) in metres (+ 0) [+ 2  = 2], returning distance/velocity/acceleration [= 2]:

Send {1,11,2,2}

Start sampling at 0.1s intervals, capturing 30 points (for a total of 3s), beginning the capture with the CBR's TRIGGER button [= 1], returning absolute time information [= 1] with medium smoothing [2]:

Send {3,.1,30,1,0,0,0,0,1,2}

Before pressing TRIGGER, check the status of the CBR and print it:

Send {7}:Get s:s[14]

... this should print 2, indicating the CBR is "armed", ready for collection.

Enter the command again but don't push ENTER yet.  Push the TRIGGER button and then press ENTER immediately (or at least within 3 seconds) — the CBR should click 30 times as it samples.  This time, the value 3 should be printed, indicating the CBR is currently sampling.

Once sampling is complete, you can run the command again and you should get 4, indicating sampling is complete and logged data available for retrieval.

There is no need to check the status of the CBR whilst doing this: I'm just showing this for completeness.

Following the above, the next call to Get will return distance data:

Get d:d

... the displayed value of d will be a 30 element list, giving the distance at each of the 30 timed data points measured, in metres.

The next calls to Get will return the velocity, acceleration and absolute time values:

Get v:v
Get a:a
Get t:t

The last one should contain the numbers from 0.1 to 3, inclusive, giving the times of the measured points.  In practice, I often get back slightly different values, such as 2.99999, which I assume is rounding in the CBR or calculator.

Real time mode example


Start by initialising the CBR:

Send {0}

Then set it up for real time mode (+ 4) in feet (+ 1) [+ 2 = 7]:

Send {1,11,7}

Now start sampling at 0.2s intervals, continuously [= -1], immediately [= 0] — the CBR should being clicking:

Send {3,0.2,-1,0,0,0,0,0,0,7}

You can now use Get to retrieve samples when you want:

Get i:i

The value returned will be the current distance in feet, velocity in ft/s, acceleration in ft/s^2 and the time since the last point retrieved with Get.  If you repeat the command quickly, the final value will shorter, since you're sampling more frequently; if you leave things a few seconds before repeating it, the interval will be longer.

When you've finished getting all the data you want, stop sampling with:

Send {1,0}

The CBR should now stop clicking.

Wednesday 17 January 2018

Spanning Tree interoperability: PVST+ and IEEE RSTP/MSTP

First post for a while — Happy 2018!

We're planning on reconfiguring the way we do Spanning Tree on our network at the moment: we don't use it on a large scale, having banished VLANs from our core, in favour of a fully-routed network with VLANs constrained to a single building/department (although there's a smidgeon of EoMPLS for the odd layer 2 point-to-point virtual circuit).

We do, however, want to use Spanning Tree on our PoP (Point-of-Presence) switches: these are our CPE (Customer Premises Equipment) layer 2 switches that we put on the border of institutional (department, college, etc.) networks.  They act as the demarcation point between our backbone and the institutional network and try to protect the next layer of the network (the distribution routers) from layer 2 issues within institutions (since as loops creating ARP or DHCP storms)

When we replaced the PoP switches back in 2008, I did some interoperability tests between the Cisco PoP equipment we were installing (Catalyst 3560G-24PS) and various other vendors that institutions may use (since they choose their own kit and pick a variety of different vendors and models).  Aside from Spanning Tree not really being geared up to work across administrative boundaries, there were questions over how (say) HP's implementation of IEEE-standard RSTP (Rapid Spanning Tree Protocol) would interoperate with Cisco's Rapid-PVST+ (that we use, and used to relied on, since we still had cross-site VLANs back then).

I did various tests and found it all didn't play well together and it all misbehaved in difficult/unpredictable ways, so ended up blocking it (via "spanning-tree bpdufilter", with "spanning-tree portfast trunk" to avoid long delays on ports going into the forwarding state) on the ports feeding institutions and having to say it was unsupported, leaving LACP or FlexLink ("switchport backup ...") as the only mechanisms for redundant links.  However, we do get the occasional loop in institutional networks (sometimes between two ports on the PoP) and I want to try and do a better job of detecting these and stopping them impacting the network further up.

Given all that, I got a new Cisco PoP switch (we selected various models from the Catalyst 3850 range) and tried hooking it up to various other bits of kit (Cisco and non-Cisco) in odd ways to see how it behaves together.

Cisco Rapid PVST+ to Rapid PVST+


OK - obvious one first: Cisco Rapid PVST+ to Cisco Rapid PVST+.  This works fine, as you'd expect: the spanning trees on each of the VLANs find each other and interconnect, ports come up quickly when bridges are linked together, redundancy/loops are detected and it all works well.

[Also, note here, when I say "Rapid PVST+", the probably all applies to (non-Rapid) PVST+, but I haven't done any testing with that and I don't think it's worth spending time on nowadays.  That said, although no-one should be using it, it is still the default in Cisco IOS!]

The PVST+ BPDUs differ from IEEE (STP/RSTP/MSTP) BPDUs in that they are of frame type SNAP (Subnetwork Access Protocol) vs a dedicated STP BPDU type (aside from a different destination multicast MAC address).  The SNAP type is then organisation "Cisco" and vendor private protocol ID of "PVST+").  Once you get into the actual frame, it looks much the same, except that it has an "Originating VLAN (PVID)" field, identifying the VLAN ID on which it's operating:

Wireshark packet capture of Rapid PVST+ BPDU
Cisco Rapid PVST+ BPDU - VLAN 1 trunk native
IEEE RSTP BPDU

If the VLAN is tagged on the port (i.e. trunk, non-native) then the PVST+ BPDU frame will also have an 802.1Q VLAN header for the appropriate VLAN:

Cisco Rapid PVST+ BPDU - VLAN 100 tagged
(There is nothing special about this being VLAN 100 here: if the VLAN is not the native/untagged VLAN, the 802.1Q header will still be included, even if VLAN 1.)

I suppose, if you do weird things like transport one VLAN over another, you might get the PVST+ PVID differing from the one in the 802.1Q header, but I don't have that situation and I can't be bothered to trying and force it.

There are a couple of gotchas, though...

PVID Inconsistent with Rapid PVST+


The PVID field in the PVST+ BPDU is what allows two switches with mismatched native VLANs configured on a port to detect that they've been incorrectly linked together and block the VLAN on that port with "PVID_Inc"[onsistent], and the VLAN on the port to block in a "BKN" (broken) state:

cat3850#show spanning-tree vlan 100

VLAN0100

  Spanning tree enabled protocol rstp
  Root ID    Priority    16484
             Address     c4b9.cd48.1980
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    16484  (priority 16384 sys-id-ext 100)

             Address     c4b9.cd48.1980
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1             Desg BKN*4         128.1    P2p *PVID_Inc

This presents us with a problem, since some institutions have never configured their main data VLAN with the "correct" 802.1Q tag (i.e. the one we use on the backbone) and just use VLAN 1 (this goes back to when only we, on the backbone, needed VLANs — institutions didn't typically have more than one, so just used the default VLAN; now they have voice, wireless, Building Management, etc.).

I don't think there's a way to override this checking, on Cisco switches.  HP ProCurve switches running Rapid PVST+ have a "spanning-tree ignore-pvid-inconsistency" command but that only fixes the HP end.

If we want to interconnect these two VLANs, I think the only solution is to filter out the BPDUs on the link between the switches with bpdufilter:

interface GigabitEthernet1/0/1
 spanning-tree bpdufilter enable

... this only needs doing at one end (since it stops BPDUs being both sent and received), although this prevents Spanning Tree from doing anything useful on the port, as well, so we can't detect loops, etc.: the port will always forward (but will still go a listening/learning phase, taking 30s to bring up the VLAN on the port, even though it's effectively deaf to other switches).

Other vendors' Rapid PVST+


Some switch vendors support Cisco Rapid PVST+.  I've done some brief testing with HP ProCurve switches in Rapid PVST+ mode ("spanning-tree mode rapid-pvst") and they've worked fine, negotiating separate spanning trees for each of the VLANs working exactly as you'd expect a Cisco to.  The HP ProCurves also handle VLAN 1 in the same way as a Cisco, including sending the additional IEEE RSTP BPDUs, as well as the Rapid PVST+ BPDUs (see below).

When I last looked at Extreme XOS-based switches (e.g. Summit X450/X460), a few years ago, these also worked perfectly fine.  However, there was a note in the documentation about VLAN 1 not being handled correctly — I'm unsure exactly what that was, but it's probably related to the VLAN 1 dual-BPDU situation that Cisco support (see below).  We don't use them enough now (and don't use VLAN 1, either), so I haven't researched this further.

Cisco Rapid PVST+ to RSTP/MSTP


Ignoring VLAN 1 (see below!), mixing Cisco Rapid PVST+ and IEEE standard RSTP/MSTP results in the switches sending BPDUs that each other ignore — there are essentially two completely separate protocols operating independently.  However, the Rapid PVST+ BPDUs appear to just flow through the non-Rapid PVST+-aware bridge like any normal frame (since they're just sent as tagged frames on their VLANs) and will end up returning to the originating switch (or another Rapid PVST+-aware bridge) for processing (a bit like having an old-fashioned repeater!).

Obviously, the BPDUs from the IEEE bridge won't get through the Rapid PVST+ bridge network, unless the ports interconnecting them back to the IEEE bridge(s) have the VLAN presented untagged/native.

If there are multiple connections from the same Rapid PVST+ switch into a segment without a Rapid PVST+-aware bridge, the redundant connections will be treated as "Backup" ports in the Spanning Tree protocol, and blocked (similar to Alternate ports).

For example, if I take a Cisco Catalyst 3850 running Rapid PVST+ and an HP ProCurve 2920 running IEEE MSTP with three ports configured: port 1 has VLANs 100 and 200 tagged/trunk, 2 has 100 tagged/trunk and 3 has 200 tagged/trunk.  I then linked the ports to their equivalent number: 1 on the Cisco to 1 on the HP, 2-2 and 3-3.  I also set the spanning tree priority on the Cisco to 16384 for both VLANs 100 and 200.  I now get this on the Cisco:

cat3850#show spanning-tree vlan 100

VLAN0100

  Spanning tree enabled protocol rstp
  Root ID    Priority    16484
             Address     c4b9.cd48.1980
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    16484  (priority 16384 sys-id-ext 100)

             Address     c4b9.cd48.1980
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1             Desg FWD 4         128.1    P2p 
Gi1/0/2             Back BLK 4         128.2    P2p 

cat3850#show spanning-tree vlan 200


VLAN0200

  Spanning tree enabled protocol rstp
  Root ID    Priority    16584
             Address     c4b9.cd48.1980
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    16584  (priority 16384 sys-id-ext 200)

             Address     c4b9.cd48.1980
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1             Desg FWD 4         128.1    P2p 
Gi1/0/3             Back BLK 4         128.3    P2p 

This situation should work fine, although if the VLAN doesn't actually go into the same switch on the IEEE side but, instead, through a number of other switches, then you get a failure between the two halves connecting back to the Rapid PVST+ bridge, it will likely take several seconds to reconverge, since this will only be detected when the BPDUs fail to arrive on the backup port.

I did see some problems with this, however, with HP Comware switches (5500-EI, running version 3 of Comware in MSTP mode): sometimes the Rapid PVST+ BPDUs wouldn't get through and both ports would stay forwarding.  This makes me think that we shouldn't rely on this as a way to build redundant topologies but more as a way to try and detect loops.

Rapid PVST+ VLAN 1 special handling


VLAN 1 gets some special treatment with Rapid PVST+: if it is present on a port, not only do you get PVST+ BPDUs, with the PVID field and sent with a 802.1Q header, if presented tagged/non-native, you also get a regular IEEE RSTP BPDU.  The to packet captures shown first, above, were actually sent consecutively by the same switch, on the same port:

IEEE RSTP BPDU and Cisco Rapid PVST+ BPDU - VLAN 1, sent consecutively

The IEEE RSTP BPDU is always sent untagged/trunk-native (without an 802.1Q header), regardless of whether the VLAN itself is untagged/trunk-native or tagged/trunk on that port.

This BPDU allows interoperation with IEEE-standard, non-PVST+ switches, such as an HP ProCurve running MSTP (Multi Spanning Tree) or regular RSTP (non-PVST+ IEEE Rapid Spanning Tree) and for a redundant topology to be built.

However, this only works for VLAN 1 — the IEEE BPDUs will never be sent if VLAN 1 is not present, regardless of whether one is untagged/native.  In addition, if other VLANs are also present, the outcome of negotiation with a partner IEEE-standard bridge will only affect the forwarding status and spanning tree of VLAN 1.  For example, the following port on a Cisco, connected to a similarly configured HP ProCurve (in terms of VLANs), with the HP running MSTP and a priority of 8192, only reports a root bridge for VLAN 1:

cat3850#show running-config interface g1/0/1
interface GigabitEthernet1/0/1
 ! (default: switchport trunk native vlan 1)
 switchport trunk allowed vlan 1,100
 switchport mode trunk

cat3850#show spanning-tree interface g1/0/1


Vlan                Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------
VLAN0001            Root FWD 4         128.1    P2p 
VLAN0100            Desg FWD 4         128.1    P2p 

... VLAN 100 will also take 30s to go through listening/learning and drop into forwarding, as if there is no partner bridge.

In our case, we don't actually use VLAN 1 anywhere on the network: we treat it as a kind of dumping ground where things end up if we haven't configured them (such as an access port with a voice VLAN but no data VLAN — there's no useful service on it, it's just used because often there has to be a VLAN of some sort specified).

[Extreme switches are nice in this area as they're able to have ports which have no VLANs present on them, and not have any untagged/trunk-native VLAN.  We had a case recently where we wanted VLAN to be tagged/non-native on an HPE Aruba wireless access point, but could only do that by making an arbitrary other, unused VLAN the untagged/native one.]

Second connection between Rapid PVST+ and IEEE RSTP/MSTP bridges with VLAN 1


If we plug in a second connection between the same two switches, using two ports configured in the same way as described above (VLAN 1 untagged/native, VLAN 100 tagged/trunk, Spanning Tree priority on IEEE bridge of 8192), VLAN 1 forms a spanning tree, but VLAN 100 just flows the BPDUs through the HP ProCurve and the Cisco sees it as a link into the same multiaccess segment and blocks it as a Backup role port:

cat3850#show spanning-tree vlan 1  

VLAN0001

  Spanning tree enabled protocol rstp
  Root ID    Priority    8192
             Address     d4c9.efb6.a680
             Cost        4
             Port        1 (GigabitEthernet1/0/1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)

             Address     c4b9.cd48.1980
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1             Root FWD 4         128.1    P2p 
Gi1/0/2             Altn BLK 4         128.2    P2p 

cat3850#show spanning-tree vlan 100

VLAN0100

  Spanning tree enabled protocol rstp
  Root ID    Priority    32868
             Address     c4b9.cd48.1980
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    24676  (priority 24576 sys-id-ext 100)

             Address     c4b9.cd48.1980
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1             Desg FWD 4         128.1    P2p 
Gi1/0/2             Back BLK 4         128.2    P2p

The HP forwards on all ports (since it's the root):

hp2920# show spanning-tree 

 Multiple Spanning Tree (MST) Information

  STP Enabled   : Yes
  Force Version : MSTP-operation
  IST Mapped VLANs : 1-4094
  Switch MAC Address : d4c9ef-b6a680
  Switch Priority    : 8192 
  Max Age  : 20
  Max Hops : 20   
  Forward Delay : 15

  Topology Change Count  : 6           
  Time Since Last Change : 11 secs     

  CST Root MAC Address : d4c9ef-b6a680
  CST Root Priority    : 8192        
  CST Root Path Cost   : 0           
  CST Root Port        : This switch is root

  IST Regional Root MAC Address : d4c9ef-b6a680
  IST Regional Root Priority    : 8192        
  IST Regional Root Path Cost   : 0           
  IST Remaining Hops            : 20          

  Root Guard Ports     : 
  Loop Guard Ports     : 
  TCN Guard Ports      : 
  BPDU Protected Ports :                                         
  BPDU Filtered Ports  :                                         
  PVST Protected Ports :                                         
  PVST Filtered Ports  :                                         

  Root Inconsistent Ports  :             
  Loop Inconsistent Ports  :             

                  |           Prio              | Designated    Hello         
  Port  Type      | Cost      rity State        | Bridge        Time PtP Edge
  ----- --------- + --------- ---- ------------ + ------------- ---- --- ----
  1     100/1000T | 20000     128  Forwarding   | d4c9ef-b6a680 2    Yes No  
  2     100/1000T | 20000     128  Forwarding   | d4c9ef-b6a680 2    Yes No  

This situation should work fine, I think, albeit a little confusing.

Now, if we set the VLAN 1 priority on the Cisco Rapid PVST+ switch to 4096 (lower/better than the HP ProCurve running IEEE MSTP), the root bridge moves over to the Cisco:

cat3850#show spanning-tree vlan 1 

VLAN0001
  Spanning tree enabled protocol rstp
  Root ID    Priority    4097
             Address     c4b9.cd48.1980
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    4097   (priority 4096 sys-id-ext 1)
             Address     c4b9.cd48.1980
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1             Desg FWD 4         128.1    P2p 
Gi1/0/2             Desg FWD 4         128.2    P2p 

cat3850#show spanning-tree vlan 100

VLAN0100
  Spanning tree enabled protocol rstp
  Root ID    Priority    32868
             Address     c4b9.cd48.1980
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32868  (priority 32768 sys-id-ext 100)
             Address     c4b9.cd48.1980
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1             Desg FWD 4         128.1    P2p 
Gi1/0/2             Desg FWD 4         128.2    P2p 

However, the HP treats both ports without regards to VLANs and so blocks one of them:

hp2920# show spanning-tree

 Multiple Spanning Tree (MST) Information


  STP Enabled   : Yes

  Force Version : MSTP-operation
  IST Mapped VLANs : 1-4094
  Switch MAC Address : d4c9ef-b6a680
  Switch Priority    : 8192 
  Max Age  : 20
  Max Hops : 20   
  Forward Delay : 15

  Topology Change Count  : 4           

  Time Since Last Change : 10 mins     

  CST Root MAC Address : c4b9cd-481980

  CST Root Priority    : 4097        
  CST Root Path Cost   : 20000       
  CST Root Port        : 1                  

  ...


                  |           Prio              | Designated    Hello         

  Port  Type      | Cost      rity State        | Bridge        Time PtP Edge
  ----- --------- + --------- ---- ------------ + ------------- ---- --- ----
  1     100/1000T | 20000     128  Forwarding   | c4b9cd-481980 2    Yes No  
  2     100/1000T | 20000     128  Blocking     | c4b9cd-481980 2    Yes No  

This situation could create some confusion, if there were different VLANs configured on each of the ports, but with VLAN 1 present: only one port would forward and the others would block, potentially breaking some of the other VLANs.

Joining two PVST+ VLANs in the IEEE STP bridge


This one allows you to get in a bit of a muddle!  Here's an example: a Rapid PVST+ switch with port 1 having VLAN 100 untagged/native and port 2 having VLAN 200 untagged/native is connected to an IEEE Spanning Tree bridge with both ports in the same VLAN untagged/native (some VLAN ID as one of the ones on the PVST+ bridge, or different - it doesn't matter), e.g.:

Cisco Rapid PVST+ bridge (assuming the VLANs are already created):

interface Gi1/0/1
 switchport mode trunk
 switchport trunk allowed vlan 100
 switchport trunk native vlan 100
!
interface Gi1/0/2
 switchport mode trunk
 switchport trunk allowed vlan 200
 switchport trunk native vlan 200

HP ProCurve (isn't this side easy!):

vlan 50 untag 1,2

The corresponding ports are then connected together (1-1, 2-2).

Once connected, the Rapid PVST+ BPDUs for the two different VLANs are interconnected through VLAN 50 in the IEEE STP bridge and find their way back to the Cisco switch, where you get a "PVID_Inc" (VLAN ID inconsistent) and both ports go into the "BKN" (broken) state and block, as described above, breaking both VLANs!

It makes no difference whether the IEEE STP bridge is running Spanning Tree or not, since the IEEE STP BPDUs pass by the Rapid PVST+ BPDUs and don't interact.  The only change in behaviour is that the broken state won't be detected until the ports on the IEEE bridge move into the Forwarding state (which will take 30s normally, or 4s if the HPs "auto-edge" mode is enabled), meaning the problem will initially not be noticed (unless there are other effects) and probably things will be OK for 30s (on the port that was connected first, at least), after which both ports will move into the Broken state and block, when the Rapid PVST+ BPDUs get bridged between the ports/VLANs on the Cisco.

This situation could confusing because both VLANs will break, when this situation occurs — perhaps by someone accidentally connecting a data VLAN to a voice VLAN.  Without spanning tree, both VLANs may continue functioning, to a certain extent.  However, this is probably a good thing as the problem is detected immediately, rather than weird things going on until the root cause is determined.

(One thing we do is have different HSRP group numbers for each SVI, meaning that the MAC addresses and other messages don't clash, when two VLANs are connected together by accident.  This also helps us spot what's happened, because the virtual MAC address of [say] the voice VLAN will appear on the data VLAN.)

Cisco Rapid PVST+ to No Spanning Tree


Last but not least, what about if there is no Spanning Tree running on the partner bridge?

There is obviously a distinction between it gobbling up the BPDUs (perhaps by running bpdufilter) and one which just lets the BPDUs flow through without processing with them, treating them like any other traffic.

HP ProCurve switches have spanning tree turned off by default and require it to be turned on with "spanning-tree" (that's it — no arguments!) command.  When turned on, it defaults of IEEE MSTP (at least on a ProCurve 2920; other switches may vary, especially the older ones, but HP have been pushing MSTP for a long while, as they're fairly keen on IEEE standards, even when they're not particularly great!).

Without Spanning Tree enabled, the situation, at least with HP ProCurve, is that the BPDUs flow through like normal traffic, creating a situation the same as the "Rapid PVST+ and RSTP/MSTP" interaction, above.

For Cisco Catalyst switches: if spanning tree is explicitly disabled on a VLAN with "no spanning-tree vlan ...", the switch will not gobble PVST+ BPDUs but allow them to flow through and the bridges on ports into that VLAN will see each other, in terms of Spanning Tree.  However, this doesn't appear to be true for IEEE STP BPDUs: they do seem to be filtered, even if a VLAN is presented untagged/native.