Friday, April 02, 2010

Cisco Nexus 5k and 2148t caveats

I ran into a few more items I think warrant a post regarding the Cisco Nexus 5k series and the 2k fabric extenders (fex.) For those that are evaluating the Nexus 5010 and 5020's with the 2148t fex for layer 2 switchport capacity there are two gotchas that I think failed to get mentioned often enough in the pre-sales engagement.

1. The 2148t fex is not capable of doing a port-channel (ether-channel) with two or more ports on a single chassis. This is not a limitation that can be fixed with software either, it is a limitation in the ASIC's themselves on the 2148t. I think this is a huge failure. That being said, I believe it is road mapped as a feature fix for the new 2248TP and 2232PP models.

2. If you do need to do a port-channel to a single host from a 2148t you need two 2148t's and you have to run a Virtual Port-channel (vPC) using the 2148t's that are each single homed to the 5k's upstream (in what is called single homed fabric extender vPC topology.) You can only do a maximum of 1 Gigabit Ethernet port to each 2148t (because of the limitation above) and then the 5k's will do the logical vPC to make them a port-channel. So a 2 Gigibit max port-channel with the 2148t single homed to the 5k's. Again, because of the limitation above I see this as poor engineering work around to the failure in design.

So, with those limitation on the 2148t fex's there is something else from a design standpoint that is frustrating. You have two recommended options when designing the 5k/2k deployment, you can do a single homed fabric extender vPC topology or a dual homed fabric extender vPC topology. Here are the diagrams from the Cisco Nexus 5000 Series NX-OS Layer 2 Switching Configuration Guide.

The single homed:

and the dual homed:

What is not mentioned or documented anywhere about this configuration is if you can mix and match the topology between single homed and dual homed 2k's on a single pair of 5k's. I can only find one reference that mentions this vPC mixed topology option. Even with this option you are still burning up a second 2148t just to get port-channeling as an option if you don't want to use the 5k's directly. Burning up the few 10/1 Gig mix use ports on the 5k's for 1 Gig port-channels seems to be a huge waste and very expensive.

So there you have it, some items to mull over while planning out your data center design with Nexus and some hard hitting questions to ask your Cisco SE's when they show up talking about cost per port and maximizing functionality and flexibility per port. On top of the above items the 2148t also only does 1Gig, it doesn't support 10/100 at all, not a show stopper but another nagging item that limits some deployments. Don't get me wrong, I'm a fan of the new data center design and topology model that Cisco is building and with OTV coming out in the 5.0 NX-OS code release some really exciting things are going to be possible. Just keep in mind you might be buying an extra one or two 2k's to accommodate some of the systems that have the port-channel requirements for higher availability.
- Ed


Anonymous said...

Obvisously you as missing something here.
Why would you take a 5k and connect it to a 2148 with only 1g or 2g at that. Remember the 2k's are essentially 6748's in a box using the Fabric extender protocol to act as a backplane to the 5k (Supervisor) in a box. This is why it is only recommended to use the 10g uplinks, and I know the 10g ports can be channeled. Look at it this way if your using a 2148T you are over subscribed 48:1, now this is something I don't want to explain when packets start disappearing.

Ed Horley said...

I don't think my post has anything about ONLY doing 1g or 2g from the 5k to the 2k's - in fact, it assumes that all the links from the 5k to the 2k are 10g. It should be for links from the host to the 2k. The issue has to do with the port-channel options available with a 2148t in a dual homed vs. single homed down to the host - not upstream to the 5k. I'll read it again to see if I mis-stated something or if I can clarify, it has been a bit since I posted this one.
- Ed

SMiTTY said...

Just to follow up on the mixing and matching of dual and single connected FEX's. You most certainly can intermix them. We just went through this with trying to create a VPC down to our Isilon nodes while dual connected. The 5010's only have 6 FEX's currently configured and we have opt'd to single connect (odd numbered fex on one 5010, evens on the other). Works fine.

Actually, we will be going through a migration to single connect all the FEX's.

Anonymous said...

What version of OS are you running?
4.2.1 cannot do vpc's from 2 5k's to a single 2K

Anonymous said...

Is this post yours too?

Anonymous said...

as the above post incines I think you are pirated by Tuam Phan at

in fact everything on seem to originate from other sources on internet.

Cheers and good luck!