Cisco Nexus 7000 - 32 port 10Gig linecard caveats

I have been working on a set of Cisco Nexus 7010's that both have 32 port 10G line cards. I have been able to get virtual PortChannel (vPC) and Virtual Device Context (VDC) set up and working on the platform, definitely some of the coolest technology out there for the data center outside of server virtualization. That being said, there are some interesting caveats that folks should pay attention to when planning out the total number of usable 10G ports on the 32 port line card.

First, the ports are "grouped" in sets of four (1,3,5,7 for instance) - each group of four is capable of doing 10G in shared mode or a single port in the group (the first one) is capable of 10G in dedicated mode and the other 3 are shutdown. Effectively the card is over subscribed 4 to 1 - not horrible. This means it has a total of 80Gbps into the backplane. All this is pretty well documented by Cisco so it's not new or an unknown.

So, what isn't obvious when you are doing planning and design? Turns out if you are designing a network to utilize VDC's in the 7k's than you need to pay closer attention to the number of usable 10G interfaces you get per VDC. Because the ports are grouped and sharing ASIC resource they cannot be individually allocated to a VDC but have to be allocated as a group.

So if you plan on needing just a single 10G interface in a VDC in reality you will have to allocate a full group of 4 10G ports (and in the correct grouping order.) Even if you decide you want to use the dedicated mode to allow full 10G for a single 10G port and leave the other 3 in the group shutdown you still have to allocate all 4 10G ports to the VDC. When you think about it this makes sense but often in the design phase while white boarding out a solution an engineer will simply allocate a single 10G interface for a VDC because that is all that will be "used" but in reality you will be allocating 4 not 1.

This can throw off your 10G port count by a lot depending on how many VDC's you plan to run. You can only run up to 4 VDC's on the 7k's today though in theory there is no reason they couldn't raise that as it is only a software limitation. So, in reality all VDC's should be allocated 4 10G ports even if you only need 1 10G port for something. That way you don't run into any surprises when trying to deploy a solution. Also, for deployments that make use of two 7k's (which really is how you should do it to gain the benefit of vPC) I would trade a dual supervisor per chassis configuration for an additional 32 port 10G line card if I had to chose. It makes more sense to have more flexibility in the number and allocation of 10G interfaces with a single supervisor per chassis solution since you will have two chassis' anyway for redundancy. Just a thought when planning and budgeting.
- Ed