Monday, December 27, 2010

End of year wrap up for IPv6

It's pretty clear that in the later part of 2010 IPv6 became a much talked about technical topic. What hasn't become clear is how many folks have followed up and implemented on it. As of this article post Hurricane Electric's IPv6 stats say 48 days of IPv4 space remaining with only 7 /8's remaining. However, since an automatic delegation happens when the number reaches 5 (1 /8 delegation goes out to each RIR) we really only have 2 /8's remaining for IANA to allocate out per RIR requests.

What is more interesting is that they have added a RIR stats showing how many /24's each RIR has and what percentage of their pool they are at. Notice that APNIC and ARIN are currently the lowest. There are several good blogs on who they thing will get the next 2 /8 delegations but it would not surprise me at all to see APNIC get one soon and for them to run out first too. Stephen Lagerholm has a blog that covers the topic in good depth and is definitely worth a read. What should scare you more are his stats and the fact that he has been scary close on most of the dates in the past. Also, his IPv6 stats show just how poorly the service providers, networking engineers and application folks are working together to get IPv6 adoption going.

I have been working on getting a new California IPv6 Task Force website put together. I will next be working on updating the North American IPv6 Task Force. There are now many sites with really good IPv6 information out there so the goal for the sites are to point to good resources and not to re-invent the wheel. The North American site will simply aggregate the content of all the other NA Task Force sites so someone can just watch that site to find out what is happening in North America. In addition, each regional task force will cover their respective area to tell you what is going on locally.

In case you don't know some of them:
California IPv6 Task Force -
Rocky Mountain IPv6 Task Force -
Texas IPv6 Task Force -
Hawaii IPv6 Task Force -
Mid-Atlantic IPv6 Task Force -
Canadian IPv6 Task Force -
Mexico IPv6 Task Force -

Task Forces are in the works for the Northwest and Southeast but nothing formal has come out as of yet. Then there is also the global IPv6 Task Force -

I hope everyone realizes that regardless of your business or residential status soon there will be both clients AND servers that will only have IPv6 available to them due to the lack of IPv4 address space. It won't happen overnight but it will happen VERY RAPIDLY. You have a choice, keep up and learn the next networking protocol that will allow everything to continue to function or fall behind and lose value and relevance. That said, there are some easy ways to start. First, check out which is a free service from Hurricane Electric. It allows you to run a dedicated tunnel to your work or home and get IPv6 connectivity. You should also check out Freenet6 from gogo6. They have a nice windows client application that allows you to have IPv6 access via their free tunnel broker.

Prediction - 2011 will be the year of IPv6 - likely the year of panic about IPv6.
- Ed

Monday, December 06, 2010

It is that Microsoft MVP renewal time of the year

I have several colleagues who are either up for a re-award for their Microsoft MVP award or others who are being considered for the award. I have to admit after all the years of being an award recipient myself I still find the process fascinating and a bit shrouded in mystery. Microsoft does the awards on a quarterly basis now to spread out the workload of reviewing people but the end of the year seems a good time to reflect on things especially since the award is given for the community work done in the "previous year."

My own experience has been really positive in terms of the nomination and re-award process and I think the Microsoft MVP leads (Shout out to Jake Grey who is my lead) do a remarkable job given the number of individuals they are doing checks on. Knowing the variation in participation and the breadth of technical topics the MVP community covers it is remarkable they can review though everyone.

I recently read a posting here by Eric Ligman and another posting here by Alessandro Teglia both on the MVP program, what it takes to become an MVP and a bit about the program also. The fact that the site list numbers like approximately 4,000 MVP's out of 100 million online community members seems to put my status in a staggeringly small category. It also means that a very small group of individuals (at least in Microsoft's opinion) influence a huge group of online and offline people.

It seems that some other companies have decided to model their community recognition after the Microsoft MVP program. For instance the VMware vExpert program seems almost a direct lift - I guess that is a great compliment to the work Microsoft has done and a validation that community really is important in a companies marketing and support relationships.

For those of you who are Microsoft MVP's I hope to see you at the Summit in Redmond. For those waiting to hear if you were re-awarded I have my fingers crossed for you and for those who are waiting to find out if they get to join the exclusive club, best of luck!
- Ed

Wednesday, November 24, 2010

California IPv6 Task Force Update

I have been working with some colleagues to get a few items moving along for the California IPv6 Task Force. The primary one has been to get the website a bit more 2010-11 in look and feel. We will be porting it to a WordPress site and we hope to have the North American IPv6 Task Force site ported over and hosted in the same way. Both will be at Hurricane Electric and available via IPv6. Our goal really is to make the site a jump site with current content and links for folks to get started on IPv6. So the first shout out goes to Jared Curtis for making this part a reality.

In addition to that goal, we are going to be starting to get some IPv6 labs up online and will be looking for individuals to help put together short technical white papers. The materials will be made available via the website and hopefully people who are trying to make the transition to IPv6 will find it useful.

On top of that, we will soon have the mailing list server up and operational. Once that is done it will be a bit easier to get some events happening throughout California. If you are interested in helping at all please don't be shy and drop me an email at ed AT cav6tf DOT org. I'll be happy to get you in the loop on what we are working on and a potential project you can do.
- Ed

Friday, October 29, 2010

gogoNET Live! IPv6 Conference - Nov 3-4

If you are in California and interested in IPv6 then you should already be signed up for the gogoNET LIVE! event being hosted at San Jose State University happening Nov 3-4th. The line up of keynotes is impressive:

Vint Cert - Honorary Chair, IPv6 Forum; Chief Internet Evangelist, Google
John Curran - President and CEO, ARIN
Elise Gerich - Vice President, IANA
Bob Hinden - Chief Internet Technologist, Nokia
Dave Ward - CTO, Infrastructure Product Group, Juniper
Latif Ladid - President, IPv6 Forum, Conference Co-Chair
Bruce Sinclair - CEO, gogo6, Conference Co-Chair

There are a ton of great speakers too. Check out who is presenting at the speakers page. Shameless plug because I will presenting on what is happening with the California IPv6 Task Force. I personally always enjoy seeing Martin Levy with Hurricane Electric present along with Tony Hain with Cisco and Stephan Lagerholm with Secure64 Software. I am excited to hear from Silva Hagen with Sunny Connection AG and Donn Lee with Facebook too.

The event is being presented by the IPv6 Forum, the California IPv6 Task Force and gogo6.

If you aren't paying attention to IPv6 here is a little bit of youtube video amusement from Cisco to get you moving.
- Ed

Tuesday, October 19, 2010

IPv6 - The Ostrich Effect

I have decided it is time to label one of the psychological effects being seen in the IT Professional community in regards to IPv6. I am calling it the Ostrich Effect and I am defining it as:

"The aversion to receiving information about IPv6 or avoiding the topic long enough in hopes it will go away"

It seems there are several different grouping of opinions regarding IPv6 and here are my quick outline of those groups below. I do think some of my group's mental states lines up with the stages of grief which are Denial, Anger, Negotiation, Depression and Acceptance however I think mine have more to do with the camps that have evolved around the issue of IPv6. I actually think that for each group defined below they will still have to go through all the stages of grief, I just think some will make it through without issue and some might not make it at all.

1. The "it will never happen" group
Typically folks in this category are more project planners, IT directors and CFO types who are justifying their comments by making generic snide references to how long have people been saying we would run out of IPv4 addresses and that it hasn't come to pass or that there is no new beneficial value from planning or even looking into IPv6. Their primary concerns are cost controls not operational issues so to them IPv6 only looks like an additional cost without any benefit. There are also people who believe IPv4 can live forever by doing even more NAT/PAT solutions but they clearly haven't looked at the bigger picture of the challenges with that for solution providers and carriers. Also, many in this group tend not to understand the larger issues with IPv4 vs IPv6 such as routing table size, the usage rate of addresses etc and seem to think that we can simply "recover" IPv4 addresses from folks who got allocated "too many" in the early days of IPv4. In a desperate plea to those folks, go take a look at what ARIN has to say about that.

2. The "I'm not convinced and this is annoying" group
Honestly, I think most business and management folks fit into this category right now. That is why there are IPv6 Task Forces all over the US to try and help people understand why they should be in the 3 or 4 category. Many professionals feel it is hard enough to keep up on what they have today in terms of technology so to learn a whole new networking protocol adds a lot of burden for the same perceived functionality they get out of IPv4 today. In addition, they truly believe they have a lot more time to work this out then they actually do. Basically they are stalling and are annoyed that people are telling them they have to pay attention.

3. The "I get it but I will wait for broader adoption" group
This group of people understand it is coming but don't want to be the first to have to do it. Unfortunately most network admins and system admins fit in this group. What is more alarming is they are horribly lacking in training and resources to get IPv6 working in their environment and unfortunately are unlikely to get budget to get the training they need. Hence their desire to wait! What is of great concern for them is when their management moves from group 2 to group 4 overnight. Then their management expects them to be able to implement overnight too. My only advise is get training any way you can so you are ready when the sleeping giant awakens.

4. The "I am totally stoked to get IPv6 rolled out" group
The rare group of IT professionals and visionary business leaders who understand that for their business model to continue working they must be able to support and communicate with everyone who might potentially be on the Internet, regardless of if they are using IPv4 or IPv6 to reach you. And with the address pool depletion happening soon (maybe before the end of 2010 if there is a run on addressing) you will have no choice - you MUST adopt IPv6 to meet that goal. So, they are already planning to design, deploy and support it, it means continued business which means everyone can keep the lights on. In other words, they get it. Many forward thinking companies have already starting doing this - kudos to you!

5. The "I've been doing IPv6 for years - what took you so long" group
The front runners and leaders who are using IPv6 as a strategic advantage as market leaders of services hoping to gain new customers and take customers from those who are unable to meet the new business requirements of those in group 4. Hate to say it but to be in group 5 means you've already been doing IPv6 for more than a year or longer. This group has left the station and it's an exclusive club and you can't join - sorry, it's just too late. For those in this group - you rock.

I hope to post up a road map that I am developing for the CAv6TF soon. It is for IT Professionals to allow them to rapidly meet the IPv6 requirements that will drop on them like a lead balloon. It is specifically targeted for folks in group 3 and 4 and has both the road map but a checklist too.
- Ed

Monday, October 18, 2010

Texas IPv6 Task Force 2010 Summit - presentations have posted

For those that are looking for IPv6 content all the recent presentations that were given at the Texas IPv6 Task Force 2010 Summit have been posted. The event was excellent (thanks to Stan Barber and Stephan Lagerholm) and I recommend download them to review when you have a moment.

Unfortunately the event was NOT video recorded but I must say that Richard Jimmerson, CIO of ARIN gave a great presentation along with Tony Hain from Cisco Systems and Martin Levy with Hurricane Electric.

It was clear from the Summit that the IPv6 message did not make it out as was hoped to allow for a smooth and structured transition. Prepare for a bumpy ride here in the next few months is my feeling.
- Ed

Sunday, October 10, 2010

Texas IPv6 Task Force 2010 Summit

I'm excited to be heading out to the TXv6TF IPv6 2010 Summit which is taking place Oct 11-12 in Richardson, TX. A good selection of IPv6 topics plus it will also be a chance to meet several of the folks who are loosely helping to run the NAv6TF and drive IPv6 adoption across North America. At the same time, I am trying to get more folks to participate in the upcoming CAv6TF / gogoNET LIVE! IPv6 Conference which is happening Nov 2-4 at San Jose State University in the California.

I hope to be able to meet a bunch of fellow IPv6 advocates and see what types of companies are currently interested in IPv6 and what stages they are in regarding deployment and planning.

If you are going to be at the event, please let me know. You can follow me on twitter or send me a note via ed AT howfunky DOT com.
- Ed
(ps - happy 101010 day)

Monday, October 04, 2010

IPv6 events for Oct and Nov 2010

There are several IPv6 events happening in October and November that are worth noting. Here is the short list of events I have on my list to attend.

The first event is the TXv6TF IPv6 Summit, Oct 11-12 in Richardson, Texas. The event is hosted at the Richardson Cisco office and has a great line up of presenters including Scott Hogg, Martin Levy, Richard Jimmerson and Stan Barber plus many more.

The next event is Owen DeLong with Hurricane Electric presenting at the East Bay Cisco's User Group on Oct 14th in Pleasanton, CA. Owen will be giving an update on the situation with IPv6 working groups, how the transition to IPv6 is going and a general state of IPv6 from a service provider view.

Then on Nov 3-4 the California IPv6 Task Force and gogoNET present the gogoNET LIVE! IPv6 conference at San Jose State University. The conference will have a pre-conference workshop day on Nov 2nd and will be the flagship IPv6 event in California. Keynotes will be given by:
Vint Cerf - Honorary Chair IPv6 Forum
John Curran - President and CEO, ARIN
Bob Hinden - Chief Internet Technologist, Nokia
Elise Gerich - Vice President, IANA

The end of the year has a lot of IPv6 events for folks to attend. I encourage everyone to participate in these great local IPv6 events to help get a better understanding of IPv6. You can either go kicking and screaming or walk through the door with confidence and knowledge regarding IPv6. IPv6 is happening, now, take the time to learn it so you know what is happening.
- Ed

Sunday, October 03, 2010

Updated look and feel for

The layout and look have been the same for Howfunky for over 5 years now and the site needed a newer updated look. I took advantage of some of the newer Blogger templates and tools to update the site. I might change and add a few more things to the site now that it is more flexible for me to do so.

I hope the updated look is a welcome improvement. None of the links should have changed so please tell me if something broke for you and I will try and see if it is something I can fix.
- Ed

Wednesday, September 22, 2010

Cisco ASA - AnyConnect for iPhone released

Cisco announced the general release of the Cisco AnyConnect client for the iPhone. While I think it is cool that Cisco has released this finally I am a bit flabbergasted at the licensing they are doing on the ASA - yet again. Not only do you need AnyConnect Essentials or Premium license but it will require the AnyConnect for Mobile license also. I still don't understand why Cisco feels the need to do this and it is frustrating when trying to sell the ASA platform to have to address each one of the licensing requirements for features that many expect from the base AnyConnect Essentials license. It makes you want to stick with the inbuilt IPSec client solutions regardless of the feature enhancements.

If you do a show version on your ASA you can see what your current license status is, the output should look like:
ASA#show version
Licensed features for this platform:
Maximum Physical Interfaces    : Unlimited
Maximum VLANs                  : 50
Inside Hosts                   : Unlimited
Failover                       : Disabled
VPN-DES                        : Enabled
VPN-3DES-AES                   : Enabled
Security Contexts              : 0
GTP/GPRS                       : Disabled
SSL VPN Peers                  : 2
Total VPN Peers                : 250
Shared License                 : Disabled
AnyConnect for Mobile          : Disabled
AnyConnect for Cisco VPN Phone : Disabled
AnyConnect Essentials          : Disabled
Advanced Endpoint Assessment   : Disabled
UC Phone Proxy Sessions        : 2
Total UC Proxy Sessions        : 2
Botnet Traffic Filter          : Disabled

So note that you will need the AnyConnect Essentials or AnyConnect Premium (SSL VPN Peers) license plus the AnyConnect for Mobile to have a working solution. Some product overview information is available here, the link might require CCO logon access.
- Ed

Tuesday, September 21, 2010

Cisco ASA - AnyConnect 3.0 features

Cisco has announced new features for AnyConnect 3.0 and will finally be supporting both IPSec VPN and SSL VPN in a single client. They have outlined some interesting things like using RDP to launch AnyConnect, IPv6 VPN access, captive portal detection and auto reconnect. All things that make AnyConnect much more useful and easier for end users to use VPN all the time.

It seems that Cisco is taking seriously Microsoft's DirectAccess solution given the features they are putting into AnyConnect. In addition, they are making the client available for platforms that Microsoft can't support such as the Apple iPad and iPhone. They are also making AnyConnect able to leverage a split VPN connectivity option to take advantage of their ScanSafe cloud security solution so that might give folks more options in terms of offloading posture assessment of VPN clients.

I still think that Microsoft's DirectAccess is an impressive solution for Win7/Server 2008R2 shops that want to provide their end users a remarkable VPN experience. You can tell the difference of something that is made from the ground up in a product and integrated so tightly in the operating system that the experience becomes seamless. The downside is that there are often shortcoming in flexibility in the number of scenarios you can deploy the solution.

I think Cisco's AnyConnect is a great solution in terms of the flexibility it offers while still providing a strong set of capabilities. I just wish a lot of these features had come out in the 1.x release instead of having to wait this long. I special note is the IPSec support in a single client - that has been an issue since AnyConnect was first launched.
- Ed

Tuesday, September 07, 2010

Relaunch of the California IPv6 Task Force

For those who are interested in IPv6 please check out the relaunched California IPv6 Task Force website. The CAv6TF will be helping with the gogonetLive! IPv6 event on Nov 2-3, 2010 in Silicon Valley (San Jose area at this point) so please keep the date reserved, it should be a great chance to interact with other IT professionals interested in IPv6.

I am excited we are finally making the CAv6TF website available via IPv6 too and also that the Task Force is now active again after a few years of hiatus due to the majority of the CAv6TF members driving the National IPv6 story. I am now serving as the Co-Chair of the Task Force handling the San Francisco Bay Area. If you have any comments or interest in helping with the CAv6TF please contact us at

If you have not yet joined gogoNET and you are interested in learning more about IPv6 this is a good place to start getting connected with other IPv6 folks so consider signing up. They also offer a free IPv6 software client service through Freenet6 so you can run IPv6 no matter where you are which is a nice option if you do not have a native IPv6 service yet. Another option is Hurricane Electric's IPv6 Tunnel Broker service which I have used for years for my home configuration.
- Ed

Monday, August 23, 2010

VMworld 2010 - August 30 - Sept 2nd

Another VMworld is around the corner, just a week away now - it is again in San Francisco at Moscone Center and should be an interesting event as always. Cisco, EMC and NetApp will all have big presence at the show for sure and it is nice to see what they have in terms of product lines and what they are doing in the virtualization space. What makes me want to attend the expo every year though is the small niche companies that have interesting products or way of solving problems that the big boys haven't figured out or haven't thought about.

Often it is the case that these folks get purchased and integrated into one of the bigger players so part of the fun is getting to see them while they are still small, still agile and still interesting plus getting their swag while they still have their own logos and tag lines and such.

I expect to see some announcements associated with the show regarding partnerships, new product launches and perhaps some heated debate too. It looks like Microsoft will once again be relegated to a regular small booth which is a shame considering the majority of deployments of VMware's products are to support Microsoft platforms. I think VMware is making a mistake not allowing Microsoft to have a bigger presence at the event and simply asking them to live within some guidelines regardings Hyper-V.
- Ed

Friday, August 06, 2010

Splunk User Conference - August 9-11, 2010

One of the tools I use to help me wade through the mass of data that networking devices spit out is Splunk. Splunk is easily one of the most useful apps to run if you have to try and figure out what is going on with regards to firewalls, ids, network logging and anything else that will output to syslog.

If you happen to live in the San Francisco Bay Area you are in luck too, next week is the first annual Splunk User Conference and you can still sign up. I wish I could make it myself, especially since I live local to the event.

Splunk has really expanded the capabilities of the product but to be honest, the simple search and filtering to help me write better access control lists for firewalls is a huge time saver, just being able to do that quickly and efficiently makes it worth it's weight in gold.
- Ed

Thursday, July 29, 2010

Why paying attention to IPv6 is now important

As a personal interest I have been working with and following developments of IPv6 for several years. I started presented on IPv6 back in 2006 because of what was happening with Windows Vista and the changes that Microsoft was doing with the OS and their new networking stack. Here we are in 2010 and I think we are past early adopters in regards to IPv6. In fact, I now think if you are not paying attention to what is happening with IPv6 it could start impacting your ability to perform your job soon, especially if you are an IT Professional.

So, who do I think will be impacted the most by the transition and more prolific use of IPv6? I think you might be surprised.

The standard answer is network engineers and granted they indeed will be rolling out and maintaining dual IPv4/IPv6 networks for years to come but I actually don't think IPv6 will be as much of a challenge for network engineers to get up and running assuming they have moderately newer network equipment. Granted, there are issues with lack of feature parity but that will be resolved over time and will be fast tracked when equipment manufactures realize they are losing sales due to the lack of the parity.

Next on the list is system admins. I think many will find IPv6 to be a bit more of a challenge in regards to the differences in behavior of the protocol and getting worked out the behavior differences of applications as a result. This is a huge issue for client machines in terms of what OS you are running on your desktop and what the server does or does not support. I would argue that the majority of system admins know enough IPv4 networking to allow them to do their job but likely will have some challenges with differences in IPv6. I know there are some great system admins out there who could run networks also so obviously this a wildly general statement but I still feel there is going to be a bigger learning curve for system admins than they care to admit. Perhaps it is time for Microsoft to bring back a dedicated networking exam - like the old MCP TCP/IP exam?

The surprise group is application developers and database admins. Just think about how much code has been written out there to account for IPv4 addresses. IPv4 addresses are 32bit and I would imagine the majority of applications out there are storing that value under a declared INTEGER. I could be wrong - maybe they are all stored as a STRING instead but I have a feeling that isn't the case. IPv6 addresses on the other hand are 128bit and likely the majority of applications will have to be modified to account for the new size, difference in how they are represented (in HEX not DEC) and also the fact that the application might potentially have to pay attention to which interface it is directing traffic through. This doesn't even cover all the databases out there that are storing IPv4 information and the SQLNET statements all based around IPv4 to query those databases.

To top it all off, IPv6 can represented an address in multiple ways due to the zero compression option. So searching through logs or analyzing output could be an additional issue unless some standards are agreed to in advance in terms of how to store and represent an IPv6 address. So imagine trying to correlate information from multiple systems and they can't match stuff because the IPv6 addresses are represented differently in each system. I think some of these issues will be the biggest road blocks to overcome in the months and years ahead for IPv6.

So, why is it important to pay attention to IPv6 now? It is important because the adoption and momentum behind the protocol has already begun. Major content providers like Facebook and network providers like Comcast and content delivery providers like Limelight have all deployed IPv6 already and are doing their trials now. If you have no knowledge or understanding of IPv6 how will you address your business needs when you need to either access content, deliver content or work with a network when you don't understand the protocol they are using to move traffic?

In short, if you don't have a working understanding of IPv6, you are already behind. Take a quick quiz. Do you know what behavior Windows 7 has when it has a public IPv4 address? What is different if it gets a public IPv6 address? Which protocol does it use for DNS resolution if it has both an IPv4 and IPv6 address? Does the type of IPv6 address it has matter to the default behavior? This is all just for Windows 7, now do this for OSX, Windows Vista, Windows XP, Windows Server 2003, 2008, 2008R2, Linux and Solaris. How did you do?
- Ed

Tuesday, July 20, 2010

Cisco ASA NetFlow configuration

I have been setting up a lot more NetFlow on Cisco ASA's recently. Mainly due to the request for more visibility into the traffic that is consuming Internet bandwidth and for compliance reasons. It seems that even with proxy services and other solutions many IT organizations still have a poor understanding of the actual traffic traversing their network.

Since NetFlow is limited in terms of platform support (specifically the Cisco Catalyst 3k/2k switches do NOT support it) but the Cisco ASA does I have been asked to turn it on the ASA to have a better idea what is going across the network. Cisco has a nice Introduction to Cisco IOS NetFlow if you need to run it on the routing or 4500/6500 platforms which is a great way to go in addition to the ASA as you can then see what is happening between devices on the network also.

Chapter 75 in the Cisco ASA 8.2 CLI Configuration Guide covers how to set up a NetFlow configuration. Here is a short script to get it up and running quickly. A couple of caveats, read the config guide because it covers the parameters for timing and limiting what you are collecting. This script is a "let's get this going, send me everything" sort of solution. Not optimal for heavily loaded ASA's. Should be good enough to get you going though.

! - NetFlow script for Cisco ASA
! - ACL to catch all IP traffic - to specify the traffic you are interested in
access-list flow_export_acl extended permit ip any any
! - set up the destination server ip and template rate
flow-export destination {interface name} {IP address} {port #}
flow-export template timeout-rate 1
! - build out the class-map for the flow that matches the ACL
class-map flow_export_class
match access-list flow_export_acl
! - or don't use an ACL by using
match any
! - build out the policy-map
policy-map flow_export_policy
class flow_export_class
flow-export event-type all destination {IP address}
! - apply the policy-map to whatever global policy you have or make one
service-policy flow_export_policy global
! - if you have an existing policy-map apply the class-map into that one
! - for instance the default ASA service-policy for global is global_policy
! - so you could add the class-map to it by doing
policy-map global_policy
class flow_export_class
flow-export event-type all destination {IP address}

You can get information about what the ASA is doing in terms of the flow output by using the following commands:
show flow-export counters
show service-policy global flow ip host {source IP} host {dest IP}
show access-list

Obviously you need some sort of NetFlow collector. There are a lot of professional and free tools to do this and there are some great vendors doing this. That being said, I have used Plixer's Scrutinizer free product to at least get folks up and working and have a functional tool to look at until they can decide what tool they want to use. It does and excellent job of showing what is possible in terms of reporting and information gathering. That being said the following companies also have NetFlow commercial products you should consider or have free offerings that can be used.
NetQoS - part of CA now
NTOP - opensource tool

I am leaving off a ton of vendors in the list but I have found if I list everyone who is doing a solution then folks who are trying it out freeze up and can't pick one. I know, not a great reason but I rather have them use something than nothing at all.

Cisco owns the NetFlow name but there is a standards version of NetFlow supported by many other networking vendors called sFlow. Basically it provides the same sort of function but on other vendor equipment. This means that almost all NetFlow collectors can work with sFlow. So if desired you can collect from non Cisco devices that support sFlow to the same collector to get a more complete view from around your network.

If you have not deployed and made use of NetFlow I really recommend doing a quick trial run. You may be surprised by what you find. I've had clients discover employee's watching video and tv shows being pulled from foreign countries (some of questionable content), others consuming high bandwidth across tunneled links they did not know they were running and lots of other interesting items. Many have been surprised how much IPv6 and tunneled IPv6 they are running on their network. It is a great tool so check it out.
- Ed

Thursday, July 15, 2010

The four datacenter horsemen - who will they be?

I've been watching some of the network manufacture market transitions happening right now and the rapid changes going on in the data center and networking market space. I think there will be four main players who I am calling the four datacenter horsemen who will be calling the shots in the future - leading players of the apocalypse I guess. This is all me having fun guessing what will go on so don't hold me to it and of course, this is only my opinion.

Team #1 is made up of Cisco/EMC/VMware due to their partnership arrangement. This partnership for taking over the data center is what caused all sorts of realignments in the industry in the first place. Specifically, with the falling out of HP and Cisco over servers and HP now having a full portfolio of network, servers and storage to go after the data center space that Cisco has traditionally shared with HP it begs the question - what next?

Granted I agree that HP, team #2, does not have a fully competitive network/storage solution when compared to the Cisco Nexus + UCS solution but they are close enough that folks are picking sides. Certainly it can be argued that HP has much more depth in the server market space and a longer deeper relationship with Microsoft to cover the hypervisor gap.

So, what other teams are out there? We can't leave IBM out of the game, they have a partner arrangement with Juniper and I would not be surprised if IBM considered buying Juniper up to cover the gap in the network portfolio. IBM has storage covered and has been doing professional services and large scale data center work since the beginning of the industry - so I say they are team #3.

Who is team #4? I think Oracle with their recent Sun acquisition is the answer. I also think since they have a storage arm from Sun that they could easily fill the networking gap by buying Brocade and then potentially developing in house (Sun team) a networking solution complement to the Foundry arm of Brocade or picking up a smaller Ethernet vendor like Force10 or Extreme to help round out the portfolio. Oracle already has a hypervisior and so does Sun so they have lots of software to leverage and they can strong arm customers into buying a "blessed" data center deployment solution that runs Oracle top to bottom and compete against everyone else.

So, who is left out of this game? I guess Dell falls back to a distant #5, and while they technically have server, storage (EqualLogic) and network their story is incomplete and their solution do not align with a complete data center story. Also marginalized are SAP, Microsoft, Novell, RedHat, Citrix and several others that used to have very strategic partnerships which now will not be as important for those four horsemen in the immediate future. And then there are folks like NetApp, F5, Riverbed, Infoblox and others who will have to fit into this ecosystem partner arrangement without being swallowed up.

I think the next 3-4 years are going to be some of the most interesting and fast moving for the data center market space. To think, we haven't even addressed the SaaS/Cloud market space that Amazon (AWS), Microsoft (Azure) and others are rapidly pushing forward. Perhaps Microsoft will leapfrog everyone at the end of the day and the four datacenter horsemen won't even be relevant because you won't need a data center anymore. Hard to imagine? Is that the real apocalypse for data center, where the four horsemen aren't even relevant to the process for most customers?
- Ed

Thursday, July 01, 2010

Microsoft MVP renewal - this time Desktop Experience

I am happy to say I have been renewed as a Microsoft MVP. This year I've been awarded in the Desktop Experience category. This is my third category so far in the program. A brief look back as I was first award in July 2004.

2004 - Windows Server - Networking
2005 - Windows Server - Networking
2006 - Windows Server - Networking
2007 - Windows Server - Networking
2008 - Enterprise Security - Networking
2009 - Enterprise Security - Networking
2010 - Desktop Experience

I still plan on being actively involved and paying attention to what is going on in Security and more specifically in networking. Most of the focus will be around IPv6 as I am now a co-chair on the California IPv6 Task Force.

I needs to extend some thanks to some Microsoft folks:
Jake Grey - MVP Lead
Emily Freet - MVP Regional Manager, Americas
Stephen Rose - Sr. Community Manager - Windows OS
Chris Avis, Harold Wong and Chris Henley - IT Pro Evangelist - DPE - West Region team

Also, thanks to Jennelle Crothers - fellow Microsoft MVP and PacITPros member for helping to make the User Group experience what it is, couldn't have made it this far without your help.
- Ed

Tuesday, June 29, 2010

Cisco Live!

I can't attend Cisco Live! this week but you can see some of the keynotes and other interesting items on the website. Looks like Cisco is announcing a bunch of stuff this year which is good news. The only thing I wish I was attending for was to get involved and be able to ask the tough questions about road map and support for IPv6. A lot of that was done at the Google IPv6 Implementors Event a few weeks ago but upper Cisco management and in particular - the product groups, were NOT at that event. Hard to see how important something is if key folks aren't paying attention.

I hope Cisco starts to push internally feature parity for IPv6 - there is a huge gap in "supporting" IPv6 and in having feature parity to their existing product portfolio. It will come over time but I think a bigger push needs to happen... and soon.
- Ed

Monday, June 14, 2010

NANOG 49 - San Francisco

For those that do not follow what is happening in the Internet Service Provider space much you might find it interesting that the North American Network Operators' Group (NANOG) is hosting their meeting in San Francisco this week. Many folks came in early to attend the Google IPv6 Implementors Conference that happened last Thursday and Friday in Mt. View and are now having fun in San Francisco.

The agenda has lots of IPv6 on it also, I am sort of bummed I will not have time to make it to event but I hope all those attending are enjoying it.

I also got to sit in on a conference call today about how the new North American IPv6 Task Force and the other regional task forces (like the California one) will be structuring and building out resources. Stay tuned for some cool events, labs facilities and meetings to happen across the country soon based out of this work. In the mean time, if you are interested in starting to hear about what will be happening with IPv6 in California you can follow and we will let you know when things are going on.
- Ed

Thursday, June 10, 2010

Google IPv6 Implementors Conference

Attending the Google IPv6 Implementors Conference - the slide decks are available on the site to download. Great presentations on what is happening in the IPv6 space.

Definitely check out the presentations, especially the Facebook presentation - they turned up their experimental IPv6 service in short order and are only using 3 IPv6 addresses to get it deployed, remarkable. To top it off they spend no new capital to make it happen.

I have a feeling the content here is a bit better than what is going on at TechEd but perhaps the gauntlet has been thrown - come on Microsoft, where are you on this?
- Ed

Friday, June 04, 2010

California IPv6 Task Force

I am excited to announce that I have been appointed a co-chair on the California IPv6 Task Force handling the Bay Area. The California IPv6 Task Force (CAv6TF) is chartered with doing advocacy and education regarding IPv6 in the state of California.

With my new appointment comes the obvious questions, what sort of events should the CAv6TF do to cover both advocacy and education? I would love to hear from people about what they feel are the best ways to go about this prior to driving off with the bus half full. Specifically, what format and topics would be the best for the Bay Area crowd who definitely have their own unique requirements.

If you are interested in participating please follow us on twitter and keep an eye out for the relaunch of our website. We believe this will be a big year for IPv6 in general and I am personally excited to start working on the events!
- Ed

IF-MAP - Infoblox and Juniper pushing a new standard

I attended an Infoblox and Juniper joint presentation on IF-MAP and it looks interesting. I think they are attempting to rally support around the standard so a great deal is self serving (asking folks to push for IF-MAP inclusion in RFP's for instance.) That being said, I do think the standard addresses a real need requirement in the industry regarding getting multi-vendor solutions to utilize a common state engineer (which is what IF-MAP really is) which anyone can leverage for different purposes. I primarily see it for a single database location for NAC and policy authentication and enforcement systems to share information in a transparent way and to subscribe to changes in the "state" of something and then take action based on those states.

Vendors who are doing policy enforcement should keep an eye on the standard to determine if they need to have a solution that will work with IF-MAP. What will be really interesting is if monitoring and alerting systems start utilizing the IF-MAP standard to learn what is happening on the network to drive alerting from layer 1 all the way up to application attributes. This already happens on the enforcement side but there is little in the way of alerting and monitoring notification that can do something similar.

I don't think this will change the landscape of the data center models being developed today by the likes of HP or Cisco but this is a critical issue to enterprise and commercial customers as a pain point and if vendors choose to implement IF-MAP then they certainly might gain an advantage in terms of interoperability.
- Ed

Thursday, May 20, 2010

Cisco ASA - failover history

If you spend time working with Cisco ASA's in a failover configuration and you want to get a history of failures on the device the you should use the "show failover history" and "show failover" commands. I typically only use the "show failover" command when working on interface status (what is being watched for failover criteria) and the state of the interfaces themselves plus which unit is active and which is standby or failed. The nice thing about the "show failover history" command is it tells you when things happened in terms of failover status. Here is some sample output:

fw-1# sh fail history
From State To State Reason
07:49:22 PST Feb 4 2010
Not Detected Negotiation No Error

07:50:07 PST Feb 4 2010
Negotiation Just Active No Active unit found

07:50:07 PST Feb 4 2010
Just Active Active Drain No Active unit found

07:50:07 PST Feb 4 2010
Active Drain Active Applying Config No Active unit found

07:50:07 PST Feb 4 2010
Active Applying Config Active Config Applied No Active unit found

07:50:07 PST Feb 4 2010
Active Config Applied Active No Active unit found

14:04:39 PST Feb 11 2010
Active Failed Interface check

14:04:41 PST Feb 11 2010
Failed Standby Ready Interface check

14:04:46 PST Feb 11 2010
Standby Ready Just Active Failover state check

14:04:46 PST Feb 11 2010
Just Active Active Drain Failover state check

14:04:46 PST Feb 11 2010
Active Drain Active Applying Config Failover state check

14:04:46 PST Feb 11 2010
Active Applying Config Active Config Applied Failover state check

14:04:46 PST Feb 11 2010
Active Config Applied Active Failover state check


So you can see that you get a history and state changes of the failover status. What is nice is that you don't have to do a debug to capture these status changes and it is in a nice summary table.
- Ed

Monday, May 17, 2010

Cisco ASA flag descriptions

I'm always trying to remember the flag codes for the ASA connection command. Turns out it is in the help for the "show connection all" command, you just have to add the keyword "detail" so that you can see it. So, the command is:
show connection all detail

or for those that like shorter commands:
sh conn all d

The output for the flags (a bit better organized then what is displayed in the help) is:
A - awaiting inside ACK to SYN
a - awaiting outside ACK to SYN
B - initial SYN from outside
b - TCP state-bypass or nailed
C - CTIQBE media
d - dump
E - outside back connection
F - outside FIN
f - inside FIN
G - group
g - MGCP
H - H.323
h - H.225.0
I - inbound data
i - incomplete
j - GTP data
K - GTP t3-response
k - Skinny media
M - SMTP data
m - SIP media
n - GUP
O - outbound data
P - inside back connection
p - Phone-proxy TFTP connection
q - SQL*Net data
R - outside acknowledged FIN
r - inside acknowledged FIN
S - awaiting inside SYN
s - awaiting outside SYN
t - SIP transient
U - up
V - VPN orphan
X - inspected by service module

I am not sure why there are two "R" values - I am assuming sunrpc is listing UDP only and an outside acknowledged FIN would only be for a TCP session so there isn't a conflict in having the same flag value in use. Anyway, got tired of looking for this all the time when debugging problems so I am posting it here so I can find it for myself later.
- Ed

Tuesday, May 11, 2010

Cisco ASA - Resource Allocation for Multiple Context

When setting up virtual contexts within an ASA it is a good practice to build out a resource allocation plan (classes) and then apply it to all the context that you have configured. By doing this you can safeguard the important FW context that need more resources ( a production FW) vs a less important FW context (think development or test.)

The reason I bring this up is after reviewing a new development FW context that I turned up for a client it was sending a huge volume of syslog data and the template that was built for dev limited the amount of syslog transactions it could send which protected the production FW. The output looked like:
fw-sys# sh resource usage all
Resource Current Peak Limit Denied Context
Syslogs [rate] 4 999 1000 449480 dev
Conns 282 2266 200000 0 dev
Syslogs [rate] 12 2558 unlimited 0 prod
Conns 20631 71951 unlimited 0 prod
Xlates 111601 145005 unlimited 0 prod

As you can see, there was a lot of denied syslog traffic. This was due to the initial configuration and leaving the dev FW context doing debug logging to the syslog server. The traffic dropped substantially after that was removed but it easily could have started consuming more than its fair share of resources on the ASA. In this particular case it is a 5580-20 so there wasn't a lot of concern but on a smaller platform (5510 or even a 5520) where memory and cpu are more constrained it could be a much bigger issue.

Your templates can be rather simple, a default is built automatically and applied to all context out of the box. This default gives maximum resources to all contexts, effectively sharing equally all the resources of the ASA. While this is ok to get thing built out and stood up I would not recommend keeping all the contexts in this configuration if you are going to use the ASA for multiple purposes like production, qa and development.

You can build it out like QoS templates. From a practical standpoint it makes sense to do a default, gold, silver and bronze configuration because I find it unlikely that most organizations would need more than that number to cover all their configuration options. Please speak up if you think I am wrong (but I doubt it.) Cisco doesn't seem to have a lot of documentation on recommendations for how to build this out. They do provide a class overview in their CLI configuration guide for 8.2, though it does not have enough details IMHO. You can apply the class to multiple contexts within the ASA so depending on your resource allocation you can easily oversubscribe resources because you set to many context as "gold" or forget to change the default and then have unlimited resources for all contexts that you build that are automatically given that default class.

Below are some sample templates to review - I set some arbitrary values based off of loads that were being observed. You should adjust the values you require for your environment since perhaps your development or qa context has a lot more hosts then what is reflected below. In other words, don't cut and copy, figure out what is right for your deployment which might require some experimenting to find out what you are actually consuming. This isn't a bad process to go through because it allows you to know where you are at, just remember to account for growth otherwise stuff will stop working or appear to have problems. Remembering that you have classes that might be limiting stuff won't be at the top of your list to check when you are troubleshooting - trust me on this one.
class default
limit-resource Xlates 100
limit-resource ASDM 2
limit-resource SSH 2
limit-resource Telnet 2
limit-resource Conns 5.0%
limit-resource Hosts 100
limit-resource rate Syslogs 1000
limit-resource rate Inspects 100

class gold
limit-resource All 0
limit-resource ASDM 5
limit-resource SSH 5
limit-resource Telnet 5

class silver
limit-resource Xlates 10000
limit-resource ASDM 2
limit-resource SSH 5
limit-resource Telnet 2
limit-resource Conns 20.0%
limit-resource Hosts 10000
limit-resource rate Syslogs 10000
limit-resource rate Inspects 5000

class bronze
limit-resource Xlates 1000
limit-resource ASDM 2
limit-resource SSH 2
limit-resource Telnet 2
limit-resource Conns 10.0%
limit-resource Hosts 1000
limit-resource rate Syslogs 1000
limit-resource rate Inspects 1000

To apply the class to the context it is as simple as:
context prod
member gold
context admin
member default
context lab
member bronze

Because the gold class has the "limit-resource All 0" command it effectively has no limit which means it can starve even the admin context in this example. This might not be ideal in all situations and I will often set limits on the gold class for production FW contexts because there will be more than one FW context that needs to run at the highest class level. In this particular case, just note it is getting everything it wants without question. Sometimes it is good to be king.
- Ed

Tuesday, May 04, 2010

Root signing and DNSSEC

Time to start paying attention to DNSSEC and its potential impact on your ability to query against root servers through your firewalls. You can keep up to date at Root DNSSEC's website regarding the status and potential issues they are seeing.

Of immediate importance is the fact that your firewall may have a default configuration to discard UDP DNS traffic larger then 512 bytes in size. This was a common practice and it is now going to be an issue with the larger payloads that DNSSEC utilizes. Basically you just need to turn off this restriction and everything should work fine, for now. Platforms like the Cisco PIX, ASA and Microsoft ISA and TMG should all be checked.

Many people haven't noticed any issues with the first few root servers being signed and doing this behavior because their servers would not get a response back from the queried root server (blocked by the firewall for the payload size issue) and the server simply went to the next root server in the list which likely wasn't doing DNSSEC and therefore everything worked as expected. This is going to change starting this month going into June and IT Pros should be aware of the issue and review their firewall configurations to make sure they are not creating a problem for themselves.

Also, some of the TLD's are starting to sign also and I believe sometime in 2011 .com should be signed. If you are running DNS on Microsoft Windows 2008 R2 then you should read the DNSSEC Deployment Guide, it can walk you through requirements of what to do to get your infrastructure signed and working.
- Ed

Thursday, April 29, 2010

Windows BranchCache OS requirements

I was asked at the Windows Intelligence event I presented at on Monday if there were specific requirements on what version of Windows 7 could be used with BranchCache. It turns out there are, you can only use Windows 7 Enterprise and Ultimate Editions to take advantage of BranchCache - even if you are doing a workgroup configuration and not a domain.

The step by step guide has the requirements for both the client and the server and can be found here.

It appears that Microsoft is assuming small and medium sized companies should be using Enterprise or Ultimate Editions to take advantage of newer technologies like BranchCache and DirectAccess. I will ask around to see if this will be changing at all in the future and post back up what I hear. I am a bit surprised to read that Windows 7 Professional didn't make the cut for BranchCache as a feature but licensing is a complicated thing that honestly I think very few folks truly understand, even within Microsoft.
- Ed

Tuesday, April 27, 2010

Windows Intelligence DirectAccess and BranchCache presentations

I had a fun time yesterday presenting on Microsoft DirectAccess and BranchCache at the Windows Intelligence NorCal event. Thank you to everyone who attended the event, I appreciate the discussion and feedback and as promised here are the slide decks for download.



If you attended my sessions you might have missed sessions by Jennelle Crothers and Steve Evens who both gave excellent presentations from what I heard (I was giving mine so I couldn't be in two places at the same time!) and should be making their presentations available on their sites also. Doug Spindler also presented but I am not sure where his presentations are posted but I will find out and update here shortly.

It was also nice to see the West Coast Microsoft DPE IT Pro Evangelists at the event, Chris Henley, Chris Avis and Harold Wong. They are all wonderful presenters and totally passionate about what they do - a big shout out to them - you guys rock!
- Ed

Friday, April 23, 2010

Cisco IPSec VPN Client - 5.0.7 Released- Win7 64-bit support

Cisco has officially released a 64-bit version (non-beta) of the IPSec VPN Client, version 5.0.7 (vpnclient-winx64-msi- available for download. Finally a Cisco supported 64-bit version of the IPSec client for Windows 7! It is available for download on CCO but requires a valid CCO login and current contract to get the code.

Do read the release notes, there are caveats for Windows Vista in the release notes so if you are planning on running it on 64-bit Vista make sure you review those. Also, it is important to note there is a specific section titled "Windows 7 and Vista Window Auto-tuning Feature Might Cause Network Timeout Problems" that states:
"Windows 7 and Vista support a feature called "Receive Window Auto-Tuning" that continually adjusts the receive Windows size, based upon the changing network conditions.

Some people reported that auto-tuning causes network timeout problems with some applications and routers. If you have experienced such problems, you can turn it off using the following procedure:

Step 1 Open an elevated command prompt.
Step 2 Enter the following command to disable auto-tuning:
netsh interface tcp set global autotuninglevel=disabled

If this solution does not fix the problem, you can turn it back on, as follows:
Step 1 Open up an elevated command prompt.
Step 2 Enter the following command to enable auto-tuning
netsh interface tcp set global autotuninglevel=normal

To view the states of the TCP global parameters, use the following command:
netsh interface tcp show global "

So for all of you that have to support Windows 7 64-bit and still have Cisco VPN Concentrators, PIX's and ASA's doing IPSec only you have a valid supported solution now! I don't know why it took so long but it is nice to know you don't have a pay the AnyConnect essentials license if you don't want to. I do like many of the features and advantages with AnyConnect, it just seemed wrong that to get VPN for a 64-bit OS you HAD to use AnyConnect only and pay the premium for the client.
- Ed

Wednesday, April 21, 2010

Cisco ASA - PAT performance and logging

It isn't a common issue for most small to medium businesses but occasionally you have to remember that when doing port address translation (PAT) or overloading a single IP address with multiple IP's behind it that there are actual limitation on how many session you can do simultaneously. Remember that for a single IP address can only do 65536 TCP sessions. If you remove the first 1024 you are down to 64512.

So why is this important? Turns out if you have enough hosts generating requests behind the ASA you can run out of ports and if you generate traffic quickly enough you will start getting the following error in the log files:

%ASA-3-305006: portmap translation creation failed for tcp src XXX dst XXX

The message description from Cisco is here and shows the format as:
%PIX|ASA-3-305006: {outbound static|identity|portmap|regular)
translation creation failed for protocol src interface_name:source_address/source_port
dst interface_name:dest_address/dest_port

I don't think this will be a common issue as it takes a lot of hosts to generate this sort of load doing thousands of requests a second. To exceed the threshold for instance you would need 250 hosts generating approximately 260 requests per second sustained per host and they would have to keep those sessions open for the ASA to start having this issue.

Luckily it is a rather fast and easy fix if you have more than one Public IP address available. You can simply add another global statement to the ASA with more public IP's to overload and the ASA will happily start spreading the load across the new IP addresses that you add. In terms of syslog, this is severity level 3 (error) so it should show up in most syslog data that is collected. Definitely recommend running Splunk, it makes finding these sort of errors so much easier!

Also, while it may be saying a portmap translation failed the ASA itself doesn't run out of resources from a CPU or memory standpoint. The ASA will continue to run along nicely it just won't be able to generate new TCP sessions as fast as your hosts may be requesting. This makes it really important to be looking at your log files to know what is happening. The problem is a function of how you are doing overloading, not a limitation of how the device performs that function.
- Ed

Friday, April 16, 2010

Cisco ASA - how to see your pre-shared-key

One of the annoying things about managing pre-shared keys for both site to site vpn tunnels and group pre-shared keys for client vpn tunnels is the fact that if you do a show run they are starred out (*) in the configuration file.

So you will see something like:
pre-shared-key *

If you need to recover back your keys because you have lots of folks running around with Cisco IPSec VPN clients with a standard PCF file and you can't remember what the group pre-shared-key is or don't have it documented you can do the following command.

more system:running-config

This will output your running-config file with the pre-shared-key variable in clear text.

Obviously this is useful for site to site keys also because you might have a tunnel set up with a third party vendor or a partner and you can't remember the key at all because you made it up on the fly with the other network engineer on the phone (That never happens... really.)

This will NOT show you the enable or passwd values because those are actually encrypted. You will have to use other tools to break those or do a standard password recovery process.
- Ed

Tuesday, April 13, 2010

Cisco ASA syslog over TCP - Potential to stop forwarding traffic!

I ran into an interesting configuration issue with the Cisco ASA and doing syslog via TCP to a syslog server. It turns out that Cisco has a feature in the ASA that when pushing logging information to a sylog server via TCP that if the server does not respond it will stop the ASA from building out new session flows and therefore it will stop forwarding traffic. It seems that existing TCP sessions continue to allow traffic to flow, the rational being that the logging of that flow being established was sent to the syslog earlier and therefore it is audited event. Apparently this was a specific request from some government agencies who did not want their firewalls to pass traffic if logging was not possible. Reasonable enough but it isn't particularly well documented and does not alert you of it's default behavior at all when you enable the syslog command using TCP.

You can find all the details in the Cisco ASA 5500 command line configuration guide (this one is for 8.2 code release) and look under "Sending Syslog Messages to a Syslog Server" to see the parameters. They do provide a optional command switch that allows the ASA to continue working even if the syslog server is down.

So how can this be a problem? It turns out that a lot of shops rotate their syslog files and will write a quick script to stop their daemon for syslog, change their logfile and then start the syslogd daemon again. This causes the TCP session to fail because the syslogd daemon is no longer keeping the socket open. Depending on the thresholds you have built on the network this can cause your ASA to stop forwarding traffic. Worse, it is difficult to figure out what is happening because existing log flow sessions (perhaps and ssh or database session) will still be working but anything new you attempt to send through the ASA will not. Also, the ASA does not consider this a flag to attempt to failover to a standby unit - in fact, there is no way to even monitor the syslog process to flag it as such. I think that is something that needs to be added so the failover unit can then test to see if it can connect to the syslog server and if so, failover so that your firewall continues to forward traffic as expected.

The error you will get in the local log is:
%ASA-3-201008: Disallowing new connections.

It tells you what is happening - it doesn't tell you why it is happening though - very frustrating.

This is not an issue if you are using UDP for syslog traffic on the ASA's as they assume if you are doing UDP to the syslog server it is not guaranteed delivery and therefore you don't have to meeting the same standards for logging or security.
- Ed

Friday, April 02, 2010

Cisco Nexus 5k and 2148t caveats

I ran into a few more items I think warrant a post regarding the Cisco Nexus 5k series and the 2k fabric extenders (fex.) For those that are evaluating the Nexus 5010 and 5020's with the 2148t fex for layer 2 switchport capacity there are two gotchas that I think failed to get mentioned often enough in the pre-sales engagement.

1. The 2148t fex is not capable of doing a port-channel (ether-channel) with two or more ports on a single chassis. This is not a limitation that can be fixed with software either, it is a limitation in the ASIC's themselves on the 2148t. I think this is a huge failure. That being said, I believe it is road mapped as a feature fix for the new 2248TP and 2232PP models.

2. If you do need to do a port-channel to a single host from a 2148t you need two 2148t's and you have to run a Virtual Port-channel (vPC) using the 2148t's that are each single homed to the 5k's upstream (in what is called single homed fabric extender vPC topology.) You can only do a maximum of 1 Gigabit Ethernet port to each 2148t (because of the limitation above) and then the 5k's will do the logical vPC to make them a port-channel. So a 2 Gigibit max port-channel with the 2148t single homed to the 5k's. Again, because of the limitation above I see this as poor engineering work around to the failure in design.

So, with those limitation on the 2148t fex's there is something else from a design standpoint that is frustrating. You have two recommended options when designing the 5k/2k deployment, you can do a single homed fabric extender vPC topology or a dual homed fabric extender vPC topology. Here are the diagrams from the Cisco Nexus 5000 Series NX-OS Layer 2 Switching Configuration Guide.

The single homed:

and the dual homed:

What is not mentioned or documented anywhere about this configuration is if you can mix and match the topology between single homed and dual homed 2k's on a single pair of 5k's. I can only find one reference that mentions this vPC mixed topology option. Even with this option you are still burning up a second 2148t just to get port-channeling as an option if you don't want to use the 5k's directly. Burning up the few 10/1 Gig mix use ports on the 5k's for 1 Gig port-channels seems to be a huge waste and very expensive.

So there you have it, some items to mull over while planning out your data center design with Nexus and some hard hitting questions to ask your Cisco SE's when they show up talking about cost per port and maximizing functionality and flexibility per port. On top of the above items the 2148t also only does 1Gig, it doesn't support 10/100 at all, not a show stopper but another nagging item that limits some deployments. Don't get me wrong, I'm a fan of the new data center design and topology model that Cisco is building and with OTV coming out in the 5.0 NX-OS code release some really exciting things are going to be possible. Just keep in mind you might be buying an extra one or two 2k's to accommodate some of the systems that have the port-channel requirements for higher availability.
- Ed

Tuesday, March 23, 2010

Cisco Nexus command - Checkpoints and rollbacks

Using the Cisco Nexus 7010, 5010 and 2148's has changed some of the habits I have traditionally used for the Cisco IOS command set. Some of the new Nexus commands have become second nature and I now miss them on IOS. Being able to use grep is one I really wish was incorporated into IOS. I am used to having it with the ASA platform and now with the Nexus platform - going back to IOS 12.x and not having it there is annoying.

A new command that is really useful on the Nexus platform is checkpoint. There are several things that are unique about checkpoints and how you can use them. First, checkpoints are primarily used for rollback situations. They allow you to make changes on the system and if required due to an error rollback to a known good configuration on the system. There are three rollback types.
Atomic rollback is done when the configuration can be applied with NO errors.
Best Effort rollback will ignore errors and push the configuration onto the system.
Stop At First Failure will process the rollback request until it hits an error and then stops.

The default rollback type is Atomic and this is likely the most common rollback method you would use on a production environment. I am not aware of many folks wanting to rollback to a "Stop At First Failure" or "Best Effort" scenario situation unless true desperation has kicked in. There might be a case of the order of rollback if you are using VDC's and moving physical resources from one VDC to the other in which case perhaps Best Effort might be useful.

Also of note, the rollback feature must be used per Virtual Device Context (VDC), in other words, you have to run the command in each VDC. This is expected behavior as each VDC is it's own NX-OS instance and you have to run all the same commands to get the desired behavior out of the NX-OS platform.

The command itself is very simple:
checkpoint {checkpoint name} description {a description} | filename {path and filename}
Example: checkpoint cp-running-config-known-good-2010-03-22 description checkpoint of running config

There are some restrictions on the checkpoint name (max length 80 characters) and there are restrictions on the filename (max length of 75 characters and filename can't start with the word "system") but otherwise it is pretty straightforward process to get this going. I am using this on NX-OS version 4.3.1, earlier versions had more restrictions on file names and such so read the documentation if you are on an earlier release.

To see what the checkpoint command does you can use the show commands. To see all the checkpoints that are in a given VDC:
show checkpoint all
show checkpoint summary

The checkpoint command basically keeps a small database of checkpoints to allow you to rollback to a specific one and calculates the differences between a current state or checkpoint and that checkpoint you want to move to. It will generate a rollback script when you use the rollback command. If you want to see the differences that are being generated you can do that too:
show diff rollback-patch {checkpoint source name | running-config | startup-config | file filename} {checkpoint destination name | running-config | startup-config | file filename}
Example: show diff rollback-patch running-config checkpoint cp-running-config-known-good-2010-03-22

To actually do a rollback:
rollback running-config {checkpoint cp name | running-config | startup-config | file filename} {atomic | best-effort | stop-at-first-failure}
Example: rollback running-config checkpoint cp-running-config-known-good-2010-03-22 atomic

To see the status of rollbacks:
show rollback log

You can also clear out the checkpoint history and files, use the command with caution.
clear checkpoint database

This is a VERY useful command to build into your scripts prior to pushing out production changes on gear. It allows you to have a well known state stored locally and be able to rollback to it quickly in case of problems in your scripts. Awesome!
- Ed

Friday, March 19, 2010

Cisco IPSec VPN Client - 5.0.7 BETA - Win7 64-bit support

Cisco has a beta version of the IPSec VPN Client out, version 5.0.7 BETA (vpnclient-winx64-msi- available for download. It appears they got the message about the need for a 64-bit version of the IPSec client for Windows 7! It is available for download on CCO but requires a valid CCO login and current contract to get the code.

This is great news for Microsoft customers that have Cisco ASA's, PIX's or VPN 3000 concentrators deployed and their IT team is migrating their client OS's to Windows 7. Many OEM's are shipping the default OS as Windows 7 64-bit to take advantage of the all the RAM systems can support today.

Cisco is still pushing their AnyConnect client + ASA platform to compete with Microsoft's DirectAccess solution on Forefront UAG. Cisco is positioning the AnyConnect client as an always ready vpn client that auto-reconnects and is seamless in that process. I don't know if I am buying that position given the flexibility of what DirectAccess can do and how you can manage the policies with GPO's in AD in the Microsoft solution. Plus the fact that once DirectAccess is deployed there is no VPN "client" per-say, it is more of a policy function which is a very unique position in terms of product positioning.

At least if you have a Cisco IPSec solution deployed today you can now leverage it with Windows 7 without requiring any third party software to get around the problem of being on a 64-bit OS.
- Ed

Wednesday, March 17, 2010

Cisco ASA 8.3 code - read the release notes!

Cisco released version 8.3(1) for the ASA on March 8th. The release notes have some interesting items. Specifically I think the important ones are outlined below and are quoted from the release notes or related documents.

Before you just upload 8.3(1) READ THE MIGRATION GUIDE!! They have made some BIG changes in how NAT and PAT work. The new code will convert your configuration, you have been warned! Save your work prior to the upgrade. Let me repeat that, save it someplace other than the ASA so you have a clean copy prior to the migration.

From the migration document:
"The major changes in Version 8.3 that require migration are:

Real IP addresses in access lists, where access lists are used in supported features—When using NAT or PAT, you used to have to specify the mapped addresses and ports in an access list for all features that use access lists. Now, for several supported features, you must use the real, untranslated IP address and ports. (Other features continue to use the mapped IP address).

NAT—The NAT feature has been redesigned for increased flexibility and functionality. All NAT and NAT-related commands have been redesigned.

Named Network and Service Objects—Network and service objects are automatically created and used for several features, including NAT and access lists that are used for access rules. "

If you have existing static entries in your ASA configuration, and I would guess most folks do, if you are allowing any inbound services then get ready for a migration. The static statements stay the same but the ACL and object-groups change! The important things is to figure out what NAT configuration you match and how it will look after the migration. The migration guide has a table that outlines this for you. Read the migration guide to make sure you understand how your configuration is going to change, this is not a minor change!

Also, in the release notes:
"To run Version 8.3 in a production environment, you need to upgrade the memory on the Cisco ASA 5505, 5510, 5520, or 5540"

The memory requirements are outlined in the doc, needless to say this is not a minor release as they are requiring a pretty big memory uplift even for the smallest of the ASA's. Make sure you order the memory upgrade kits if you plan to run the 8.3x version code at all.

"You can now create named network objects that you can use in place of a host, a subnet, or a range of IP addresses in your configuration and named service objects that you can use in place of a protocol and port in your configuration. You can then change the object definition in one place, without having to change any other part of your configuration. This release introduces support for network and service objects in the following features:


Access lists

Network object groups

The following commands were introduced or modified: object network, object service, show running-config object, clear configure object, access-list extended, object-group network."

The change to allow the use of objects is actually a good thing but it is going to take awhile before people are used to using it in NAT statements for instance. I am actually excited about this one as it will make life much easier for those that plan out their network and resources well. It also allows for much easier change control and scripting to get hosts in and out of rules on the firewall.

"Displays the timestamp, along with the hash value and hit count, for a specified access list. The following command was modified: show access-list. "

For those of us who spend a lot of time looking at access-list hit counts having the timestamp and hash value is a huge plus. It is the simple things. Now if only they would allow multiple | and grep commands.

There are some items about licensing that are worth noting and if you are doing UC they have made changes to improve UC-IME. There is plenty to read up on, definitely a lot to keep up on.
- Ed

Friday, March 12, 2010

New Windows networking blog and other blogs to check out

Joe Davies is maintaining a new blog location that I think will be useful for those that are into network topics about the Windows platform. The blog is up on technet and is The Windows Server User Assistance Networking (WSUAN) team's blog. It covers all things networking related including subjects like DNS, DHCP, DirectAccess, BranchCache, Windows Firewall and my personal favorite, IPv6.

Also worth checking out is the Springboard Series blog that Stephen Rose runs - good stuff there for IT Pros.

There are several other personal blogs worth mentioning. My short list, for those that are interested, of ones I look at.
Jennelle Crothers at
Susan Bradley at
Colin McNamara at
Scott Lowe at
Steve Riley at
Jeremy Stretch at
Jessica Deen at

There are a ton more out there that are really good, when I get a moment I will dump down my list to another post.
- Ed

Thursday, March 11, 2010

Cisco Nexus vPC port channel notes

Ran into an interesting issue with turning up vPC's between a Cisco Nexus 7010 and Cisco Nexus 5010's in regards to have the underlying port channels are configured. It turns out that if the 7k and 5k pairs have different port channel configurations they do not appropriately block the way you would think they should but instead continue to bounce the port. The mistake in my configuration was simple, the 5k was configured as such:
int e2/1
channel-group 10 mode active

int e2/2
channel-group 10 mode active

int p10
switchport mode trunk
vpc 10

The 7k was set as:
int e1/1
channel-group 10

int e1/2
channel-group 10

int p10
switchport mode trunk
vpc 10

The 5k was going LACP and the 7k was forced on. What was odd was that the vPC reported down as expected but the ports kept bouncing up and down which then forced spanning-tree to go into effect, not something I desired to have happen in this misconfiguration situation. I would have been ok with the port(s) staying down due to a port channel type mismatch but it kept trying to bring the ports up which was causing loops and causing spanning-tree to do its thing.

What I found interesting was that if I simply disabled one of the 7k ports, for instance int e1/2 everything would stabilize out since there was not a loop (no spanning tree) and the vPC still reported down. Expected behavior when you think about it but not intuitive when debugging the problem.

Once I corrected the port-channel configuration and forced everything on the vPC came up and I was able to fail interfaces with no interruption of traffic. My concern regarding this is the impact it had on all the vlans associated with that trunk, so it is critical to make sure that adding any new 5k's to the 7k's that the port configurations are double checked or an outage is possible due to this problem. I sort of wish the ports would error disable out due to vPC determining that the port channel type doesn't match to help prevent the L2 loop situation. This would keep the ports from bouncing and therefore prevent spanning-tree from having to go into effect to do loop prevention.
- Ed

Friday, March 05, 2010

Cisco Nexus 7000 - 32 port 10Gig linecard caveats

I have been working on a set of Cisco Nexus 7010's that both have 32 port 10G line cards. I have been able to get virtual PortChannel (vPC) and Virtual Device Context (VDC) set up and working on the platform, definitely some of the coolest technology out there for the data center outside of server virtualization. That being said, there are some interesting caveats that folks should pay attention to when planning out the total number of usable 10G ports on the 32 port line card.

First, the ports are "grouped" in sets of four (1,3,5,7 for instance) - each group of four is capable of doing 10G in shared mode or a single port in the group (the first one) is capable of 10G in dedicated mode and the other 3 are shutdown. Effectively the card is over subscribed 4 to 1 - not horrible. This means it has a total of 80Gbps into the backplane. All this is pretty well documented by Cisco so it's not new or an unknown.

So, what isn't obvious when you are doing planning and design? Turns out if you are designing a network to utilize VDC's in the 7k's than you need to pay closer attention to the number of usable 10G interfaces you get per VDC. Because the ports are grouped and sharing ASIC resource they cannot be individually allocated to a VDC but have to be allocated as a group.

So if you plan on needing just a single 10G interface in a VDC in reality you will have to allocate a full group of 4 10G ports (and in the correct grouping order.) Even if you decide you want to use the dedicated mode to allow full 10G for a single 10G port and leave the other 3 in the group shutdown you still have to allocate all 4 10G ports to the VDC. When you think about it this makes sense but often in the design phase while white boarding out a solution an engineer will simply allocate a single 10G interface for a VDC because that is all that will be "used" but in reality you will be allocating 4 not 1.

This can throw off your 10G port count by a lot depending on how many VDC's you plan to run. You can only run up to 4 VDC's on the 7k's today though in theory there is no reason they couldn't raise that as it is only a software limitation. So, in reality all VDC's should be allocated 4 10G ports even if you only need 1 10G port for something. That way you don't run into any surprises when trying to deploy a solution. Also, for deployments that make use of two 7k's (which really is how you should do it to gain the benefit of vPC) I would trade a dual supervisor per chassis configuration for an additional 32 port 10G line card if I had to chose. It makes more sense to have more flexibility in the number and allocation of 10G interfaces with a single supervisor per chassis solution since you will have two chassis' anyway for redundancy. Just a thought when planning and budgeting.
- Ed

Sunday, February 28, 2010

Update - Windows 7 - Microphone control - where is it?

Ok, now that I have the RTM of Windows 7 installed I realized they have changed the microphone controls and made it a bit easier but still confusing. So, here are some screenshots with a brief write up so you can find it. Crazy frustrating for those of us who do livemeeting, gotomeeting and webex stuff on a regular basis!

You will need to right mouse click on the speaker icon in the taskbar in the lower right and it will pop up a menu option. The third one down is "Recording devices" - select this one.

From there it will pop up a "Sound" window - pick the second tab which is "Recording" and it should display a microphone. Highlight it and then pick properties in the lower right of the window.

When that window pops up you will finally be able to adjust the properties of the microphone. You will need to go to the third tab labeled "Levels" in order to set the microphone gain or adjust it at all with the exception of muting, sort of. Even though it is shown muted in this window if you use the "Volume Mixer" window it will not display it as "muted!" but it is actually muted. Am I the only one confused here?

You get to the Volume Mixer window by left mouse clicking on the speaker in the lower right of the task bar and selecting "Mixer." It will pop up a window that shows all your current speaker and microphone devices as shown. I have selected to mute it here and if I toggle it back and forth it does what I expect but it does not match what is in the Microsoft Properties tab! I think that is a bug.

Glad they made it a bit easier to at least mute the microphone but weird that the two items don't display the same "state" for the microphone.
- Ed