Howfunky.com

Relaunch of the California IPv6 Task Force

noreply@blogger.com (Ed Horley) — Tue, 07 Sep 2010 17:00:00 +0000

For those who are interested in IPv6 please check out the relaunched California IPv6 Task Force website. The CAv6TF will be helping with the gogonetLive! IPv6 event on Nov 2-3, 2010 in Silicon Valley (San Jose area at this point) so please keep the date reserved, it should be a great chance to interact with other IT professionals interested in IPv6.

I am excited we are finally making the CAv6TF website available via IPv6 too and also that the Task Force is now active again after a few years of hiatus due to the majority of the CAv6TF members driving the National IPv6 story. I am now serving as the Co-Chair of the Task Force handling the San Francisco Bay Area. If you have any comments or interest in helping with the CAv6TF please contact us at info@cav6tf.org.

If you have not yet joined gogoNET and you are interested in learning more about IPv6 this is a good place to start getting connected with other IPv6 folks so consider signing up. They also offer a free IPv6 software client service through Freenet6 so you can run IPv6 no matter where you are which is a nice option if you do not have a native IPv6 service yet. Another option is Hurricane Electric's IPv6 Tunnel Broker service which I have used for years for my home configuration.
- Ed

VMworld 2010 - August 30 - Sept 2nd

noreply@blogger.com (Ed Horley) — Mon, 23 Aug 2010 17:00:00 +0000

Another VMworld is around the corner, just a week away now - it is again in San Francisco at Moscone Center and should be an interesting event as always. Cisco, EMC and NetApp will all have big presence at the show for sure and it is nice to see what they have in terms of product lines and what they are doing in the virtualization space. What makes me want to attend the expo every year though is the small niche companies that have interesting products or way of solving problems that the big boys haven't figured out or haven't thought about.

Often it is the case that these folks get purchased and integrated into one of the bigger players so part of the fun is getting to see them while they are still small, still agile and still interesting plus getting their swag while they still have their own logos and tag lines and such.

I expect to see some announcements associated with the show regarding partnerships, new product launches and perhaps some heated debate too. It looks like Microsoft will once again be relegated to a regular small booth which is a shame considering the majority of deployments of VMware's products are to support Microsoft platforms. I think VMware is making a mistake not allowing Microsoft to have a bigger presence at the event and simply asking them to live within some guidelines regardings Hyper-V.
- Ed

Splunk User Conference - August 9-11, 2010

noreply@blogger.com (Ed Horley) — Fri, 06 Aug 2010 17:00:00 +0000

One of the tools I use to help me wade through the mass of data that networking devices spit out is Splunk. Splunk is easily one of the most useful apps to run if you have to try and figure out what is going on with regards to firewalls, ids, network logging and anything else that will output to syslog.

If you happen to live in the San Francisco Bay Area you are in luck too, next week is the first annual Splunk User Conference and you can still sign up. I wish I could make it myself, especially since I live local to the event.

Splunk has really expanded the capabilities of the product but to be honest, the simple search and filtering to help me write better access control lists for firewalls is a huge time saver, just being able to do that quickly and efficiently makes it worth it's weight in gold.
- Ed

Why paying attention to IPv6 is now important

noreply@blogger.com (Ed Horley) — Thu, 29 Jul 2010 17:00:00 +0000

As a personal interest I have been working with and following developments of IPv6 for several years. I started presented on IPv6 back in 2006 because of what was happening with Windows Vista and the changes that Microsoft was doing with the OS and their new networking stack. Here we are in 2010 and I think we are past early adopters in regards to IPv6. In fact, I now think if you are not paying attention to what is happening with IPv6 it could start impacting your ability to perform your job soon, especially if you are an IT Professional.

So, who do I think will be impacted the most by the transition and more prolific use of IPv6? I think you might be surprised.

The standard answer is network engineers and granted they indeed will be rolling out and maintaining dual IPv4/IPv6 networks for years to come but I actually don't think IPv6 will be as much of a challenge for network engineers to get up and running assuming they have moderately newer network equipment. Granted, there are issues with lack of feature parity but that will be resolved over time and will be fast tracked when equipment manufactures realize they are losing sales due to the lack of the parity.

Next on the list is system admins. I think many will find IPv6 to be a bit more of a challenge in regards to the differences in behavior of the protocol and getting worked out the behavior differences of applications as a result. This is a huge issue for client machines in terms of what OS you are running on your desktop and what the server does or does not support. I would argue that the majority of system admins know enough IPv4 networking to allow them to do their job but likely will have some challenges with differences in IPv6. I know there are some great system admins out there who could run networks also so obviously this a wildly general statement but I still feel there is going to be a bigger learning curve for system admins than they care to admit. Perhaps it is time for Microsoft to bring back a dedicated networking exam - like the old MCP TCP/IP exam?

The surprise group is application developers and database admins. Just think about how much code has been written out there to account for IPv4 addresses. IPv4 addresses are 32bit and I would imagine the majority of applications out there are storing that value under a declared INTEGER. I could be wrong - maybe they are all stored as a STRING instead but I have a feeling that isn't the case. IPv6 addresses on the other hand are 128bit and likely the majority of applications will have to be modified to account for the new size, difference in how they are represented (in HEX not DEC) and also the fact that the application might potentially have to pay attention to which interface it is directing traffic through. This doesn't even cover all the databases out there that are storing IPv4 information and the SQLNET statements all based around IPv4 to query those databases.

To top it all off, IPv6 can represented an address in multiple ways due to the zero compression option. So searching through logs or analyzing output could be an additional issue unless some standards are agreed to in advance in terms of how to store and represent an IPv6 address. So imagine trying to correlate information from multiple systems and they can't match stuff because the IPv6 addresses are represented differently in each system. I think some of these issues will be the biggest road blocks to overcome in the months and years ahead for IPv6.

So, why is it important to pay attention to IPv6 now? It is important because the adoption and momentum behind the protocol has already begun. Major content providers like Facebook and network providers like Comcast and content delivery providers like Limelight have all deployed IPv6 already and are doing their trials now. If you have no knowledge or understanding of IPv6 how will you address your business needs when you need to either access content, deliver content or work with a network when you don't understand the protocol they are using to move traffic?

In short, if you don't have a working understanding of IPv6, you are already behind. Take a quick quiz. Do you know what behavior Windows 7 has when it has a public IPv4 address? What is different if it gets a public IPv6 address? Which protocol does it use for DNS resolution if it has both an IPv4 and IPv6 address? Does the type of IPv6 address it has matter to the default behavior? This is all just for Windows 7, now do this for OSX, Windows Vista, Windows XP, Windows Server 2003, 2008, 2008R2, Linux and Solaris. How did you do?
- Ed

Cisco ASA NetFlow configuration

noreply@blogger.com (Ed Horley) — Tue, 20 Jul 2010 17:00:00 +0000

I have been setting up a lot more NetFlow on Cisco ASA's recently. Mainly due to the request for more visibility into the traffic that is consuming Internet bandwidth and for compliance reasons. It seems that even with proxy services and other solutions many IT organizations still have a poor understanding of the actual traffic traversing their network.

Since NetFlow is limited in terms of platform support (specifically the Cisco Catalyst 3k/2k switches do NOT support it) but the Cisco ASA does I have been asked to turn it on the ASA to have a better idea what is going across the network. Cisco has a nice Introduction to Cisco IOS NetFlow if you need to run it on the routing or 4500/6500 platforms which is a great way to go in addition to the ASA as you can then see what is happening between devices on the network also.

Chapter 75 in the Cisco ASA 8.2 CLI Configuration Guide covers how to set up a NetFlow configuration. Here is a short script to get it up and running quickly. A couple of caveats, read the config guide because it covers the parameters for timing and limiting what you are collecting. This script is a "let's get this going, send me everything" sort of solution. Not optimal for heavily loaded ASA's. Should be good enough to get you going though.

!#####
! - NetFlow script for Cisco ASA
!
! - ACL to catch all IP traffic - to specify the traffic you are interested in
access-list flow_export_acl extended permit ip any any
!
! - set up the destination server ip and template rate
flow-export destination {interface name} {IP address} {port #}
flow-export template timeout-rate 1
!
! - build out the class-map for the flow that matches the ACL
class-map flow_export_class
match access-list flow_export_acl
! - or don't use an ACL by using
match any
!
! - build out the policy-map
policy-map flow_export_policy
class flow_export_class
flow-export event-type all destination {IP address}
!
! - apply the policy-map to whatever global policy you have or make one
service-policy flow_export_policy global
!
! - if you have an existing policy-map apply the class-map into that one
! - for instance the default ASA service-policy for global is global_policy
! - so you could add the class-map to it by doing
policy-map global_policy
class flow_export_class
flow-export event-type all destination {IP address}
!
!#####

You can get information about what the ASA is doing in terms of the flow output by using the following commands:
show flow-export counters
show service-policy global flow ip host {source IP} host {dest IP}
show access-list flow_export_acl

Obviously you need some sort of NetFlow collector. There are a lot of professional and free tools to do this and there are some great vendors doing this. That being said, I have used Plixer's Scrutinizer free product to at least get folks up and working and have a functional tool to look at until they can decide what tool they want to use. It does and excellent job of showing what is possible in terms of reporting and information gathering. That being said the following companies also have NetFlow commercial products you should consider or have free offerings that can be used.
AdventNet
NetQoS - part of CA now
NTOP - opensource tool
SolarWinds

I am leaving off a ton of vendors in the list but I have found if I list everyone who is doing a solution then folks who are trying it out freeze up and can't pick one. I know, not a great reason but I rather have them use something than nothing at all.

Cisco owns the NetFlow name but there is a standards version of NetFlow supported by many other networking vendors called sFlow. Basically it provides the same sort of function but on other vendor equipment. This means that almost all NetFlow collectors can work with sFlow. So if desired you can collect from non Cisco devices that support sFlow to the same collector to get a more complete view from around your network.

If you have not deployed and made use of NetFlow I really recommend doing a quick trial run. You may be surprised by what you find. I've had clients discover employee's watching video and tv shows being pulled from foreign countries (some of questionable content), others consuming high bandwidth across tunneled links they did not know they were running and lots of other interesting items. Many have been surprised how much IPv6 and tunneled IPv6 they are running on their network. It is a great tool so check it out.
- Ed

The four datacenter horsemen - who will they be?

noreply@blogger.com (Ed Horley) — Thu, 15 Jul 2010 17:00:00 +0000

I've been watching some of the network manufacture market transitions happening right now and the rapid changes going on in the data center and networking market space. I think there will be four main players who I am calling the four datacenter horsemen who will be calling the shots in the future - leading players of the apocalypse I guess. This is all me having fun guessing what will go on so don't hold me to it and of course, this is only my opinion.

Team #1 is made up of Cisco/EMC/VMware due to their partnership arrangement. This partnership for taking over the data center is what caused all sorts of realignments in the industry in the first place. Specifically, with the falling out of HP and Cisco over servers and HP now having a full portfolio of network, servers and storage to go after the data center space that Cisco has traditionally shared with HP it begs the question - what next?

Granted I agree that HP, team #2, does not have a fully competitive network/storage solution when compared to the Cisco Nexus + UCS solution but they are close enough that folks are picking sides. Certainly it can be argued that HP has much more depth in the server market space and a longer deeper relationship with Microsoft to cover the hypervisor gap.

So, what other teams are out there? We can't leave IBM out of the game, they have a partner arrangement with Juniper and I would not be surprised if IBM considered buying Juniper up to cover the gap in the network portfolio. IBM has storage covered and has been doing professional services and large scale data center work since the beginning of the industry - so I say they are team #3.

Who is team #4? I think Oracle with their recent Sun acquisition is the answer. I also think since they have a storage arm from Sun that they could easily fill the networking gap by buying Brocade and then potentially developing in house (Sun team) a networking solution complement to the Foundry arm of Brocade or picking up a smaller Ethernet vendor like Force10 or Extreme to help round out the portfolio. Oracle already has a hypervisior and so does Sun so they have lots of software to leverage and they can strong arm customers into buying a "blessed" data center deployment solution that runs Oracle top to bottom and compete against everyone else.

So, who is left out of this game? I guess Dell falls back to a distant #5, and while they technically have server, storage (EqualLogic) and network their story is incomplete and their solution do not align with a complete data center story. Also marginalized are SAP, Microsoft, Novell, RedHat, Citrix and several others that used to have very strategic partnerships which now will not be as important for those four horsemen in the immediate future. And then there are folks like NetApp, F5, Riverbed, Infoblox and others who will have to fit into this ecosystem partner arrangement without being swallowed up.

I think the next 3-4 years are going to be some of the most interesting and fast moving for the data center market space. To think, we haven't even addressed the SaaS/Cloud market space that Amazon (AWS), Microsoft (Azure) and others are rapidly pushing forward. Perhaps Microsoft will leapfrog everyone at the end of the day and the four datacenter horsemen won't even be relevant because you won't need a data center anymore. Hard to imagine? Is that the real apocalypse for data center, where the four horsemen aren't even relevant to the process for most customers?
- Ed

Microsoft MVP renewal - this time Desktop Experience

noreply@blogger.com (Ed Horley) — Thu, 01 Jul 2010 16:14:00 +0000

I am happy to say I have been renewed as a Microsoft MVP. This year I've been awarded in the Desktop Experience category. This is my third category so far in the program. A brief look back as I was first award in July 2004.

2004 - Windows Server - Networking
2005 - Windows Server - Networking
2006 - Windows Server - Networking
2007 - Windows Server - Networking
2008 - Enterprise Security - Networking
2009 - Enterprise Security - Networking
2010 - Desktop Experience

I still plan on being actively involved and paying attention to what is going on in Security and more specifically in networking. Most of the focus will be around IPv6 as I am now a co-chair on the California IPv6 Task Force.

I needs to extend some thanks to some Microsoft folks:
Jake Grey - MVP Lead
Emily Freet - MVP Regional Manager, Americas
Stephen Rose - Sr. Community Manager - Windows OS
Chris Avis, Harold Wong and Chris Henley - IT Pro Evangelist - DPE - West Region team

Also, thanks to Jennelle Crothers - fellow Microsoft MVP and PacITPros member for helping to make the User Group experience what it is, couldn't have made it this far without your help.
- Ed

Cisco Live!

noreply@blogger.com (Ed Horley) — Tue, 29 Jun 2010 17:41:00 +0000

I can't attend Cisco Live! this week but you can see some of the keynotes and other interesting items on the website. Looks like Cisco is announcing a bunch of stuff this year which is good news. The only thing I wish I was attending for was to get involved and be able to ask the tough questions about road map and support for IPv6. A lot of that was done at the Google IPv6 Implementors Event a few weeks ago but upper Cisco management and in particular - the product groups, were NOT at that event. Hard to see how important something is if key folks aren't paying attention.

I hope Cisco starts to push internally feature parity for IPv6 - there is a huge gap in "supporting" IPv6 and in having feature parity to their existing product portfolio. It will come over time but I think a bigger push needs to happen... and soon.
- Ed

NANOG 49 - San Francisco

noreply@blogger.com (Ed Horley) — Tue, 15 Jun 2010 02:36:00 +0000

For those that do not follow what is happening in the Internet Service Provider space much you might find it interesting that the North American Network Operators' Group (NANOG) is hosting their meeting in San Francisco this week. Many folks came in early to attend the Google IPv6 Implementors Conference that happened last Thursday and Friday in Mt. View and are now having fun in San Francisco.

The agenda has lots of IPv6 on it also, I am sort of bummed I will not have time to make it to event but I hope all those attending are enjoying it.

I also got to sit in on a conference call today about how the new North American IPv6 Task Force and the other regional task forces (like the California one) will be structuring and building out resources. Stay tuned for some cool events, labs facilities and meetings to happen across the country soon based out of this work. In the mean time, if you are interested in starting to hear about what will be happening with IPv6 in California you can follow http://twitter.com/CAv6TF and we will let you know when things are going on.
- Ed

Google IPv6 Implementors Conference

noreply@blogger.com (Ed Horley) — Thu, 10 Jun 2010 19:58:00 +0000

Attending the Google IPv6 Implementors Conference - the slide decks are available on the site to download. Great presentations on what is happening in the IPv6 space.

Definitely check out the presentations, especially the Facebook presentation - they turned up their experimental IPv6 service in short order and are only using 3 IPv6 addresses to get it deployed, remarkable. To top it off they spend no new capital to make it happen.

I have a feeling the content here is a bit better than what is going on at TechEd but perhaps the gauntlet has been thrown - come on Microsoft, where are you on this?
- Ed

California IPv6 Task Force

noreply@blogger.com (Ed Horley) — Fri, 04 Jun 2010 23:00:00 +0000

I am excited to announce that I have been appointed a co-chair on the California IPv6 Task Force handling the Bay Area. The California IPv6 Task Force (CAv6TF) is chartered with doing advocacy and education regarding IPv6 in the state of California.

With my new appointment comes the obvious questions, what sort of events should the CAv6TF do to cover both advocacy and education? I would love to hear from people about what they feel are the best ways to go about this prior to driving off with the bus half full. Specifically, what format and topics would be the best for the Bay Area crowd who definitely have their own unique requirements.

If you are interested in participating please follow us on twitter and keep an eye out for the relaunch of our website. We believe this will be a big year for IPv6 in general and I am personally excited to start working on the events!
- Ed

IF-MAP - Infoblox and Juniper pushing a new standard

noreply@blogger.com (Ed Horley) — Fri, 04 Jun 2010 17:00:00 +0000

I attended an Infoblox and Juniper joint presentation on IF-MAP and it looks interesting. I think they are attempting to rally support around the standard so a great deal is self serving (asking folks to push for IF-MAP inclusion in RFP's for instance.) That being said, I do think the standard addresses a real need requirement in the industry regarding getting multi-vendor solutions to utilize a common state engineer (which is what IF-MAP really is) which anyone can leverage for different purposes. I primarily see it for a single database location for NAC and policy authentication and enforcement systems to share information in a transparent way and to subscribe to changes in the "state" of something and then take action based on those states.

Vendors who are doing policy enforcement should keep an eye on the standard to determine if they need to have a solution that will work with IF-MAP. What will be really interesting is if monitoring and alerting systems start utilizing the IF-MAP standard to learn what is happening on the network to drive alerting from layer 1 all the way up to application attributes. This already happens on the enforcement side but there is little in the way of alerting and monitoring notification that can do something similar.

I don't think this will change the landscape of the data center models being developed today by the likes of HP or Cisco but this is a critical issue to enterprise and commercial customers as a pain point and if vendors choose to implement IF-MAP then they certainly might gain an advantage in terms of interoperability.
- Ed

Cisco ASA - failover history

noreply@blogger.com (Ed Horley) — Thu, 20 May 2010 16:50:00 +0000

If you spend time working with Cisco ASA's in a failover configuration and you want to get a history of failures on the device the you should use the "show failover history" and "show failover" commands. I typically only use the "show failover" command when working on interface status (what is being watched for failover criteria) and the state of the interfaces themselves plus which unit is active and which is standby or failed. The nice thing about the "show failover history" command is it tells you when things happened in terms of failover status. Here is some sample output:

fw-1# sh fail history
==========================================================================
From State To State Reason
==========================================================================
07:49:22 PST Feb 4 2010
Not Detected Negotiation No Error

07:50:07 PST Feb 4 2010
Negotiation Just Active No Active unit found

07:50:07 PST Feb 4 2010
Just Active Active Drain No Active unit found

07:50:07 PST Feb 4 2010
Active Drain Active Applying Config No Active unit found

07:50:07 PST Feb 4 2010
Active Applying Config Active Config Applied No Active unit found

07:50:07 PST Feb 4 2010
Active Config Applied Active No Active unit found

14:04:39 PST Feb 11 2010
Active Failed Interface check

14:04:41 PST Feb 11 2010
Failed Standby Ready Interface check

14:04:46 PST Feb 11 2010
Standby Ready Just Active Failover state check

14:04:46 PST Feb 11 2010
Just Active Active Drain Failover state check

14:04:46 PST Feb 11 2010
Active Drain Active Applying Config Failover state check

14:04:46 PST Feb 11 2010
Active Applying Config Active Config Applied Failover state check

14:04:46 PST Feb 11 2010
Active Config Applied Active Failover state check

==========================================================================

So you can see that you get a history and state changes of the failover status. What is nice is that you don't have to do a debug to capture these status changes and it is in a nice summary table.
- Ed

Cisco ASA flag descriptions

noreply@blogger.com (Ed Horley) — Tue, 18 May 2010 00:49:00 +0000

I'm always trying to remember the flag codes for the ASA connection command. Turns out it is in the help for the "show connection all" command, you just have to add the keyword "detail" so that you can see it. So, the command is:
show connection all detail

or for those that like shorter commands:
sh conn all d

The output for the flags (a bit better organized then what is displayed in the help) is:
Flags:
A - awaiting inside ACK to SYN
a - awaiting outside ACK to SYN
B - initial SYN from outside
b - TCP state-bypass or nailed
C - CTIQBE media
D - DNS
d - dump
E - outside back connection
F - outside FIN
f - inside FIN
G - group
g - MGCP
H - H.323
h - H.225.0
I - inbound data
i - incomplete
J - GTP
j - GTP data
K - GTP t3-response
k - Skinny media
M - SMTP data
m - SIP media
n - GUP
O - outbound data
P - inside back connection
p - Phone-proxy TFTP connection
q - SQL*Net data
R - outside acknowledged FIN
R - UDP SUNRPC
r - inside acknowledged FIN
S - awaiting inside SYN
s - awaiting outside SYN
T - SIP
t - SIP transient
U - up
V - VPN orphan
W - WAAS
X - inspected by service module

I am not sure why there are two "R" values - I am assuming sunrpc is listing UDP only and an outside acknowledged FIN would only be for a TCP session so there isn't a conflict in having the same flag value in use. Anyway, got tired of looking for this all the time when debugging problems so I am posting it here so I can find it for myself later.
- Ed

Cisco ASA - Resource Allocation for Multiple Context

noreply@blogger.com (Ed Horley) — Tue, 11 May 2010 17:55:00 +0000

When setting up virtual contexts within an ASA it is a good practice to build out a resource allocation plan (classes) and then apply it to all the context that you have configured. By doing this you can safeguard the important FW context that need more resources ( a production FW) vs a less important FW context (think development or test.)

The reason I bring this up is after reviewing a new development FW context that I turned up for a client it was sending a huge volume of syslog data and the template that was built for dev limited the amount of syslog transactions it could send which protected the production FW. The output looked like:
fw-sys# sh resource usage all
Resource Current Peak Limit Denied Context
(...)
Syslogs [rate] 4 999 1000 449480 dev
Conns 282 2266 200000 0 dev
(...)
Syslogs [rate] 12 2558 unlimited 0 prod
Conns 20631 71951 unlimited 0 prod
Xlates 111601 145005 unlimited 0 prod
(...)

As you can see, there was a lot of denied syslog traffic. This was due to the initial configuration and leaving the dev FW context doing debug logging to the syslog server. The traffic dropped substantially after that was removed but it easily could have started consuming more than its fair share of resources on the ASA. In this particular case it is a 5580-20 so there wasn't a lot of concern but on a smaller platform (5510 or even a 5520) where memory and cpu are more constrained it could be a much bigger issue.

Your templates can be rather simple, a default is built automatically and applied to all context out of the box. This default gives maximum resources to all contexts, effectively sharing equally all the resources of the ASA. While this is ok to get thing built out and stood up I would not recommend keeping all the contexts in this configuration if you are going to use the ASA for multiple purposes like production, qa and development.

You can build it out like QoS templates. From a practical standpoint it makes sense to do a default, gold, silver and bronze configuration because I find it unlikely that most organizations would need more than that number to cover all their configuration options. Please speak up if you think I am wrong (but I doubt it.) Cisco doesn't seem to have a lot of documentation on recommendations for how to build this out. They do provide a class overview in their CLI configuration guide for 8.2, though it does not have enough details IMHO. You can apply the class to multiple contexts within the ASA so depending on your resource allocation you can easily oversubscribe resources because you set to many context as "gold" or forget to change the default and then have unlimited resources for all contexts that you build that are automatically given that default class.

Below are some sample templates to review - I set some arbitrary values based off of loads that were being observed. You should adjust the values you require for your environment since perhaps your development or qa context has a lot more hosts then what is reflected below. In other words, don't cut and copy, figure out what is right for your deployment which might require some experimenting to find out what you are actually consuming. This isn't a bad process to go through because it allows you to know where you are at, just remember to account for growth otherwise stuff will stop working or appear to have problems. Remembering that you have classes that might be limiting stuff won't be at the top of your list to check when you are troubleshooting - trust me on this one.
!
class default
limit-resource Xlates 100
limit-resource ASDM 2
limit-resource SSH 2
limit-resource Telnet 2
limit-resource Conns 5.0%
limit-resource Hosts 100
limit-resource rate Syslogs 1000
limit-resource rate Inspects 100
!

class gold
limit-resource All 0
limit-resource ASDM 5
limit-resource SSH 5
limit-resource Telnet 5
!

class silver
limit-resource Xlates 10000
limit-resource ASDM 2
limit-resource SSH 5
limit-resource Telnet 2
limit-resource Conns 20.0%
limit-resource Hosts 10000
limit-resource rate Syslogs 10000
limit-resource rate Inspects 5000
!

class bronze
limit-resource Xlates 1000
limit-resource ASDM 2
limit-resource SSH 2
limit-resource Telnet 2
limit-resource Conns 10.0%
limit-resource Hosts 1000
limit-resource rate Syslogs 1000
limit-resource rate Inspects 1000
!

To apply the class to the context it is as simple as:
!
context prod
member gold
!
context admin
member default
!
context lab
member bronze
!

Because the gold class has the "limit-resource All 0" command it effectively has no limit which means it can starve even the admin context in this example. This might not be ideal in all situations and I will often set limits on the gold class for production FW contexts because there will be more than one FW context that needs to run at the highest class level. In this particular case, just note it is getting everything it wants without question. Sometimes it is good to be king.
- Ed

Root signing and DNSSEC

noreply@blogger.com (Ed Horley) — Tue, 04 May 2010 22:03:00 +0000

Time to start paying attention to DNSSEC and its potential impact on your ability to query against root servers through your firewalls. You can keep up to date at Root DNSSEC's website regarding the status and potential issues they are seeing.

Of immediate importance is the fact that your firewall may have a default configuration to discard UDP DNS traffic larger then 512 bytes in size. This was a common practice and it is now going to be an issue with the larger payloads that DNSSEC utilizes. Basically you just need to turn off this restriction and everything should work fine, for now. Platforms like the Cisco PIX, ASA and Microsoft ISA and TMG should all be checked.

Many people haven't noticed any issues with the first few root servers being signed and doing this behavior because their servers would not get a response back from the queried root server (blocked by the firewall for the payload size issue) and the server simply went to the next root server in the list which likely wasn't doing DNSSEC and therefore everything worked as expected. This is going to change starting this month going into June and IT Pros should be aware of the issue and review their firewall configurations to make sure they are not creating a problem for themselves.

Also, some of the TLD's are starting to sign also and I believe sometime in 2011 .com should be signed. If you are running DNS on Microsoft Windows 2008 R2 then you should read the DNSSEC Deployment Guide, it can walk you through requirements of what to do to get your infrastructure signed and working.
- Ed

Windows BranchCache OS requirements

noreply@blogger.com (Ed Horley) — Thu, 29 Apr 2010 17:02:00 +0000

I was asked at the Windows Intelligence event I presented at on Monday if there were specific requirements on what version of Windows 7 could be used with BranchCache. It turns out there are, you can only use Windows 7 Enterprise and Ultimate Editions to take advantage of BranchCache - even if you are doing a workgroup configuration and not a domain.

The step by step guide has the requirements for both the client and the server and can be found here.

It appears that Microsoft is assuming small and medium sized companies should be using Enterprise or Ultimate Editions to take advantage of newer technologies like BranchCache and DirectAccess. I will ask around to see if this will be changing at all in the future and post back up what I hear. I am a bit surprised to read that Windows 7 Professional didn't make the cut for BranchCache as a feature but licensing is a complicated thing that honestly I think very few folks truly understand, even within Microsoft.
- Ed

Windows Intelligence DirectAccess and BranchCache presentations

noreply@blogger.com (Ed Horley) — Tue, 27 Apr 2010 16:04:00 +0000

I had a fun time yesterday presenting on Microsoft DirectAccess and BranchCache at the Windows Intelligence NorCal event. Thank you to everyone who attended the event, I appreciate the discussion and feedback and as promised here are the slide decks for download.

DirectAccess

BranchCache

If you attended my sessions you might have missed sessions by Jennelle Crothers and Steve Evens who both gave excellent presentations from what I heard (I was giving mine so I couldn't be in two places at the same time!) and should be making their presentations available on their sites also. Doug Spindler also presented but I am not sure where his presentations are posted but I will find out and update here shortly.

It was also nice to see the West Coast Microsoft DPE IT Pro Evangelists at the event, Chris Henley, Chris Avis and Harold Wong. They are all wonderful presenters and totally passionate about what they do - a big shout out to them - you guys rock!
- Ed

Cisco IPSec VPN Client - 5.0.7 Released- Win7 64-bit support

noreply@blogger.com (Ed Horley) — Fri, 23 Apr 2010 17:00:00 +0000

Cisco has officially released a 64-bit version (non-beta) of the IPSec VPN Client, version 5.0.7 (vpnclient-winx64-msi-5.0.07.0290-k9.exe) available for download. Finally a Cisco supported 64-bit version of the IPSec client for Windows 7! It is available for download on CCO but requires a valid CCO login and current contract to get the code.

Do read the release notes, there are caveats for Windows Vista in the release notes so if you are planning on running it on 64-bit Vista make sure you review those. Also, it is important to note there is a specific section titled "Windows 7 and Vista Window Auto-tuning Feature Might Cause Network Timeout Problems" that states:
"Windows 7 and Vista support a feature called "Receive Window Auto-Tuning" that continually adjusts the receive Windows size, based upon the changing network conditions.

Some people reported that auto-tuning causes network timeout problems with some applications and routers. If you have experienced such problems, you can turn it off using the following procedure:

Step 1 Open an elevated command prompt.
Step 2 Enter the following command to disable auto-tuning:
netsh interface tcp set global autotuninglevel=disabled

If this solution does not fix the problem, you can turn it back on, as follows:
Step 1 Open up an elevated command prompt.
Step 2 Enter the following command to enable auto-tuning
netsh interface tcp set global autotuninglevel=normal

To view the states of the TCP global parameters, use the following command:
netsh interface tcp show global "

So for all of you that have to support Windows 7 64-bit and still have Cisco VPN Concentrators, PIX's and ASA's doing IPSec only you have a valid supported solution now! I don't know why it took so long but it is nice to know you don't have a pay the AnyConnect essentials license if you don't want to. I do like many of the features and advantages with AnyConnect, it just seemed wrong that to get VPN for a 64-bit OS you HAD to use AnyConnect only and pay the premium for the client.
- Ed

Cisco ASA - PAT performance and logging

noreply@blogger.com (Ed Horley) — Wed, 21 Apr 2010 17:00:00 +0000

It isn't a common issue for most small to medium businesses but occasionally you have to remember that when doing port address translation (PAT) or overloading a single IP address with multiple IP's behind it that there are actual limitation on how many session you can do simultaneously. Remember that for a single IP address can only do 65536 TCP sessions. If you remove the first 1024 you are down to 64512.

So why is this important? Turns out if you have enough hosts generating requests behind the ASA you can run out of ports and if you generate traffic quickly enough you will start getting the following error in the log files:

%ASA-3-305006: portmap translation creation failed for tcp src XXX dst XXX

The message description from Cisco is here and shows the format as:
%PIX|ASA-3-305006: {outbound static|identity|portmap|regular)
translation creation failed for protocol src interface_name:source_address/source_port
dst interface_name:dest_address/dest_port

I don't think this will be a common issue as it takes a lot of hosts to generate this sort of load doing thousands of requests a second. To exceed the threshold for instance you would need 250 hosts generating approximately 260 requests per second sustained per host and they would have to keep those sessions open for the ASA to start having this issue.

Luckily it is a rather fast and easy fix if you have more than one Public IP address available. You can simply add another global statement to the ASA with more public IP's to overload and the ASA will happily start spreading the load across the new IP addresses that you add. In terms of syslog, this is severity level 3 (error) so it should show up in most syslog data that is collected. Definitely recommend running Splunk, it makes finding these sort of errors so much easier!

Also, while it may be saying a portmap translation failed the ASA itself doesn't run out of resources from a CPU or memory standpoint. The ASA will continue to run along nicely it just won't be able to generate new TCP sessions as fast as your hosts may be requesting. This makes it really important to be looking at your log files to know what is happening. The problem is a function of how you are doing overloading, not a limitation of how the device performs that function.
- Ed

Cisco ASA - how to see your pre-shared-key

noreply@blogger.com (Ed Horley) — Fri, 16 Apr 2010 17:00:00 +0000

One of the annoying things about managing pre-shared keys for both site to site vpn tunnels and group pre-shared keys for client vpn tunnels is the fact that if you do a show run they are starred out (*) in the configuration file.

So you will see something like:
pre-shared-key *

If you need to recover back your keys because you have lots of folks running around with Cisco IPSec VPN clients with a standard PCF file and you can't remember what the group pre-shared-key is or don't have it documented you can do the following command.

more system:running-config

This will output your running-config file with the pre-shared-key variable in clear text.

Obviously this is useful for site to site keys also because you might have a tunnel set up with a third party vendor or a partner and you can't remember the key at all because you made it up on the fly with the other network engineer on the phone (That never happens... really.)

This will NOT show you the enable or passwd values because those are actually encrypted. You will have to use other tools to break those or do a standard password recovery process.
- Ed

Cisco ASA syslog over TCP - Potential to stop forwarding traffic!

noreply@blogger.com (Ed Horley) — Tue, 13 Apr 2010 17:00:00 +0000

I ran into an interesting configuration issue with the Cisco ASA and doing syslog via TCP to a syslog server. It turns out that Cisco has a feature in the ASA that when pushing logging information to a sylog server via TCP that if the server does not respond it will stop the ASA from building out new session flows and therefore it will stop forwarding traffic. It seems that existing TCP sessions continue to allow traffic to flow, the rational being that the logging of that flow being established was sent to the syslog earlier and therefore it is audited event. Apparently this was a specific request from some government agencies who did not want their firewalls to pass traffic if logging was not possible. Reasonable enough but it isn't particularly well documented and does not alert you of it's default behavior at all when you enable the syslog command using TCP.

You can find all the details in the Cisco ASA 5500 command line configuration guide (this one is for 8.2 code release) and look under "Sending Syslog Messages to a Syslog Server" to see the parameters. They do provide a optional command switch that allows the ASA to continue working even if the syslog server is down.

So how can this be a problem? It turns out that a lot of shops rotate their syslog files and will write a quick script to stop their daemon for syslog, change their logfile and then start the syslogd daemon again. This causes the TCP session to fail because the syslogd daemon is no longer keeping the socket open. Depending on the thresholds you have built on the network this can cause your ASA to stop forwarding traffic. Worse, it is difficult to figure out what is happening because existing log flow sessions (perhaps and ssh or database session) will still be working but anything new you attempt to send through the ASA will not. Also, the ASA does not consider this a flag to attempt to failover to a standby unit - in fact, there is no way to even monitor the syslog process to flag it as such. I think that is something that needs to be added so the failover unit can then test to see if it can connect to the syslog server and if so, failover so that your firewall continues to forward traffic as expected.

The error you will get in the local log is:
%ASA-3-201008: Disallowing new connections.

It tells you what is happening - it doesn't tell you why it is happening though - very frustrating.

This is not an issue if you are using UDP for syslog traffic on the ASA's as they assume if you are doing UDP to the syslog server it is not guaranteed delivery and therefore you don't have to meeting the same standards for logging or security.
- Ed

Cisco Nexus 5k and 2148t caveats

noreply@blogger.com (Ed Horley) — Fri, 02 Apr 2010 17:00:00 +0000

I ran into a few more items I think warrant a post regarding the Cisco Nexus 5k series and the 2k fabric extenders (fex.) For those that are evaluating the Nexus 5010 and 5020's with the 2148t fex for layer 2 switchport capacity there are two gotchas that I think failed to get mentioned often enough in the pre-sales engagement.

1. The 2148t fex is not capable of doing a port-channel (ether-channel) with two or more ports on a single chassis. This is not a limitation that can be fixed with software either, it is a limitation in the ASIC's themselves on the 2148t. I think this is a huge failure. That being said, I believe it is road mapped as a feature fix for the new 2248TP and 2232PP models.

2. If you do need to do a port-channel to a single host from a 2148t you need two 2148t's and you have to run a Virtual Port-channel (vPC) using the 2148t's that are each single homed to the 5k's upstream (in what is called single homed fabric extender vPC topology.) You can only do a maximum of 1 Gigabit Ethernet port to each 2148t (because of the limitation above) and then the 5k's will do the logical vPC to make them a port-channel. So a 2 Gigibit max port-channel with the 2148t single homed to the 5k's. Again, because of the limitation above I see this as poor engineering work around to the failure in design.

So, with those limitation on the 2148t fex's there is something else from a design standpoint that is frustrating. You have two recommended options when designing the 5k/2k deployment, you can do a single homed fabric extender vPC topology or a dual homed fabric extender vPC topology. Here are the diagrams from the Cisco Nexus 5000 Series NX-OS Layer 2 Switching Configuration Guide.

The single homed:

and the dual homed:

What is not mentioned or documented anywhere about this configuration is if you can mix and match the topology between single homed and dual homed 2k's on a single pair of 5k's. I can only find one reference that mentions this vPC mixed topology option. Even with this option you are still burning up a second 2148t just to get port-channeling as an option if you don't want to use the 5k's directly. Burning up the few 10/1 Gig mix use ports on the 5k's for 1 Gig port-channels seems to be a huge waste and very expensive.

So there you have it, some items to mull over while planning out your data center design with Nexus and some hard hitting questions to ask your Cisco SE's when they show up talking about cost per port and maximizing functionality and flexibility per port. On top of the above items the 2148t also only does 1Gig, it doesn't support 10/100 at all, not a show stopper but another nagging item that limits some deployments. Don't get me wrong, I'm a fan of the new data center design and topology model that Cisco is building and with OTV coming out in the 5.0 NX-OS code release some really exciting things are going to be possible. Just keep in mind you might be buying an extra one or two 2k's to accommodate some of the systems that have the port-channel requirements for higher availability.
- Ed

Cisco Nexus command - Checkpoints and rollbacks

noreply@blogger.com (Ed Horley) — Tue, 23 Mar 2010 17:00:00 +0000

Using the Cisco Nexus 7010, 5010 and 2148's has changed some of the habits I have traditionally used for the Cisco IOS command set. Some of the new Nexus commands have become second nature and I now miss them on IOS. Being able to use grep is one I really wish was incorporated into IOS. I am used to having it with the ASA platform and now with the Nexus platform - going back to IOS 12.x and not having it there is annoying.

A new command that is really useful on the Nexus platform is checkpoint. There are several things that are unique about checkpoints and how you can use them. First, checkpoints are primarily used for rollback situations. They allow you to make changes on the system and if required due to an error rollback to a known good configuration on the system. There are three rollback types.
Atomic rollback is done when the configuration can be applied with NO errors.
Best Effort rollback will ignore errors and push the configuration onto the system.
Stop At First Failure will process the rollback request until it hits an error and then stops.

The default rollback type is Atomic and this is likely the most common rollback method you would use on a production environment. I am not aware of many folks wanting to rollback to a "Stop At First Failure" or "Best Effort" scenario situation unless true desperation has kicked in. There might be a case of the order of rollback if you are using VDC's and moving physical resources from one VDC to the other in which case perhaps Best Effort might be useful.

Also of note, the rollback feature must be used per Virtual Device Context (VDC), in other words, you have to run the command in each VDC. This is expected behavior as each VDC is it's own NX-OS instance and you have to run all the same commands to get the desired behavior out of the NX-OS platform.

The command itself is very simple:
checkpoint {checkpoint name} description {a description} | filename {path and filename}
Example: checkpoint cp-running-config-known-good-2010-03-22 description checkpoint of running config

There are some restrictions on the checkpoint name (max length 80 characters) and there are restrictions on the filename (max length of 75 characters and filename can't start with the word "system") but otherwise it is pretty straightforward process to get this going. I am using this on NX-OS version 4.3.1, earlier versions had more restrictions on file names and such so read the documentation if you are on an earlier release.

To see what the checkpoint command does you can use the show commands. To see all the checkpoints that are in a given VDC:
show checkpoint all
show checkpoint summary

The checkpoint command basically keeps a small database of checkpoints to allow you to rollback to a specific one and calculates the differences between a current state or checkpoint and that checkpoint you want to move to. It will generate a rollback script when you use the rollback command. If you want to see the differences that are being generated you can do that too:
show diff rollback-patch {checkpoint source name | running-config | startup-config | file filename} {checkpoint destination name | running-config | startup-config | file filename}
Example: show diff rollback-patch running-config checkpoint cp-running-config-known-good-2010-03-22

To actually do a rollback:
rollback running-config {checkpoint cp name | running-config | startup-config | file filename} {atomic | best-effort | stop-at-first-failure}
Example: rollback running-config checkpoint cp-running-config-known-good-2010-03-22 atomic

To see the status of rollbacks:
show rollback log

You can also clear out the checkpoint history and files, use the command with caution.
clear checkpoint database

This is a VERY useful command to build into your scripts prior to pushing out production changes on gear. It allows you to have a well known state stored locally and be able to rollback to it quickly in case of problems in your scripts. Awesome!
- Ed

Cisco IPSec VPN Client - 5.0.7 BETA - Win7 64-bit support

noreply@blogger.com (Ed Horley) — Fri, 19 Mar 2010 18:07:00 +0000

Cisco has a beta version of the IPSec VPN Client out, version 5.0.7 BETA (vpnclient-winx64-msi-5.0.07.0240-k9-BETA.exe) available for download. It appears they got the message about the need for a 64-bit version of the IPSec client for Windows 7! It is available for download on CCO but requires a valid CCO login and current contract to get the code.

This is great news for Microsoft customers that have Cisco ASA's, PIX's or VPN 3000 concentrators deployed and their IT team is migrating their client OS's to Windows 7. Many OEM's are shipping the default OS as Windows 7 64-bit to take advantage of the all the RAM systems can support today.

Cisco is still pushing their AnyConnect client + ASA platform to compete with Microsoft's DirectAccess solution on Forefront UAG. Cisco is positioning the AnyConnect client as an always ready vpn client that auto-reconnects and is seamless in that process. I don't know if I am buying that position given the flexibility of what DirectAccess can do and how you can manage the policies with GPO's in AD in the Microsoft solution. Plus the fact that once DirectAccess is deployed there is no VPN "client" per-say, it is more of a policy function which is a very unique position in terms of product positioning.

At least if you have a Cisco IPSec solution deployed today you can now leverage it with Windows 7 without requiring any third party software to get around the problem of being on a 64-bit OS.
- Ed