Thursday, May 20, 2010

Cisco ASA - failover history

If you spend time working with Cisco ASA's in a failover configuration and you want to get a history of failures on the device the you should use the "show failover history" and "show failover" commands. I typically only use the "show failover" command when working on interface status (what is being watched for failover criteria) and the state of the interfaces themselves plus which unit is active and which is standby or failed. The nice thing about the "show failover history" command is it tells you when things happened in terms of failover status. Here is some sample output:

fw-1# sh fail history
From State To State Reason
07:49:22 PST Feb 4 2010
Not Detected Negotiation No Error

07:50:07 PST Feb 4 2010
Negotiation Just Active No Active unit found

07:50:07 PST Feb 4 2010
Just Active Active Drain No Active unit found

07:50:07 PST Feb 4 2010
Active Drain Active Applying Config No Active unit found

07:50:07 PST Feb 4 2010
Active Applying Config Active Config Applied No Active unit found

07:50:07 PST Feb 4 2010
Active Config Applied Active No Active unit found

14:04:39 PST Feb 11 2010
Active Failed Interface check

14:04:41 PST Feb 11 2010
Failed Standby Ready Interface check

14:04:46 PST Feb 11 2010
Standby Ready Just Active Failover state check

14:04:46 PST Feb 11 2010
Just Active Active Drain Failover state check

14:04:46 PST Feb 11 2010
Active Drain Active Applying Config Failover state check

14:04:46 PST Feb 11 2010
Active Applying Config Active Config Applied Failover state check

14:04:46 PST Feb 11 2010
Active Config Applied Active Failover state check


So you can see that you get a history and state changes of the failover status. What is nice is that you don't have to do a debug to capture these status changes and it is in a nice summary table.
- Ed

Monday, May 17, 2010

Cisco ASA flag descriptions

I'm always trying to remember the flag codes for the ASA connection command. Turns out it is in the help for the "show connection all" command, you just have to add the keyword "detail" so that you can see it. So, the command is:
show connection all detail

or for those that like shorter commands:
sh conn all d

The output for the flags (a bit better organized then what is displayed in the help) is:
A - awaiting inside ACK to SYN
a - awaiting outside ACK to SYN
B - initial SYN from outside
b - TCP state-bypass or nailed
C - CTIQBE media
d - dump
E - outside back connection
F - outside FIN
f - inside FIN
G - group
g - MGCP
H - H.323
h - H.225.0
I - inbound data
i - incomplete
j - GTP data
K - GTP t3-response
k - Skinny media
M - SMTP data
m - SIP media
n - GUP
O - outbound data
P - inside back connection
p - Phone-proxy TFTP connection
q - SQL*Net data
R - outside acknowledged FIN
r - inside acknowledged FIN
S - awaiting inside SYN
s - awaiting outside SYN
t - SIP transient
U - up
V - VPN orphan
X - inspected by service module

I am not sure why there are two "R" values - I am assuming sunrpc is listing UDP only and an outside acknowledged FIN would only be for a TCP session so there isn't a conflict in having the same flag value in use. Anyway, got tired of looking for this all the time when debugging problems so I am posting it here so I can find it for myself later.
- Ed

Tuesday, May 11, 2010

Cisco ASA - Resource Allocation for Multiple Context

When setting up virtual contexts within an ASA it is a good practice to build out a resource allocation plan (classes) and then apply it to all the context that you have configured. By doing this you can safeguard the important FW context that need more resources ( a production FW) vs a less important FW context (think development or test.)

The reason I bring this up is after reviewing a new development FW context that I turned up for a client it was sending a huge volume of syslog data and the template that was built for dev limited the amount of syslog transactions it could send which protected the production FW. The output looked like:
fw-sys# sh resource usage all
Resource Current Peak Limit Denied Context
Syslogs [rate] 4 999 1000 449480 dev
Conns 282 2266 200000 0 dev
Syslogs [rate] 12 2558 unlimited 0 prod
Conns 20631 71951 unlimited 0 prod
Xlates 111601 145005 unlimited 0 prod

As you can see, there was a lot of denied syslog traffic. This was due to the initial configuration and leaving the dev FW context doing debug logging to the syslog server. The traffic dropped substantially after that was removed but it easily could have started consuming more than its fair share of resources on the ASA. In this particular case it is a 5580-20 so there wasn't a lot of concern but on a smaller platform (5510 or even a 5520) where memory and cpu are more constrained it could be a much bigger issue.

Your templates can be rather simple, a default is built automatically and applied to all context out of the box. This default gives maximum resources to all contexts, effectively sharing equally all the resources of the ASA. While this is ok to get thing built out and stood up I would not recommend keeping all the contexts in this configuration if you are going to use the ASA for multiple purposes like production, qa and development.

You can build it out like QoS templates. From a practical standpoint it makes sense to do a default, gold, silver and bronze configuration because I find it unlikely that most organizations would need more than that number to cover all their configuration options. Please speak up if you think I am wrong (but I doubt it.) Cisco doesn't seem to have a lot of documentation on recommendations for how to build this out. They do provide a class overview in their CLI configuration guide for 8.2, though it does not have enough details IMHO. You can apply the class to multiple contexts within the ASA so depending on your resource allocation you can easily oversubscribe resources because you set to many context as "gold" or forget to change the default and then have unlimited resources for all contexts that you build that are automatically given that default class.

Below are some sample templates to review - I set some arbitrary values based off of loads that were being observed. You should adjust the values you require for your environment since perhaps your development or qa context has a lot more hosts then what is reflected below. In other words, don't cut and copy, figure out what is right for your deployment which might require some experimenting to find out what you are actually consuming. This isn't a bad process to go through because it allows you to know where you are at, just remember to account for growth otherwise stuff will stop working or appear to have problems. Remembering that you have classes that might be limiting stuff won't be at the top of your list to check when you are troubleshooting - trust me on this one.
class default
limit-resource Xlates 100
limit-resource ASDM 2
limit-resource SSH 2
limit-resource Telnet 2
limit-resource Conns 5.0%
limit-resource Hosts 100
limit-resource rate Syslogs 1000
limit-resource rate Inspects 100

class gold
limit-resource All 0
limit-resource ASDM 5
limit-resource SSH 5
limit-resource Telnet 5

class silver
limit-resource Xlates 10000
limit-resource ASDM 2
limit-resource SSH 5
limit-resource Telnet 2
limit-resource Conns 20.0%
limit-resource Hosts 10000
limit-resource rate Syslogs 10000
limit-resource rate Inspects 5000

class bronze
limit-resource Xlates 1000
limit-resource ASDM 2
limit-resource SSH 2
limit-resource Telnet 2
limit-resource Conns 10.0%
limit-resource Hosts 1000
limit-resource rate Syslogs 1000
limit-resource rate Inspects 1000

To apply the class to the context it is as simple as:
context prod
member gold
context admin
member default
context lab
member bronze

Because the gold class has the "limit-resource All 0" command it effectively has no limit which means it can starve even the admin context in this example. This might not be ideal in all situations and I will often set limits on the gold class for production FW contexts because there will be more than one FW context that needs to run at the highest class level. In this particular case, just note it is getting everything it wants without question. Sometimes it is good to be king.
- Ed

Tuesday, May 04, 2010

Root signing and DNSSEC

Time to start paying attention to DNSSEC and its potential impact on your ability to query against root servers through your firewalls. You can keep up to date at Root DNSSEC's website regarding the status and potential issues they are seeing.

Of immediate importance is the fact that your firewall may have a default configuration to discard UDP DNS traffic larger then 512 bytes in size. This was a common practice and it is now going to be an issue with the larger payloads that DNSSEC utilizes. Basically you just need to turn off this restriction and everything should work fine, for now. Platforms like the Cisco PIX, ASA and Microsoft ISA and TMG should all be checked.

Many people haven't noticed any issues with the first few root servers being signed and doing this behavior because their servers would not get a response back from the queried root server (blocked by the firewall for the payload size issue) and the server simply went to the next root server in the list which likely wasn't doing DNSSEC and therefore everything worked as expected. This is going to change starting this month going into June and IT Pros should be aware of the issue and review their firewall configurations to make sure they are not creating a problem for themselves.

Also, some of the TLD's are starting to sign also and I believe sometime in 2011 .com should be signed. If you are running DNS on Microsoft Windows 2008 R2 then you should read the DNSSEC Deployment Guide, it can walk you through requirements of what to do to get your infrastructure signed and working.
- Ed