Splunk User Conference - August 9-11, 2010

One of the tools I use to help me wade through the mass of data that networking devices spit out is Splunk. Splunk is easily one of the most useful apps to run if you have to try and figure out what is going on with regards to firewalls, ids, network logging and anything else that will output to syslog.

If you happen to live in the San Francisco Bay Area you are in luck too, next week is the first annual Splunk User Conference and you can still sign up. I wish I could make it myself, especially since I live local to the event.

Splunk has really expanded the capabilities of the product but to be honest, the simple search and filtering to help me write better access control lists for firewalls is a huge time saver, just being able to do that quickly and efficiently makes it worth it's weight in gold.
- Ed

Cisco ASA - PAT performance and logging

It isn't a common issue for most small to medium businesses but occasionally you have to remember that when doing port address translation (PAT) or overloading a single IP address with multiple IP's behind it that there are actual limitation on how many session you can do simultaneously. Remember that for a single IP address can only do 65536 TCP sessions. If you remove the first 1024 you are down to 64512.

So why is this important? Turns out if you have enough hosts generating requests behind the ASA you can run out of ports and if you generate traffic quickly enough you will start getting the following error in the log files:

%ASA-3-305006: portmap translation creation failed for tcp src XXX dst XXX

The message description from Cisco is here and shows the format as:
%PIX|ASA-3-305006: {outbound static|identity|portmap|regular)
translation creation failed for protocol src interface_name:source_address/source_port
dst interface_name:dest_address/dest_port

I don't think this will be a common issue as it takes a lot of hosts to generate this sort of load doing thousands of requests a second. To exceed the threshold for instance you would need 250 hosts generating approximately 260 requests per second sustained per host and they would have to keep those sessions open for the ASA to start having this issue.

Luckily it is a rather fast and easy fix if you have more than one Public IP address available. You can simply add another global statement to the ASA with more public IP's to overload and the ASA will happily start spreading the load across the new IP addresses that you add. In terms of syslog, this is severity level 3 (error) so it should show up in most syslog data that is collected. Definitely recommend running Splunk, it makes finding these sort of errors so much easier!

Also, while it may be saying a portmap translation failed the ASA itself doesn't run out of resources from a CPU or memory standpoint. The ASA will continue to run along nicely it just won't be able to generate new TCP sessions as fast as your hosts may be requesting. This makes it really important to be looking at your log files to know what is happening. The problem is a function of how you are doing overloading, not a limitation of how the device performs that function.
- Ed

Cisco ASA syslog over TCP - Potential to stop forwarding traffic!

I ran into an interesting configuration issue with the Cisco ASA and doing syslog via TCP to a syslog server. It turns out that Cisco has a feature in the ASA that when pushing logging information to a sylog server via TCP that if the server does not respond it will stop the ASA from building out new session flows and therefore it will stop forwarding traffic. It seems that existing TCP sessions continue to allow traffic to flow, the rational being that the logging of that flow being established was sent to the syslog earlier and therefore it is audited event. Apparently this was a specific request from some government agencies who did not want their firewalls to pass traffic if logging was not possible. Reasonable enough but it isn't particularly well documented and does not alert you of it's default behavior at all when you enable the syslog command using TCP.

You can find all the details in the Cisco ASA 5500 command line configuration guide (this one is for 8.2 code release) and look under "Sending Syslog Messages to a Syslog Server" to see the parameters. They do provide a optional command switch that allows the ASA to continue working even if the syslog server is down.

So how can this be a problem? It turns out that a lot of shops rotate their syslog files and will write a quick script to stop their daemon for syslog, change their logfile and then start the syslogd daemon again. This causes the TCP session to fail because the syslogd daemon is no longer keeping the socket open. Depending on the thresholds you have built on the network this can cause your ASA to stop forwarding traffic. Worse, it is difficult to figure out what is happening because existing log flow sessions (perhaps and ssh or database session) will still be working but anything new you attempt to send through the ASA will not. Also, the ASA does not consider this a flag to attempt to failover to a standby unit - in fact, there is no way to even monitor the syslog process to flag it as such. I think that is something that needs to be added so the failover unit can then test to see if it can connect to the syslog server and if so, failover so that your firewall continues to forward traffic as expected.

The error you will get in the local log is:
%ASA-3-201008: Disallowing new connections.

It tells you what is happening - it doesn't tell you why it is happening though - very frustrating.

This is not an issue if you are using UDP for syslog traffic on the ASA's as they assume if you are doing UDP to the syslog server it is not guaranteed delivery and therefore you don't have to meeting the same standards for logging or security.
- Ed