Wednesday, April 21, 2010

Cisco ASA - PAT performance and logging

It isn't a common issue for most small to medium businesses but occasionally you have to remember that when doing port address translation (PAT) or overloading a single IP address with multiple IP's behind it that there are actual limitation on how many session you can do simultaneously. Remember that for a single IP address can only do 65536 TCP sessions. If you remove the first 1024 you are down to 64512.

So why is this important? Turns out if you have enough hosts generating requests behind the ASA you can run out of ports and if you generate traffic quickly enough you will start getting the following error in the log files:

%ASA-3-305006: portmap translation creation failed for tcp src XXX dst XXX

The message description from Cisco is here and shows the format as:
%PIX|ASA-3-305006: {outbound static|identity|portmap|regular)
translation creation failed for protocol src interface_name:source_address/source_port
dst interface_name:dest_address/dest_port

I don't think this will be a common issue as it takes a lot of hosts to generate this sort of load doing thousands of requests a second. To exceed the threshold for instance you would need 250 hosts generating approximately 260 requests per second sustained per host and they would have to keep those sessions open for the ASA to start having this issue.

Luckily it is a rather fast and easy fix if you have more than one Public IP address available. You can simply add another global statement to the ASA with more public IP's to overload and the ASA will happily start spreading the load across the new IP addresses that you add. In terms of syslog, this is severity level 3 (error) so it should show up in most syslog data that is collected. Definitely recommend running Splunk, it makes finding these sort of errors so much easier!

Also, while it may be saying a portmap translation failed the ASA itself doesn't run out of resources from a CPU or memory standpoint. The ASA will continue to run along nicely it just won't be able to generate new TCP sessions as fast as your hosts may be requesting. This makes it really important to be looking at your log files to know what is happening. The problem is a function of how you are doing overloading, not a limitation of how the device performs that function.
- Ed

No comments: