It isn't a common issue for most small to medium businesses but occasionally you have to remember that when doing port address translation (PAT) or overloading a single IP address with multiple IP's behind it that there are actual limitation on how many session you can do simultaneously. Remember that for a single IP address can only do 65536 TCP sessions. If you remove the first 1024 you are down to 64512.
So why is this important? Turns out if you have enough hosts generating requests behind the ASA you can run out of ports and if you generate traffic quickly enough you will start getting the following error in the log files:
%ASA-3-305006: portmap translation creation failed for tcp src XXX dst XXX
The message description from Cisco is here and shows the format as:
%PIX|ASA-3-305006: {outbound static|identity|portmap|regular)
translation creation failed for protocol src interface_name:source_address/source_port
dst interface_name:dest_address/dest_port
I don't think this will be a common issue as it takes a lot of hosts to generate this sort of load doing thousands of requests a second. To exceed the threshold for instance you would need 250 hosts generating approximately 260 requests per second sustained per host and they would have to keep those sessions open for the ASA to start having this issue.
Luckily it is a rather fast and easy fix if you have more than one Public IP address available. You can simply add another global statement to the ASA with more public IP's to overload and the ASA will happily start spreading the load across the new IP addresses that you add. In terms of syslog, this is severity level 3 (error) so it should show up in most syslog data that is collected. Definitely recommend running Splunk, it makes finding these sort of errors so much easier!
Also, while it may be saying a portmap translation failed the ASA itself doesn't run out of resources from a CPU or memory standpoint. The ASA will continue to run along nicely it just won't be able to generate new TCP sessions as fast as your hosts may be requesting. This makes it really important to be looking at your log files to know what is happening. The problem is a function of how you are doing overloading, not a limitation of how the device performs that function.
- Ed
Wednesday, April 21, 2010
Friday, April 16, 2010
Cisco ASA - how to see your pre-shared-key
One of the annoying things about managing pre-shared keys for both site to site vpn tunnels and group pre-shared keys for client vpn tunnels is the fact that if you do a show run they are starred out (*) in the configuration file.
So you will see something like:
pre-shared-key *
If you need to recover back your keys because you have lots of folks running around with Cisco IPSec VPN clients with a standard PCF file and you can't remember what the group pre-shared-key is or don't have it documented you can do the following command.
more system:running-config
This will output your running-config file with the pre-shared-key variable in clear text.
Obviously this is useful for site to site keys also because you might have a tunnel set up with a third party vendor or a partner and you can't remember the key at all because you made it up on the fly with the other network engineer on the phone (That never happens... really.)
This will NOT show you the enable or passwd values because those are actually encrypted. You will have to use other tools to break those or do a standard password recovery process.
- Ed
So you will see something like:
pre-shared-key *
If you need to recover back your keys because you have lots of folks running around with Cisco IPSec VPN clients with a standard PCF file and you can't remember what the group pre-shared-key is or don't have it documented you can do the following command.
more system:running-config
This will output your running-config file with the pre-shared-key variable in clear text.
Obviously this is useful for site to site keys also because you might have a tunnel set up with a third party vendor or a partner and you can't remember the key at all because you made it up on the fly with the other network engineer on the phone (That never happens... really.)
This will NOT show you the enable or passwd values because those are actually encrypted. You will have to use other tools to break those or do a standard password recovery process.
- Ed
Tuesday, April 13, 2010
Cisco ASA syslog over TCP - Potential to stop forwarding traffic!
I ran into an interesting configuration issue with the Cisco ASA and doing syslog via TCP to a syslog server. It turns out that Cisco has a feature in the ASA that when pushing logging information to a sylog server via TCP that if the server does not respond it will stop the ASA from building out new session flows and therefore it will stop forwarding traffic. It seems that existing TCP sessions continue to allow traffic to flow, the rational being that the logging of that flow being established was sent to the syslog earlier and therefore it is audited event. Apparently this was a specific request from some government agencies who did not want their firewalls to pass traffic if logging was not possible. Reasonable enough but it isn't particularly well documented and does not alert you of it's default behavior at all when you enable the syslog command using TCP.
You can find all the details in the Cisco ASA 5500 command line configuration guide (this one is for 8.2 code release) and look under "Sending Syslog Messages to a Syslog Server" to see the parameters. They do provide a optional command switch that allows the ASA to continue working even if the syslog server is down.
So how can this be a problem? It turns out that a lot of shops rotate their syslog files and will write a quick script to stop their daemon for syslog, change their logfile and then start the syslogd daemon again. This causes the TCP session to fail because the syslogd daemon is no longer keeping the socket open. Depending on the thresholds you have built on the network this can cause your ASA to stop forwarding traffic. Worse, it is difficult to figure out what is happening because existing log flow sessions (perhaps and ssh or database session) will still be working but anything new you attempt to send through the ASA will not. Also, the ASA does not consider this a flag to attempt to failover to a standby unit - in fact, there is no way to even monitor the syslog process to flag it as such. I think that is something that needs to be added so the failover unit can then test to see if it can connect to the syslog server and if so, failover so that your firewall continues to forward traffic as expected.
The error you will get in the local log is:
%ASA-3-201008: Disallowing new connections.
It tells you what is happening - it doesn't tell you why it is happening though - very frustrating.
This is not an issue if you are using UDP for syslog traffic on the ASA's as they assume if you are doing UDP to the syslog server it is not guaranteed delivery and therefore you don't have to meeting the same standards for logging or security.
- Ed
You can find all the details in the Cisco ASA 5500 command line configuration guide (this one is for 8.2 code release) and look under "Sending Syslog Messages to a Syslog Server" to see the parameters. They do provide a optional command switch that allows the ASA to continue working even if the syslog server is down.
So how can this be a problem? It turns out that a lot of shops rotate their syslog files and will write a quick script to stop their daemon for syslog, change their logfile and then start the syslogd daemon again. This causes the TCP session to fail because the syslogd daemon is no longer keeping the socket open. Depending on the thresholds you have built on the network this can cause your ASA to stop forwarding traffic. Worse, it is difficult to figure out what is happening because existing log flow sessions (perhaps and ssh or database session) will still be working but anything new you attempt to send through the ASA will not. Also, the ASA does not consider this a flag to attempt to failover to a standby unit - in fact, there is no way to even monitor the syslog process to flag it as such. I think that is something that needs to be added so the failover unit can then test to see if it can connect to the syslog server and if so, failover so that your firewall continues to forward traffic as expected.
The error you will get in the local log is:
%ASA-3-201008: Disallowing new connections.
It tells you what is happening - it doesn't tell you why it is happening though - very frustrating.
This is not an issue if you are using UDP for syslog traffic on the ASA's as they assume if you are doing UDP to the syslog server it is not guaranteed delivery and therefore you don't have to meeting the same standards for logging or security.
- Ed
Friday, April 02, 2010
Cisco Nexus 5k and 2148t caveats
I ran into a few more items I think warrant a post regarding the Cisco Nexus 5k series and the 2k fabric extenders (fex.) For those that are evaluating the Nexus 5010 and 5020's with the 2148t fex for layer 2 switchport capacity there are two gotchas that I think failed to get mentioned often enough in the pre-sales engagement.
1. The 2148t fex is not capable of doing a port-channel (ether-channel) with two or more ports on a single chassis. This is not a limitation that can be fixed with software either, it is a limitation in the ASIC's themselves on the 2148t. I think this is a huge failure. That being said, I believe it is road mapped as a feature fix for the new 2248TP and 2232PP models.
2. If you do need to do a port-channel to a single host from a 2148t you need two 2148t's and you have to run a Virtual Port-channel (vPC) using the 2148t's that are each single homed to the 5k's upstream (in what is called single homed fabric extender vPC topology.) You can only do a maximum of 1 Gigabit Ethernet port to each 2148t (because of the limitation above) and then the 5k's will do the logical vPC to make them a port-channel. So a 2 Gigibit max port-channel with the 2148t single homed to the 5k's. Again, because of the limitation above I see this as poor engineering work around to the failure in design.
So, with those limitation on the 2148t fex's there is something else from a design standpoint that is frustrating. You have two recommended options when designing the 5k/2k deployment, you can do a single homed fabric extender vPC topology or a dual homed fabric extender vPC topology. Here are the diagrams from the Cisco Nexus 5000 Series NX-OS Layer 2 Switching Configuration Guide.
The single homed:

and the dual homed:

What is not mentioned or documented anywhere about this configuration is if you can mix and match the topology between single homed and dual homed 2k's on a single pair of 5k's. I can only find one reference that mentions this vPC mixed topology option. Even with this option you are still burning up a second 2148t just to get port-channeling as an option if you don't want to use the 5k's directly. Burning up the few 10/1 Gig mix use ports on the 5k's for 1 Gig port-channels seems to be a huge waste and very expensive.
So there you have it, some items to mull over while planning out your data center design with Nexus and some hard hitting questions to ask your Cisco SE's when they show up talking about cost per port and maximizing functionality and flexibility per port. On top of the above items the 2148t also only does 1Gig, it doesn't support 10/100 at all, not a show stopper but another nagging item that limits some deployments. Don't get me wrong, I'm a fan of the new data center design and topology model that Cisco is building and with OTV coming out in the 5.0 NX-OS code release some really exciting things are going to be possible. Just keep in mind you might be buying an extra one or two 2k's to accommodate some of the systems that have the port-channel requirements for higher availability.
- Ed
1. The 2148t fex is not capable of doing a port-channel (ether-channel) with two or more ports on a single chassis. This is not a limitation that can be fixed with software either, it is a limitation in the ASIC's themselves on the 2148t. I think this is a huge failure. That being said, I believe it is road mapped as a feature fix for the new 2248TP and 2232PP models.
2. If you do need to do a port-channel to a single host from a 2148t you need two 2148t's and you have to run a Virtual Port-channel (vPC) using the 2148t's that are each single homed to the 5k's upstream (in what is called single homed fabric extender vPC topology.) You can only do a maximum of 1 Gigabit Ethernet port to each 2148t (because of the limitation above) and then the 5k's will do the logical vPC to make them a port-channel. So a 2 Gigibit max port-channel with the 2148t single homed to the 5k's. Again, because of the limitation above I see this as poor engineering work around to the failure in design.
So, with those limitation on the 2148t fex's there is something else from a design standpoint that is frustrating. You have two recommended options when designing the 5k/2k deployment, you can do a single homed fabric extender vPC topology or a dual homed fabric extender vPC topology. Here are the diagrams from the Cisco Nexus 5000 Series NX-OS Layer 2 Switching Configuration Guide.
The single homed:

and the dual homed:

What is not mentioned or documented anywhere about this configuration is if you can mix and match the topology between single homed and dual homed 2k's on a single pair of 5k's. I can only find one reference that mentions this vPC mixed topology option. Even with this option you are still burning up a second 2148t just to get port-channeling as an option if you don't want to use the 5k's directly. Burning up the few 10/1 Gig mix use ports on the 5k's for 1 Gig port-channels seems to be a huge waste and very expensive.
So there you have it, some items to mull over while planning out your data center design with Nexus and some hard hitting questions to ask your Cisco SE's when they show up talking about cost per port and maximizing functionality and flexibility per port. On top of the above items the 2148t also only does 1Gig, it doesn't support 10/100 at all, not a show stopper but another nagging item that limits some deployments. Don't get me wrong, I'm a fan of the new data center design and topology model that Cisco is building and with OTV coming out in the 5.0 NX-OS code release some really exciting things are going to be possible. Just keep in mind you might be buying an extra one or two 2k's to accommodate some of the systems that have the port-channel requirements for higher availability.
- Ed
Tuesday, March 23, 2010
Cisco Nexus command - Checkpoints and rollbacks
Using the Cisco Nexus 7010, 5010 and 2148's has changed some of the habits I have traditionally used for the Cisco IOS command set. Some of the new Nexus commands have become second nature and I now miss them on IOS. Being able to use grep is one I really wish was incorporated into IOS. I am used to having it with the ASA platform and now with the Nexus platform - going back to IOS 12.x and not having it there is annoying.
A new command that is really useful on the Nexus platform is checkpoint. There are several things that are unique about checkpoints and how you can use them. First, checkpoints are primarily used for rollback situations. They allow you to make changes on the system and if required due to an error rollback to a known good configuration on the system. There are three rollback types.
Atomic rollback is done when the configuration can be applied with NO errors.
Best Effort rollback will ignore errors and push the configuration onto the system.
Stop At First Failure will process the rollback request until it hits an error and then stops.
The default rollback type is Atomic and this is likely the most common rollback method you would use on a production environment. I am not aware of many folks wanting to rollback to a "Stop At First Failure" or "Best Effort" scenario situation unless true desperation has kicked in. There might be a case of the order of rollback if you are using VDC's and moving physical resources from one VDC to the other in which case perhaps Best Effort might be useful.
Also of note, the rollback feature must be used per Virtual Device Context (VDC), in other words, you have to run the command in each VDC. This is expected behavior as each VDC is it's own NX-OS instance and you have to run all the same commands to get the desired behavior out of the NX-OS platform.
The command itself is very simple:
checkpoint {checkpoint name} description {a description} | filename {path and filename}
Example: checkpoint cp-running-config-known-good-2010-03-22 description checkpoint of running config
There are some restrictions on the checkpoint name (max length 80 characters) and there are restrictions on the filename (max length of 75 characters and filename can't start with the word "system") but otherwise it is pretty straightforward process to get this going. I am using this on NX-OS version 4.3.1, earlier versions had more restrictions on file names and such so read the documentation if you are on an earlier release.
To see what the checkpoint command does you can use the show commands. To see all the checkpoints that are in a given VDC:
show checkpoint all
show checkpoint summary
The checkpoint command basically keeps a small database of checkpoints to allow you to rollback to a specific one and calculates the differences between a current state or checkpoint and that checkpoint you want to move to. It will generate a rollback script when you use the rollback command. If you want to see the differences that are being generated you can do that too:
show diff rollback-patch {checkpoint source name | running-config | startup-config | file filename} {checkpoint destination name | running-config | startup-config | file filename}
Example: show diff rollback-patch running-config checkpoint cp-running-config-known-good-2010-03-22
To actually do a rollback:
rollback running-config {checkpoint cp name | running-config | startup-config | file filename} {atomic | best-effort | stop-at-first-failure}
Example: rollback running-config checkpoint cp-running-config-known-good-2010-03-22 atomic
To see the status of rollbacks:
show rollback log
You can also clear out the checkpoint history and files, use the command with caution.
clear checkpoint database
This is a VERY useful command to build into your scripts prior to pushing out production changes on gear. It allows you to have a well known state stored locally and be able to rollback to it quickly in case of problems in your scripts. Awesome!
- Ed
A new command that is really useful on the Nexus platform is checkpoint. There are several things that are unique about checkpoints and how you can use them. First, checkpoints are primarily used for rollback situations. They allow you to make changes on the system and if required due to an error rollback to a known good configuration on the system. There are three rollback types.
Atomic rollback is done when the configuration can be applied with NO errors.
Best Effort rollback will ignore errors and push the configuration onto the system.
Stop At First Failure will process the rollback request until it hits an error and then stops.
The default rollback type is Atomic and this is likely the most common rollback method you would use on a production environment. I am not aware of many folks wanting to rollback to a "Stop At First Failure" or "Best Effort" scenario situation unless true desperation has kicked in. There might be a case of the order of rollback if you are using VDC's and moving physical resources from one VDC to the other in which case perhaps Best Effort might be useful.
Also of note, the rollback feature must be used per Virtual Device Context (VDC), in other words, you have to run the command in each VDC. This is expected behavior as each VDC is it's own NX-OS instance and you have to run all the same commands to get the desired behavior out of the NX-OS platform.
The command itself is very simple:
checkpoint {checkpoint name} description {a description} | filename {path and filename}
Example: checkpoint cp-running-config-known-good-2010-03-22 description checkpoint of running config
There are some restrictions on the checkpoint name (max length 80 characters) and there are restrictions on the filename (max length of 75 characters and filename can't start with the word "system") but otherwise it is pretty straightforward process to get this going. I am using this on NX-OS version 4.3.1, earlier versions had more restrictions on file names and such so read the documentation if you are on an earlier release.
To see what the checkpoint command does you can use the show commands. To see all the checkpoints that are in a given VDC:
show checkpoint all
show checkpoint summary
The checkpoint command basically keeps a small database of checkpoints to allow you to rollback to a specific one and calculates the differences between a current state or checkpoint and that checkpoint you want to move to. It will generate a rollback script when you use the rollback command. If you want to see the differences that are being generated you can do that too:
show diff rollback-patch {checkpoint source name | running-config | startup-config | file filename} {checkpoint destination name | running-config | startup-config | file filename}
Example: show diff rollback-patch running-config checkpoint cp-running-config-known-good-2010-03-22
To actually do a rollback:
rollback running-config {checkpoint cp name | running-config | startup-config | file filename} {atomic | best-effort | stop-at-first-failure}
Example: rollback running-config checkpoint cp-running-config-known-good-2010-03-22 atomic
To see the status of rollbacks:
show rollback log
You can also clear out the checkpoint history and files, use the command with caution.
clear checkpoint database
This is a VERY useful command to build into your scripts prior to pushing out production changes on gear. It allows you to have a well known state stored locally and be able to rollback to it quickly in case of problems in your scripts. Awesome!
- Ed
Subscribe to:
Posts (Atom)