Tuesday, April 13, 2010

Cisco ASA syslog over TCP - Potential to stop forwarding traffic!

I ran into an interesting configuration issue with the Cisco ASA and doing syslog via TCP to a syslog server. It turns out that Cisco has a feature in the ASA that when pushing logging information to a sylog server via TCP that if the server does not respond it will stop the ASA from building out new session flows and therefore it will stop forwarding traffic. It seems that existing TCP sessions continue to allow traffic to flow, the rational being that the logging of that flow being established was sent to the syslog earlier and therefore it is audited event. Apparently this was a specific request from some government agencies who did not want their firewalls to pass traffic if logging was not possible. Reasonable enough but it isn't particularly well documented and does not alert you of it's default behavior at all when you enable the syslog command using TCP.

You can find all the details in the Cisco ASA 5500 command line configuration guide (this one is for 8.2 code release) and look under "Sending Syslog Messages to a Syslog Server" to see the parameters. They do provide a optional command switch that allows the ASA to continue working even if the syslog server is down.

So how can this be a problem? It turns out that a lot of shops rotate their syslog files and will write a quick script to stop their daemon for syslog, change their logfile and then start the syslogd daemon again. This causes the TCP session to fail because the syslogd daemon is no longer keeping the socket open. Depending on the thresholds you have built on the network this can cause your ASA to stop forwarding traffic. Worse, it is difficult to figure out what is happening because existing log flow sessions (perhaps and ssh or database session) will still be working but anything new you attempt to send through the ASA will not. Also, the ASA does not consider this a flag to attempt to failover to a standby unit - in fact, there is no way to even monitor the syslog process to flag it as such. I think that is something that needs to be added so the failover unit can then test to see if it can connect to the syslog server and if so, failover so that your firewall continues to forward traffic as expected.

The error you will get in the local log is:
%ASA-3-201008: Disallowing new connections.

It tells you what is happening - it doesn't tell you why it is happening though - very frustrating.

This is not an issue if you are using UDP for syslog traffic on the ASA's as they assume if you are doing UDP to the syslog server it is not guaranteed delivery and therefore you don't have to meeting the same standards for logging or security.
- Ed


Anonymous said...

logging permit-hostdown is the configuration command that will turn off this feature.

JaroS said...

Thanks for the article as well as for the fixing command ;)

Anonymous said...

Hi, we recently had the same issue on a ASA5580, it caused intermittent issues for trasversing traffic. Did that command "logging permit-hostdown enabled" fixed the issue?.

Ed Horley said...

You have two options, make syslog work over UDP then you won't run into this issue or you have to do the logging host permit-hostdown command to have the ASA still work properly when the syslog server is down or unavailable.

Anonymous said...

Thank you Ed Horley for your quick response.

Unknown said...

This bit us pretty hard. I have an active CCNA-Security, and I'm sure it was not covered.

Anonymous said...

You will also get bit by this if you try to use TLS/SSL based logging and the firewall can't validate the certificate of the receiving syslog server. (i.e. If you try to use a self-signed certificate on the syslog server.)