HOWTO: Use Splunk as Your Remote Syslog Server

So I’ve been messing with Splunk> a bit recently, and as part of that I’ve been sending logs from iptables, snort, and apache–not to mention the other stuff that naturally lands within /var/log/messages.

As you can see, the reason I’m doing this is to get a brutally powerful data view in one interface. Here I’m showing some GET requests within my Apache logs, but I currently have saved searches for all these various types of information:

• drops on my firewall
• accepts on my firewall
• successful SSH logins (password or key)
• failed SSH logins (password or key)
• associations to my wireless
• incoming GET requests to Apache
• user agents

The key with Splunk> is the quickness in which you can search raw data, and create powerful visualizations of the results.

firewall drops by port within 3 hours

Syslog Setup

So this all requires that Splunk> see your log data; here’s how to set up syslog-ng to forward your various log types to an arbitrary destination.

netfilter/iptables

Log your desired traffic (this is my default-deny at the bottom of my ruleset)

/sbin/iptables -A INPUT -i eth0 -d $SENECA -j LOG --log-level 7 --log-prefix "Firewall: Default Deny: "

This will automatically go to syslog on most systems.

Apache

You don’t do anything specific in Apache, other than make sure you’re logging the stuff you want. I prefer to get user-agent and such in my logs:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-Agent}i" agent
LogFormat "%v %h %l %u %t \"%r\" %>s %b %T" script
LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" VLOG=%{VLOG}e" vhost

syslog

Then for the most important piece you have to:

1. Tell syslog-ng to parse your Apache logs
2. Tell syslog-ng to send logs to your remote system (Splunk, in this case)

First, here’s how you get arbitrary, quickly expanding logs into syslog-ng:

source access {
    file("/var/log/apache2/access_log" <em>follow_freq</em>(1)
    flags(no-parse));
};

This names a source access (for access_log) that will be harvested from a file. The file is my main Apache log. The important bit is the follow_freq(1), as it keeps you from having to do crazy tail / pipe tricks to get access_log's input into syslog-ng. The 1 says to parse the file for new content every second.

Then you need to define a destination for your logs:

destination logserver { udp("your.remote.logserver.dns" port(514)); };

And then give the log command, which calls your custom source and your custom destination:

log { source(access); destination(logserver); };

[ ** Don't forget to also add log lines for your default syslog source as well. ]

And that's pretty much it. Configure Splunk to listen on UDP/514 and you will have some decent data to start playing with. ::

Links

[ Splunk Search Syntax | splunk.com ]