Daniel Miessler

Appearance

Use Splunk as Your Remote Syslog Server

May 26, 2023

Use Splunk as Your Remote Syslog Server ​

So I’ve been messing with Splunk> a bit recently, and as part of that I’ve been sending logs from iptables, snort, and apache–not to mention the other stuff that naturally lands within /var/log/messages.

get_requests

As you can see, the reason I’m doing this is to get a brutally powerful data view in one interface. Here I’m showing some GET requests within my Apache logs, but I currently have saved searches for all these various types of information:

  • drops on my firewall
  • accepts on my firewall
  • successful SSH logins (password or key)
  • failed SSH logins (password or key)
  • associations to my wireless
  • incoming GET requests to Apache
  • user agents

The key with Splunk> is the quickness in which you can search raw data, and create powerful visualizations of the results.

firewall drops by port within 3 hours

dports2

Syslog Setup ​

So this all requires that Splunk> see your log data; here’s how to set up syslog-ng to forward your various log types to an arbitrary destination.

netfilter/iptables ​

Log your desired traffic (this is my default-deny at the bottom of my ruleset)

bash
/sbin/iptables -A INPUT -i eth0 -d $SENECA -j LOG –log-level 7 –log-prefix "Firewall: Default Deny: "

This will automatically go to syslog on most systems.

Apache ​

You don’t do anything specific in Apache, other than make sure you’re logging the stuff you want. I prefer to get user-agent and such in my logs:

bash
LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" combined
LogFormat "%h %l %u %t "%r" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-Agent}i" agent
LogFormat "%v %h %l %u %t "%r" %>s %b %T" script
LogFormat "%v %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" VLOG=%{VLOG}e" vhost

syslog ​

Then for the most important piece you have to:

Unsupervised Learning — Security, Tech, and AI in 10 minutes…

Get a weekly breakdown of what's happening in security and tech—and why it matters.

  1. Tell syslog-ng to parse your Apache logs
  2. Tell syslog-ng to send logs to your remote system (Splunk, in this case)

First, here’s how you get arbitrary, quickly expanding logs into syslog-ng:

bash
source access { file("/var/log/apache2/access_log" follow_freq(1) flags(no-parse));};

This names a source access (for accesslog) that will be harvested from a _file. The file is my main Apache log. The important bit is the follow_freq(1), as it keeps you from having to do crazy tail / pipe tricks to get access_log’s input into syslog-ng. The 1 says to parse the file for new content every second.

Then you need to define a destination for your logs:

bash
destination logserver { udp("your.remote.logserver.dns" port(514)); };

And then give the log command, which calls your custom source and your custom destination:

bash
log { source(access); destination(logserver); };

** Don’t forget to also add log lines for your default syslog source as well. **

And that’s pretty much it. Configure Splunk to listen on UDP/514 and you will have some decent data to start playing with. ::

Splunk Search Syntax | splunk.com