Easy Unified2 File Reading in Python

July 20, 2013

I recently consolidated my Python code bits for dealing with Snort and Suricata unified2 log files into a project called idstools. While I'll be adding more than just unified2 reading support, that is about it for now. While it can be installed with pip (pip install idstools), if you just want to play around with it I suggest cloning the repo (git clone https://github.com/jasonish/idstools.py). You can then use the REPL or write test scripts from within the idstools.py directory without having to install the library (yeah, basic stuff for Python developers). idstools does come with a few example programs that demonstrate unified2 file reading, namely, u2fast.py, u2tail.py and u2spewfoo.py (a simple clone of the Snort provided u2spewfoo).

Basic Unified2 File Reading

from idstools import unified2

reader = unified2.FileEventReader("tests/merged.log")
for event in reader:
    print("Event:n%s" % str(event))

This few lines of code will iterate through each record in the specified unified2 log files, aggregate the records into events and return each event as a dict. If straight up record is reading is more what you are after then check out unified2.FileRecordReader, or the lower level unified2.read_record function. Each event is represented as a dict containing the fields of a unified2 event record, with the associated packets represented as a list in event["packets"] and extra data records represented as a list in event["extra-data"].

Resolving Event Message and Classification Names

To make event reading just a little more useful, code to map signature and classifications IDs to descriptions is provided.

from idstools import maps

# Create and populate the signature message map.
sigmap = maps.MsgMap()
sigmap.load_genmsg_file("gen-msg.map")
sigmap.load_sidmsg_file("sid-msg.map")

# Get the description for 1:498.
print("Message for 1:498: %s" % (sigmap.get(1, 498).msg))

# Create and populate the classification map.
classmap = maps.ClassificationMap()
classmap.load_classification_file("classification.config")
print("The description for classification id 9 is %s, with priority %d." % (
        classmap.get(9).description, classmap.get(9).priority))

The example program u2fast.py is a complete example of reading events from one or more files, resolving event descriptions and classification names and printing the event in a "fast" like style.

Spool Reading

idstools also contains a spool reader for processing a unified2 spool directory as commonly used by Snort and Suricata. It supports bookmarking, deleting files, and open and close hooks which can be used to implement custom archiving.

from idstools import spool

def my_open_hook(reader, filename):
    print("File %s has been opened." % (filename))

def my_close_hook(reader, filename):
    print("File %s has been closed." % (filename))

reader = spool.Unified2EventSpoolReader(
    "/var/log/snort", "merged.log", delete_on_close=False,
    bookmark=True,
    open_hook=my_open_hook,
    close_hook=my_close_hook)

for event in reader:
    print("Read event with generator-id %d, signature-id %d." % (
            event["signature-id"], event["generator-id"]))

To see a more complete directory spool process, check out the u2tail.py example program.To learn more checkout idstools over at GitHub, PyPI, or the work-in-progress documentation on Read the Docs.