Category: Tutorials

Suricata EVE JSON format is becoming the de-facto standard for this IDS. All type of events are now exported to this format. The JSON format allows a nice handling of data in external tool like Elasticsearch or even DOM. The output is readable by human but as an event/record can contain a lot of data it can be difficult to do a by-eye analysis when looking at a file. The following screenshot give you an idea of the possible output:

Tailing EVE

Using standard unix tools like grep on the EVE JSON file is not the perfect idea. For example if you want to extract a field to get some statistics you may want to try using grep, cut or awk but you may find it painful. And it is worthed to mention here that JSON fields are not ordered.

Here to the rescue comes the jq utility. jq is a tool dedicated to the transformation/parsing of a JSON entry. It is Debian packaged, so a simple apt-get install jq is enough for the install.

Some jq examples

The most basic usage is to colorize the entry. To do that, just do something like

$ tail -n100 eve.json| jq '.'

The output is done the pretty way:
JQ displaying an event
To get a one line per event output, just add the -c flag to the command:
One line

To extract a single field from the JSON events, one can do:

$ jq '.src_ip' eve.json
"58.218.211.155"
"58.218.211.155"
"58.218.211.155"

The point to remember is that the point in .src_ip is a place holder for the current entry.

By default when a field is not present null is displayed in the output. To fix that, it is possible to filter the event to only get the one we are interested in. This is done via the select keyword. For instance to select the SSH events and extract the information about the client part one can do:

$ tail eve.json | jq -c 'select(.event_type == "ssh")|.ssh.client'
{"proto_version":"2.0","software_version":"PUTTY"}
{"proto_version":"2.0","software_version":"PUTTY"}

Far more things can be done with jq. Good starting points are the jq manual and wiki.

Introduction

SELKS 1.0 is featuring a privacy dashboard. This is a dashboard focusing on HTTP and TLS protocols. The used data source is events generated by Suricata for these two protocols. The goal of this dashboard is to show the different interaction between website. For example, you will see on the following video that opening elysee.fr which is the French president website is triggering the opening of page on Facebook and Google Analytics. This means that both Facebook and Google knows you’ve went to the presidential website.

Setup

The setup of the demonstration is simple as we are connecting to the web on the virtual machine. This has been done because it was easier to record the screencast in that case. But the most interesting setup consists in sniffing the traffic of the physical host from SELKS running on the virtual machine. This way, SELKS will analyse your local traffic and you will be able to see in SELKS all the events coming from your real internet life.

The setup is simple. In Virtualbox, go to the machine details and click on network. Then choose to bridge your physical network interface and allow promiscuous mode on the interface:

Screenshot from 2014-10-19 12:10:43

Demonstration

Watch the following video to discover how this dashboard can be used:

An other way to use this privacy dashboard is to use one of the filter. For instance, if we filter on http.http_refer:"http://www.whitehouse.gov" we get a dashboard containing all HTTP events with a referrer being the US president website. So if you look at the hostname on the following screenshot, you will see that going on whitehouse.gov also lead you to external websites

Whitehouse links

My favorite in this list is www.youtube-nocookie.com but something like cloud.typography.com is really interesting too. Even a website like whitehouse.gov is not anymore hosting is own fonts.

The privacy dashboard is also containing TLS information extracted by Suricata. It lists TLS connections done on well know wesbite such as Facebook, Twitter or Google. For example, we can see that going on CNN cause some TLS hits on Twitter and Facebook.
Screenshot from 2014-10-19 12:00:45
TLS being encrypted we can’t prove this link and that’s the short time frame that stand for a proof of the link between websites.

Conclusion

SELKS privacy dashboard is just an example of what you can achieve in SELKS by using Suricata network security monitoring capabilities. The demonstration shown here is local but don’t forget you can do it at the level of a whole network.

0

The Ubuntu used in this tutorial:

root@LTS-64-1:~/opt#uname -a
Linux LTS-64-1 3.5.0-45-generic #68~precise1-Ubuntu SMP Wed Dec 4 16:18:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

If you have these packages installed you need to remove them so that Scirius would work with the latest python dependencies.
Please be careful so that this actually does not affect your current running services. It is always best to test first 🙂

root@LTS-64-1:~/opt#apt-get remove django-tables python-django python-django-south python-git

Install the needed dependencies:

root@LTS-64-1:~/opt#aptitude install python-pip git
root@LTS-64-1:~/opt#pip install django django-tables2 South GitPython pyinotify daemon

Clone the latest version

root@LTS-64-1:~/opt#git clone https://github.com/StamusNetworks/scirius.git
root@LTS-64-1:~/opt#cd scirius/
root@LTS-64-1:~/opt/scirius# python manage.py syncdb

Start Scirius

root@LTS-64-1:~/opt/scirius#python manage.py runserver
Validating models…
Failed to setup thread-interrupt handler. This is usually not critical
0 errors found
May 20, 2014 – 19:51:27
Django version 1.6.4, using settings ‘scirius.settings’
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

If you need to connect to the server remotely (provide your ip)  –

root@LTS-64-1:~/opt/scirius#python manage.py runserver 10.0.10.5:8000
Validating models…
Failed to setup thread-interrupt handler. This is usually not critical
0 errors found
May 20, 2014 – 19:51:58
Django version 1.6.4, using settings ‘scirius.settings’
Starting development server at http://10.0.10.5:8000/
Quit the server with CONTROL-C.

Now lets have a walk through registering and adding a ruleset

For example (for the latest stable and dev Suricata) from  http://rules.emergingthreats.net/open/suricata/emerging.rules.tar.gz since Emerging Threats create, write and distribute specially tuned for Suricata rulesets that make  use of the advanced features of that IDS engine.

First we need to add a source:AddRuleset-1

AddRuleset-0

 

Then we add a ruleset:

 

AddRuleset-2

We need to edit the ruleset and select the categories we want from that ruleset:

 

AddRuleset-3

 

AddRuleset-4

Select categories:

AddRuleset-5

Validate changes:

AddRuleset-6
If you have already Elasticsearch, Logstash, Kibana installed on the same server,
you could do the following -> put in the values as on the picture – except the host name – chose the hostname to be exactly as your hostkey in Elasticsearch/Kibana , like in the pics below.

 

hostkey1 hostkey2

 

Then in Scirius ->

CreatingSuricata-1 CreatingSuricata-2

 

Now you should be able to see the hits and which rules are making the most noise 🙂

That’s it for a quick intro.