Our Blog

0

Introduction

Elasticsearch and Kibana are wonderful tools but as all tools you need to know their limits. This article will try to explain how you must be careful when reading data and explain how to improve this situation by using an existing Elastisearch feature.

The Problem

All did start with the analysis of an SSH bruteforce attack coming from Vietnam. This attack was interesting because of the announced SSH client “PuTTY-Local: Mar 19 2005 07:19:17” which really looks like a correct PuTTY software version when most attack don’t spoof their software version and reveal what they are using.

The Kibana dashboard was showing all information needed to get a good idea of attacks:

Screenshot from 2014-12-03 16:36:47

But when looking at less used and most used passwords, there was something really strange:

Screenshot from 2014-12-02 08:59:41

For example, webmaster is seen in the two panels with different values which is not logical.

By adding a filter on this value, the result was a bit surprising:

Screenshot from 2014-12-02 09:08:59

When looking at the detail of events, it was obvious this last result was correct. This SSH bruteforce has tried 10 different logins and has always used the same dictionary of 23 passwords.

To a solution

So the panels with top passwords and less seen passwords are displaying incorrect data in some circumstances. They have been setup in Kibana using the terms type.

This corresponds in Elasticsearch to a facets query. Here’s is the content of the query with the filter removed for readability:

{
 "facets": {
    "terms": {
      "terms": {
        "field": "password.raw",
        "size": 10,
        "order": "count",
        "exclude": []
      },
  }
}

So we have a simple request and it is not returning the correct data. The explanation of this problem can be found in Elasticsearch Issue #1305.

Adrien Grand is explaining that a algorithm returning possibly inaccurate values has been chosen to avoid a too memory intensive and network intensive search. The per-default algorithm is mainly wrong when they are more different values than searched values.

We can confirm that behavior in our case by asking for 30 values (on the 23 different passwords we have):

Screenshot from 2014-12-02 09:30:30

The result is correct this time.

If we continue reading Adrien Grand comment on the issue, we see that a shard_size parameter has been introduced to be able improve the algorithm accuracy.

So we can use this parameters to improve the accuracy of the queries. Patching this in Kibana is trivial:

diff --git a/src/vendor/elasticjs/elastic.js b/src/vendor/elasticjs/elastic.js
index ba9c8ee..8daa72a 100644
--- a/src/vendor/elasticjs/elastic.js
+++ b/src/vendor/elasticjs/elastic.js
@@ -3085,6 +3085,7 @@
         }
 
         facet[name].terms.size = facetSize;
+        facet[name].terms.shard_size = 10 * facetSize;
         return this;
       },

Here we just choose a far larger shard_size than the number of elements asked in the query. We could also have used the special value 0 (or Integer.MAX_VALUE) for shard_size to get perfect result. But in our test setup, Elasticsearch is failing to honor the request with this parameter. And furthermore, the result was already correct:

Screenshot from 2014-12-02 10:10:10

This patch has been proposed to Elasticsearch as PR 2106.

That was a small patch but this fixed our dashboard as the value in the terms panels are now correct:

Screenshot from 2014-12-03 16:44:57

0

Stamus Networks is proud to announce the availability of SELKS 1.1 stable release. SELKS is both Live and installable Network Security Management ISO based on Debian implementing and focusing on a complete and ready to use Suricata IDS/IPS ecosystem with its own graphic rule manager. Stamus Networks is a proud member of the Open Source community and SELKS is released under GPLv3 license.

New features:

  • Suricata 2.1beta2 with SMTP support as main new feature
  • Optimized admin scripts
  • Scirius 1.0-beta1 rule manager
  • Authentication for remote access via HTTPS with user based role access
  • Improved Kibana dashboards and an addition of the SMTP dashboard
SMTP-Attachments

SELKS dashboard showing SMTP Attachments

Rule detail

Rule detail in scirius

You can download SELKS 1.1 from Stamus Networks’ open source page. Happy users of SELKS 1.0 can upgrade to SELKS 1.1 by using the traditional apt-get update && apt-get dist-upgrade. Please note that default login/password for HTTPS access is selks-user/selks-user.

More information: Howto and README

Follow us on Twitter, Google+ and Github

Get help at Freenode IRC on the #SELKS channel and/or  Google Mailing list.

Stamus Networks is proud to announce the availability of version 1.0-beta1 of Scirius, our web interface for Suricata ruleset management. This new release is a huge step toward 1.0 release as it contains a lot of new features and improvements. You can download it from Github download page.

The most visible update is the new design of the interface. It has been been completely changed thanks to Bootstrap CSS framework.

Screenshot from 2014-11-12 10:45:55

But the first change for user is that authentication and user management is now by default. Scirius is now multi user and features three level of permissions from read-only to superuser.

Another new feature is the display of graphics in some page. They are using Elasticsearch data. For example, the next screenshot is showing detail of a rule. A graph has been added to show the activity for that specific rules:

Screenshot from 2014-11-12 10:46:46

The interface is now more responsive as asynchronous requests are used to interact with Elasticsearch. This guarantee a responsive interface even if your Elasticsearch is slow.

SELKS user can upgrade to Scirius 1.0-beta1 via apt-get update && apt-get dist-upgrade. Please note that the default user/password on SELKS is selks-user/selks-user. Do not forget to change it after first login.

0

Stamus Networks supports its own generic and standard Debian Wheezy 64 bit packaging repositories for

These repositories provide Debian package for the newest Suricata IDS/IPS , htp releases and newest long-term stable kernel level version. SELKS already includes those repositories under /etc/apt/sources.list.d/selks.list.

You can use as follows:

wget -O – -q http://packages.stamus-networks.com/packages.stamus-networks.com.gpg.key | apt-key add – && \
apt-get update

Then  you can add the following :

deb http://packages.stamus-networks.com/debian/ wheezy main
deb http://packages.stamus-networks.com/debian-kernel/ wheezy main

in /etc/apt/sources.list.d/stamus.list for example.

The repositories contain packages for the long-term stable kernel level version. So if you would like to upgrade to the latest long-term supported kernel you can just do (on Debian):

apt-get update && apt-get upgrade
apt-get install linux-libc-dev linux-headers-3.14.19-stamus linux-image-3.14.19-stamus

 

UpgradeKernel

Kernel Packages

UpgradeKernel2

Kernel Upgrade

UpgradeKernel3

Verification

Those repos are included by default in SELKS.

Anther example:

apt-get install suricata

After giving a talk about malware detection and suricata, Eric Leblond gave a lightning talk to present SELKS at hack.lu conference.

Screenshot from 2014-10-23 13:46:02

You can download the slides here: 2014 hacklu selks

Introduction

SELKS 1.0 is featuring a privacy dashboard. This is a dashboard focusing on HTTP and TLS protocols. The used data source is events generated by Suricata for these two protocols. The goal of this dashboard is to show the different interaction between website. For example, you will see on the following video that opening elysee.fr which is the French president website is triggering the opening of page on Facebook and Google Analytics. This means that both Facebook and Google knows you’ve went to the presidential website.

Setup

The setup of the demonstration is simple as we are connecting to the web on the virtual machine. This has been done because it was easier to record the screencast in that case. But the most interesting setup consists in sniffing the traffic of the physical host from SELKS running on the virtual machine. This way, SELKS will analyse your local traffic and you will be able to see in SELKS all the events coming from your real internet life.

The setup is simple. In Virtualbox, go to the machine details and click on network. Then choose to bridge your physical network interface and allow promiscuous mode on the interface:

Screenshot from 2014-10-19 12:10:43

Demonstration

Watch the following video to discover how this dashboard can be used:

An other way to use this privacy dashboard is to use one of the filter. For instance, if we filter on http.http_refer:"http://www.whitehouse.gov" we get a dashboard containing all HTTP events with a referrer being the US president website. So if you look at the hostname on the following screenshot, you will see that going on whitehouse.gov also lead you to external websites

Whitehouse links

My favorite in this list is www.youtube-nocookie.com but something like cloud.typography.com is really interesting too. Even a website like whitehouse.gov is not anymore hosting is own fonts.

The privacy dashboard is also containing TLS information extracted by Suricata. It lists TLS connections done on well know wesbite such as Facebook, Twitter or Google. For example, we can see that going on CNN cause some TLS hits on Twitter and Facebook.
Screenshot from 2014-10-19 12:00:45
TLS being encrypted we can’t prove this link and that’s the short time frame that stand for a proof of the link between websites.

Conclusion

SELKS privacy dashboard is just an example of what you can achieve in SELKS by using Suricata network security monitoring capabilities. The demonstration shown here is local but don’t forget you can do it at the level of a whole network.

0

Stamus Networks is proud to announce the availability of SELKS 1.0 stable release. SELKS is both Live and installable Network Security Management ISO based on Debian implementing and focusing on a complete and ready to use Suricata IDS/IPS ecosystem with its own graphic rule manager. Stamus Networks is a proud member of the Open Source community and SELKS is released under GPLv3 license.

Screenshot from 2014-10-15 21:39:11
You can download SELKS from SELKS main page.

SELKS is comprised of the following major components:

It offers proven, powerful, innovative and scalable open source multi-threading technologies in a bundle.

SELKS 1.0 comes with 10 pre-installed Kibana IDS/NSM dashboards. They cover analysis of the Suricata alerts and events with per-protocol dashboards (Alerts, HTTP, Flow, SSH, TLS,DNS …). Some dashboards are also dedicated to more specific tasks – like the PRIVACY dashboard:
Screenshot from 2014-10-15 21:28:27
It shows privacy related information such as which page are leading to well know personal data providers such as Facebook, Twitter or Google.

SELKS provides Scirius – a rules management interface for Suricata. Scirius has been developed by Stamus Networks to provide interaction with Kibana and Elasticsearch. It displays for example statistics on rules and links to existing Kibana dashboards:
Screenshot from 2014-10-15 21:17:37

Scirius provides up-to-date signatures via EmergingThreats Open (or PRO ) ruleset and SSL abuse.ch signatures
Screenshot from 2014-10-15 21:18:29

Scirius can be upgraded via standard Debian method (apt-get upgrade). Stamus Networks is also determined to provide the latest stable Debian kernel release for SELKS. Upgrade to the latest stable kernel is easy via the package system. For example, it is possible for the user running the installed version to upgrade the kernel to the latest 3.14 version:

kernel-upgrade-3.14.21
Scirius 1.0rc1 can upgrade to the 1.0 version by running apt-get dist-upgrade

The list of provided Kibana dashboards will be augmented in the future and this will be done seamlessly via the Debian packaging system and Kibana autodiscovery:

Kibana-dashboards

We really hope you will enjoy SELKS  an enterprise-grade IDS and Network Security Monitoring system in 30 seconds.

How to and README

Follow us on Twitter, Google+ and Github

Lets talk about SELKS…

0

Stamus Networks is proud to announce the availability of SELKS 1.0 RC1. This is the first release candidate of our live and installable ISO based on Debian implementing a ready to use Suricata IDS/IPS. More about SELKS you could read on our Open Source page.

This release includes major overhaul and improvements:

  • Introducing for the first time the new Stamus Networks package repositories developed especially for SELKS – Kibana, Scirius
  • Update and upgrade all software and SELKS the Debian way (apt-get or aptitude)
  • 9 ready to use out of the box IDS/IPS dashboards
  • Over 150 fields to search,select,filter and easily analyze upon right out of the box
  • Fully enabled logging
  • Suricata 2.1beta1 (adding flow and alert payload logging to the NSM arsenal)
  • Scirius 0.8  (latest release of our graphic Suricata ruleset manager)

A better interface

SELKS 1.0 RC1 comes with preloaded dashboards and a modified version of Kibana:

Dashboards

Screenshot from 2014-09-09 20:44:42
This allows interaction with Scirius, our open-source Suricata ruleset management interface:

Screenshot from 2014-09-09 20:26:15

SELKS 1.0 RC1 contains Suricata 2.1beta1 which brings flow and alert payload logging – available right out of the box on the predefined dashboards:

Screenshot from 2014-09-09 22:45:00

Alert-SELKS-Payload1

Easy upgrade

Stamus is dedicated to provide the latest releases of Suricata, htp and kernel level. That’s why we provide generic Debian packaging for the newest Suricata IDS/IPS , htp releases and newest long-term kernel level version (3.14.18 at the time of this writing).

SELKS comes with a standard Debian Wheezy distribution with 3.2 kernel – if you would like to upgrade to the latest long-term supported kernel you can just do (for example kernel 3.14.18):

apt-get update && apt-get upgrade
apt-get install linux-headers-3.14.18-stamus linux-image-3.14.18-stamus

For everything else you can just do:

apt-get update && apt-get upgrade

As easy as that!

DOWNLOAD SELKS HERE

 

Stamus Networks is proud to announce the availability of the version 0.8 of Scirius, the web management interface for Suricata. This new release contains a lot of new features as well as bug fixes.

On the functional side, the main new features are:

  • Support for content such as IP reputation list
  • Changelog support: display change on sources after update
  • Global search: text search in all objects
  • The changelog on source is really useful to know what signatures have been added or modified:
    Screenshot from 2014-09-03 16:51:18

    The global search is accessible from the top bar in all pages. It allows you to quickly access to the matching objects:
    Screenshot from 2014-09-03 16:53:23

    Among the other features, one can also mention the syntax highlighting for the rule. Rule detail now comes with information about rule status in rulesets and rule stats:
    Screenshot from 2014-09-03 16:36:58

    We hope you will enjoy this new release. As usual it can be downloaded from Github. Happy NIDSing!

Thanks to the EVE JSON events and alerts format that appear in Suricata 2.0, it is now easy to import Suricata generated data into a running Splunk.

To ease the first steps of integration, Stamus Networks is providing a Splunk application: Suricata by Stamus Networks

It can be installed like any other applications and it just requires that a Suricata EVE JSON file is known and parsed by Splunk.

Current version is providing a dashboard and a few searches:

Screenshot from 2014-07-30 15:39:11

This post describes how to import the application and if you don’t have already done it how to import data from a Suricata EVE file.

Importing the application

Importing the application is done via the Apps menu on top of Splunk starting page:

Screenshot from 2014-07-30 15:33:39

Suricata by Stamus Networks application is currently provided as a file, so you need to download it: Suricata by Stamus Networks. Once done, you can add the application:

Screenshot from 2014-07-30 15:33:50

You need to select the file

Screenshot from 2014-07-30 15:34:05

Importing a Suricata EVE JSON file

Since splunk 6.1.x, the recognition of the file format is automatic. If you are using an older version of Splunk, you may need to refer to this page to import Suricata EVE file.

Here’s the detailed procedure to import Suricata EVE data into Splunk. From the starting page, we click on Add Data:

Screenshot from 2014-07-30 15:27:48

Then we click an Files & Directories to tell Splunk to import data from Suricata EVE JSON file:

Screenshot from 2014-07-30 15:28:08

Once done, we click on the New button:

Screenshot from 2014-07-30 15:28:21

Now, we only need to give the complete path to the eve.json file:

Screenshot from 2014-07-30 15:28:47

Once this is done, we just need to click on all Continue buttons to be done.

Using the application

Now, we can go to the application by clicking on Suricata by Stamus Networks:

Screenshot from 2014-07-30 15:34:42

Next step can be to to go the dashboard:

Screenshot from 2014-07-30 15:35:02

The dashboard contains some interesting panels like the following one who displays the destination IP addresses that are using a self-signed certificate for TLS connections:
Screenshot from 2014-07-30 14:37:52

Conclusion

This application should evolve with time, so stay tuned and follow us on twitter for more information.