Author: regit

Following the release of Scirius Community Edition 2.0, Stamus Networks is happy to announce the availability of Scirius Enterprise Edition U29. It is using the ruleset management capabilities of Scirius CE 2.0 so new features such as transformations and public sources are available.

This release continues on the redesign of the interface done with Scirius CE. The landing page for the appliances management has been modified to offer a list of appliances with a number of filtering and ordering options.

This list has expandable items so it is easy to get information about one specific probe:

The asynchronous tasks display has also been redesigned with the same consistent approach:

If we have been busy in the design, the U29 release also comes with three exciting new functional features: REST API, VPN based probes and device monitoring dashboards.

The REST API is allowing third party application to query and modify the objects defined in scirius:

Applications like SIEM would benefit of that as it will enable powerful integration.

The VPN based probes is a big change as it allows to have probes that can connect to SEE from behind private networks/NAT/Firewalls. There is no need anymore of direct connectivity from Scirius to the probe.

The monitoring dashboard is available for Scirius Enterprise itself and for the managed Stamus Probes. It gives key indicator of the health of the devices:

Feel free to contact us if ever you want more information about our products. We will be happy to set up a demo and answer any of your questions.

Stamus Networks is proud to announce the availability of Scirius Community Edition 2.0. This is the first release of the 2.0 branch that features a brand new user interface and new features such as lateral movement and target transformations. Both modify signatures to improve them. Lateral movement uses an algorithm to enlarge the signature IP address filter to detect attacks in the internal networks. Target transformation implement an other algorithm to add target keyword to signatures thus helping to find and visualize attack paths.

Scirius 2.0.0 now features an automated addition of any of the sources defined in the public ruleset list published by the OISF:

So you can now add to your ruleset a new feed/source in two clicks. That’s really easier compared to the form based method where a series of fields as to be entered. The addition process itself is also faster. The parsing and update time of a ruleset like ET Pro has been improved to be three times faster in this version.

As you may have noticed, Scirius 2.0.0 interface is really different from one from the previous versions:

Scirius is now using the Patternfly framework to provide a consistent interface and usability oriented components. Usability has also been improved by the integration of the documentation in the interface.

On Suricata related side, the most important change is the handling of transformations. Scirius can now modify the signatures through a transformation:

Currently two transformations are available and they aim at making Suricata’s detection capabilities stronger:

Lateral Movement

Lateral movement transformation modifies signatures to have them detect lateral movement. As signatures are often written with the EXTERNAL_NET and HOME_NET variables, this means they won’t match if both sides of a flow are in the HOME_NET. Thus, lateral movements are not detected. This transformation changes EXTERNAL_NET to any to be able to detect lateral movements. Scirius propose per ruleset, per categories and per signature changes. One of the value proposed is auto that use an algorithm that trigger the substitution if the signature verifies some properties.

Target Keyword

The second substitution is the addition of the target keyword donated by Stamus Networks. Available since Suricata 4.0, the target keyword can be used to tell which side of a flow triggering a signature is the target. If this key is present then related events are enhanced to contain the source and target of the attack. Once more the user can choose the value of the option or let Scirius determine what side to use via an algorithm using signature properties.

For the eye candy fans, pktcity is now part of Scirius. This 3D webGL visualization interface is now available as part of the new dashboards:

Finally, for the list addicts, here is Scirius 2.0.0 changelog:

  • Rule transformation with lateral movement and target
  • Support of OISF public sources for easier setup
  • Convert documentation to sphinx and integrate it in interface
  • Rework of interface with Patternfly components
  • Link to Onyphe to get IP informations
  • Rules parsing optimization
  • More dashboards including pktcity webGL visualization
  • Initial REST API to interact with Scirius from outside

Scirius 2.0.0 is available on github. Debian packages for SELKS are also available. Users of Scirius Enterprise Edition will get access to this feature in the upcoming 29 release.

0

Suricata 4.0 is out and this switch from 3.x to 4.x is not marketing driven because the changes are really important. This post is not exhaustive on changes. It is Stamus Networks’ take on some of the important changes that have been introduced in this version.

Rust addition

This is the big step forward on the technology side. Suricata is written in C language. This gives performances and a good control over memory. But it goes with a series of well known problems. I name here buffer overflows, use after free, …

And the worse is that Suricata is parsing traffic content which is a kind of vice supercharged user input. If one should not trust user input, guess how careful we should be with network traffic. At Suricon 2016, Pierre Chifflier did present a proof of concept implementation of protocol parsers in Rust. The idea is to use the property of Rust that has been designed to avoid complete class of attacks on memory handling. But there is more in the approach as the implementation is using Nom which is a Rust parser combinator framework. It allows you to write protocol parser easily and in a reusable way. Thus the addition of Rust is two things at the same time: more security and easier code. Which means a lot of new protocols should be added in the near future.

Suricata 4.0 Rust support comes with NFS, DNS and NTP. NTP support is implemented via an external crate (read library): ntp-parser.

As mentioned before, the code uses Nom and the syntax is very different from traditional code. For instance, here is the code of ntp-parser parsing NTP extension:

named!(pub parse_ntp_extension,
    do_parse!(
           ty: be_u16
        >> len: be_u16 // len includes the padding
        >> data: take!(len)
        >> (
            NtpExtension{
                field_type:ty,
                length:len,
                value:data,
            }
        ))
);

This define a parsing function that read the stream of data. The code says, take 16 bits, store them as unsigned integer in ty. Then store the next 16 bits as unsigned integer in len. Then store in data a chunck of data of length len. And with that build a NTP extension structure. If the writing is concise and efficient, the best thing with Nom is under the hood. Nom is taking care of detecting the invalidities. For instance we could have a chunck of data of length 50, and len being set to 1000 (remember Heartbleed ?). Nom will see that there is not enough data available in the chunck and return it wants more data.

Better alerts

As you may know, the preferred output of Suricata is the EVE JSON format. It is flexible, easy to extend and easy to read by human and tools. Suricata 4.0 is introducing some major changes here:

  • ‘vars’ extraction mechanism
  • The new target keyword
  • HTTP bodies logging
HTTP body output

Suricata is able to uncompressed HTTP body on the fly and match on the uncompressed content. This means that if you get the payload of the stream triggering the alert in your event, you will just see compression noise and won’t be able to analyze why the alert was triggered. Suricata is now able to include the HTTP bodies in the alert. The analyst can then directly see from the event the content that did trigger the alert.

The following event shows how payload_printable is completely compression noise and the http_response_body_printable is readable:

Target keyword

The new target keyword is a fix on a very old problem. It is not possible to know in an alert event which side of the source or destination is the target of the attack. This is a problem as it is not possible to automate things due to that lack of information. The target keyword allow the rules writer to specify which is side is the target. Doing so automated analysis and better visualization can be made.

Usage is simple, signature has to contain the target keyword with value dest_ip or src_ip. For example, in a simple scan alert we have:

alert tcp $EXTERNAL_NET any -> $HOME_NET 3306 (msg:"ET POLICY Suspicious inbound to mySQL port 3306"; flow:to_server; flags:S; threshold: type limit, count 5, seconds 60, track by_src; reference:url,doc.emergingthreats.net/2010937; classtype:bad-unknown; target: dest_ip; sid:2010937; rev:2;)

If target is present in a signature, the alert is added an alert.source and alert.target field:

For example, on a visualization where node are IP address and links are alerts between the two, we can get an idea of the possible compromised path. With the target addition, we can switch from a non oriented graph:

To an oriented graph that show which paths were really possible:

If you know French, you can learn more about this subject with Eric Leblond’s talk at SSTIC 2017.

Vars extraction

This is one of the most expected feature of Suricata 4.0. This has been described by Victor Julien in an extensive blog post. The concept is to be able to define in signature data to extract and store them in a key value form. There is a lot of possible usage ranging from application version extraction to getting exfiltred data. For example, let’s consider there is a domain we are interested in. One interesting information is the list of email addresses where mail are sent to. To do so we can use the following signature:

alert smtp any any -> any any (msg:"Mail to stamus"; content:"rcpt to|3A|"; nocase; content:"stamus-networks.com"; within: 200; fast_pattern; pcre:"/^RCPT TO\x3a\s*<([\w-\.]+@stamus-networks.com)>/ism pkt:email"; flow:established,to_server; sid:1; rev:1;)

The magix here is the groupe in the regular expression ([\w-\.]+@stamus-networks.com) that is save in a packet var named email by the pkt:email in the regular expression definition.

Using that signature we get this kind of alerts:

The key point here is the vars sub object:

  "vars": {
    "pktvars": [
      {
        "email": "eleblond@stamus-networks.com"
      }
    ]
  },

We have an extraction of the data and this can be easily search by tool like Elasticsearch or Splunk.

Conclusion

Suricata 4.0 is really an important milestone for the project. Introduction of Rust is opening a really interesting path. The alerts improvement may change the way signatures are written and it will help to provide really accurate information to the analysts.

Suricata 4.0 is already available in SELKS and it will be available in Stamus Probe by the end of August. To conclude on a personal note, we, Stamus Networks, are really happy to have contributed to this release with features such as via HTTP body logging and target keyword.

Stamus Networks is proud to announce the availability of Scirius 1.2.0. This release of our Suricata ruleset management interface comes after 4 months of development bringing two new major features: rules transformations to manage IPS and users activity logging to ease collaboration.

Rules transformation

With rules transformations, Scirius can now manage Suricata in IPS mode but also add the filestore option to specific rules allowing the user to transform existing rules coming from feed in rules realizing file extraction.

A signature can be transformed per ruleset to a drop or reject rule as shown in the following capture:

The filestore transformation will trigger file extraction by Suricata in case of alert. This allows user to have file extraction without the need of cloning existing rules.

User activity logging

The second big new feature is user activity logging. It is now possible to comment actions. A team collaboring on the same Scirius can now comment actions such as disabling a rule or adding a threshold.

It is also possible to simply comment on a rule.

All these features are already available in Scirius Enterprise and Amsterdam and will be available in SELKS in the coming days.

Eric Leblond gave a talk entitled “The adventures of a Suricata in eBPF land” at netdev 1.2, the Technical Conference on Linux Networking. This talk reviewed Stamus Networks’ work in the field of bypass and showed how the eBPF technology can be used to implement this feature.

eBPF is a technology that extends the traditional Berkeley Packet Filter that you can for example use with tcpdump. For instance eBPF filter can be written in a subset of C and allows kernel and userspace to share data via maps that can be for example an array or hash table. This technology has been used to implement a kernel bypass in Suricata. The idea is that Suricata is asking the Linux kernel to stop sending  it (bypass) packets for particular flow once it has decided that no further inspection is needed to be done.

For detailed information on the subject, you can get the Slides of “Suricata and eBPF” or watch the video that is already available thanks to the great work of Netdev team:

0

Introduction

Stamus Networks was working on a new Suricata feature named bypass. It has just been merged into Suricata sources and will be part of the upcoming 3.2 release. Stamus team did initially present his work on Suricata bypass code at Netdev 1.1, the technical conference on Linux networking that took place in Sevilla in February 2016.

In most cases an attack is done at start of TCP session and generation of requests prior to attack is not common. Furthermore multiple requests are often not even possible on same TCP session. Suricata reassembles TCP sessions till a configurable size (stream.reassembly.depth in bytes). Once the limit is reached the stream is not analyzed.

Considering that Suricata is not really inspecting anymore the traffic, it could be interesting to stop receiving the packets of a flow which enter in that state. This is the main idea behind bypass.

The second one consist in doing the same with encrypted flows. Once Suricata sees a traffic is encrypted it stops inspecting it so it is possible to bypass the packets for these flows in the same way it is done for packets after stream depth.

In some cases, network traffic is mostly due to session we don’t really care about on the security side. This is for example the case of Netflix or Youtube traffic. This is why we have added the bypass keywords to Suricata rules language. A user can now write a signature using this keyword and all packets for the matching flow will be bypassed. For instance to bypass all traffic to Stamus Networks website, one can use:

alert http any any -> any any (msg="Stamus is good"; content:"www.stamus-networks.com"; http_host; bypass; sid:1; rev:1;)

This is for sure just an example and as you may have seen our website is served only on HTTPS protocol.

Currently, Netfilter IPS mode is the only capture supporting the bypass. Stamus team represented by Eric Leblond will be at Netdev 1.2, first week of October 2016, to present an implementation of bypass for the Linux AF_PACKET capture method based on extended Berkeley Packet Filter.

And if you can’t make it to Japan, you will have another chance to hear about that during suricon, the Suricata user conference that will take place in Washington DC beginning of November.

Suricata bypass concepts

Suricata bypass technics

Suricata is now implementing two bypass methods:

  • A suricata only bypass called local bypass
  • A capture handled bypass called capture bypass

The idea is simply to stop treating packets of a flow that we don’t want to inspect anymore as fast as possible. Local bypass is doing it internally and capture bypass is using the capture method to do so.

Test with iperf on localhost with a MTU of 1500:

  • standard IPS mode: 669Mbps
  • IPS with local bypass: 899Mbps
  • IPS with NFQ bypass: 39 Gbps
Local bypass

The concept of local bypass is simple: Suricata reads a packet, decodes it, checks it in the flow table. If the corresponding flow is local bypassed then it simply skips all streaming, detection and output and the packet goes directly out in IDS mode and to verdict in IPS mode.

Once a flow has been local bypassed it is applied a specific timeout strategy. Idea is that we can’t handle cleanly the end of the flow as we are not doing the streaming reassembly anymore. So Suricata can just timeout the flow when seeing no packets. As the flow is supposed to be really alive we can set a timeout which is shorter than the established timeout. That’s why the default value is equal to the emergency established timeout value.

Capture bypass

In capture bypass, when Suricata decides to bypass it calls a function provided by the capture method to declare the bypass in the capture. For NFQ this is a simple mark that will be used by the ruleset. For AF_PACKET this will be a call to add an element in an eBPF hash table stored in kernel.

If the call to capture bypass is successful, then we set a short timeout on the flow to let time of already queued packets to get out of suricata without creating a new entry and once timeout is reached we remove the flow from the table and log the entry.

If the call to capture bypass is not successful then we switch to local bypass.

The difference between local and capture bypass

When Suricata is used with capture methods that do not offer the bypass functionality of eBPF/NFQ mark – pcap, netmap, pfring – it will switch to local bypass mode as explained above. Bypass is available for Suricata’s IDS/IPS and NSM modes alike.

Handling capture bypass failure

Due to misconfiguration or to other unknown problems it is possible that a capture bypassed flow is sending us packets. In that case, suricata is switching back the flow to local bypass so we handle it more correctly.

Stamus Networks is proud to announce the availability of version 1.0, nicknamed “glace à la vanille”, of Amsterdam, our container based ready to use Suricata IDS. Amsterdam is a fully web managed software appliance that is using Docker to provide:

  • Network Intrusion Detection and Network Security Monitoring via Suricata
  • Log storage and analysis via the Elastic stack: latest Logstash, Elasticsearch and Kibana are part of the Amsterdam
  • Suricata ruleset management and basic reporting via Scirius our web interface
  • Alerts listing and acknowledgement via Evebox

Scirius homepage

Each component is running in its own container and Amsterdam is using by default the official image on Docker Hub. This guarantees you fast update and heavily tested software. The orchestration of the different containers is done via Docker compose but all the details are hidden to you and Amsterdam should be your only interface in daily usage.

Installation is just a few commands:

pip install amsterdam
amsterdam -d ams -i wlan0 setup
amsterdam -d ams start

Once every containers are running, you can simply point your browser to https://localhost/ to start analyzing the traffic and fine tune the system. Kibana is coming with a set of predefined dashboards so you don’t have to build your own before starting to work.

Kibana Alert dashboard

Amsterdam offers you really easy upgrade via integrated commands:

amsterdam -d ams upgrade
amsterdam -d ams restart

Amsterdam is multi instances. For example, let’s say you have two customers where you analyzed the traffic when on site. You can set up two instances:

amsterdam -d customer1 -i wlan0 setup
amsterdam -d customer2 -i eth0 setup

and start the first one when at customer 1

amsterdam -d customer1 start

and second one when at customer 2

amsterdam -d customer2 start

The two different instances are not sharing any data, so you can freely show the interface to any of the customer if running the good instance. All data and configuration files are in customer1 directory for first customer and customer2 for the second one.

Amsterdam can digest any JSON formatted data. For that is is enough to copy a file to analyzed in the suricata directory inside the instance:

cp /path/to/passwords.json customer1/suricata/

This method makes it really easy to combine different sources of information into Kibana dashboards:
Pshitt and Suricata information

Amsterdam is also really easy to tune. The configuration files are stored for each components in the config directory so you can easily update Suricata, Logstash or Nginx configuration.

Stamus Networks is really excited by this first stable release of Amsterdam and we think that it has never been so easy to sniff and understand your network.

This release is dedicated to the memory of Edith Leblond.

Stamus Networks is proud to announce the availability of Scirius 1.1.6. This new release brings interesting new features and a lot of bugfixes to our Suricata ruleset manager.

Rule page in scirius 1.1.6

The main new features in release are:

  • Backup support
  • Threshold support
  • Xbits and hostbits support
  • Down detection of scirius
  • Top src and destination in rule page
  • Fix of test system that takes Suricata local config into account

The backup system adds a set of new commands to manage.py to backup and restore completely a Scirius instance. scbackup will do a backup and screstore will erase everything and restore latest backup. Backup can be done locally but it is also possible to use FTP, Dropbox or Amazon AWS to store and fetch backups.

On the usability feature side the most important is the support of thresholding. Scirius is now managing a threshold.config that is used by Suricata to limit or suppress alert(s) for a signature under certain conditions. Easiest way to access this feature is to start from a rule page and look at new top source and destination tables:

Top src and dest IP for a signature

The arrow down and the cross can be clicked to trigger edition of a form for a threshold (limit) or a suppression. For instance if you click on the cross, you will get something like:
Suppression
If there is already a suppression activated for the network/IP, you get a warning:
Adding a suppression

Latest ruleset management feature is the handling of the new xbits and hostbits. When a rule is disable, all the rule sharing a flowbits, a xbits or a hostbits are also deactivated.

At last, browser is now detecting that Scirius is down allowing you to avoid to navigate away from a form you were editing till connection is not restored:
Scirius down

Scirius 1.1.6 may be a minor release for the number in term of features it adds a lots of things users were asking for. You can already get scirius 1.1.6 in latest amsterdam. And it will be part of SELKS 3.0 that will be available really soon.

0

Suricata stats in EVE JSON format

Suricata 3.0 will come with a lot of improvements on the output side. One of them is the ability to output Suricata internal statistics in the EVE JSON format.

Stats event in EVE JSON format

This functionality is already used by scirius to display statistics graphs of the Suricata running in SELKS, Amsterdam or Stamus Networks’ appliances:

Stats in scirius

This statistic sometimes help to visualize the impact of configuration change. For example, in the next screenshot the generic receive offloading on the capture interface has been disable at 23:33:

Impact of iface offloading

Impact is cristal clear as the counter of invalid decoding did stop increasing.

Using Kibana Timelion plugin

Amsterdam came with Kibana 4 and the Timelion plugin is preinstalled. Timelion is a plugin providing a new interface and language to graph timeline.

As Suricata stats data are fed into Elasticsearch, we can use it to graph Suricata performance data.

For example to graph DNS and HTTP memory usage, one can use the following syntax:

.es(metric=’avg:stats.dns.memuse’).label(‘DNS’) .es(metric=’avg:stats.http.memuse’).label(‘HTTP’)

Result is the following graph:
Screenshot from 2016-01-07 11-01-48

If you have a counter and want to graph rate, then you can use:

.es(metric=’avg:stats.capture.kernel_packets’).derivative().label(‘PPS’) .es(metric=’avg:stats.capture.kernel_drops’).derivative().label(‘Drops’)

And you get the following graph:

Screenshot from 2016-01-07 10-59-01

One interesting thing with Timelion is that you can use Lucene query to get a count of something really easily. For example to get a view on the rate of different event type, one can use:

.es(q=’event_type:http’) .es(q=’event_type:tls’) .es(q=’event_type:dns’)

Rate of different event types

Both method can be mixed so, if you have different probes (let’s say probe-1 and probe-2) you can do something like:

.es(q=’host.raw:”probe-1″‘, metric=’avg:stats.dns.memuse’).label(‘Probe-1 DNS’) .es(q=’host.raw:”probe-2″‘, metric=’avg:stats.dns.memuse’).label(‘Probe-2 DNS’)

Conclusion

The new Suricata statistic output is really improving the information we can use when doing performance analysis. Combined with timelion, we get a really easy and powerful solution. If you want to give a try to all these technologies one of the easiest way is to use Amsterdam which comes with latest Suricata and a pre installed timelion.

Stamus Networks is proud to announce the availability of the first technology preview of Amsterdam.

Amsterdam is a ready-to-use Suricata IDS/NSM based system running on Docker. It features the same components as SELKS our live and installable Suricata distibution. So by running Amsterdam you get:

  • Suricata: latest version of the IDS/NSM engine
  • Elasticsearch: the powerful search engine
  • Logstash: the data pipeline application injecting Suricata events in the database
  • Kibana: Version 4 of the famous dashboard interface
  • Scirius: Stamus Networks’ Suricata ruleset management interface

Each component runs in a separate container and Amsterdam is using the official images for ELK stack to provide you an always up-to-date experience. Amsterdam is available under GPLv3 license.

Using Amsterdam is really simple. Once installed, you need to setup a directory that will contain data and configuration. For example to create and use a directory named ams-wlan0 and sniff the wlan0 interface on host, one can run once:

amsterdam -d ams-wlan0 -i wlan0 setup

You can then start Amsterdam by running:

amsterdam -d ams-wlan0 start

Scirius running in Amsterdam

Once all containers are built and started, you can point your browser to http://localhost:8000 to get access to the management interface.

It is possible to run multiple instances of Amsterdam on the same system by using different data directories. Each data directory contains the configuration files of the components so you can easily tune your installation.

More information and source code available on Github.