As I mentioned in the introductory article in this series (see here >>), Suricata produces not only IDS alerts but also produces protocol transaction logs, flow records, full packet capture and extracted files.

In this article we attempt to clear up a couple of misconceptions (myths) and illustrate these lesser-known capabilities through real-world examples.

Data used in this article

The example that we will use in this article to showcase Suricata’s capabilities is from https://www.malware-traffic-analysis.net/2021/02/08/index.html

NOTE: The PCAP and the artifacts extracted or present (files/domains etc) in this example contain live malware. If you chose to explore or visit any of the domains/servers/files/ listed here, you should do so at your own risk, in a controlled manner and only in a sandbox environment.

Suricata Myth 1 - Alerts Only

Myth: Suricata produces only alerts/alarms and you need another tool to produce the network protocol log evidence (network protocol transaction or network security logs) for that alert in order to confirm if it is real or a false positive.

Fact: This is simply not true.

Suricata produces all relevant network security monitoring logs: protocol, flow, file transaction, and anomaly logs, including the ones related to an alert - but also independent of alerts. In the regular JSON logs that Suricata generates (eve.json), you will find something called “flow_id” that correlates the network protocol data and evidence that Suricata has logged - to an alert event and that alert’s metadata. So for example for a HTTP based alert, all the relevant HTTP and or file transactions and flow logs would be available.

Suricata-Generated-Events v2 source
The ability to correlate any existing evidence/logs to an alert (“flow_id”) was introduced in 2014 by Suricata lead developer, Victor Julien

https://github.com/OISF/suricata/commit/f1185d051c210ca0daacdddbe865a51af24f4ea3

There are, of course, other and many more network security and protocol logs produced by Suricata before and after an alert occurred that are also accessible to the user.

Classic IDS alert

Here is an example of a legacy/classic IDS alert

Alert log:
02/08/2021-17:00:13.183792 [**] [1:2014819:3] ET INFO Packed Executable Download [**] [Classification: Misc activity] [Priority: 3] {TCP} 8.208.10.147:80 -> 10.2.8.101:49757

You can see that it does not contain much information beyond source and destination IP address. In short, it is distinctly lacking in context that might be useful for incident investigation or threat hunting.

The Suricata alert difference

The Suricata difference can be seen below.

NOTE: for easier reading, check these out on Pastebin here: https://pastebin.com/vXF59n6u

Notice the areas highlighted below in blue which demonstrate the additional data and indicate there are flow and file records associated with the alert.

Alert log:
{"timestamp":"2021-02-08T17:00:13.183792+0100","flow_id":1890654805068045,"pcap_cnt":4153,"event_type":"alert","src_ip":"8.208.10.147","src_port":80,"dest_ip":"10.2.8.101","dest_port":49757,"proto":"TCP","metadata":{"flowbits":["exe.no.referer"]},"alert":{"action":"allowed","gid":1,"signature_id":2014819,"rev":3,"signature":"ET INFO Packed Executable Download","category":"Misc activity","severity":3,"metadata":{"created_at":["2012_05_30"],"updated_at":["2012_05_30"]}},"http":{},"app_proto":"http","flow":{"pkts_toserver":9,"pkts_toclient":11,"bytes_toserver":1042,"bytes_toclient":9759,"start":"2021-02-08T17:00:12.297229+0100"}}
{"timestamp":"2021-02-08T17:00:13.408464+0100","flow_id":1890654805068045,"pcap_cnt":4208,"event_type":"alert","src_ip":"8.208.10.147","src_port":80,"dest_ip":"10.2.8.101","dest_port":49757,"proto":"TCP","metadata":{"flowbits":["exe.no.referer","ET.http.binary"]},"tx_id":2,"alert":{"action":"allowed","gid":1,"signature_id":2018959,"rev":4,"signature":"ET POLICY PE EXE or DLL Windows file download HTTP","category":"Potential Corporate Privacy Violation","severity":1,"metadata":{"created_at":["2014_08_19"],"former_category":["POLICY"],"updated_at":["2017_02_01"]}},"http":{"hostname":"roanokemortgages[.]com","url":"/6lhjgfdghj.exe","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko","http_content_type":"application/octet-stream","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":41029},"files":[{"filename":"/6lhjgfdghj.exe","sid":[],"gaps":false,"state":"UNKNOWN","stored":false,"size":41029,"tx_id":2}],"app_proto":"http","flow":"pkts_toserver":24,"pkts_toclient":37,"bytes_toserver":1852,"bytes_toclient":46939,"start":"2021-02-08T17:00:12.297229+0100"}}

And Suricata’s Network Security Monitoring (NSM)

Below are a few of the network security and protocol logs produced by Suricata associated with the alert shown above. Note they are all tied together by a common flow_id value (highlighted in blue):

HTTP Log:
{"timestamp":"2021-02-08T17:00:12.648463+0100","flow_id":1890654805068045,"pcap_cnt":3883,"event_type":"http","src_ip":"10.2.8.101","src_port":49757,"dest_ip":"8.208.10.147","dest_port":80,"proto":"TCP","tx_id":0,"http":{"hostname":"roanokemortgages[.]com","url":"/0801.bin","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko","http_content_type":"application/octet-stream","accept":"*/*","cache_control":"no-cache","connection":"keep-alive","content_length":"876","content_type":"application/octet-stream","date":"Mon, 08 Feb 2021 16:00:13 GMT","last_modified":"Mon, 08 Feb 2021 13:20:38 GMT","server":"nginx","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":876,"request_headers":[{"name":"Accept","value":"*/*"},{"name":"User-Agent","value":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko"},{"name":"Host","value":"roanokemortgages[.]com"},{"name":"Cache-Control","value":"no-cache"}],"response_headers":[{"name":"Server","value":"nginx"},{"name":"Date","value":"Mon, 08 Feb 2021 16:00:13 GMT"},{"name":"Content-Type","value":"application/octet-stream"},{"name":"Content-Length","value":"876"},{"name":"Connection","value":"keep-alive"},{"name":"Last-Modified","value":"Mon, 08 Feb 2021 13:20:38 GMT"},{"name":"ETag","value":"\"60213aa6-36c\""},{"name":"Accept-Ranges","value":"bytes"}]}}

File Transaction log:
{"timestamp":"2021-02-08T17:00:12.648463+0100","flow_id":1890654805068045,"pcap_cnt":3883,"event_type":"fileinfo","src_ip":"8.208.10.147","src_port":80,"dest_ip":"10.2.8.101","dest_port":49757,"proto":"TCP","http":{"hostname":"roanokemortgages[.]com","url":"/0801.bin","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko","http_content_type":"application/octet-stream","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":876},"app_proto":"http","fileinfo":{"filename":"/0801.bin","sid":[],"magic":"data","gaps":false,"state":"CLOSED","sha256":"ee33a8fa2ae6f6b9366c97ed4c00c2796d98a371249dca725a01aca03caf747b","stored":false,"size":876,"tx_id":0}}

HTTP log:
{"timestamp":"2021-02-08T17:00:12.878417+0100","flow_id":1890654805068045,"pcap_cnt":3908,"event_type":"http","src_ip":"10.2.8.101","src_port":49757,"dest_ip":"8.208.10.147","dest_port":80,"proto":"TCP","tx_id":1,"http":{"hostname":"roanokemortgages[.]com","url":"/0801s.bin","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko","http_content_type":"application/octet-stream","accept":"*/*","cache_control":"no-cache","connection":"keep-alive","content_length":"913","content_type":"application/octet-stream","date":"Mon, 08 Feb 2021 16:00:13 GMT","last_modified":"Mon, 08 Feb 2021 13:20:37 GMT","server":"nginx","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":913,"request_headers":[{"name":"Accept","value":"*/*"},{"name":"User-Agent","value":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko"},{"name":"Host","value":"roanokemortgages[.]com"},{"name":"Cache-Control","value":"no-cache"}],"response_headers":[{"name":"Server","value":"nginx"},{"name":"Date","value":"Mon, 08 Feb 2021 16:00:13 GMT"},{"name":"Content-Type","value":"application/octet-stream"},{"name":"Content-Length","value":"913"},{"name":"Connection","value":"keep-alive"},{"name":"Last-Modified","value":"Mon, 08 Feb 2021 13:20:37 GMT"},{"name":"ETag","value":"\"60213aa5-391\""},{"name":"Accept-Ranges","value":"bytes"}]}}

File Transaction log:
{"timestamp":"2021-02-08T17:00:12.878417+0100","flow_id":1890654805068045,"pcap_cnt":3908,"event_type":"fileinfo","src_ip":"8.208.10.147","src_port":80,"dest_ip":"10.2.8.101","dest_port":49757,"proto":"TCP","http":{"hostname":"roanokemortgages[.]com","url":"/0801s.bin","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko","http_content_type":"application/octet-stream","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":913},"app_proto":"http","fileinfo":{"filename":"/0801s.bin","sid":[],"magic":"data","gaps":false,"state":"CLOSED","sha256":"7af0dc117d2dcd112f50889c4c8a14ac9ee55c2525a24fa66ff9a89b480b7e99","stored":false,"size":913,"tx_id":1}}

HTTP log:
{"timestamp":"2021-02-08T17:00:13.699055+0100","flow_id":1890654805068045,"pcap_cnt":4506,"event_type":"http","src_ip":"10.2.8.101","src_port":49757,"dest_ip":"8.208.10.147","dest_port":80,"proto":"TCP","metadata":{"flowbits":["exe.no.referer","ET.http.binary"]},"tx_id":2,"http":{"hostname":"roanokemortgages[.]com","url":"/6lhjgfdghj.exe","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko","http_content_type":"application/octet-stream","accept":"*/*","cache_control":"no-cache","connection":"keep-alive","content_length":"273422","content_type":"application/octet-stream","date":"Mon, 08 Feb 2021 16:00:14 GMT","last_modified":"Wed, 20 Jan 2021 09:59:19 GMT","server":"nginx","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":273422,"request_headers":[{"name":"Accept","value":"*/*"},{"name":"User-Agent","value":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko"},{"name":"Host","value":"roanokemortgages[.]com"},{"name":"Cache-Control","value":"no-cache"}],"response_headers":[{"name":"Server","value":"nginx"},{"name":"Date","value":"Mon, 08 Feb 2021 16:00:14 GMT"},{"name":"Content-Type","value":"application/octet-stream"},{"name":"Content-Length","value":"273422"},{"name":"Connection","value":"keep-alive"},{"name":"Last-Modified","value":"Wed, 20 Jan 2021 09:59:19 GMT"},{"name":"ETag","value":"\"6007fef7-42c0e\""},{"name":"Accept-Ranges","value":"bytes"}]}}

File Transaction log:
{"timestamp":"2021-02-08T17:00:13.699055+0100","flow_id":1890654805068045,"pcap_cnt":4506,"event_type":"fileinfo","src_ip":"8.208.10.147","src_port":80,"dest_ip":"10.2.8.101","dest_port":49757,"proto":"TCP","metadata":{"flowbits":["exe.no.referer","ET.http.binary"]},"http":{"hostname":"roanokemortgages[.]com","url":"/6lhjgfdghj.exe","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko","http_content_type":"application/octet-stream","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":273422},"app_proto":"http","fileinfo":{"filename":"/6lhjgfdghj.exe","sid":[],"magic":"PE32 executable (GUI) Intel 80386 (stripped to external PDB), for MS Windows","gaps":false,"state":"CLOSED","sha256":"94e60de577c84625da69f785ffe7e24c889bfa6923dc7b017c21e8a313e4e8e1","stored":false,"size":273422,"tx_id":2}}

Flow record:
{"timestamp":"2021-02-08T16:58:15.247118+0100","flow_id":1890654805068045,"event_type":"flow","src_ip":"10.2.8.101","src_port":49757,"dest_ip":"8.208.10.147","dest_port":80,"proto":"TCP","app_proto":"http","flow":{"pkts_toserver":105,"pkts_toclient":207,"bytes_toserver":6226,"bytes_toclient":287136,"start":"2021-02-08T17:00:12.297229+0100","end":"2021-02-08T17:01:56.764859+0100","age":104,"state":"closed","reason":"shutdown","alerted":true},"metadata":{"flowbits":["exe.no.referer","ET.http.binary"]},"tcp":{"tcp_flags":"1b","tcp_flags_ts":"1b","tcp_flags_tc":"1b","syn":true,"fin":true,"psh":true,"ack":true,"state":"closed"}}

Suricata Myth 2 - NSM requires signatures/rules

Myth: Suricata can only function with signatures/rules.

Fact: This is also not true. And it could be one of the most misunderstood capabilities of Suricata.

Suricata actually produces network security monitoring logs - protocol transactions and network flow data without signatures. In fact, if you were to switch off the rules/signatures in the example above, Suricata will still produce the exact same logs as above without the alert itself. That is, everything else will still be logged.

Here’s an example on Pastebin here:
https://pastebin.com/PxKDx2NF

In the next article, we will provide a more detailed review of the NSM capabilities of Suricata.

Suricata Myth Busting: Alerts and NSM

Data used in this article

Suricata Myth 1 - Alerts Only

Classic IDS alert

The Suricata alert difference

And Suricata’s Network Security Monitoring (NSM)

Suricata Myth 2 - NSM requires signatures/rules

Further Reading on Suricata

Peter Manev

Schedule a Demo of Stamus Security Platform

Suricata Myth Busting: Alerts and NSM

Data used in this article

Suricata Myth 1 - Alerts Only

Classic IDS alert

The Suricata alert difference

And Suricata’s Network Security Monitoring (NSM)

Suricata Myth 2 - NSM requires signatures/rules

Further Reading on Suricata

Peter Manev

Schedule a Demo of Stamus Security Platform

Related posts

Suricata Threat Hunting Fundamentals

Threat Hunting with Suricata and Newly-Registered Domain Threat Intel (Open NRD) Part 4

Threat Hunting with Suricata and Newly-Registered Domain Threat Intel (Open NRD) Part 3