<img height="1" width="1" style="display:none;" alt="" src="https://px.ads.linkedin.com/collect/?pid=2180921&amp;fmt=gif">

Suricata Myth Busting: Alerts and NSM

As I mentioned in the introductory article in this series (see here >>), Suricata produces not only IDS alerts but also produces protocol transaction logs, flow records, full packet capture and extracted files.

In this article we attempt to clear up a couple of misconceptions (myths) and illustrate these lesser-known capabilities through real-world examples.

Data used in this article

The example that we will use in this article to showcase Suricata’s capabilities is from https://www.malware-traffic-analysis.net/2021/02/08/index.html

NOTE: The PCAP and the artifacts extracted or present (files/domains etc) in this example contain live malware. If you chose to explore or visit any of the domains/servers/files/ listed here, you should do so at your own risk, in a controlled manner and only in a sandbox environment.


Suricata Myth 1 - Alerts Only

Myth: Suricata produces only alerts/alarms and you need another tool to produce the network protocol log evidence (network protocol transaction or network security logs) for that alert in order to confirm if it is real or a false positive.

Fact: This is simply not true.

Suricata produces all relevant network security monitoring logs: protocol, flow, file transaction, and anomaly logs, including the ones related to an alert - but also independent of alerts. In the regular JSON logs that Suricata generates (eve.json), you will find something called “flow_id” that correlates the network protocol data and evidence that Suricata has logged - to an alert event and that alert’s metadata. So for example for a HTTP based alert, all the relevant HTTP and or file transactions and flow logs would be available.

Suricata-Generated-Events v2 source
The ability to correlate any existing evidence/logs to an alert (“flow_id”) was introduced in 2014 by Suricata lead developer, Victor Julien

https://github.com/OISF/suricata/commit/f1185d051c210ca0daacdddbe865a51af24f4ea3

There are, of course, other and many more network security and protocol logs produced by Suricata before and after an alert occurred that are also accessible to the user.

Classic IDS alert

Here is an example of a legacy/classic IDS alert

Alert log:
02/08/2021-17:00:13.183792 [**] [1:2014819:3] ET INFO Packed Executable Download [**] [Classification: Misc activity] [Priority: 3] {TCP} 8.208.10.147:80 -> 10.2.8.101:49757

You can see that it does not contain much information beyond source and destination IP address. In short, it is distinctly lacking in context that might be useful for incident investigation or threat hunting.

The Suricata alert difference

The Suricata difference can be seen below. 

NOTE: for easier reading, check these out on Pastebin here: https://pastebin.com/vXF59n6u

Notice the areas highlighted below in blue which demonstrate the additional data and indicate there are flow and file records associated with the alert. 

Alert log:
{"timestamp":"2021-02-08T17:00:13.183792+0100","flow_id":1890654805068045,"pcap_cnt":4153,"event_type":"alert","src_ip":"8.208.10.147","src_port":80,"dest_ip":"10.2.8.101","dest_port":49757,"proto":"TCP","metadata":{"flowbits":["exe.no.referer"]},"alert":{"action":"allowed","gid":1,"signature_id":2014819,"rev":3,"signature":"ET INFO Packed Executable Download","category":"Misc activity","severity":3,"metadata":{"created_at":["2012_05_30"],"updated_at":["2012_05_30"]}},"http":{},"app_proto":"http","flow":{"pkts_toserver":9,"pkts_toclient":11,"bytes_toserver":1042,"bytes_toclient":9759,"start":"2021-02-08T17:00:12.297229+0100"}}
{"timestamp":"2021-02-08T17:00:13.408464+0100","flow_id":1890654805068045,"pcap_cnt":4208,"event_type":"alert","src_ip":"8.208.10.147","src_port":80,"dest_ip":"10.2.8.101","dest_port":49757,"proto":"TCP","metadata":{"flowbits":["exe.no.referer","ET.http.binary"]},"tx_id":2,"alert":{"action":"allowed","gid":1,"signature_id":2018959,"rev":4,"signature":"ET POLICY PE EXE or DLL Windows file download HTTP","category":"Potential Corporate Privacy Violation","severity":1,"metadata":{"created_at":["2014_08_19"],"former_category":["POLICY"],"updated_at":["2017_02_01"]}},"http":{"hostname":"roanokemortgages[.]com","url":"/6lhjgfdghj.exe","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko","http_content_type":"application/octet-stream","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":41029},"files":[{"filename":"/6lhjgfdghj.exe","sid":[],"gaps":false,"state":"UNKNOWN","stored":false,"size":41029,"tx_id":2}],"app_proto":"http","flow":"pkts_toserver":24,"pkts_toclient":37,"bytes_toserver":1852,"bytes_toclient":46939,"start":"2021-02-08T17:00:12.297229+0100"}}

And Suricata’s Network Security Monitoring (NSM)

Below are a few of the network security and protocol logs produced by Suricata associated with the alert shown above. Note they are all tied together by a common flow_id value (highlighted in blue):

HTTP Log:
{"timestamp":"2021-02-08T17:00:12.648463+0100","flow_id":1890654805068045,"pcap_cnt":3883,"event_type":"http","src_ip":"10.2.8.101","src_port":49757,"dest_ip":"8.208.10.147","dest_port":80,"proto":"TCP","tx_id":0,"http":{"hostname":"roanokemortgages[.]com","url":"/0801.bin","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko","http_content_type":"application/octet-stream","accept":"*/*","cache_control":"no-cache","connection":"keep-alive","content_length":"876","content_type":"application/octet-stream","date":"Mon, 08 Feb 2021 16:00:13 GMT","last_modified":"Mon, 08 Feb 2021 13:20:38 GMT","server":"nginx","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":876,"request_headers":[{"name":"Accept","value":"*/*"},{"name":"User-Agent","value":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko"},{"name":"Host","value":"roanokemortgages[.]com"},{"name":"Cache-Control","value":"no-cache"}],"response_headers":[{"name":"Server","value":"nginx"},{"name":"Date","value":"Mon, 08 Feb 2021 16:00:13 GMT"},{"name":"Content-Type","value":"application/octet-stream"},{"name":"Content-Length","value":"876"},{"name":"Connection","value":"keep-alive"},{"name":"Last-Modified","value":"Mon, 08 Feb 2021 13:20:38 GMT"},{"name":"ETag","value":"\"60213aa6-36c\""},{"name":"Accept-Ranges","value":"bytes"}]}}

File Transaction log:
{"timestamp":"2021-02-08T17:00:12.648463+0100","flow_id":1890654805068045,"pcap_cnt":3883,"event_type":"fileinfo","src_ip":"8.208.10.147","src_port":80,"dest_ip":"10.2.8.101","dest_port":49757,"proto":"TCP","http":{"hostname":"roanokemortgages[.]com","url":"/0801.bin","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko","http_content_type":"application/octet-stream","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":876},"app_proto":"http","fileinfo":{"filename":"/0801.bin","sid":[],"magic":"data","gaps":false,"state":"CLOSED","sha256":"ee33a8fa2ae6f6b9366c97ed4c00c2796d98a371249dca725a01aca03caf747b","stored":false,"size":876,"tx_id":0}}

HTTP log:
{"timestamp":"2021-02-08T17:00:12.878417+0100","flow_id":1890654805068045,"pcap_cnt":3908,"event_type":"http","src_ip":"10.2.8.101","src_port":49757,"dest_ip":"8.208.10.147","dest_port":80,"proto":"TCP","tx_id":1,"http":{"hostname":"roanokemortgages[.]com","url":"/0801s.bin","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko","http_content_type":"application/octet-stream","accept":"*/*","cache_control":"no-cache","connection":"keep-alive","content_length":"913","content_type":"application/octet-stream","date":"Mon, 08 Feb 2021 16:00:13 GMT","last_modified":"Mon, 08 Feb 2021 13:20:37 GMT","server":"nginx","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":913,"request_headers":[{"name":"Accept","value":"*/*"},{"name":"User-Agent","value":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko"},{"name":"Host","value":"roanokemortgages[.]com"},{"name":"Cache-Control","value":"no-cache"}],"response_headers":[{"name":"Server","value":"nginx"},{"name":"Date","value":"Mon, 08 Feb 2021 16:00:13 GMT"},{"name":"Content-Type","value":"application/octet-stream"},{"name":"Content-Length","value":"913"},{"name":"Connection","value":"keep-alive"},{"name":"Last-Modified","value":"Mon, 08 Feb 2021 13:20:37 GMT"},{"name":"ETag","value":"\"60213aa5-391\""},{"name":"Accept-Ranges","value":"bytes"}]}}

File Transaction log:
{"timestamp":"2021-02-08T17:00:12.878417+0100","flow_id":1890654805068045,"pcap_cnt":3908,"event_type":"fileinfo","src_ip":"8.208.10.147","src_port":80,"dest_ip":"10.2.8.101","dest_port":49757,"proto":"TCP","http":{"hostname":"roanokemortgages[.]com","url":"/0801s.bin","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko","http_content_type":"application/octet-stream","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":913},"app_proto":"http","fileinfo":{"filename":"/0801s.bin","sid":[],"magic":"data","gaps":false,"state":"CLOSED","sha256":"7af0dc117d2dcd112f50889c4c8a14ac9ee55c2525a24fa66ff9a89b480b7e99","stored":false,"size":913,"tx_id":1}}

HTTP log:
{"timestamp":"2021-02-08T17:00:13.699055+0100","flow_id":1890654805068045,"pcap_cnt":4506,"event_type":"http","src_ip":"10.2.8.101","src_port":49757,"dest_ip":"8.208.10.147","dest_port":80,"proto":"TCP","metadata":{"flowbits":["exe.no.referer","ET.http.binary"]},"tx_id":2,"http":{"hostname":"roanokemortgages[.]com","url":"/6lhjgfdghj.exe","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko","http_content_type":"application/octet-stream","accept":"*/*","cache_control":"no-cache","connection":"keep-alive","content_length":"273422","content_type":"application/octet-stream","date":"Mon, 08 Feb 2021 16:00:14 GMT","last_modified":"Wed, 20 Jan 2021 09:59:19 GMT","server":"nginx","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":273422,"request_headers":[{"name":"Accept","value":"*/*"},{"name":"User-Agent","value":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko"},{"name":"Host","value":"roanokemortgages[.]com"},{"name":"Cache-Control","value":"no-cache"}],"response_headers":[{"name":"Server","value":"nginx"},{"name":"Date","value":"Mon, 08 Feb 2021 16:00:14 GMT"},{"name":"Content-Type","value":"application/octet-stream"},{"name":"Content-Length","value":"273422"},{"name":"Connection","value":"keep-alive"},{"name":"Last-Modified","value":"Wed, 20 Jan 2021 09:59:19 GMT"},{"name":"ETag","value":"\"6007fef7-42c0e\""},{"name":"Accept-Ranges","value":"bytes"}]}}

File Transaction log:
{"timestamp":"2021-02-08T17:00:13.699055+0100","flow_id":1890654805068045,"pcap_cnt":4506,"event_type":"fileinfo","src_ip":"8.208.10.147","src_port":80,"dest_ip":"10.2.8.101","dest_port":49757,"proto":"TCP","metadata":{"flowbits":["exe.no.referer","ET.http.binary"]},"http":{"hostname":"roanokemortgages[.]com","url":"/6lhjgfdghj.exe","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko","http_content_type":"application/octet-stream","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":273422},"app_proto":"http","fileinfo":{"filename":"/6lhjgfdghj.exe","sid":[],"magic":"PE32 executable (GUI) Intel 80386 (stripped to external PDB), for MS Windows","gaps":false,"state":"CLOSED","sha256":"94e60de577c84625da69f785ffe7e24c889bfa6923dc7b017c21e8a313e4e8e1","stored":false,"size":273422,"tx_id":2}}

Flow record:
{"timestamp":"2021-02-08T16:58:15.247118+0100","flow_id":1890654805068045,"event_type":"flow","src_ip":"10.2.8.101","src_port":49757,"dest_ip":"8.208.10.147","dest_port":80,"proto":"TCP","app_proto":"http","flow":{"pkts_toserver":105,"pkts_toclient":207,"bytes_toserver":6226,"bytes_toclient":287136,"start":"2021-02-08T17:00:12.297229+0100","end":"2021-02-08T17:01:56.764859+0100","age":104,"state":"closed","reason":"shutdown","alerted":true},"metadata":{"flowbits":["exe.no.referer","ET.http.binary"]},"tcp":{"tcp_flags":"1b","tcp_flags_ts":"1b","tcp_flags_tc":"1b","syn":true,"fin":true,"psh":true,"ack":true,"state":"closed"}}

Suricata Myth 2 - NSM requires signatures/rules 

Myth: Suricata can only function with signatures/rules.

Fact: This is also not true. And it could be one of the most misunderstood capabilities of Suricata.

Suricata actually produces network security monitoring logs - protocol transactions and network flow data without signatures. In fact, if you were to switch off the rules/signatures in the example above, Suricata will still produce the exact same logs as above without the alert itself. That is, everything else will still be logged.

Here’s an example on Pastebin here:
https://pastebin.com/PxKDx2NF

In the next article, we will provide a more detailed review of the NSM capabilities of Suricata.

Further Reading on Suricata

Stamus Networks has produced a number of articles and papers on Suricata. Please see the list below for more information:

White Paper: Scaling Suricata for Enterprise Deployment
https://www.stamus-networks.com/en-us/landing/wp-scaling-suricata?hsLang=en

White Paper: Introduction to eBPF and XDP support in Suricata
https://www.stamus-networks.com/en-us/landing/introduction-to-ebpf-and-xdp-support-in-suricata?hsLang=en

Blog: Suricata - The First 12 Years of Innovation
https://www.stamus-networks.com/blog/suricata-the-first-12-years-of-innovation

Blog: The Other Side of Suricata
https://www.stamus-networks.com/blog/the-other-side-of-suricata

Blog: Advanced Deployment & Configuration with Suricata at SuriCon 2021
https://www.stamus-networks.com/blog/advanced-deployment-configuration-with-suricata-at-suricon-2021

Blog: Introducing the Stamus Networks App for Splunk®
https://www.stamus-networks.com/blog/introducing-the-stamus-networks-app-for-splunk

Blog: Just Released - Suricata 6
https://www.stamus-networks.com/hubfs/Library/Documents%20(PDFs)/StamusNetworks_TB-U37SURI-062021-2.pdf

Tech Brief: Supercharging Suricata Sensors with SSP
https://www.stamus-networks.com/hubfs/Library/Documents%20(PDFs)/StamusNetworks_TB-SUPERSURI-032022-2a.pdf

Solutions Page - Simplify Your Suricata Deployment
https://www.stamus-networks.com/simplifying-suricata

 

 

 

Peter Manev

Peter Manev is the co-founder and chief strategy officer (CSO) at Stamus Networks. He is a member of the executive team at Open Network Security Foundation (OISF). Peter has over 15 years of experience in the IT industry, including enterprise-level IT security practice. He is a passionate user, developer, and explorer of innovative open-source security software, and he is responsible for training as well as quality assurance and testing on the development team of Suricata – the open-source threat detection engine. Peter is a regular speaker and educator on open-source security, threat hunting, and network security at conferences and live-fire cyber exercises, such as Crossed Swords, DeepSec, Troopers, DefCon, RSA, Suricon, SharkFest, and others. Peter resides in Gothenburg, Sweden.

Schedule a Demo of Stamus Security Platform

REQUEST A DEMO

Related posts

Introducing U40: Powerful New Edition of Stamus Security Platform

Today, we are proud to announce the general availability of Update 40 (U40) - the latest release of...