<img height="1" width="1" style="display:none;" alt="" src="https://px.ads.linkedin.com/collect/?pid=2180921&amp;fmt=gif">

Streamlining Suricata Development with NixOS and Custom Build Targets

tl;dr

This article explores the integration of NixOS, a declarative Linux distribution, with Suricata, a high-performance network security engine, to create an efficient development environment. It details how NixOS's functional package manager simplifies dependency management for Suricata's C-based development, addressing the challenges of setting up a reproducible build environment. A key focus is on overcoming issues with Language Server Protocol (LSP) integration due to NixOS's shell wrappers for compilers, introducing a custom `make compile-commands` target within Suricata's autotools-based build system to generate the necessary `compile_commands.json` file. This innovative solution not only streamlines Suricata development on NixOS but also offers a generalizable technique for other C projects using autotools, promoting more reproducible and efficient software development workflows.

NixOS: A Declarative Linux Distribution

NixOS is a unique Linux distribution that stands out due to its declarative configuration model. Unlike traditional Linux distributions where system configuration is managed through imperative scripts and manual modifications, NixOS uses a purely functional package manager called Nix. This allows the entire system, including the kernel, applications, system services, and configuration files, to be described in a declarative language. This approach ensures that a NixOS system is highly reproducible, meaning that you can reliably rebuild the exact same system configuration on different machines. But being disruptive comes with some difficulties, and this was the case when trying to set up a working development environment for Suricata.

Suricata: A High-Performance Network IDS/IPS/NSM Engine

Suricata is a free and open-source, high-performance network intrusion detection system (IDS), intrusion prevention system (IPS), and network security monitoring (NSM) engine. It is widely used for real-time threat detection, protecting networks from various cyberattacks, and providing deep insights into network traffic. A key enabler of Suricata's high performance and efficiency lies in its development methodology. Suricata is primarily developed in C, a programming language known for its low-level memory management and direct hardware access.

Building the Environment

Suricata requires many dependencies to compile properly. And installing them manually is a pain. NixOS fixes that as you can have a shell.nix file that describes what are the needed packages and that can be activated as a user by simply running nix-shell in the directory. 

The shell.nix file has been added to the source and it basically list all components needed for the build and for the development:

let

  pkgs = import <nixpkgs> {};

 

in with pkgs;

  pkgs.mkShell {

    name = "suri-shell";

 

    buildInputs = [

      bash

      cargo

      rustc

      rust-cbindgen

      clang

      libclang.python

      libllvm

      automake

      autoconf

      autogen

      libtool

      pkg-config

      elfutils

      jansson

      ...

      libpcap

      libyaml

      lz4

      pcre2

      vectorscan

      zlib

    ];

}

 

For example, compiler rustc and clang are installed in the temporary environment when running nix-shell and do not need to be installed on the system. This is the same for  libclang.python that is here to get the Language Server Protocol clangd installed. 

 

Getting Language Server Protocol to work with autoconf

Suricata uses autotools for its build system. This is the oldest generation of build system, and we welcome contributions from the community to port it to a more modern build system, such as cmake.

To get the clangd Language Server Protocol work correctly, a compile_commands.json file is needed. In most distributions, running bear tool is a good way to obtain a valid file:

bear -- make

But this does not work on NixOS because it uses a shell wrapper for clang, and because of that, bear can not detect the correct option. This is a real issue because it causes clangd to generate excessive errors and is not usable.

To address that, there is a set of complex solutions that can be found. Complexity is caused by the fact that – without modifying the build system – it is complicated to generate the compile command file.

At Stamus Networks, we knew that the Suricata development team could accept a patch to generate the compile_commands.json without any tool, so we proposed modifications to address the issue. Once the modifications were in, we had a new make target to build the needed file.

make compile-commands

The change takes advantage of the fact that clang can generate individual entries for each build file if it is given the -MJ option. This option takes one parameter: the destination JSON file. In our case, we are using -MJ $$TMP_DIR/\$$*.json.

The target is defined like so:

compile-commands:                                                                                                                                                                                                                                                                                                    

       $(MAKE) -C src clean                                                                                                                                                    

       TMP_DIR=$$(mktemp -d --suffix=.suricata-compile-commands) && \                                                                                                          

       $(MAKE) EXTRA_CFLAGS="-MJ $$TMP_DIR/\$$*.json" && \                                                                                                                     

       echo '[' > $(top_builddir)/compile_commands.json && \                                                                                                                   

       cat $$TMP_DIR/*.json >> $(top_builddir)/compile_commands.json && \                                                                                                      

       echo ']' >> $(top_builddir)/compile_commands.json && \                                                                                                                  

       rm -rf $$TMP_DIR                                                                                                                                                        

 

We first perform a make clean in the C code of Suricata, then we create a temporary directory TMP_DIR. Then we run a make passing the EXTRA_CFLAGS that modifies clang options  to add the -MJ option triggering the creation of the intermediate files. Then we build the final compile_commands.json by creating a JSON object and then concatenating individual JSON entries. 

The commit introducing the EXTRA_FLAGS is https://github.com/OISF/suricata/commit/20371dbdf6a0f65beeeb791e4d22c5b595c88353

… and the one introducing the new Makefile target is https://github.com/OISF/suricata/commit/2a2f38ff88961b17b4f9a9b1c0cf52196cd48af6.

This technique is interesting as it can be used in any other software that uses autotools and the C programming language.

Conclusion

The integration of NixOS with Suricata development, as detailed in this article, showcases a robust solution for managing complex build environments. By leveraging NixOS's declarative nature and introducing a custom `make compile-commands` target within Suricata's build system, the challenges of dependency management and Language Server Protocol integration are effectively overcome. This innovative approach not only streamlines development for Suricata on NixOS but also provides a transferable technique applicable to other C-based projects utilizing autotools, paving the way for more reproducible and efficient software development workflows.

Eric Leblond

Éric Leblond is the co-founder and chief technology officer (CTO) at Stamus Networks. He sits on the board of directors at Open Network Security Foundation (OISF). Éric has more than 15 years of experience as co-founder and technologist of cybersecurity software companies and is an active member of the security and open-source communities. He has worked on the development of Suricata – the open-source network threat detection engine – since 2009 and is part of the Netfilter Core team, responsible for the Linux kernel's firewall layer. Eric is a respected expert and speaker on all things network security. Éric resides in Escalles, France.

Schedule a Demo of Clear NDR

REQUEST A DEMO

Related posts

Declarations of Compromise®: Cutting Through the Noise to Pinpoint Serious and Imminent Threats

Security teams are often overwhelmed by a flood of alerts, leading to alert fatigue and missed...

NEW with Suricata 8.0: Threat Context for IOC Matching

Indicators of compromise (IOCs) have become a key component of the threat detection practice....

Uncovered: Real-Time Policy Violation Detection in a Zero Trust Environment

Modern IT infrastructure, whether traditional or hybrid, faces persistent challenges: staff...