Wellcome to Clickdetect blog.
Many SIEM platforms invest significant resources into building functional detection engines. I developed this project independently in my free time. Clickdetect is designed to be a practical, agnostic, and versatile engine for log detection and alerting.
I will update this blog with interesting information about new updates, fixes and usage of clickdetect.
Feel free to update and contrib with clickdetect!
alertmanager webhook integration by @souzomainalertmanager documentation by @souzomainlock to permit multiples instances by @souzomainmax_detector_time option to cancel large time queries by @souzomainredis dependency by @souzomain
Hey, do you like DuckDB and want to generate security alerts using it? I have a solution for you.
Recently I started studying Delta Lake from Databricks, and I started looking for SQL engines to search logs in the data lake.
During my study I reached some conclusions: the concept of a data lake is not explored much right now, so I'm here to explain why this is a good option for a SIEM database and how you can generate security alerts with it.
This blog is not meant to teach you how to create a data lake with security data, but you can do this with Wazuh... I did it ;)
So, Clickdetect is a tool that I'm developing with many data source integrations, and I recently created the DuckDB data source.
DuckDB can integrate with many data sources too, so in this blog post I will show you why and how you can use DuckDB to generate security alerts using Delta Lake.
I have some thoughts that make me consider this advanced.
DuckDB is a SQL engine; right now it is not a database. You don't need to deploy or create large servers to use DuckDB.
You can think of DuckDB as a connector for different data sources you have, like PostgreSQL.
Why did I choose DuckDB for this blog post? Because it is lightweight. With Clickdetect you don't need to deploy a large server to run and get security analytics data.
If you want to deploy large servers, you may need to think of another option, like ClickHouse. ClickHouse can search large amounts of data in S3 using a cluster, which is the right option for large environments.
Delta Lake is a storage framework on steroids. It's not a file format.
Delta Lake was created by Databricks and is governed by the Linux Foundation. You can store terabytes/petabytes in S3 storage and use Delta Lake as the framework to do that.
You can use Delta Lake to store petabytes in S3 storage and still have good search performance.
Delta Lake stores metadata about the files, ensuring file skipping. You can use partitions to store logs in logically separated directories for better performance, and many other things.
So, let's get hands on. We will deploy Clickdetect with the DuckDB data source and use DuckDB's delta_scan function to search on Delta Lake.
In this example, I will use my local Delta Lake on my disk, but you can use an S3 database too.
First of all, you need to create your rule. Clickdetect rules are simple; if you are familiar with Sigma rules, this will be easy.
First, create your rule directory.
Create your rule inside the rules/ directory; in this example it will be rule.yml.
As the example for this post, I will use DuckDB's delta_scan function. Here I filter on Windows Event ID 4720 (A user account was created).
id: "3f2b1c4a-9d7e-4a1b-8c2d-5e6f7a8b9c0d"
name: "Windows admin created"
level: 5
size: ">0" # trigger if the query returns more than 0 rows
active: true
author:
- Vinicius Morais <[email protected]>
rule: |- # this is the rule data
SELECT * FROM delta_scan("tmp/test") where timestamp >= now() - interval 5 minute and win.data.eventID = '4720'
The size: ">0" field is what decides when the alert fires: if the query returns more than 0 rows within the interval, the rule triggers. You can use other comparisons (for example ">10") to alert only above a threshold.
Now you have your first DuckDB rule!
Note
delta_scan requires DuckDB's delta extension. DuckDB auto-installs and loads official extensions on first use when it has internet access. If you run in an offline/air-gapped environment, install it beforehand with INSTALL delta; LOAD delta;.
Now you need to define the runner.
The runner is the main configuration that defines: scheduler, rule path, data source, plugins and webhooks.
Go back to the parent directory and create runner.yml outside the rules directory.
This is the example runner.yml
datasource:
type: duckdb
database: ":memory:"
webhooks:
generic_webhook:
type: generic
url: http://localhost:3000/alerts/create/clickdetect
headers:
X-Type: test
detectors:
my_detector_1:
name: "5m interval"
for: "5m"
description: "detect rules with 5 min interval"
rules:
- "/app/rules/*.yml"
webhooks:
- generic_webhook
sigma: false
The easiest way is using Docker; you just need to deploy the latest package.
docker run -v ./rules/:/app/rules/ -v ./runner.yml:/app/runner.yml -v ./tmp/test/:/app/tmp/test/ -p 8080:8080 ghcr.io/clicksiem/clickdetect:latest --api -p 8080 -r /app/runner.yml
The extra volume (./tmp/test/) mounts your local Delta Lake into the container so the delta_scan("tmp/test") in the rule can reach it. If your data lake lives in S3 instead, drop this volume and point delta_scan at your s3:// path.
With this, you now have a fully functional Clickdetect running your rule.
This is the result of running the rule with the DuckDB search in the Delta Lake repository.
As you can see, Clickdetect can search data using DuckDB, and based on the rule match the rule will trigger and send the alert to the webhook.
You can use Clickdetect with an AI agent to automatically analyze your alerts and generate context.
This is the configuration for runner.yml
This is the result from my last post.
I did the same in my last blog post about OpenSearch PPL.
With this, you can use Clickdetect as the alerting engine for your data lake and DuckDB as the SQL engine for it.
My Social * Github: https://github.com/souzomain * E-mail: [email protected] * Matrix: @souzo:matrix.org * Twitter/X: https://x.com/souzomain * Linkedin: https://www.linkedin.com/in/vinicius-m-a76ba51b5/ * Reddit: https://www.reddit.com/user/_souzo/
--dry-run option to test in cicd by @souzomainHey, souzo here. If you've ever wanted alerting rules that actually work in Wazuh without fighting OpenSearch's detection engine, this post is for you.
Repository: https://github.com/clicksiem/clickdetect
In this blog post I will guide you to: - Install and configure Opensearch PPL in an existing Wazuh environment - Install and configure clickdetect - Write Opensearch PPL - Write Sigma rules with Opensearch PPL - Detect threats with your Wazuh data extending wazuh detetion capabilities
After working many years with wazuh and opensearch, I wanted some features that currently not exists or are so broken to work with.
OpenSearch has been working to transform its product into a complete SIEM with a detection engine, however... it's VERY buggy. I tested it several times with real data and always ended up with a corrupted index.
I looked into Elastalert, but I didn't like its engine; I found the code and maintenance too confusing. Also, why create a rules system when I can use the datasource's own language? So instead of forking, I created my own solution.
I created ClickDetect to help security teams around the world have an additional tool for generating alerts.
Clickdetect is an alerting system tool created to help you to create your detection strategy in whatever datasource you want.
Clickdetect has many datasources supported like: - Clickhouse (+sigma) - Opensearch + Opensearch PPL (+sigma) - Elasticsearch - Victorialogs - PostgreSQL - Loki (+sigma) - Databricks
Clickdetect is multi-tenant by default, you can specify tenant in rules too.
Clickdetect v1.4.0 actually supports sigma backend. Check out the documentation https://clickdetect.souzo.me/sigma/
I created ClickDetect to work primarily with ClickHouse because, in my opinion, it's a better alternative, as it allows for magnificent data compression, which directly impacts the price of SOC services. Furthermore, wazuh's data is structured, so the ClickHouse JSON column makes more sense (My opinion).
But in this post I will show that it's possible to use ClickDetect with OpenSearch.
Opensearchppl is used to search, filter, and analyze data in an easy and intuitive way. It's very similar to the LogQL language of Loki or Splunk SPL.
This greatly increases the possibility of turning OpenSearch into a SIEM instead of running queries in the standard DSL format.
If you want SQL, try clickhouse or postgresql of tigerdata. PPL makes more sense for Opensearch environment.
Here's how all the pieces fit together:
Wazuh ships events into the OpenSearch indexer. Clickdetect queries the indexer using PPL (or compiled Sigma rules), evaluates the configured rules on a schedule, and fires alerts to your webhooks when a condition is met.
Following the documentation, we first need to install SQL plugin https://docs.opensearch.org/latest/sql-and-ppl/ppl/index/.
Your openseach configurations and binaries are in the directory /usr/share/wazuh-indexer/. Change your directory
First, verify if your opensearch does not have Opensearch SQL Plugin installed. The plugin usually comes pre-installed with Wazuh.
If the plugin is not installed, install it.
Restart your wazuh indexer
Now It's fully operational.
mkdir -p rules/
cat << EOF > rules/manager_started.yml
id: $(cat /proc/sys/kernel/random/uuid)
name: "Wazuh opensearch sigma test - Manager Started"
level: 10
size: ">0"
active: true
author:
- Vinicius Morais <[email protected]>
group: base_rule
tags:
- base
rule: |-
search source=wazuh-alerts-* | where rule.id='502' and `@timestamp` >= DATE_SUB(NOW(), INTERVAL 5 HOUR )
EOF
You need to configure a runner, runner is a file that configure the schedulers, webhooks and the datasource.
For this example, we will use "teams" as the webhook and configure the detector to run every 5 minute.
cat << EOF > runner.yml
datasource:
type: opensearch-ppl
url: https://127.0.0.1:9200
username: wazuh
password: wazuh-adm
verify: false
webhooks:
teams_alert:
type: teams
url: https://<your_companie>.webhook.office.com/...
timeout: 10
verify: false
detectors:
my_detector:
name: "5m interval"
for: "5m"
description: "detect rules with 5 min interval"
rules:
- "/app/rules/*"
webhooks:
- teams_alert
Now you can run with docker.
docker run -v ./runner.yml:/app/runner.yml -v ./rules/:/app/rules/ ghcr.io/clicksiem/clickdetect:latest
Running
Clickdetect starts up, loads the rule from rules/, and schedules the detector to run every 5 minutes.
Results in terminal
The terminal output shows the rule matched at least one event. Each match includes the rule name, level, and the raw document returned by the PPL query.
Results in teams
The Teams webhook delivers the alert card with the rule name, severity level, and a summary of the matched events.
This is an extra to you understand the potential.
You can use sigma rules with clickdetect, check out the documentation. https://clickdetect.souzo.me/sigma/
WARNING: opensearch PPL sigma backend I'ts aligned with the latest opensearch version, "earliest" and "latest" I'ts not recorinized in wazuh indexer 2.19
Let's create the sigma rule directory and save the sigma rule.
Save your rule inside the created directory.
cat << EOF > rules/sigma/manager_started.yml
title: "wazuh opensearch sigma test - Manager Started"
status: test
id: 32bc608c-67ab-4d58-8361-35d3baac726c
logsource:
product: wazuh
category: indexer
detection:
sel:
rule.id: '502'
condition: 1 of sel
custom:
opensearch_ppl_min_time: "-5m"
opensearch_ppl_max_time: "now"
EOF
Change the directory of rule in you runner
cat << EOF > runner.yml
datasource:
type: opensearch-ppl
url: https://127.0.0.1:9200
username: wazuh
password: wazuh-adm
verify: false
webhooks:
teams_alert:
type: teams
url: https://<your_companie>.webhook.office.com/...
timeout: 10
verify: false
detectors:
my_sigma_detector:
name: "5m interval"
for: "5m"
description: "detect sigma rule with 5 min interval"
rules:
- "/app/rules/sigma/*"
sigma: true
webhooks:
- teams_alert
Now you can run.
docker run -v ./runner.yml:/app/runner.yml -v ./rules/:/app/rules/ ghcr.io/clicksiem/clickdetect:latest
Clickdetect + AI SOC Agent. You can specify an AI to auto analyze and generate score using clickdetect agentic plugin.
Example with deepseek.
This is the result of clickagentic with teams webhook.
Check out the documentation: https://clickdetect.souzo.me/plugin/clickagentic/
With this blog post now you can: - Run Sigma rules inside your wazuh environment - Perform multi tenant search and cross site search - Perform correlation between different logs sources - Improve your SOC team.
Clickdetect is not affiliated to ClickHouse, and I'm not sponsored (yet).
If this post helped you, consider giving clickdetect a star on GitHub — it helps the project reach more security teams. :star:
Follow my social: * E-mail: [email protected] * Matrix: @souzo:matrix.org - Twitter/X: https://x.com/souzomain - Linkedin: https://www.linkedin.com/in/vinicius-m-a76ba51b5/ - Reddit: https://www.reddit.com/user/_souzo/