I Added ML Anamoly Detection to My Cybersecurity Tool - Here's What the Numbers Actually Showed

A few weeks ago, I built Log Sentinel, a log analysis dashboard that detects brute force attacks, directory scanning, and error spikes in Apache server logs using rule-based detectors. The rules worked well, but I kept wondering: what would a machine learning model catch that my rules missed? And more importantly, would ML actually be better? I spent two weeks adding an Isolation Forest model to find out. The results were more interesting than I expected.

What I built

The ML upgrade adds a second detection layer on top of the existing rule-based system. Instead of checking specific conditions like "did this IP make 10+ failed logins?", the Isolation Forest learns what normal traffic looks like from the data itself and flags anything statistically unusual.

Before the model can run, I transform each IP address into a 5-feature behavioural vector:

• request_count - total requests made
• error_rate - fraction of requests returning 4xx or 5xx errors
• unique_paths - how many different URLs they visited
• night_traffic_ratio - fraction of requests made between midnight and 6 am
• avg_bytes - average response size

All five features get normalized to mean=0, std=1 using StandardScaler. Without normalization, an IP with 1000 requests would dominate an IP with 10 requests purely because of scale.

The model is configured with contamination=0.15; I tuned this by testing values from 0.05 to 0.50 and picking the one with the best F1 score on a labelled evaluation dataset.

The evaluation setup

To measure performance properly, I generated a labelled dataset of 230 IPs: 200 normal and 30 attackers (10 brute force, 10 directory scanners, 10 error spikers). Each IP has a ground truth label so I can compute precision, recall, and F1 for both detectors.

Printing accuracy on training data proves nothing. You need labelled ground truth to know if your model actually works.

The results

The rule-based detectors achieved perfect scores. The Isolation Forest caught every single attacker (recall=1.0) but flagged 5 innocent IPs along the way (precision=0.857).

What this actually means

My first instinct was to be disappointed that ML didn't win. But the more I thought about it, the more interesting the result became.

The rule-based detectors achieved perfect scores because the evaluation dataset contains well-defined, structured attack patterns, exactly the patterns the rules were written to catch. Of course they're perfect: I designed both the rules and the data.

The Isolation Forest's false positives tell a different story. It flagged 5 normal IPs that happened to have slightly unusual behaviour; maybe they made a burst of requests, or visited an uncommon path. The model found something statistically interesting about them even though they weren't attackers.

In a real production environment, this is actually the more valuable behaviour. Rules only catch what you anticipated. ML catches things you didn't think to write rules for.

The real insight is that both detectors belong in the same system. Rules handle known attack signatures with zero false alarms. ML handles the unknown. My dashboard now runs both and shows you exactly where they agree and where they disagree, which is where the interesting cases live.

What I'd do differently

The biggest limitation of my evaluation is that I generated the labelled dataset myself. Real attack patterns are messier and more diverse than synthetic data. A more honest evaluation would use real labelled log data from a production environment.

I'd also experiment with other unsupervised algorithms; Local Outlier Factor and One-Class SVM are worth comparing against Isolation Forest on the same dataset.

The live dashboard

Both detectors run on every analysis. The dashboard has three views:

• rule-based results
• ML results
• a comparison showing which IPs were caught by both, rules only, or ML only

There's also a Model Performance section showing the confusion matrices and F1 scores side by side.

Live demo: atenahfr.github.io/log-sentinel/frontend/index.html
Source code: github.com/atenahfr/log-sentinel