Monitoring, or 'Why is the internet constantly asking for my .php files?

A Grafana dashboard sounds like an enterprise thing, far too much hassle for a private homelab. I used to think so too. Terms like Prometheus, Exporter, Loki, and Promtail sound like a lot of overhead and a steep learning curve. But I started playing around with it for fun, and thanks to Docker the whole thing was up and running faster than you can spell "monitoring". This is how I ended up monitoring my setup and turning away bots at the front door. What do I actually want to know? You might remember my setup from the article about Cloudflare Tunnels and Docker Stacks. I want visibility into my Nginx instances, the Proxmox host, the Ubuntu server (including SSH logs), and my Docker containers. That's a fair number of endpoints, but there's a matching exporter for almost all of it: - pve-exporter fetches metrics directly from the Proxmox API. - node-exporter is the classic for Linux host metrics (CPU, RAM, disk). - cAdvisor looks inside Docker containers and cgroups. - promtail is the log vacuum that ships everything to Loki. The backend is Loki for logs and Prometheus for time-series metrics. It looks like this: graph LR classDef grafana fill:#F47A20,stroke:#fff,stroke-width:2px,color:#fff; classDef prometheus fill:#E6522C,stroke:#fff,stroke-width:2px,color:#fff; classDef loki fill:#000,stroke:#fff,stroke-width:2px,color:#fff; classDef exporter fill:#00a3cc,stroke:#fff,stroke-width:2px,color:#fff; classDef source fill:#444,stroke:#fff,stroke-width:1px,color:#fff,stroke-dasharray: 5 5; subgraph DataSources ["🔍 Data Sources"] LogFiles["Logfiles (Journal, Fail2ban, Docker Socket)"]:::source HostOS["Host OS (/proc, /sys, rootfs)"]:::source DockerEngine["Docker Engine (Containers, cgroups)"]:::source ProxmoxAPI["Proxmox VE (API)"]:::source end subgraph Agents ["🤖 Agents & Exporters"] Promtail["Promtail"]:::exporter NodeExporter["Node-Exporter"]:::exporter cAdvisor["cAdvisor"]:::exporter PVEExporter["PVE-Exporter"]:::exporter end subgraph Databases ["💾 Storage Databases"] Loki["Loki (Log-Aggregation)"]:::loki Prometheus["Prometheus (Time-Series Metrics)"]:::prometheus end Grafana["Grafana (Dashboards)"]:::grafana LogFiles -.-> Promtail Promtail --> Loki HostOS -.-> NodeExporter DockerEngine -.-> cAdvisor ProxmoxAPI -.-> PVEExporter Prometheus --> NodeExporter Prometheus --> cAdvisor Prometheus --> PVEExporter Grafana --> Loki Grafana --> Prometheus Under Docker the configuration is mostly a copy-paste job. Only the Proxmox exporter needs manual work, since you have to create an API token with the right permissions in the PVE interface. Dashboards At grafana.com/dashboards there's a large selection of community dashboards you can import by ID. My favorites: - Node Exporter Full: everything about the host. - Nginx Web Analytics: who's coming from where? - cAdvisor Exporter: what are my containers eating? - SSH Logs: who is trying to log in via SSH? - CrowdSec Metrics: who's already been kicked out? Provisioning runs from YAML files, so everything comes back instantly after a fresh install. The details are in my GitHub repo. First discoveries: hello, botnet As soon as the first Nginx logs showed up in Grafana, there was a lot more traffic than I expected. A second look made it clear those weren't readers, they were bots. I was getting hammered with 404s for paths like /tool.php , /admin-footer.php , or /abc.php . My sites are static HTML built with Astro, so there is nothing here for a PHP exploit to find, but the bots keep trying every second anyway. Log excerpt (anonymized): request for /tool.php with HTTP status: 404 from 172.22.0.8 located in IT request for /jga.php with HTTP status: 404 from 172.22.0.8 located in IT The fix: Traefik and CrowdSec I thought Cloudflare's free plan would catch all of this. It doesn't, plenty of these scans still get through. And since Traefik sits behind the Cloudflare proxy, it normally only sees the Cloudflare IP, so a ban would block the proxy itself. The answer is CrowdSec, the modern successor to fail2ban, plus a Traefik plugin. The key is the label forwardedHeadersCustomName=CF-Connecting-IP , which tells the CrowdSec plugin to read the real visitor IP from the Cloudflare header. Here is my docker-compose.yml snippet for the two of them: version: "3.8" services: crowdsec: image: crowdsecurity/crowdsec:latest container_name: crowdsec restart: unless-stopped # Entrypoint handles the bouncer key from secrets entrypoint: ["/bin/sh", "-c", "export BOUNCER_KEY_traefik=$(cat /run/secrets/crowdsec_bouncer_key) && exec /docker_start.sh"] environment: COLLECTIONS: "crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/base-http-scenarios" volumes: - crowdsec-config:/etc/crowdsec - crowdsec-data:/var/lib/crowdsec/data - traefik-access-logs:/var/log/traefik:ro networks: - apps - monitoring_net secrets: - crowdsec_bouncer_key traefik: image: traefik:v3.6.6 container_name: traefik restart: unless-stopped depends_on: - crowdsec command: - "--accesslog=true" - "--accesslog.filepath=/var/log/traefik/access.log" - "--experimental.plugins.crowdsec-bouncer.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin" - "--experimental.plugins.crowdsec-bouncer.version=v1.4.1" labels: - "traefik.enable=true" - "traefik.http.middlewares.crowdsec.plugin.crowdsec-bouncer.crowdseclapihost=crowdsec:8080" - "traefik.http.middlewares.crowdsec.plugin.crowdsec-bouncer.forwardedHeadersCustomName=CF-Connecting-IP" networks: - cloudflare - apps The middleware in action From here I just add the middleware label to every web service (portfolio, blog, and so on). Every request gets checked against the CrowdSec database, and if an IP is flagged as malicious, Traefik blocks it immediately. blog-astro-web-prd: image: nginx:alpine labels: - "traefik.enable=true" - "traefik.http.routers.blog-astro-prd.rule=Host(www.slashgordon.link)" - "traefik.http.routers.blog-astro-prd.middlewares=crowdsec@docker" Is it worth the effort? For me, yes. It's reassuring to open Grafana and watch the banned-IPs count climb while the CPU load of my 35-watt roommate stays flat. The dashboard also makes the effect obvious: as soon as a bot starts scanning my .php paths, it's blocked after a few attempts. This post was originally published on www.slashgordon.link. Top comments (0)