On-Premise Cyber Fortress: OPNsense High Availability, Graylog SIEM, Zabbix, and Sec-Sentinel Forensic AI

---

⚡ Key Defense Infrastructure Metrics

---

🗺️ Distributed Defense Architectural Blueprint

The architecture fuses hardware redundancy and autonomous software intelligence into a continuous stream: from the boundary FTTH gateway through isolated DMZ segments to local on-premise AI orchestrators.

---

1. The Perimeter Fortress: OPNsense High Availability Cluster

At the bedrock of the defensive infrastructure is the systematic elimination of every Single Point of Failure (SPOF). Traffic filtering is anchored across two high-availability firewall appliances built on FreeBSD 15.1 and OPNsense 26.7, arranged in a stateful active-passive cluster:

Dual-Node Topology and Real-Time State Synchronization

• Alpha Node (fw-core-01 – Preferred Master): advskew 1, WAN IP 172.16.100.11, LAN IP 10.10.1.11.
• Beta Node (fw-core-02 – Standby Backup): advskew 101, WAN IP 172.16.100.12, LAN IP 10.10.1.12.
• Dedicated Sync Backplane (vtnet2 / opt1): Isolated point-to-point sub-millisecond link (10.99.0.0/24) carrying two mission-critical telemetry channels:
  • pfsync: Instantly synchronizes every established connection in the kernel state table (syncok: 1). During failovers or rolling kernel maintenance, the backup node takes over transparently without dropping active downloads, VoIP calls, SSH terminals, or VPN tunnels.
  • XMLRPC Sync: Synchronizes rule changes, threat alias blocks, NAT tables, and TLS certificates from Master to Standby, ensuring 100% configuration symmetry.
• Kea DHCP High Availability: Inter-node lease replication over port 8111 with automated scheduled nightly synchronization of static MAC reservations.

CARP Virtual IPs and Advanced Outbound NAT

Physical nodes never expose individual interfaces directly to public peers, communicating via shared virtual abstractions (CARP Virtual IPs):

• CARP WAN VIP (172.16.100.254, vhid 1, vMAC 00:00:5e:00:01:01): DMZ Exposed Host destination configured on upstream edge CPE gateway.
• CARP LAN VIP (10.10.1.1, vhid 3, vMAC 00:00:5e:00:01:03): Default gateway and resilient DNS resolver for all internal workloads.
• Advanced Manual Outbound NAT: Internal subnets masquerade outbound traffic strictly via the CARP WAN VIP rather than node-specific interface IPs.
• High-Availability WireGuard: VPN daemons dynamically bind to CARP state events (carp_depend_on) with explicit omission of the reply-to flag, preventing asymmetric routing for roaming mobile clients.

---

2. Engineering Case Study: Resolving Physical MAC Leak for Sub-2-Second Failover

During failover chaos testing, an insidious routing bottleneck emerged, characteristic of FreeBSD network stacks fronted by consumer-grade ISP CPE hardware:

Root Cause of Inbound Routing Blackout

When failover occurred (Alpha degraded to Backup, Beta promoted to Master), outbound LAN masquerading continued smoothly, but all inbound exposed services (HTTP/HTTPS reverse proxies, WireGuard VPN) instantly blacked out.

• FreeBSD Physical MAC Leak: When generating Outbound NAT traffic, the FreeBSD kernel encapsulated egress Ethernet frames with the physical hardware MAC of the transmitting interface (Alpha: 02:fe:88:11:22:a1) rather than the virtual CARP MAC (00:00:5e:00:01:01).
• Upstream CPE ARP Poisoning: The edge gateway committed to its ARP cache that virtual IP 172.16.100.254 belonged to Alpha’s physical MAC.
• Drop by Standby Node: Following failover, the edge router persisted in forwarding inbound frames to Alpha’s physical MAC. Because Alpha was now in Standby/Backup status, the kernel silently dropped the packets. Beta received zero inbound frames.

The Engineering Fix: Transition Syshook (50-carp-arp-sync)

To circumvent the edge router’s sluggish ARP cache timeout (which could persist for several minutes), we engineered and deployed an automated syshook across both cluster nodes at /usr/local/etc/rc.syshook.d/carp/50-carp-arp-sync:

#!/bin/sh
# /usr/local/etc/rc.syshook.d/carp/50-carp-arp-sync
# Hook istantaneo invocato al cambio di stato CARP ($1 = subsys, $2 = event)

SUBSYS="$1"
EVENT="$2"

ROUTER_IP="172.16.100.1"
WAN_VIP="172.16.100.254"
LAN_VIP="10.10.1.1"
LAN_BACKEND="10.10.1.20"

/usr/bin/logger -t carp_hook "CARP transition detected: subsystem=${SUBSYS} event=${EVENT}"

if [ "${EVENT}" = "MASTER" ]; then
    # 1. Force immediate ARP cache flushing on edge gateway via unicast ICMP from CARP WAN VIP
    /sbin/ping -c 4 -S "${WAN_VIP}" "${ROUTER_IP}" >/dev/null 2>&1 &
    # 2. Refresh ARP cache across internal LAN reverse proxy backends
    /sbin/ping -c 2 -S "${LAN_VIP}" "${LAN_BACKEND}" >/dev/null 2>&1 &
    /usr/bin/logger -t carp_hook "CARP MASTER: Triggered ARP refresh pings for ${WAN_VIP} -> ${ROUTER_IP}"
fi

Risultato verificato: The moment Beta assumes the MASTER role, it emits four targeted ICMP frames sourced directly from the CARP VIP. The gateway ARP table is flushed and refreshed instantaneously: inbound failover recovery dropped to less than 2 seconds flat.

---

3. Deep Packet Inspection: Suricata IDS/IPS with EVE-JSON Streaming

Perimeter filtering is bolstered by the Suricata IDS/IPS engine operating in promiscuous pcap mode across both WAN and internal subnets to intercept Layer 7 threat signatures:

• Emerging Threats (ET) Signatures: Continuous updates across curated rule catalogs:
  • emerging-current_events.rules
  • emerging-malware.rules
  • emerging-exploit.rules
  • emerging-scan.rules
• EVE-JSON Standard: Threat events are formatted into rich, structured EVE-JSON telemetry (syslog_eve = 1). Each entry embeds signature IDs (SID), threat classification, TLS metadata, correlated DNS queries, and raw packet headers, primed for SIEM ingestion.

---

4. Central SIEM Command Center: Graylog Pipeline & Threat Intelligence

All security telemetry generated by the OPNsense cluster (filterlog and suricata) streams in real time via Syslog RFC 5424 over dedicated sockets (UDP/TCP 1514 / 5140 at 10.10.50.15) into our self-hosted Graylog SIEM.

Multi-Stage Processing Pipeline

• Stage 0 (CSV Parser): Graylog’s pipeline engine unpacks the native CSV schema of filterlog, distilling ingress interface, firewall action (pass / block), Layer 4 protocol, IP tuples, port numbers, and TCP state flags.
• Stage 1 (GeoIP & Threat Tagging): Public source IPs are correlated against local MaxMind GeoIP2 databases and live IP reputation feeds to flag botnet probes and automated vulnerability scanners.
• Stage 2 (Heuristics and Traffic Anomalies):
  • Port Scan Detection: Automated detection of external probes targeting more than 25 distinct ports within a 60-second window.
  • Brute Force & Unauthorized Access: Heuristic detection of credential-stuffing and repeated failures against VPN portals, management interfaces, and DMZ endpoints.
  • Martian & Spoofed IP: Immediate high-priority alert for spoofed private IP packets (RFC 1918) arriving at the WAN interface.

---

5. Hardware Correlation & Predictive Telemetry via Zabbix 7.0 LTS

In production operations, elevated firewall block counts do not inherently signal a crisis: they often represent ambient background Internet noise. To prevent alert fatigue while rapidly identifying genuine volumetric attacks or appliance exhaustion, the infrastructure relies on a custom Template Security Graylog - OPNsense deployed on Zabbix 7.0 LTS (host 10.10.50.10), monitored by OpenClaw’s Systems Specialist:

Multidimensional Correlation: Threat Velocity vs Physical Resources

Severity Level	Trigger Condition & Threshold	Firewall Hardware Health	Operational Action
Warning 🟡	Isolated block or baseline noise anomaly	CPU < 30%, Stati pf < 40%	Passive aggregation, zero alert fatigue
High 🟠	Coordinated port scan (>25 ports/60s) or >300% block velocity spike	Moderate CPU utilization, accelerating pf state rate	Threat tagging, operations room push notification
Disaster 🔴	Martian IP on WAN, loss of CARP quorum, or volumetric DoS flood	CPU exhaustion (>90%) or pf state table saturation (>90%)	Immediate automated escalation, dynamic null-routing, and cluster failover

---

6. The On-Premise Forensic Analyst: Sec-Sentinel in OpenClaw

At the architectural zenith operates Sec-Sentinel, the autonomous forensic sub-agent of OpenClaw. Its core purpose is breaking past the operational constraints imposed by commercial cloud LLMs:

Why an Uncensored On-Premise Model?

Cloud LLMs hosted on external platforms (OpenAI, Anthropic, Google) enforce rigid alignment filters that routinely refuse to analyze hex payloads, shellcode fragments, SQL injection vectors, or exploit strings, triggering automated refusals like “I cannot assist with malicious content”.

Sec-Sentinel operates entirely on-premise via Arcane (host 10.10.50.25:8088) powered by an unconstrained local model (Qwen-Uncensored):

• Deep Forensic Inspection: Directly evaluates raw hexadecimal dumps and Suricata payloads extracted from Graylog, classifying Common Weakness Enumerations (CWE), identifying exploit mechanics, and delivering actionable mitigation steps.
• Privacy Gatekeeper & PII Sanitization: Before any diagnostic snippet, log segment, or forensic summary is pushed to external alert webhooks (Telegram or storage), Sec-Sentinel programmatically scrubs private IP addresses, credentials, and API keys, replacing them with deterministic redaction tags (e.g. [REDACTED_API_KEY], [REDACTED_INTERNAL_IP]).

---

7. Zero-Secret Architecture and Deterministic Validation

All defensive layers adhere strictly to Safe-by-Default and Zero-Secret operational doctrines:

• Zero Destructive Hot-Patching: Autonomous agents are barred from executing unverified live mutations on production firewalls; every rule modification undergoes rigorous offline linting and validation via test harnesses (validate_security_event.py and audit_opnsense_graylog.py).
• Isolated Vector Knowledge Stores: Each agent preserves its private Qdrant collection to prevent context contamination and transactional deadlocks.
• Absolute Data Sovereignty: The entire telemetry lifecycle — from wire transit through deep packet inspection to forensic AI dissection — remains enclosed within private on-premise hardware, immune to external cloud surveillance or third-party breaches.

---

⚡ Conclusion: The Unified Defensive Cyber-Organism

The convergence of OPNsense HA, Suricata, Graylog, Zabbix, and Sec-Sentinel elevates separate tools into a unified, self-healing cyber-organism. A defensive architecture that does not merely block threats, but dissects, isolates, and adapts to them in real time — proving that elite modern cybersecurity is forged at the intersection of deterministic systems engineering and private on-premise artificial intelligence.