In modern production AI engineering, the monolithic “single all-knowing agent” pattern has rapidly reached its structural limits: uncontrolled hallucinations, operational fragility, choking rate-limits, and critical security vulnerabilities. To break through these bottlenecks, our infrastructure implements a distributed separation-of-concerns architecture: OpenClaw orchestrates and executes operational system tasks; Hermes acts as an independent read-only technical auditor; OmniRoute manages intelligent routing and zero-downtime failover across dozens of frontier LLMs; while two dedicated on-premise AIs govern persistent semantic memory (Mem0 with local embeddings) and uncensored forensic security analysis (Sec-Sentinel).
⚡ Key Infrastructure Metrics
🗺️ Complete Architectural Blueprint
The infrastructure bridges bare-metal nodes, LXC containers on Proxmox VE, and 2 Cloud VMs (2 vCPU, 12GB RAM), delivering high availability, multi-tier redundancy, and continuous cryptographic auditing.
1. OpenClaw: The Executive Orchestrator
OpenClaw operates as the system’s primary executive engine and operational orchestrator, hosted on a dedicated instance (10.100.1.15). Rather than a plain chat generator, it functions as a comprehensive runtime equipped with deterministic tool calling, direct filesystem operations, network daemon administration, and dynamic sub-agent delegation.
Specialized Sub-Agent Roles
- Global_Router: The central routing brain. Ingests user intent, inspects hierarchical knowledge trees, and dynamically routes tasks to the appropriate specialized sub-agent or handles them directly.
- Sistemista: Oversees systems administration across Proxmox VE hypervisors, OPNsense HA/CARP firewall clusters, Zabbix 7.0 LTS telemetry, and container lifecycle via Arcane.
- Media-Manager, CFO & Musician: Verticalized domain specialists handling financial telemetry, lossless audio processing via Lidarr, and media libraries.
“Audit-Ready” Core Philosophy
OpenClaw enforces an immutable protocol codified in AUDIT_EVENTS.md: whenever an agent initiates an infrastructure mutation (CHANGE) or remediation (CORRECTION), the task remains uncommitted until verifiable empirical evidence (exit status, code diffs, cryptographic hash) is published to the local audit outbox.
2. Hermes: The Independent Read-Only Technical Auditor
In autonomous workflows, allowing the executing entity to self-certify its output inevitably creates dangerous confirmation bias and blind spots. Hermes was purposefully introduced to enforce a strict separation of powers.
Read-Only Sandboxed Isolation
Hermes resides in an isolated LXC container on Proxmox VE (10.100.1.14, Debian 13 Trixie). By strict architectural decree:
- Zero Mutation Rights: Hermes holds strictly zero write credentials across production servers, firewalls, and network configurations.
- Evidence Evaluation: Evaluates structured audit reports, immutable system logs, and cryptographic snapshot hashes exclusively.
- Formal Feedback Loop: Upon detecting anomalies, it never attempts direct intervention; it issues a formal remediation request back to OpenClaw.
The Audit Pipeline (FastAPI + HMAC-SHA256)
Inter-agent telemetry between OpenClaw and Hermes traverses a hardened, tamper-proof channel:
- OpenClaw compiles the task execution manifest (intent, evidence, bash logs, code diffs).
- The Python client (
tools/openclaw_audit_client.py) signs the JSON payload using HMAC-SHA256 with monotonic anti-replay timestamps. - The
hermes-audit-receivermicroservice (port8644) validates the signature, queues the entry, and executes dual deterministic and semantic validation viaomniroute/hermes-review. - Verdict states are strictly categorized:
approved,approved_with_notes,needs_correction,insufficient_evidence. - If issues are flagged, an automated remediation cycle is triggered, hard-capped at 2 iterative rounds to avoid infinite loops.
3. OmniRoute: Unified Intelligent Gateway for LLM Inferences
Every inference request from OpenClaw and Hermes must route through OmniRoute, deployed on a high-availability cloud VM (2 vCPU, 12GB RAM) at 10.200.0.12:20128.
| Virtual Profile | Primary Models & Fallback Chain | Designated Workload |
|---|---|---|
omniroute/openclaw |
Codestral → Copilot GPT-4.1 → Groq Qwen 3.6 27B → Copilot GPT-4o → Gemini Flash | Primary runtime for general systems and execution tasks |
omniroute/openclaw-smart |
Codestral → Copilot GPT-4.1 → Copilot GPT-4o → Groq GPT-OSS 120B → Nemotron Reasoning | Deep analytical reasoning, complex architectural refactoring |
omniroute/openclaw-fast |
Copilot GPT-4o Mini → Groq Qwen 3.6 → Mistral Code → Gemini Flash Lite → Gemma 4 | Preliminary classification, entity extraction, health probes |
omniroute/hermes-review |
Codestral → Copilot GPT-4.1 → Groq Qwen 3.6 → Gemini Flash → Fallback Cloud | Hermes audit review and log verification (high precision) |
omniroute/emergency |
Copilot Mini → Gemini Flash Lite → Groq OSS | Emergency fallback under multi-provider rate limits or outages |
4. The Two Local AIs: Data Sovereignty and Uncensored Security
Local AI 1: Mem0 & Private Semantic Vector Indexing
Production agents demand contextual continuity: architectural patterns, topology maps, diagnostic history, and operational quirks. Shipping persistent memory fragments to external cloud endpoints introduces severe data sovereignty risks.
- Engine Locale: Dedicated Ollama instance deployed on the secondary Cloud VM (2 vCPU, 12GB RAM) at
10.200.0.13:11434. - Embedding Model:
nomic-embed-text:latest, generating 768-dimensional high-fidelity vector representations. - Integrazione con OpenClaw: Via the integrated
openclaw-mem0bridge, facts distilled from sessions are recorded to local SQLite storage (/root/.mem0/history.db) and vectorized into private Qdrant collections (opnsense_kb,zabbix_kb).
Local AI 2: Sec-Sentinel (Arcane & Qwen3-8B Uncensored)
When investigating live SQL injections, reverse shells, or binary payloads extracted from Suricata or OPNsense logs, commercial cloud LLMs routinely reject processing under false-positive “harmful content” filters. Sec-Sentinel was designed to overcome this bottleneck:
- Local Engine: Private Arcane inference runtime on LAN (
192.168.20.9:8088/v1). - Specialized Model:
Qwen3-8B-Uncensored(optimized quantizationQwen3-8B-Uncensor-v2.Q4_K_M.gguf). - Uncensored Forensic Inspection: Objectively analyzes raw threat telemetry from Suricata, Graylog, and OPNsense firewalls, assessing attack vectors and tactical remediation.
- Zero-Leak PII Sanitization: Strips passwords, API tokens, and internal subnet IPs with
[REDACTED_*]tokens before any telemetry ever leaves the private network.
5. Key Takeaways & Architectural Advantages
- Vendor Lock-in Immunity: With OmniRoute acting as an abstracted gateway, swapping or load-balancing LLM providers is a single-line configuration change with zero agent code refactoring.
- Deterministic Engineering Rigor: The absolute separation between Executor (OpenClaw) and Auditor (Hermes) guarantees every mutation is verified against objective evidence and cryptographic signatures.
- Absolute Data Sovereignty: Semantic memory vectors (Mem0) and raw exploit inspection (Sec-Sentinel) remain fully air-gapped on premise within private infrastructure.