Architecture Overview

System Goals

• Local-first inference and control — no external API dependency for core operation
• Deterministic execution pathways for critical actions (HID injection, file writes)
• Service composition with explicit trust boundaries
• Operator authority at all layers — OtherU acts, humans approve or override

Major Planes

1) Perception

Captures relevant state from the environment:

• JetKVM HDMI capture — real-time desktop screenshots via hardware bridge
• Vision resize pipeline — raw capture letterboxed to model input resolution
• Optical flow cursor tracking — Lucas-Kanade tracker, confidence-scored position
• Memory retrieval — Hindsight (semantic + keyword + entity-graph recall) across Hermes Agent

2) Reasoning

A single inference engine handles all reasoning tiers. The model is selected by prompt complexity and tool requirements:

Routing path:

User Message (Telegram)
    │
    ▼
Hermes Agent
    │  ┌── custom provider: http://127.0.0.1:8000/v1
    │  │   model: deepseek-v4-flash (ds4-server)
    │  │   backend: ROCm 7.2.4 on AMD Radeon 8060S
    │  ▼
ds4-Server (Podman container, port 8000)
    │
    ├── FAST → single-turn response
    ├── DEFAULT → streaming chat completion
    ├── REASONING → high-temperature chain-of-thought
    └── COMPLEX → tool-calling agent loop with Hermes tool dispatch

The inference server runs as a Podman container with ROCm GPU passthrough (/dev/kfd, /dev/dri). KV-cache is disk-persisted (/srv/kvcache, 2 TB budget) for fast session resume across restarts.

3) Action

Hermes Agent executes constrained operations through typed tools:

• Desktop/HID actions — JetKVM (WebRTC capture + USB HID) driven by Hermes computer-use tools
• Coordinate translation — vision model coordinates inverse-transformed to real screen coordinates before HID injection
• Code execution — sandboxed shell commands
• File operations — workspace read/write
• Messaging — post to Telegram topics, send alerts, create summaries
• Web & research — web search, page extraction
• Delegation — spawn sub-agents for parallel work

4) Memory

Hermes Agent provides three memory systems:

Hindsight (Long-Term Fact Storage)

Hindsight is the durable memory layer. It stores structured facts extracted from conversations and retrieves them on future turns via three parallel mechanisms:

• Semantic search — embedding-based similarity across all stored entries
• Keyword matching — full-text search over entity names and descriptions
• Entity graph traversal — follows relationships between entities (people, projects, systems) to surface connected facts

Three query modes:

Hindsight recall is on-demand: the agent calls hindsight_recall or hindsight_reflect when it needs long-term context (automatic per-turn injection is disabled to keep the inference server's KV cache stable). Writing remains automatic — auto_retain captures durable facts from conversations, and the retain tool is also called proactively when the agent learns a durable fact about the user, their environment, or a project.

Configuration:

Session Search (Conversation History)

FTS5-indexed archive of every past conversation session across all Hermes profiles. Accessed via session_search:

Session search is not automatically injected — it is called on demand when the user references past work.

Skills (Procedural Memory)

Reusable markdown files stored in ~/.hermes/skills/ that encode workflows for recurring task types. Each skill has a YAML frontmatter block (trigger conditions, description, category) and a markdown body with numbered steps, commands, and pitfalls. Skills are loaded explicitly via skill_view(name) and listed via skills_list().

Reliability Model

• ds4-server runs as a Podman container with restart on failure
• KV-cache on disk survives container restarts
• Hermes Agent handles channel disconnection and retry
• Cascade routing — if primary model is unavailable, Hermes falls through the configured model chain

Inference Stack

The reference deployment runs a single model via ds4-server on AMD ROCm:

OS & Storage

Container Stack

The old multi-container OtherU Core stack (coordinator, orchestrator, responder, reasoner, coder, Fara, hardware-bridge, LightRAG, Redis, GSD) has been retired in favour of this simpler architecture. All agent roles are now handled by Hermes Agent's tool dispatch against the single ds4-server model.