Architecture Overview

System Goals

• Local-first inference and control — no external API dependency for core operation
• Deterministic execution pathways for critical actions (HID injection, file writes)
• Service composition with explicit trust boundaries
• Operator authority at all layers — OtherU acts, humans approve or override

Major Planes

1) Perception

Captures relevant state from the environment:

• JetKVM HDMI capture — real-time desktop screenshots via hardware bridge
• Vision resize pipeline — raw capture letterboxed to model input resolution for Fara
• Optical flow cursor tracking — Lucas-Kanade tracker, confidence-scored position
• RAG memory retrieval — LEANN (HNSW vector index) + LightRAG + GraphRAG for context

2) Reasoning

Builds plans from user intent and system state using a three-tier model selection system:

Routing path:

User Message
    │
    ▼
OpenClaw (Telegram / Signal / WhatsApp)
    │
    ▼
Coordinator (OpenAI-compatible API, port 8080)
    │
    ├── preroute_by_pattern()     ← rule-based fast-path
    │     ├── screenshot request  → execute directly
    │     ├── desktop/KVM task    → execute_desktop_tool()
    │     ├── record request      → execute directly
    │     └── code / web / etc    → route to specialist agent
    │
    ├── openclaw_tool_decision()  ← 8B orchestrator selects external tools
    │
    └── council_streaming()
          ├── needs_tools() → COMPLEX (orchestrator loop)
          │     └── orchestrator selects tool → execute_tool_call()
          └── otherwise    → DEFAULT / REASONING / FAST

3) Action

Executes constrained operations through typed adapters:

• Desktop/HID actions — JetKVM WebRTC bridge → mouse move/click, keyboard type/press
• Coordinate translation — vision model coordinates inverse-transformed to real screen coordinates before HID injection
• Code execution — sandboxed shell commands via coordinator exec tool
• File operations — workspace read/write via local filesystem tool
• Messaging — post to Telegram topics, send alerts, create summaries

4) Memory

Stores context for continuity and retrieval:

• LEANN — HNSW vector index using facebook/contriever embeddings
• LightRAG — Graph + vector hybrid retrieval, integrated in coordinator
• GraphRAG — Custom knowledge graph implementation
• Redis — Operational state: routing session pins, model memory tracking, GSD loop state
• Daily log files — raw session notes for rolling context
• Long-term memory file — curated knowledge, updated periodically

Reliability Model

• Health and dependency endpoints on every service (/health, /dependencies/health)
• All containers configured with --restart unless-stopped
• Routing policy with session pinning (configurable TTL) for conversational coherence
• Cascade routing — primary path failure falls back through candidate tiers
• Coordinator files bind-mounted from host — edit and restart to apply changes

WMMA Ops Integration (AMD gfx1151)

OtherU Core integrates the wmma_ops kernel stack for AMD gfx1151 (RDNA3.5 / Strix Halo):

• Architecture target: gfx1151 with ROCm WMMA paths
• Achieved throughput: 21.6 TFLOPS on AMD Radeon 8060S iGPU
• Kernel variants: matmul_adaptive, matmul_zerocopy, matmul_asmOpt — selected by workload shape
• Critical ROCm flags: -fa 1 (flash attention), --no-mmap, ROCBLAS_USE_HIPBLASLT=1
• Result: +40% prefill improvement over baseline ROCm configuration

Container Stack