Setup and Deployment

Edited by Lawrence Beckwith on July 21, 2026 at 7:54 PM UTC

System Requirements

The reference deployment runs on a single Linux host with sufficient unified or discrete GPU memory.

Recommended:

  • AMD Ryzen AI Max+ (Strix Halo) with 128 GB unified memory, or equivalent
  • Linux (Fedora recommended) with ROCm (AMD GPU) or CUDA (NVIDIA GPU)
  • Podman (or Docker) for container runtime
  • 200 GB+ storage for models

Minimum:

  • 32 GB+ GPU/unified memory
  • Linux with Podman

Inference Runtime: ds4-server

The reference deployment runs a single model via ds4-server. This replaces the old multi-container OtherU Core stack.

Install ds4-server Image

podman pull docker.io/kyuz0/strix-halo-ds4-toolbox:rocm-7.2.4

Place the Model

Download a GGUF model and place it in /srv/models/:

# Example for DeepSeek-V4-Flash abliterated imatrix-quantized:
# Place at /srv/models/DeepSeek-V4-Flash-abliterated-imatrix-ds4.gguf

Start the Server

podman run --rm --replace --name ds4-server \
  --device /dev/kfd --device /dev/dri \
  --security-opt seccomp=unconfined --security-opt label=disable \
  --network host \
  -v /srv:/srv \
  docker.io/kyuz0/strix-halo-ds4-toolbox:rocm-7.2.4 \
  ds4-server \
  -m /srv/models/<your-model>.gguf \
  --ctx 393216 \
  --host 127.0.0.1 \
  --port 8000 \
  --power 100 \
  --kv-disk-dir /srv/kvcache \
  --kv-disk-space-mb 2097152

Key flags:

Flag Purpose
--device /dev/kfd ROCm GPU device passthrough
--device /dev/dri DRM render device for GPU access
--network host Bind to host network namespace
-v /srv:/srv Bind-mount models and KV-cache from host
--ctx 393216 Context length (tokens)
--power 100 GPU power limit (percentage)
--kv-disk-dir Persist KV-cache to disk for fast resume
--kv-disk-space-mb KV-cache disk budget

Hermes Agent Integration

Configure Hermes Agent as a custom provider pointing to ds4-server:

model:
  default: deepseek-v4-flash
  provider: custom
  base_url: http://127.0.0.1:8000/v1

Hermes Agent handles Telegram, Signal, and WhatsApp channels.

Validation Checklist

  • [ ] curl http://127.0.0.1:8000/v1/chat/completions -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"ping"}]}' → returns a chat completion
  • [ ] podman ps → ds4-server container running and healthy
  • [ ] Send a test message through Hermes Agent → response arrives in the chat

Troubleshooting

Symptom Likely Cause Fix
Container exits immediately Model not found at specified path Check /srv/models/<model>.gguf exists
GPU not detected Missing --device flags Verify --device /dev/kfd --device /dev/dri
OOM at load Model too large for GPU memory Use smaller model or imatrix quantization
Slow first response KV-cache building on first access Normal; subsequent requests are fast
Connection refused ds4-server not running podman ps → restart with podman run

Old Stack Migration

If migrating from the old multi-container OtherU Core stack:

  1. Stop old containers: docker compose down in the otheru-core directory
  2. Start ds4-server per the instructions above
  3. Update Hermes Agent config to point to port 8000 instead of 8080
  4. Verify chat completion and channel delivery