Operations Runbook

Edited by Lawrence Beckwith on July 22, 2026 at 11:47 PM UTC

Daily Health Checks

# ds4-server liveness (chat completion)
curl http://127.0.0.1:8000/v1/chat/completions \
  -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"ping"}]}'

# ds4-server model list (liveness check)
curl http://127.0.0.1:8000/v1/models


# Container health
podman ps

ds4-Server

On the reference deployment ds4-server is managed by a root user-level systemd unit — prefer sudo XDG_RUNTIME_DIR=/run/user/0 systemctl --user restart ds4-server.service. The commands below are the unit's ExecStart equivalent, for debugging or non-systemd hosts.

# Restart (container was created with --rm, so re-run the same command)
podman kill ds4-server
podman run --rm --replace --name ds4-server \
  --device /dev/kfd --device /dev/dri \
  --security-opt seccomp=unconfined --security-opt label=disable \
  --network host \
  -v /srv:/srv \
  docker.io/kyuz0/strix-halo-ds4-toolbox:rocm-7.2.4 \
  ds4-server \
  -m /srv/models/DeepSeek-V4-Flash-abliterated-imatrix-ds4.gguf \
  --ctx 393216 --host 127.0.0.1 --port 8000 --power 100 \
  --kv-disk-dir /srv/kvcache --kv-disk-space-mb 2097152

# Follow logs
podman logs -f ds4-server

# Verify model loaded
curl http://127.0.0.1:8000/v1/chat/completions \
  -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"ping"}]}'

Desktop Control (JetKVM)

The legacy hardware bridge on port 8005 is retired. Desktop capture and HID control run through Hermes Agent's computer-use tools, which drive the JetKVM directly (WebRTC capture + USB HID). Any step referencing 127.0.0.1:8005 is historical.

Incident Triage

  1. Identify scope — is the model responding? Is the bridge alive?
  2. Check logs — podman logs --since=10m ds4-server
  3. Validate model — curl http://127.0.0.1:8000/v1/chat/completions with a ping prompt
  4. Apply targeted fix — restart container, patch model path, reload
  5. Verify recovery — re-run health checks
  6. Document — update runbook with root cause and fix

Common Issues

ds4-server doesn't start

Check the model path and GPU device availability:

# Verify model exists
ls -lh /srv/models/*.gguf

# Verify GPU devices
ls -la /dev/kfd /dev/dri

If the model path changed, update the podman run command and restart.

Display goes dark after EDID change

Custom low-resolution EDIDs may lack fallback timing modes. Reset to the native display EDID from the JetKVM web UI (Settings → Display → EDID).

GPU memory pressure

Check overall memory with free -h. If ds4-server is OOMing, reduce context size (--ctx) or switch to a smaller quantized model.

free -h
podman logs --since=10m ds4-server | grep -i 'oom\|memory'

KV-cache corruption

If responses degrade or hallucinate, clear the disk cache and restart:

rm -rf /srv/kvcache/*
podman kill ds4-server
podman run --rm --replace --name ds4-server \
  --device /dev/kfd --device /dev/dri \
  --security-opt seccomp=unconfined --security-opt label=disable \
  --network host \
  -v /srv:/srv \
  docker.io/kyuz0/strix-halo-ds4-toolbox:rocm-7.2.4 \
  ds4-server \
  -m /srv/models/DeepSeek-V4-Flash-abliterated-imatrix-ds4.gguf \
  --ctx 393216 --host 127.0.0.1 --port 8000 --power 100 \
  --kv-disk-dir /srv/kvcache --kv-disk-space-mb 2097152

Hermes Agent not responding

Check Hermes Agent's provider config points to http://127.0.0.1:8000/v1 with the correct model name:

# Verify config
grep -A5 'base_url\|default' /root/.hermes/config.yaml

Release Hygiene

  • Keep Podman run commands and model paths version-controlled
  • The model is baked into /srv/models/ — never edit inside the container
  • Bridge changes require container rebuild or podman cp
  • KV-cache is on disk (/srv/kvcache) — clear on model swap or corruption

WMMA Ops Profiling

Standard workflow

  1. Run timing benchmarks for kernel variants
  2. Capture HIP traces for launch behavior and synchronization overhead
  3. Use rocprofv3 where available to compare instruction and memory patterns

gfx1151 limitation

Many hardware performance counters are unavailable on consumer gfx1151 (aqlprofile-backed counters fail). Prioritize:

  • Wall-clock benchmark stability
  • PyTorch profiler traces
  • HIP API trace analysis

Rule: Promote a kernel variant only when it passes both correctness checks and repeatable timing benchmarks. Keep the adaptive fallback enabled.