DwarfStar (ds4) — Inference Engine

DwarfStar (repo: antirez/ds4) is the inference engine that powers OtherU. It is a narrow, self-contained C-based runtime optimized for DeepSeek V4 Flash and PRO, not a general GGUF loader. It runs on the reference deployment via Podman with ROCm passthrough.

Source & Upstream

The project is developed with strong GPT 5.5 assistance and builds on the foundations of llama.cpp / GGML.

Reference Deployment (Strix-Halo)

Start Command

podman run --rm --replace --name ds4-server \
  --device /dev/kfd --device /dev/dri \
  --security-opt seccomp=unconfined --security-opt label=disable \
  --network host \
  -v /srv:/srv \
  docker.io/kyuz0/strix-halo-ds4-toolbox:rocm-7.2.4 \
  ds4-server \
  -m /srv/models/DeepSeek-V4-Flash-abliterated-imatrix-ds4.gguf \
  --ctx 393216 --host 127.0.0.1 --port 8000 --power 100 \
  --kv-disk-dir /srv/kvcache --kv-disk-space-mb 2097152

Architecture

DwarfStar is not a wrapper or generic runner — it is a purpose-built C program with its own graph execution engine. Key design choices:

• Single-model narrowness — only DeepSeek V4 Flash and PRO GGUFs with the expected tensor layout, quantization mix, and metadata
• Self-contained — no dependency on llama.cpp/GGML at runtime; kernels, quant formats, and prompt rendering are native
• Asymmetric quantization — routed MoE experts at IQ2_XXS, down at Q2_K; shared experts, projections, and routing left unquantized. This is what makes 2-bit quants viable: most of the model space is aggressively compressed but the critical paths stay accurate
• Disk-first KV cache — checkpoints are written to SSD at four moments (cold, continued, evict, shutdown), enabling fast session resume across restarts without re-prefilling from token zero
• Single-threaded inference — one graph worker serializes all requests; client threads handle HTTP parsing in parallel

Endpoints

Model aliases: deepseek-v4-flash and deepseek-v4-pro — both serve the loaded GGUF.

Key CLI Flags

Disk KV Cache Behavior

Checkpoints are written at four moments:

• cold — after a long first prompt reaches a stable prefix, before generation
• continued — when prefill or generation reaches the next aligned frontier
• evict — before an unrelated request replaces the live in-memory session
• shutdown — when the server exits cleanly

Cold saves trim the last 32 tokens and align down to a 2048-token chunk boundary to avoid BPE retokenization misses.

Model Quantization

The project provides pre-built GGUFs from HuggingFace (antirez/deepseek-v4-gguf):

Download script: download_model.sh q2-imatrix fetches from HuggingFace.

Performance Benchmarks

Upstream Reference Numbers (from antirez/ds4 repo)

These are single-run Metal CLI numbers with --ctx 32768, --nothink, greedy decoding, -n 256.

Flash (2-bit Quant) — Short Prompt

Flash (2-bit Quant) — Long Prompt (\~11.7k tokens)

Flash (4-bit Quant) — M3 Ultra 512 GB Only

PRO Model (2-bit) — Mac Studio M3 Ultra 512 GB

Reference Deployment (AMD Strix-Halo) — Measured

Measured under production Hermes Agent traffic (not ds4-bench): \~14 t/s sustained decode at 20k–140k context (14.1 t/s average over 300+ second generations; \~16 t/s at smaller contexts), and \~250 t/s compute prefill on long prompts. Disk KV-cache restores replay much faster than compute prefill (\~2,400 t/s effective). Short-prompt ds4-bench numbers to be added.

(Charts from the upstream repo. Our own benchmark chart to be added once *ds4-bench* data is collected on the Strix-Halo.)

SSD Streaming

When the model does not fit in GPU-addressable memory, DwarfStar can run in SSD streaming mode. Non-routed weights stay resident; routed MoE experts are cached and loaded from the GGUF file on miss. The automatic budget takes 80% of Metal recommended working set, subtracts non-routed weights, then uses the rest for expert cache.

# Automatic cache budget
ds4-server --ssd-streaming

# Explicit cache size
ds4-server --ssd-streaming --ssd-streaming-cache-experts 32GB

Tooling

Additional tools under gguf-tools/: GGUF generation, imatrix collection, quantization, quality testing against official DeepSeek logits.