This reference covers routine diagnosis and recovery for the standard Ember Docker Compose deployment. Run Compose commands from the checkout used to start the service. Adapt service names and ports for an existing custom installation.
Check service and inference separately
docker compose ps
docker compose logs --since=10m ember
curl --fail --show-error http://127.0.0.1:8080/health
curl --fail --show-error http://127.0.0.1:8080/v1/models
HTTP availability does not prove that generation works. Once the model has loaded, run scripts/smoke_test.sh --generate from the release checkout. Test a messaging channel separately after direct inference succeeds.
Diagnose a failure
| Symptom | Investigation |
|---|---|
| Container exits during startup | Read the first error in the logs; check configuration, model integrity, and mounted paths |
| GPU initialization fails | Verify /dev/kfd, /dev/dri, and the host's supported driver setup; run scripts/preflight.sh |
| Model load runs out of memory | Check free -h, competing workloads, and the selected model and context settings |
| API connection is refused | Confirm the service is running and that the client uses the correct host and port |
| First request is slow | Distinguish model loading and prompt prefill from token generation |
| Agent channel is silent | Test direct inference, then inspect the agent's provider and channel logs |
Preserve the relevant error, release version, and configuration before changing the deployment. Redact prompts, credentials, and private data before sharing logs.
Restart or stop
For a process restart:
docker compose restart ember
After changing Compose or environment configuration, use docker compose up -d to apply it. To stop the deployment, use docker compose down. These operations preserve the configured host model and cache directories.
Stopping inference does not cancel already-running tools or scheduled agent work. For unwanted actions, stop the responsible agent or integration as well and review its credentials.
Cache investigation
An unexpected response is not sufficient evidence of cache corruption. Check the prompt, loaded model, release, and logs first. If a cache issue is suspected, stop the service and preserve the affected cache before testing with a separate empty cache directory. Avoid deleting the only copy during diagnosis.
Updates and rollback
Record the working release and configuration before an update. Use a release's matching image and model artifacts, then verify health, generation, and dependent integrations. If validation fails, return to the previous release and configuration while preserving evidence of the failure.
For KVM-specific failures, diagnose display capture and input delivery separately from inference. Use the device's documentation for display and connection settings.