The 408-Second Answer
A single configuration variable was turning a 3-second answer into a 7-minute one. This is how we found it.
The Railway deployment looked fine at first. Health check passed, the model loaded, queries returned answers. Then we ran the benchmark. A verified answer that took 3-4 seconds locally was taking 21 seconds on Railway. Sometimes 60. Sometimes 408 seconds. The benchmark wasn’t even completing within a reasonable timeout.
The Clue in the Metrics
The first thing we checked was the obvious stuff: network latency, model size, available RAM. All fine. The container was getting the resources it asked for.
The clue was in the metrics. Prefill speed, the part where the model processes the prompt in parallel, was running at about 19 tokens per second. That’s slow but not broken. Decode speed, the part where the model generates one token at a time, was at 0.09 tokens per second. A 34-token answer at 0.09 tok/s takes 378 seconds. That’s where the 408-second answer came from.
Prefill can parallelize across the prompt length, so it tolerates thread contention reasonably well. Decode is sequential. Each token depends on the previous one. So at each token boundary, all the threads have to sync. If you have too many threads competing for CPU time, they spend most of their time waiting for each other at that barrier rather than doing work.
Thread Thrashing on Shared Compute
The Railway container had a CPU quota of about 4 vCPUs. But llama.cpp, by default, spawns one thread per host core. The host machine had 32 cores. So the model was running 32 threads on a container that only had quota for 4.
Prefill tolerated it because batched computation over the prompt has enough work to distribute. Decode thrashed.
The fix was one environment variable: METIS_NUM_THREAD=4, which gets plumbed through to the ollama num_thread option, which tells llama.cpp how many threads to actually use.
| metric | before | after |
|---|---|---|
| prefill | 19.7 tok/s | 218 tok/s |
| decode | 0.09 tok/s | 35.9 tok/s |
| verified answer latency | ~408 s | ~3.4 s |
Two Segfaults
We also found two segfaults during deployment.
The first was from OLLAMA_KV_CACHE_TYPE=q8_0. Quantized KV cache needs flash attention; without it, llama-server segfaults on load. The fix was to drop it. Default FP16 KV at 2048 context is only 224 MB anyway, which is fine for our budget.
The second segfault was more interesting. Earlier deploys had been fine, but a later one crashed on load with a segfault that had nothing to do with memory. The logs showed a 380 GB host machine with AMX instructions (Intel Sapphire Rapids, a recent server-grade architecture). The ollama AMX and flash-attention codepath had a bug loading Qwen3 on those hosts. The fix was OLLAMA_FLASH_ATTENTION=0 plus OLLAMA_LLM_LIBRARY=cpu_avx2 to force the older AVX2 path. That’s slower than AMX by some margin, but it doesn’t crash.
Persistent Volumes
We also added a persistent volume for the model files. Every deploy was re-pulling 1.1 GB of model weights, which took a couple of minutes and re-exposed us to the AMX lottery. Mount a persistent volume at /root/.ollama, pull the model once, and subsequent deploys skip the download.
Three env vars and a volume mount. That’s the difference between “unusable” and “responsive” on a cheap shared container.
The lesson is that shared PaaS infrastructure lies to you about available CPU in a way that doesn’t matter for latency-tolerant workloads but completely destroys per-token sequential operations. Any LLM deployment on shared compute should be pinning threads explicitly.
Key takeaways:
- Decode is sequential; too many threads on a CPU-quoted container causes catastrophic thrashing at each token boundary
- One env var (METIS_NUM_THREAD=4) turned a 408-second answer into a 3.4-second one
- Persistent model volumes eliminate re-pull time on every deploy and remove exposure to host hardware variation
Next: the research behind why verifying answers is fundamentally easier than generating them.