NVIDIA RTX 4000 SFF Ada Local LLM Inference Performance Test

Overview

This article documents a hands-on benchmark of the NVIDIA RTX 4000 SFF Ada Generation 20 GB — a 70 W card with no external power connector that fits the ZimaCube — running local LLM inference on ZimaOS 1.7.1 through a containerized llama.cpp server. The 20 GB of VRAM is the key difference from the 16 GB RTX PRO 2000: it unlocks the model’s native 256K context window.

For the five-step deployment walkthrough, see Local LLM Inference.

TL;DR

In one sentence: with 20 GB of VRAM, the RTX 4000 SFF Ada runs the same 35B MoE model as the 16 GB tier but with the full 256K context window — the recommended setup fits in 14.5 GB with room to spare.

Test Environment

Unless noted otherwise, benchmarks use llama-bench with -p 512 -n 256 -r 3 (prompt 512 tokens, generate 256 tokens, mean of 3 runs).

Benchmark Results

Quantization Comparison (llama-bench, 512/256)

The quantization source decides whether the model keeps its MTP (next-token prediction) layer: the prithivMLmods/...-MTP-GGUF quants keep MTP, while the unsloth/...-GGUF UD quants strip it (loading reports model doesn't contain MTP layers).

The 20 GB card should take IQ3_XXS: it leaves room for 256K context with KV q4. Q3_K_S at 15.55 GB cannot even load on a 16 GB card.

The model choice itself is the same as on the 16 GB tier: Qwen3.6-35B-A3B is the strongest model that fits this class of card, and its hybrid MoE architecture is built for agent and coding scenarios — the full bandwidth analysis is on the RTX PRO 2000 page. For serious coding and knowledge-base workloads, the 122B / Flash tier is the next step and requires larger cards.

Optimization Switches A/B (llama-bench, 512/256, mean of 3 runs)

Flash-Attention and Long Context

With fa=off, even 4096 tokens fail with failed to create context. Flash attention is a hard requirement for long contexts with this model.

256K Context Memory Accounting

MTP Speculative Decoding

MTP is rejected for the 20 GB tier: acceptance is high, but decode is bandwidth-locked — the MoE model must read about 3B active experts per token regardless. The gain is small and unstable, so the recommendation is to skip MTP.

Agent Workload: DeepSeek Harness at Long Context

Measured with a real DeepSeek Harness agent session as the workload:

Decode slows as context grows (larger KV reads plus more attention). This is the inherent cost of long context, not a fault — no overflow, no truncation. Even at ~100K context, ~33 t/s is still fluid for a streaming agent. For DeepSeek Harness and Codex-style coding agents, this 256K window is the reason to pick this card over the 16 GB tier.

Power and Temperature (full load)

Why MTP Does Not Help Here

On the 16 GB RTX PRO 2000, decode is capped by memory bandwidth at about 65–70 t/s. The 4000 SFF Ada has more bandwidth (320 GB/s), which lifts decode to 74–83 t/s — but the same MoE bandwidth logic still applies: every token must read the ~3B active experts. MTP adds a draft model on top, and with 62–72% acceptance the real gain is a few percent on natural text, negative under greedy sampling. The memory is better spent on KV q4 and the 256K window.

Recommended Configuration (20 GB Tier)

Start the server in one command:

docker run -d --name llm-server --gpus all -p 8080:8080 \
  -v "$PWD/models/llm:/models" \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf \
  --host 0.0.0.0 --port 8080 \
  -c 262144 --n-gpu-layers 999 \
  -fa on --cache-type-k q4_0 --cache-type-v q4_0

The service exposes an OpenAI-compatible endpoint at http://<host-ip>:8080/v1 (API key is a placeholder, the server does not authenticate). Any OpenAI-format client or agent can connect directly.

Tier Comparison

Limitations

1. MTP data points and long-context decay are single samples; variance was not strictly evaluated. The optimization matrix (A/B table) is a mean of 3 runs.
2. Only the 20 GB single-card tier was verified on hardware. The 16/24/32 GB tiers follow the theoretical compatibility matrix and are not covered by these measurements.
3. NVFP4 (Blackwell-only), EAGLE, and power-limit tuning are not available on this card and were not included.