Qwen 3.8 27B: Hardware requirements and how to deploy locally

Qwen3.8-27B is small enough to run on a desktop, but the memory math punishes rough estimates. The model card describes a 27B-parameter vision-language model with a 262,144-token native context, image and video input, thinking controls, and support for Transformers, vLLM, and SGLang. Its raw BF16 checkpoint is about 54.7 GB in the published GGUF inventory. A 4-bit build is roughly 16 to 18 GB, and the official FP8 checkpoint is about 28.75 GiB. Those figures cover the weights only. Context cache, vision inputs, runtime buffers, operating-system memory, and concurrent requests add to the total.

For a consumer machine, a 4-bit GGUF or MLX build with a bounded context window is the practical default. On newer NVIDIA or AMD accelerators, the official FP8 checkpoint is a good choice when throughput and fidelity matter. On a data-center GPU, BF16 works when you need a reference configuration or have memory for long context and concurrency. The platform decides which artifact and runtime make sense.

Qwen3.8-27B's memory envelope

Treat the 262K context as a capacity figure. Running at that length costs memory, so measure it before enabling it. The architecture lists 16 gated-attention layers, four key-value heads, a head dimension of 256, and separate key and value tensors. A simple calculation gives 16 × 4 × 256 × 2 × 2 bytes per token, or about 64 KiB of FP16 KV cache per token. At 262,144 tokens, that is roughly 16 GiB before allocator overhead, image or video state, batching, and framework-specific buffers. The figure comes from the published architecture, so treat it as an estimate. Quantizing the KV cache to 8-bit or FP8 roughly halves this part of the budget when the chosen runtime supports it.

The published model and quantized repositories give a useful starting range:

Weight format	Approximate size for Qwen3.8-27B	Where it fits	What remains to budget
BF16	54.7 GB	64 GB-class accelerator with a short context, or larger data-center GPU	KV cache, vision tower, runtime, and concurrency can push this beyond one card
Official FP8	28.75 GiB	32 GB GPU for shorter contexts; 48 GB or larger for comfortable serving	FP8 kernel support, KV cache, and framework workspace
Q8_0 GGUF	29.0 GB	32 GB GPU or 64 GB unified memory with a bounded context	Cache and application memory
Q6_K GGUF	22.9 GB	24 GB GPU for short context; larger memory for long context	Cache, vision input, and batching
Q5_K_M or dynamic Q5	19.3 to 20.2 GB	24 GB GPU or 32 to 64 GB unified memory	Cache and runtime headroom
Q4_K variants	16.1 to 17.9 GB	24 GB GPU, 32 GB unified memory, or larger	Cache, vision input, and application headroom
Q3 variants	11.9 to 13.8 GB	16 GB GPU for short sessions	Quality evaluation and cache capacity
Q2 variants	9.0 to 10.7 GB	12 to 16 GB GPU when memory is the hard limit	Larger quality risk and limited working memory

The sizes above come from the Qwen3.8 GGUF repository, so they describe that publisher's tensor choices and packaging. Other repositories may use different mixed-precision policies, split files, or metadata. Treat the file size as the first line of a capacity plan.

Quantization changes how the weights are stored, but the KV cache and the vision encoder still use memory. A 17 GB Q4 file plus a 16 GB FP16 cache already needs about 33 GB before framework overhead. On a 24 GB card, that means a shorter context, a smaller cache dtype, lower concurrency, or a lower-bit artifact.

Apple silicon deployment: unified memory and MLX

Apple silicon works well for local deployments because the CPU, GPU, and Neural Engine share unified memory. Apple lists the 2025 Mac Studio with M4 Max configurations up to 128 GB and M3 Ultra configurations from 96 GB up to 512 GB. The Apple specifications also list 546 GB/s memory bandwidth for the 40-core M4 Max configuration and 819 GB/s for M3 Ultra.

Recommended Apple configurations

• A 48 GB or 64 GB Mac is a workable starting point for a 4-bit model with a modest context budget and light application load.
• A 128 GB M4 Max handles 4-bit and 8-bit deployments with room for long documents, image inputs, and local tools.
• A 96 GB M3 Ultra can run BF16 weights, but the cache and application budget become tight as context grows. The 192 GB, 256 GB, or 512 GB configurations suit BF16 experiments and larger concurrent workloads.

The native MLX Community 4-bit build is listed at 16.1 GB. The 8-bit build is listed at 29.5 GB. Both model cards show mlx-vlm commands for image input, which matters because Qwen3.8-27B is a vision-language model. MLX's own quantization API supports affine 2-, 3-, 4-, 5-, 6-, and 8-bit weights, as well as MXFP4, MXFP8, and NVFP4 modes, but a particular Qwen artifact is only available when somebody has converted and validated it.

Install the MLX vision runtime and start with the published model path:

pip install -U mlx-vlm
python -m mlx_vlm.generate \
  --model mlx-community/Qwen3.8-27B-4bit \
  --max-tokens 512 \
  --temperature 0.0 \
  --prompt "Describe this image." \
  --image ./sample.png

For an OpenAI-compatible local endpoint, mlx_lm.server --model mlx-community/Qwen3.8-27B-4bit is documented in the model card and in the MLX LM server guide. The server is useful for development and local agents. The MLX guide says its basic server is not intended to provide production-grade security controls, so put authentication, authorization, rate limits, and network exposure outside it.

GGUF remains useful on a Mac when the same model file must run across Apple, NVIDIA, and AMD machines. llama.cpp enables Metal by default on macOS, supports Apple silicon through Metal and Accelerate, and exposes 1.5- to 8-bit integer quantization. Its build documentation and README describe that path. For Apple-specific work, native MLX 4-bit or 8-bit models are usually the simpler starting point, while GGUF is the portability option.

NVIDIA RTX deployment: CUDA, VRAM, and consumer quantization

Consumer RTX cards divide into distinct memory envelopes. NVIDIA lists 24 GB for the GeForce RTX 4090 and 32 GB for the RTX 5090. That difference changes which quantization and context budgets are comfortable.

RTX 4090 and other 24 GB cards

A Q4_K_M file around 17.1 GB leaves approximately 7 GB on a 24 GB card before the runtime, vision tensors, and cache. It works well for text-heavy sessions at 8K to 32K context, followed by a peak-memory test. Q3 files leave more room for the cache; Q5 and Q6 files preserve more weight precision at the cost of context capacity. The official FP8 file at 28.75 GiB does not fit on a 24 GB card without offload or sharding.

RTX 5090 and other 32 GB cards

The 32 GB envelope fits Q4 and many Q5 files with room to spare. Q6 at roughly 22.9 GB works for shorter contexts. The official FP8 checkpoint fits by file size, although a full context cache, vision input, and framework workspace can exceed the remaining memory. Blackwell also makes NVFP4 worth considering. TensorRT supports FP4, FP8, INT8, and INT4, and a community Qwen3.8 NVFP4 checkpoint is available. Validate the checkpoint and loader together before using it for an application.

Match the artifact to the CUDA runtime

For Hugging Face Transformers, vLLM, or SGLang, the official Qwen3.8-27B-FP8 checkpoint is a convenient low-precision reference. Qwen describes it as fine-grained FP8 with a block size of 128 and documents compatibility with Transformers, vLLM, SGLang, and TokenSpeed. vLLM's current quantization compatibility chart lists FP8 W8A8 for Ada and Hopper, AWQ and GPTQ for Turing through Hopper, and GGUF across NVIDIA GPU generations. The table changes as kernels change, so check the exact vLLM release and GPU architecture.

Use a serving engine for the safetensors artifacts:

vllm serve Qwen/Qwen3.8-27B-FP8 \
  --dtype auto \
  --max-model-len 32768

AWQ and GPTQ INT4 checkpoints reduce the weight footprint and use dedicated weight-only kernels when the GPU and loader support them. They work on 24 GB cards and on older NVIDIA generations. GGUF has the widest support across local apps. The GGUF inventory includes 2-, 3-, 4-, 5-, 6-, 8-, and 16-bit entries, with Q4_K_M, Q5_K_M, Q6_K, and Q8_0 as convenient points on the size and fidelity curve.

llama.cpp with its CUDA backend is a good fit for GGUF and for CPU-plus-GPU offload when the model exceeds VRAM. Two RTX cards do not combine into one 48 GB or 64 GB pool. The runtime must shard the model, and PCIe topology affects the result. Use tensor parallelism in vLLM or an equivalent multi-GPU setting, then measure prompt processing and decode separately:

vllm serve Qwen/Qwen3.8-27B \
  --tensor-parallel-size 2 \
  --dtype bfloat16 \
  --max-model-len 65536

Treat this command as a starting configuration. Lower --max-model-len until the server starts with the intended vision inputs and concurrency, then increase it in controlled steps.

NVIDIA DGX deployment: multi-GPU serving

A DGX is a data-center system, a different category from an RTX card. NVIDIA's DGX B200 guide lists eight B200 GPUs, 1,440 GB of aggregate GPU memory, and 14.4 TB/s of aggregate NVLink bandwidth. The DGX H100 reference architecture lists 640 GB of aggregate HBM3 across eight H100 GPUs.

You do not need a full DGX to hold the model. A single 80 GB-class H100 can fit the 54.7 GB BF16 weights and the estimated 16 GiB FP16 cache at native context, leaving little space for vision state, framework buffers, and concurrent requests. FP8 or tensor parallelism is the safer production choice. A B200 has more memory per GPU, so BF16 with long context is easier to place, and the eight-GPU NVLink fabric supports higher concurrency with lower communication overhead than a pair of consumer cards.

Serving engines and parallelism

Qwen's model card recommends current SGLang, vLLM, or TokenSpeed releases for production and high-throughput inference. TensorRT-LLM is the NVIDIA-specific path when the team wants built engines, Triton integration, and explicit parallelism controls. The TensorRT-LLM backend documentation covers tensor, pipeline, and expert parallelism. TensorRT's multi-device inference guide explains how NCCL distributes execution across GPUs.

For Qwen3.8-27B, tensor parallelism is useful for three reasons:

1. It creates room for the KV cache and multimodal buffers when one GPU is tight.
2. It raises the concurrency ceiling for a shared endpoint.
3. It keeps the model in a predictable memory envelope while the service grows batch size.

Pipeline parallelism becomes useful when the serving topology or model size warrants it. A 27B dense model rarely needs pipeline parallelism for weight capacity on DGX B200, but the runtime supports it for a larger deployment graph.

DGX quantization choices

• BF16: use it for a quality reference, calibration, or a workload where memory is abundant and reproducibility matters.
• Official FP8: use Qwen's fine-grained block-128 FP8 artifact with a runtime that has tested kernels for the installed GPU generation. It is a reasonable production starting point for throughput.
• INT8 W8A16: use it after measuring the target workload. TensorRT supports INT8; W8A16 is a weight-only path that can suit older accelerators or fidelity-sensitive deployments.
• INT4 and FP4: TensorRT supports these as lower-memory options, with INT4 described as weight-only and FP4 supported as a low-precision floating-point type. A Qwen-specific engine or checkpoint still needs validation for vision, thinking controls, and long context.
• AWQ, GPTQ, and GGUF: these are useful portability or consumer-GPU formats. They fit a DGX when the chosen engine supports them, but an engine-native FP8 or TensorRT path usually gives the operations team a clearer support boundary.

Keep the context limit explicit in the server configuration. A service that accepts 262K tokens on every request reserves a different cache budget from one that caps requests at 32K or 64K. Record prompt length, generation length, batch size, cache dtype, and image or video token counts in the capacity test.

AMD deployment: Radeon for local work, Instinct for servers

AMD deployments split into consumer Radeon and data-center Instinct. The software path is ROCm and HIP, and the right quantization depends on the GPU family and the exact runtime release.

Radeon RX 7900 XTX and similar cards

AMD lists 24 GB of GDDR6 on the Radeon RX 7900 XTX, and ROCm's current GPU table lists the card as gfx1100 with 24 GB. It is a practical Q4 GGUF target for short or medium contexts. Q3 gives more cache headroom, while Q5 and Q6 require a larger memory budget or a shorter context. The official FP8 file and Q8 GGUF file exceed the card's raw memory envelope before runtime overhead.

llama.cpp supports AMD GPUs through HIP, so the GGUF path is a simple cross-platform starting point. Build or install the HIP backend, load a Q4_K_M or Q5_K_M file, and begin with a bounded context. The application should report whether layers are on the GPU or CPU, because silent offload changes latency and host-memory use.

AMD Instinct MI300X

The MI300X is a different class of deployment. AMD lists 192 GB of HBM3 and 5.3 TB/s peak bandwidth per accelerator. An eight-GPU MI300X platform exposes 1.5 TB of HBM3. One MI300X has room for BF16 Qwen3.8-27B, the estimated native-context cache, and substantial serving workspace. Multiple accelerators raise throughput and support larger batches.

ROCm's vLLM optimization guide lists GPTQ at 4 or 8 bits, FP8 KV cache, AMD Quark FP8 and MXFP4, and compressed-tensors W8A8 INT8 or FP8. It also describes AWQ support on MI300 and MI350 GPUs. The generic vLLM compatibility table and ROCm-specific guidance do not always expose the same support boundary, so pin the ROCm, PyTorch, vLLM, and model versions in the deployment manifest.

For MI300X, start with BF16 or the official Qwen FP8 checkpoint through a ROCm-tested Transformers or vLLM path. Move to AMD Quark FP8 or MXFP4 when its kernels and model conversion are validated for the exact accelerator. Use GGUF through HIP when portability is the priority.

Which quantization should you choose?

Quantization is a deployment decision with three separate dimensions: how much memory the weights occupy, which kernels the runtime can execute, and how the model behaves on your workload.

Choice	Weight and activation pattern	Best first use	Main caveat
BF16 or FP16	16-bit weights and activations	Reference quality, fine-tuning, and high-memory servers	About 55 GB for weights before cache and buffers
Official FP8	Fine-grained 8-bit weights, block size 128	Ada, Hopper, Blackwell, and supported AMD data-center paths	Requires FP8-capable kernels and validation of vision and cache behavior
INT8 W8A16	8-bit weights with 16-bit activations	Ampere or other GPUs where native FP8 execution is unavailable	Loader and kernel support vary; it is not the same as W8A8
INT8 W8A8	8-bit weights and activations	Server inference with calibrated kernels	Calibration and activation support affect quality and speed
AWQ or GPTQ INT4	Weight-only 4-bit formats	Consumer NVIDIA cards and some dedicated loaders	Hardware compatibility is narrower than the file name suggests
GGUF Q4 to Q8	Portable 4- to 8-bit packed weights	Apple, NVIDIA, AMD, llama.cpp, and local desktop apps	The same label can hide different tensor mixes and calibration quality
MLX 4-bit or 8-bit	Apple-native quantized safetensors	Apple silicon with mlx-vlm	The artifact is tied to MLX; GGUF interoperability is separate
NVFP4 or MXFP4	4-bit floating-point weights with scale formats	Blackwell or AMD Instinct paths with validated kernels	Qwen-specific conversion and multimodal behavior need testing

For a first local test, use Q4_K_M or a native MLX 4-bit artifact on a consumer machine. Compare it with Q5 or Q6 if coding, tool arguments, long documents, or image grounding are sensitive to weight error. Use Q8 or FP8 as a reference when you have the memory. Keep the same prompts, context limit, sampling settings, and evaluation harness across every comparison.

A deployment sequence for realistic capacity planning

1. Choose the runtime first. Pick MLX for an Apple-native multimodal path, llama.cpp for portable GGUF, vLLM or SGLang for CUDA and ROCm server endpoints, or TensorRT-LLM for an NVIDIA engine workflow.
2. Select the matching artifact. Pair MLX with MLX safetensors, vLLM with the official Transformers or FP8 artifact, and llama.cpp with GGUF. Do not assume a format conversion preserves the vision tower or chat template.
3. Budget the full memory envelope. Add weight size, KV cache at the intended context, vision and video state, runtime workspace, operating system, and one or more concurrent sequences.
4. Start at 16K or 32K context. Run text, image, and video smoke tests. Increase the limit only after peak memory and latency are recorded.
5. Measure the workload. Capture time to first token, prompt processing rate, decode rate, peak memory, cache dtype, invalid tool calls, and completed workflow rate.
6. Test the quantization against a higher-precision reference. A low KLD or a fast token rate does not establish tool reliability, JSON validity, visual grounding, or long-context recall.
7. Put authority in the application. Validate tool arguments, enforce permissions and tenant boundaries, require confirmation for irreversible actions, and retain an audit trail.

The practical recommendation

Choose the platform from the context and serving requirement:

• Apple silicon: MLX 4-bit on 64 GB is a reasonable starting point; 128 GB or M3 Ultra memory tiers suit 8-bit and long-context work.
• NVIDIA RTX: Q4 GGUF or AWQ/GPTQ INT4 fits 24 GB cards; 32 GB cards add Q5, Q6, FP8, and NVFP4 options with different context limits.
• NVIDIA DGX: BF16 is a useful reference; official FP8 is the first throughput configuration; TensorRT-LLM or vLLM tensor parallelism handles concurrency and cache pressure.
• AMD: Q4 GGUF through HIP is a workable Radeon path; MI300X supports BF16, FP8, INT8, and AMD Quark options in a larger HBM envelope.

The next step is a smoke test with one text task, one image task, and one tool call. Keep the model version, artifact, runtime, context cap, cache dtype, and generation settings fixed. The test will tell you whether the bottleneck is weights, cache, kernels, or the surrounding application.

Sources

• Qwen3.8-27B official model card (accessed August 15, 2026)
• Qwen3.8-27B official FP8 checkpoint (accessed August 15, 2026)
• Qwen3.8-27B GGUF quantization inventory (accessed August 15, 2026)
• MLX Community Qwen3.8-27B 4-bit and 8-bit checkpoints (accessed August 15, 2026)
• Apple Mac Studio technical specifications (accessed August 15, 2026)
• MLX LM and MLX quantization API (accessed August 15, 2026)
• llama.cpp README and build documentation (accessed August 15, 2026)
• vLLM quantization and hardware compatibility (accessed August 15, 2026)
• NVIDIA RTX 4090 specifications and RTX 5090 specifications (accessed August 15, 2026)
• NVIDIA DGX B200 User Guide and DGX H100 reference architecture (accessed August 15, 2026)
• NVIDIA TensorRT quantized types and multi-device inference (accessed August 15, 2026)
• AMD Radeon RX 7900 XTX specifications, ROCm GPU specifications, and AMD Instinct MI300X platform (accessed August 15, 2026)
• ROCm vLLM inference optimization and quantization guidance (accessed August 15, 2026)