Eagle vs DFlash: Best speculative decoding methods

Modern speculative decoding methods differ in how much target-model state the drafter can reuse and how much sequential work remains in drafting. Those choices determine whether the method can turn accepted draft tokens into lower end-to-end latency.

For a compatible target model and serving stack, DFlash is the better first experiment when the objective is maximum decode speed. Its block-diffusion drafter predicts a whole block in parallel and injects target-model features into every draft layer. EAGLE-3 is the better default when integration breadth, an established tree-drafting design, or a familiar training path matters most. Paper results cannot supply a universal production multiplier. Choose the method with the lowest end-to-end latency on your exact target checkpoint, context distribution, concurrency, hardware, quantization, and decoding policy.

Target-model decoding is serial

During ordinary autoregressive decoding, the target model produces one token, appends it to the sequence, and runs again. A token at position (t+1) cannot be produced until the token at position (t) is known. The GPU may execute the matrix multiplications efficiently, but the application still pays a sequence of synchronization points.

Speculative decoding changes the division of labor. A cheap drafter proposes (gamma) future tokens. The target model evaluates the whole proposal in one forward pass, then accepts a prefix of it. If the target accepts several draft tokens, one target invocation advances the sequence by several tokens at once.

The method is lossless in a precise sense. Let (p_i) be the target distribution for draft position (i), and let (q_i) be the drafter distribution. Under the standard rejection-sampling rule, a proposed token (x_i) is accepted with probability

min(1, p_i(x_i) / q_i(x_i))

If a token is rejected, the system samples a replacement from the normalized residual distribution max(0, p_i - q_i), and discards later draft tokens. This preserves the target distribution under the assumptions of the algorithm. The claim is not that every draft token is correct, nor that the accelerated run is bit-for-bit identical to one particular vanilla run.

The EAGLE papers describe this procedure and its distribution-preservation guarantee in detail in their speculative sampling preliminaries and original EAGLE method. The target model remains the authority. The drafter affects how much work the target can verify at once.

A useful performance model

Accepted tokens per draft-and-verify cycle determine the useful gain. Draft-token count alone does not.

Let:

• (T_d) be drafter latency;
• (T_v) be target verification latency;
• ( au) be the expected number of accepted tokens, including the target’s bonus token; and
• (L = (T_d + T_v)/ au) be average latency per generated token.

The resulting speedup is the vanilla target-model token latency divided by (L). This model comes directly from the DFlash paper’s speedup analysis.

That equation explains why “draft 16 tokens” is not enough information. Increasing the speculation budget can raise ( au), but it can also increase verification work, memory use, tree size, or rejection waste. A drafter that accepts 8 tokens but costs almost as much as the target can lose to a drafter that accepts 5 tokens at a fraction of the cost.

The hardware regime matters too. Speculative decoding is most attractive when the target is memory-bandwidth-bound and the workload has low or moderate batch size. At high concurrency, the target is already using more of the GPU’s parallel capacity, and the extra draft model and verification bookkeeping can reduce or erase the gain. Measure throughput and tail latency separately from single-request token speed.

How EAGLE works

EAGLE means Extrapolation Algorithm for Greater Language-model Efficiency. The name covers a family of implementations. The relevant production comparison is usually EAGLE-3, with EAGLE-2’s dynamic draft-tree logic included in the inference design.

EAGLE-1: predict target features, then recover tokens

Vanilla speculative decoding uses a separate small language model that predicts tokens autoregressively. EAGLE instead reuses representations from the frozen target model, especially the feature immediately before the target language-model head. A small autoregression head predicts a future feature; the target model’s own LM head converts that predicted feature into a token distribution.

The drafter also receives a token sequence shifted one position forward. This matters because a sampled token determines which feature should be predicted next. Without the shifted token, two different sampled continuations can look identical at the feature input and create uncertainty in the next predicted feature.

EAGLE’s draft is tree-shaped. Several candidate continuations can share a prefix, and target verification uses a tree attention mask so tokens from one branch cannot see tokens from another. The tree lets the verifier explore alternatives without requiring a separate full target pass for every branch. The EAGLE paper’s drafting description gives the feature, token, and tree relationships.

EAGLE-2: spend the tree budget where confidence is higher

EAGLE-1 uses a fixed draft-tree shape. That wastes nodes when the next token is obvious and may underspeculate when the context is ambiguous. EAGLE-2 uses draft confidence as a proxy for acceptance probability, expands promising nodes, and reranks candidates by a path value before verification.

The resulting tree is context-dependent. A high-confidence prefix can remain narrow and deep. An uncertain position can receive more branches. The verifier still needs an ancestor-only attention mask, and the selected nodes must form a connected tree so the target evaluates valid causal paths. EAGLE-2 describes the expansion, reranking, and mask construction.

EAGLE-2 changes how the verification budget is allocated across a dynamic tree. Simply increasing draft length only raises the maximum size.

EAGLE-3: remove the feature-prediction bottleneck

EAGLE-3 changes how the drafter is trained. The original feature-prediction objective constrains the draft output to resemble a target hidden feature, even though the final objective is to predict tokens that the target will accept. EAGLE-3 removes that feature-prediction constraint and predicts tokens directly.

The input combines low-, middle-, and high-level target features. A fully connected layer fuses the selected representations into the target hidden size. The draft decoder then predicts a token, samples it, and feeds the sampled result into the next draft step.

The training problem is that inference feeds the drafter’s own sampled tokens back into itself. A normal teacher-forced training sequence does not reproduce that condition. EAGLE-3’s “training-time test” simulates draft steps during training, feeding predicted outputs into later draft positions and using attention masks that match the inference dependency pattern. That reduces the train-test mismatch and lets the drafter benefit from more training data.

The trade-off is structural: EAGLE-3 still drafts autoregressively at the feature or token-step level. A tree can produce many candidate nodes with a small number of sequential draft passes, but its drafting path still has sequential dependencies. The official paper reports up to 6.5x speedup in its own experiments, with results varying by target, task, temperature, and runtime. For example, its LLaMA 3.1 8B results average 4.44x at temperature 0 and 3.45x at temperature 1 across the listed tasks. These paper measurements carry no general service-level guarantee. The EAGLE-3 results table shows the full comparison.

How DFlash works

DFlash attacks the sequential part that remains in EAGLE. It uses a lightweight block-diffusion drafter that predicts the masked positions in a token block in parallel, while the target model verifies the resulting block.

Target features are persistent conditioning, not a one-time hint

At the start of a speculative cycle, the target model has already processed the prompt or accepted prefix. DFlash extracts hidden representations from several target layers, concatenates them, and projects them into a compact conditioning representation.

The reference configuration extracts features from five target layers selected between the second layer and the third-to-last layer. Treat these as configuration choices. The paper’s Qwen3 experiments use a five-layer draft and a block size of 16; LLaMA 3.1 uses a block size of 10, and Qwen3 Coder uses eight draft layers.

The crucial mechanism is KV injection. DFlash projects the fused target feature into the key and value projections of every draft layer and stores those projections in the draft KV cache. This keeps target context available throughout the draft network; a one-time input concatenation can fade in deeper layers. The paper identifies this persistent conditioning as a reason acceptance length continues to scale with draft depth. DFlash’s inference section describes the feature extraction and KV injection.

Block diffusion removes sequential draft passes

The drafter starts with a clean anchor token supplied by the accepted target prefix and masked positions for the future block. It predicts the masked positions together in one forward pass. Within the block, positions can use bidirectional information permitted by the diffusion-style mask; across blocks, attention is restricted so the drafter does not use future information that would be unavailable during target verification.

This changes the draft-cost term in the latency equation. For an autoregressive drafter, (T_d) grows roughly with (gamma), the number of speculative positions. For a block-diffusion drafter, the paper models (T_d) as the cost of one parallel block-generation pass. The pass still becomes more expensive as the block or draft network grows, but its cost is much less tied to the number of tokens produced.

The paper reports that a five-layer DFlash drafter generating 16 tokens can have lower drafting latency and higher acceptance length than EAGLE-3 with an 8-token draft in its measured setup. That is the fundamental reason DFlash can move the speed frontier: it can spend more capacity on draft quality without paying one sequential forward pass per drafted token.

DFlash is trained for the verification loop

DFlash training freezes the target model and trains the draft layers to imitate target-generated responses under block-masked conditions. It shares the target’s token embedding and LM head, leaving those components frozen and updating the lightweight draft transformer layers.

The training data is arranged around random clean anchor tokens. The remaining positions in a block are masked, which matches inference where the previous accepted target token is known but future draft tokens are not. Multiple blocks can be processed with a sparse attention mask in one training pass.

The loss is position-weighted. An error at the first position of a block prevents later positions from being accepted, so DFlash uses an exponentially decaying weight (w_k = exp(-(k-1)/gamma)). Early draft positions receive more weight than later positions. This is a practical alignment between the training objective and the verification process, where the first rejection terminates the accepted prefix.

EAGLE vs DFlash at a glance

Criterion	EAGLE-3	DFlash
Draft generation	Autoregressive draft steps, often organized as a tree	Parallel block-diffusion draft
Target-model information	Fused multi-layer target features	Fused target features injected into every draft layer’s KV projections
Verification	Target verifies a chain or tree with causal/tree attention	Target verifies a block in parallel with the normal speculative acceptance rule
Main strength	Mature tree design, broad ecosystem support, adaptable draft shapes	Lower drafting overhead and a higher speed ceiling when supported
Main coupling	Must match the target family’s features, tokenizer, and checkpoint contract	Must match target features, layer selection, tokenizer, checkpoint, and DFlash drafter weights
Main tuning knobs	Tree shape, expansion width, max draft length, temperature	Block size, draft depth, selected target layers, KV injection, serving overlap
Failure mode	Sequential draft cost and error accumulation reduce gains at longer speculation	Target-specific implementation and draft-model availability can block adoption
Best initial setting	When support and integration risk dominate	When a compatible DFlash drafter exists and decode latency dominates

Both methods need a target-specific contract. A drafter trained for one target family is not a drop-in replacement for another model. The tokenizer must match, hidden sizes and layer semantics must match, and the serving runtime must capture and route the target features expected by the drafter. NVIDIA’s TensorRT-LLM speculative-decoding documentation explicitly lists tokenizer compatibility for draft-target decoding and target-layer configuration for DFlash.

Where DFlash is the better choice

Choose DFlash first when all of these conditions are true:

• The exact target checkpoint has an available or trainable DFlash drafter.
• Your runtime supports the required DFlash path and target-feature capture.
• You serve long enough responses that draft latency is a material part of total decode time.
• Your hardware benefits from parallel block work and has enough memory for the target and draft KV state.
• You can benchmark at the concurrency and quantization level you will actually deploy.

The published Qwen3 results illustrate the upside. In the DFlash paper’s temperature-0 table, DFlash on Qwen3-8B with a block size of 16 averages 4.86x speedup and 6.49 accepted tokens per cycle across the listed math, code, and chat tasks. The compared EAGLE-3 configurations average 1.76x and 2.02x with 16- and 60-node settings. The same paper reports DFlash speedups of roughly 4.5x and 3.9x for Qwen3 reasoning models with thinking enabled.

Those numbers describe specific Qwen3 checkpoints, datasets, decoding settings, and hardware. They cannot predict your service. The serving-framework tests use a single NVIDIA B200 with SGLang and FlashAttention-4, and the reported speedup falls as concurrency rises in several rows. DFlash’s experiment tables and serving setup defines the scope.

Where EAGLE-3 is the better choice

Choose EAGLE-3 first when DFlash’s target-specific dependencies are not available or when you value an established integration path. The EAGLE project lists integrations across serving and acceleration systems including SGLang, vLLM, TensorRT-LLM, NVIDIA NeMo, MLC-LLM, AMD ROCm, AWS Neuron, and others. Its official repository also includes EAGLE-2 and EAGLE-3 checkpoints, training guidance, and model-specific support information.

EAGLE-3 is also a reasonable choice when dynamic trees suit the workload. A tree can allocate candidate budget unevenly: narrow on predictable syntax, wider when the next token is uncertain. That can be useful for code and structured output, where local patterns often make the first few continuations easy but branch later around identifiers, arguments, or values.

EAGLE-3 still delivers substantial acceleration in its paper, and its direct-token, multi-layer design is materially stronger than a generic small draft model. Its drafting stage retains sequential work. DFlash’s parallel block generation removes more of that cost when the hardware and target-specific integration can exploit it.

The limitations that decide a production rollout

Long context can reduce acceptance

The draft model is trained for a context distribution. DFlash’s paper reports that a base drafter trained on 4K contexts degrades beyond that range, while lightweight long-context fine-tuning improves acceptance through 32K on the tested tasks. A long context window in the target model does not automatically mean the drafter will remain well calibrated at that length.

For a retrieval-heavy agent, benchmark short, medium, and worst-case contexts separately. Include the actual system prompt, tool schemas, retrieved records, previous tool results, and conversation history. The acceptance behavior of a clean benchmark prompt may not represent the context assembled by your product.

Quantization changes the bottleneck

Quantizing the target can make verification much faster. That is good for baseline latency but can make the drafter a larger fraction of the total cycle. A DFlash configuration that wins with a high-latency target may regress after the target is quantized. The same issue can appear when the target and drafter use different precisions or when feature capture requires conversions.

Measure the target-only baseline, draft time, verification time, feature-extraction time, KV-cache allocation, and synchronization overhead. Do not infer the result from model parameter counts.

“Lossless” does not mean “free” or “always faster”

Exact acceptance correction protects the target distribution, but it does not eliminate the compute used to produce rejected drafts. At low acceptance, the target still pays verification cost and the drafter adds overhead. At high concurrency, scheduling and memory pressure can make the speculative path slower than ordinary decoding.

The acceptance length ( au) must be measured together with wall-clock latency. A high acceptance rate can coexist with poor speedup if the drafter is too expensive. A modest acceptance length can still win if the drafter is extremely cheap.

Target dependence limits portability

EAGLE and DFlash both use target internals, but DFlash makes that dependency especially explicit. Its drafter consumes selected target hidden layers and uses target-compatible embeddings and LM-head behavior. A target-model upgrade can require a new drafter checkpoint, layer mapping, feature projection, or training run.

Treat the drafter as part of the target-model release artifact. Version the pair together, pin the exact model revision, and invalidate the pairing when tokenizer, vocabulary, hidden size, layer layout, or forward-pass outputs change.

Production frameworks are part of the method

Paper pseudocode is only one part of the implementation. The runtime must support paged KV caches, tree or block attention, feature capture, CUDA graph or compilation behavior, scheduling, cancellation, batching, and fallback to ordinary decoding. The DFlash repository documents Transformers, SGLang, vLLM, and MLX installation paths and currently states that vLLM 0.20.1 and later includes core DFlash support for most models. Verify that statement against the runtime release you intend to ship. DFlash’s implementation repository contains the current support matrix and model list.

A fair evaluation protocol

Start with one target checkpoint and one serving runtime. Establish a vanilla autoregressive baseline with the same tokenizer, precision, sampling parameters, max output length, prompt set, and stopping rules.

Then evaluate EAGLE-3 and DFlash with the same workload matrix:

1. Decoding policy: greedy or temperature 1, top-p, repetition controls, stop tokens, and reasoning mode.
2. Context length: short, median, p95, and worst-case assembled context.
3. Output length: short responses and long generations. Speculation has more opportunity to amortize overhead on long outputs.
4. Concurrency: single request, expected operating concurrency, and overload conditions.
5. Hardware and precision: the exact GPU, tensor parallelism, target precision, draft precision, and attention backend.
6. Runtime behavior: continuous batching, prefix caching, scheduling overlap, cancellation, and fallback.

Record at least:

• time to first token;
• time per output token and p50/p95/p99 tail latency;
• end-to-end tokens per second;
• target verification time;
• draft time, feature-capture time, and synchronization time;
• accepted tokens per cycle and acceptance by position;
• rejected-token rate and number of discarded suffixes;
• GPU memory, utilization, and inter-GPU traffic;
• throughput at each concurrency level; and
• fallback frequency and error rate.

Use the same prompt traces for both methods, including tool-heavy and structured-output cases if the target serves an agent. For an embedded product, add workflow metrics: time until a verified final state, tool-call validity, retries, user edits, and approval wait. Faster token emission can still produce slower product completion once those costs are included.

Run correctness tests independently from performance tests. For greedy decoding, compare the accepted output against a target-only run under identical state. For stochastic decoding, verify that the implementation uses the exact acceptance and residual-sampling rule; a confidence threshold alone is insufficient. Test stop-token handling, rejected prefixes, batch padding, cache reuse, cancellation, and fallback after a draft or feature-capture error.

A practical rollout sequence

First, make the model pair work without optimization. Load the exact target and drafter revisions, confirm tokenizer identity, run a short prompt, inspect the target-layer tensors or feature metadata, and verify that a rejected draft falls back to the target sampler.

Next, profile a small fixed trace. Tune block size for DFlash or tree size and expansion policy for EAGLE-3. Start with a conservative speculation budget. A large budget can increase memory and verification cost before it improves accepted length.

Then test long contexts and real concurrency. The single-request result should not be the release gate. A method that is 5x faster at concurrency 1 but 1.2x slower at the service’s p95 concurrency is not the better production method for that service.

Finally, add fallback. If the draft checkpoint is unavailable, target feature extraction fails, memory pressure crosses a threshold, or acceptance collapses for a request class, route the request to ordinary target decoding. Log the reason. Speculative decoding is an optimization layer; it should not become a correctness dependency.

Where Ginger Labs fits

For a customer-facing product, the hard part is turning the customer’s API and data model into a responsive, reliable agent experience. That includes retrieval, tool execution, state management, model routing, evaluation, observability, and the serving decisions that determine latency and cost. A speculative decoder optimizes one part of that backend.

At Ginger Labs, the embedded agent or copilot can live in a SaaS product’s side panel, inline surface, or modal and reason over the customer’s schemas, stages, records, and data to progress defined multi-step work. The implementation fits the product’s existing API, data model, permissions, and workflow rules, keeping the inference system behind a product capability. The SDK includes retrieval, evaluations, self-learning loops, and observability.

Keep the backend flexible enough to evaluate EAGLE-3, DFlash, or another serving optimization against the same product workflow. The team can then adjust the model, drafter, runtime, context strategy, and fallback behavior as the workload changes. Optimize responsiveness and cost per verified result for the actual product; no method will remain optimal for every workload.

The customer still owns the product API and data model, domain rules, tenant boundaries, user permissions, allowed actions, approval policy, customer experience, and definition of a correct result. Ginger Labs handles the agent implementation and the surrounding backend optimization; the customer’s product remains the authority for what the agent may do and what counts as correct.

If the product also needs to expose selected capabilities to external AI clients, Ginger Labs’ managed MCP server is a separate distribution option. It can take on the MCP infrastructure while the customer continues to choose the exposed tools and govern access. MCP does not make every tool compatible with every client, and it does not replace server-side authorization or tenant isolation.

For the workflow contract that should surround any inference optimization, see What Matters Most When Building AI Agents for Business Workflows. For the product-side rollout sequence, see How to Turn Your SaaS Into a Customer-Facing AI Agent.

Recommendation

Use DFlash as the leading candidate when the target-specific drafter exists, your runtime supports it, and decode latency is the bottleneck. Its parallel block drafting and persistent target-feature conditioning address the main weakness of autoregressive speculation: the drafter itself remains sequential.

Use EAGLE-3 when you need the more established ecosystem, a supported target family without a DFlash checkpoint, dynamic tree allocation, or a training path your team already understands. Those are primary selection criteria and make EAGLE-3 a strong method in its own right.

If both are available, do not choose by the largest number in either abstract. Run the same target, trace, hardware, runtime, concurrency, and sampling policy through both. Choose the method with the lowest tail latency and cost per verified output, while preserving exact target behavior and a clean ordinary-decoding fallback.

For an embedded agent, the final metric is even more concrete: time from the user’s request to a verified product result. A speculative decoder is successful when it improves that number without changing who is allowed to act, what the product considers correct, or how the user can inspect the work.

Sources

• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty, Li et al. Accessed August 6, 2026.
• EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees, Li et al. Accessed August 6, 2026.
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test, Li et al. Accessed August 6, 2026.
• DFlash: Block Diffusion for Flash Speculative Decoding, Chen, Liang, and Liu. ICML 2026 camera-ready version. Accessed August 6, 2026.
• DFlash implementation and supported models, z-lab. Accessed August 6, 2026.
• TensorRT-LLM speculative decoding, NVIDIA. Accessed August 6, 2026.