Dpine

Draft in Parallel, Condition Through Depth

Adjacent Causal Injection for Speculative Decoding

Haohui Zhang^1,2†, Keyu Chen^2†§, Haocheng Sun³, Bo Ke², Weibo Gu², Ruizhi Qiao², Xing Sun², Bo Jiang^1*

¹Shanghai Jiao Tong University²Tencent YouTu Lab³Xiamen University

† Equal contribution § Project lead * Corresponding author

DSpine drafts are accepted longest on every benchmark. Mean acceptance length with Qwen3-8B on seven math, code, and chat benchmarks at temperatures 0 and 1; the strip gives DSpine's margin over the strongest baseline. At temperature 0, DSpine raises the mean from DFlash's 3.77 to 4.82 (+27.8%).

Where does the predecessor come in?

A parallel drafter predicts a whole block in one pass, so each position commits before its predecessors do. DSpine passes each predecessor's prediction into its successor at every layer, and reuses the same gate once the token is decoded.

Verified context “… the flight from Boston lands in”, then draft three tokens

Parallel drafting

DFlash

Predecessor: never used

Condition through depth

DSpine

Predecessor: from layer 1 on

Serial post-backbone head

Domino · DSpark

Predecessor: after the last layer

Gated injection of the predecessor's prediction Conditioning on the decoded predecessor Transition cache: all K×K pair scores in one parallel pass

Decode bars are to scale with the measured post-backbone head time per round (Qwen3-8B, SGLang): Domino 0.97 ms and DSpark 0.88 ms (bar at their mean), DSpine 0.36 ms.

Abstract

Parallel speculative drafting generates multiple candidates in one backbone pass, but independent token selection can produce inconsistent continuations that shorten the accepted prefix. Existing methods mostly leave conditional decoding to a lightweight module after the backbone, which limits the flow of predecessor information to successors. Our analysis of DFlash shows that early positions already form recoverable predictions in shallow layers, and that accurate adjacent predecessors help successors more when they enter earlier.

We therefore propose DSpine, a drafter with causal conditioning injection throughout the backbone: at every layer, gated adjacent injection writes each predecessor's predicted feature into its successor, so the causal conditioning chain unfolds over network depth while all positions update in parallel. A unified transfer space built from the target model's output embeddings unifies layer-wise injection with predecessor-conditioned decoding, and layer-wise output-embedding supervision promotes the formation of predicted features in shallow layers. Fused kernels and a transition cache execute both efficiently in parallel within SGLang.

Across seven math, code, and chat benchmarks, DSpine achieves the longest acceptance length at both temperatures on Qwen3-4B and Qwen3-8B. At temperature zero on Qwen3-8B, it raises the seven-benchmark mean from DFlash's 3.77 to 4.82 (+27.8%); in SGLang serving tests, it delivers 23.3% higher throughput than DFlash on average.

Motivation

We probe the official Qwen3-8B DFlash drafter, using the target's greedy continuations as the correct tokens.

Three panels: readout accuracy by layer and block position; gain from correct prefixes by layer; gain in successor accuracy when the adjacent predecessor comes from the correct token at different layers. — (a) Readout accuracy of each layer by block position. (b) Gain in consecutive correct tokens when the drafted prefix is replaced by the correct prefix at the filled layers. (c) Gain in successor accuracy when the adjacent predecessor's features come from the correct token at the marked layers.

Earlier positions form predictions at shallower depth. At the same shallow layer, early block positions are read out far more accurately than distant ones, so a prefix could condition successors that are still forming. Correct prefix features from an early layer on raise consecutive correct tokens by 6.7%, and moving the same number of injections earlier lifts the gain from 1.2% to 4.3%.

The adjacent predecessor matters, and so does when it arrives. With correct earlier history, swapping only the predecessor for its draft prediction lowers successor accuracy by 31.8 points; supplying the same correct predecessor earlier rather than later gains 14.8 points.

Method

One injection carries predecessor information from the first layer to the last write.

DSpine overview: (a) injection passes predecessor features to successors at every layer, and transition-cached decoding selects each token conditioned on its selected predecessor; (b) one injection layer with gate, message, and write-back. — (a) Injection passes predecessor features to successors at every layer (green); after the final layer, transition-cached decoding selects each token conditioned on its selected predecessor (orange). (b) One injection layer: predecessor and successor features jointly gate how much of the predecessor's message enters the successor's residual stream.

Unified transfer space. Hidden states mix a predecessor's prediction with its context, so DSpine passes the prediction itself, in a fixed space built from the target LM head's output embeddings. An undecided predecessor sends a predicted feature; a decoded token enters as its embedding. Both take the same form, so one pathway carries either.

Adjacent causal injection. After every layer, each position receives a gated message from its immediate predecessor only. With block-causal attention, the dependency chain unfolds over depth while every layer still updates all positions in parallel.

Predecessor-conditioned decoding. Once a predecessor's token is decoded, its embedding replaces the predicted feature and the last write is redone, with no new module. A transition cache scores all K×K candidate pairs in parallel, so selection becomes a lookup.

Layer-wise output-embedding supervision. Each layer's predicted feature is aligned by cosine with the correct token's embedding, which helps features form in shallow layers and keeps them close to the embeddings used in decoding.

Results

All drafters are retrained on the same ShareGPT data with the same budget (five layers, block size 16, six epochs) and evaluated in SGLang on GSM8K, MATH-500, HumanEval, MBPP, LiveCodeBench, MT-Bench, and Arena-Hard.

	Math		Code			Chat
Method	GSM8K	MATH-500	HumanEval	MBPP	LCB	MT-Bench	Arena-Hard	Avg.

Serving throughput in SGLang, as the gain over DFlash (k tokens/s, temperature 0). The dashed zero line is DFlash; shading and the strip give DSpine's lead over DFlash2, the strongest baseline. DSpine is fastest in all 40 settings; on Qwen3-8B it averages +23.3% over DFlash and +11.1% over DFlash2, reaching up to 4.0× over autoregressive decoding.

Decoding time breakdown for Qwen3-8B. Injection adds at most 0.15 ms per round, and transition-cached decoding costs less than half the post-backbone heads of DSpark and Domino; DSpine spends 10.9% less time per generated token than DFlash2.

BibTeX

@article{zhang2026dspine,
  title   = {Draft in Parallel, Condition Through Depth: Adjacent Causal Injection for Speculative Decoding},
  author  = {Zhang, Haohui and Chen, Keyu and Sun, Haocheng and Gu, Weibo and Qiao, Ruizhi and Sun, Xing and Jiang, Bo},
  journal = {arXiv preprint arXiv:2609.36173},
  year    = {2026}
}