Speculative decoding accelerates autoregressive generation by letting a faster draft model propose multiple tokens that a target model verifies in parallel without changing the target distribution.

Geng et al., Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning (NeurIPS 2023) — https://arxiv.org/abs/2305.13971. Applies the per-step valid ...

Cache-based speculative decoding: draft a short continuation by looking for a matching suffix in the committed token history, then let the target model verify the whole draft before any token becomes ...