Prefill vs decode

Also: prefill · decode · inference phases

Inference has two phases. Prefill processes the whole prompt in parallel and is compute-bound (high MFU); decode then generates output tokens one at a time and is memory-bandwidth-bound (sets MBU and latency). Because their bottlenecks differ, input and output tokens are priced — and should be sized — differently.

← All terms · Definitions maintained by Mintok. Methodology.