Quantization (served precision)

Also: precision · fp8 · int4

Quantization represents a model’s weights and activations at lower numeric precision (FP16 → FP8 → INT4) to cut memory footprint and speed inference, usually with some quality trade-off. Because a provider’s served precision affects both cost and quality, it is worth knowing what precision a hosted model actually runs at.

← All terms · Definitions maintained by Mintok. Methodology.