Tokens per second (throughput)

Also: throughput · tok/s

Tokens per second is the steady-state speed at which a model generates output during decode, after the first token. For a single stream it is bounded by memory bandwidth (MBU); total system throughput rises with batching, which is how providers serve many users on one chip economically.

← All terms · Definitions maintained by Mintok. Methodology.