Trying to Understand Inference From Scratch

I have been reading a lot about inference over the last few weeks and honestly I still get tripped up on things that should probably be simple. So I want to write down what I think I understand right now, mostly so I can look back later and see where I was wrong.

For a long time I thought of inference as a single event. You give the model a prompt, it thinks for a second, it gives you an answer. That was my mental model. It turns out that is not really what is happening.

What is actually happening is a long chain of small decisions. Every token the model outputs is its own forward pass. The model looks at everything so far, produces a distribution over possible next tokens, picks one, and does the whole thing again. It keeps going until it decides to stop.

Once I started seeing it that way, a lot of the engineering around inference started to make more sense. Why is latency so hard to reduce? Because you are not making one thing faster. You are making hundreds of forward passes faster, one after the other, and each one depends on the last.

The other thing that surprised me is how much of inference is memory-bound rather than compute-bound. I always assumed the bottleneck was math. But most of the time the GPU is waiting on weights and cached values to move around. The math itself is fast. The bookkeeping is what slows things down.

I think this is why a lot of clever inference tricks are less about doing the math faster and more about doing less of it, or reusing what you already computed. Caching, batching, and speculative decoding all seem to share this idea. The compute is expensive, so avoid repeating yourself when you can.

I still do not fully understand why some optimizations work in one setting and not another. Batch sizes, sequence lengths, and prefill versus decode all seem to interact in ways I have not internalized yet. I keep trying to build intuition by reading benchmarks and writing tiny scripts, but I have a feeling this is one of those things that only clicks after seeing it break a few times in production.

For now I am just trying to be careful about my assumptions. Inference is not one thing. It is a pipeline of small steps, and each step has its own personality. The more I treat it that way, the less confused I feel.