Why Batching Bends My Brain

Batching is one of those ideas that seems obvious until you try to reason about it carefully. The idea is that instead of running the model once per request, you group requests together and run them at the same time. More work per pass. Better use of the hardware. Faster overall.

But the deeper I go, the weirder it gets.

The first thing that confused me is that batching is basically free from the model side. A batch of eight sequences takes almost the same time as a batch of one, up to a point. That is because the GPU spends most of its time moving weights around, and once the weights are loaded, running them on more data does not cost much extra. This is the part that makes batching look magical.

The second thing that confused me is that batching in inference is not the same as batching in training. In training everyone is doing the same thing at the same time, so grouping is natural. In inference, each user is at a different point in their sequence. Some are just starting. Some are near the end. Some are short. Some are long. Grouping them cleanly is much harder than it sounds.

Continuous batching, from what I understand, is the trick that makes this work in practice. Instead of waiting for a batch to finish, you swap requests in and out as they finish. New ones join. Old ones leave. The batch is always full, more or less. This feels obvious in hindsight but I have no idea how I would have thought of it on my own.

The part I still do not fully grasp is how batching interacts with latency. A larger batch is more efficient, but a request in a larger batch might wait longer before it starts. If you want low latency for one user, sometimes you want a smaller batch. If you want high throughput across all users, you want a bigger one. The optimal choice depends on what you are actually optimizing for, and that answer is rarely obvious.

I also keep running into the fact that batching changes memory patterns in ways I did not expect. Bigger batches need more memory for activations. Longer sequences push you into different regimes entirely. Sometimes adding one more request to a batch makes everything slower because of a cache eviction I did not see coming.

I think the honest summary is that batching taught me that inference is not really about the model. It is about scheduling. It is about figuring out which pieces of work to do together, in what order, on which hardware. The model is almost a detail inside a larger scheduling problem. That is not how I used to think about it.