The T4 was not a product anyone got excited about. It arrived in 2018, drew about as much power as a bright lightbulb, and slotted into the kind of server a company already owned, without new cooling, new power supplies or a rebuilt room. Alongside NVIDIA’s large accelerators, which had begun consuming several hundred watts each and demanded infrastructure to match, it looked like the cautious option.
It was, in hindsight, the piece of hardware that had correctly guessed where the money in artificial intelligence would actually go.
Two different kinds of expense
There are two things a computer does with a machine-learning model, and the difference between them is the whole story.
The first is training: the process of building the model by pushing enormous quantities of data through it until its internal settings produce useful answers. Training is expensive, spectacular, and finite. It happens on thousands of processors, it runs for weeks, and then it stops. What comes out is an asset.
The second is inference: using the finished model to answer an actual question. Classify this photograph. Translate this sentence. Recommend the next thing to watch. Each answer costs a small amount of computing, and unlike training, it never stops. Every user, every query, every day, for as long as the service exists.
One is a construction project. The other is a utility bill that arrives forever.
For the first decade of modern machine learning, almost all the attention went to the construction project, because that was where the research happened and the records were broken. But a company running an AI service discovers quickly that the model it spent a fortune to build is not the expensive part. The expensive part is answering.
This is a genuinely awkward fact for the technology industry, because it reverses the economics that made software such an extraordinary business in the first place. A copy of a spreadsheet program costs nothing to deliver to the ten millionth customer, which is why software companies have historically earned margins that manufacturers can only envy. An AI answer is not like that. It consumes electricity and a slice of an expensive machine every single time, which means costs rise with popularity rather than flattening against it. Success gets more expensive as it arrives.
What that does to the value of hardware
Once inference dominates the budget, the question you ask of a processor changes.
Peak speed stops being the number that matters. Cost per answer takes its place, and cost per answer depends on things that never appear in a headline. How much power the card draws. How many fit in a rack before the room runs out of cooling. Whether it works in the building you already have. How well it copes with a workload that arrives unevenly through the day instead of as one enormous scheduled job.
A smaller processor that can be deployed in a thousand existing servers can be worth more to a business than a faster one that requires a new data centre to house it. This is an unglamorous kind of superiority and it is the kind that tends to win in production.
The trick the T4 relied on
The T4’s most consequential design choice was its support for lower-precision arithmetic, which is worth explaining because it sounds technical and is not.
When a model is being trained, its internal numbers are handled with considerable precision, because small errors accumulate over billions of adjustments. When the finished model is answering a question, that precision is often unnecessary. The numbers can be rounded aggressively, stored in a fraction of the space, and processed several times faster, and the answer the user sees stays the same. A photograph is still identified correctly. The translation still reads properly.
This is not universally true, and getting it wrong degrades the product in ways that are difficult to detect. But where it holds, it is close to free money: the same result for a fraction of the electricity. The T4 was built to exploit it, with hardware dedicated to those rounded formats and 16 gigabytes of memory, which in 2018 was a comfortable amount.
Cloud providers took to it for a reason beyond efficiency. Because the card was not specialised for one task, they could rent the same machines to a customer running image analysis, another running a recommendation system, and a third doing video encoding. Infrastructure that serves several purposes is easier to buy than infrastructure that serves one, since it is still useful when the fashion changes.
What broke it
Large language models arrived and violated the T4’s central assumption, which was that the model would be small enough to sit comfortably in the card’s memory.
Modern conversational models are not small. They want far more memory than 16 gigabytes, they hold long conversations in that memory while they work, and serving many users at once multiplies the requirement. On this class of work the T4 is not slow so much as insufficient. Newer accelerators are built with much larger memory and much faster connections between cards precisely because the shape of the workload changed.
The card remains genuinely useful for the enormous quantity of AI that is not a chatbot. Computer vision, recommendation, classification, video analytics: these run on models that have not grown in the same way, and they represent more of the AI actually deployed in businesses than the current conversation would suggest.
The part that outlived the hardware
What the T4 established was a way of thinking that the industry keeps having to relearn.
The most valuable processor is not the one at the top of the specification chart. It is the one that delivers an acceptable answer at the lowest total cost inside the building you actually have. That was true when the workload was image classification on a 70-watt card, and it is true now, at a scale that would have seemed absurd in 2018, as companies build sites whose entire purpose is answering questions rather than training the models that answer them.
The industry spent years fascinated by the construction project. The bill was always going to be in the operating costs.
