The six AI model families and what they're good for

Part two of AI Infrastructure 101, a seven-part series. Part one covered how AI became an infrastructure problem.

The question this post answers: Which kind of AI actually solves my problem?

Nearly every model you will encounter is a neural network underneath, but the families differ in the data they consume, the tasks they perform and the compute they demand. Knowing which family your business problem belongs to is the first sizing decision you will make, because the family determines the hardware profile.

Language models (NLP and LLMs)

Natural language processing went through several generations, from statistical word-counting methods to recurrent networks, before the transformer architecture arrived in 2017 and changed everything. The transformer's self-attention mechanism lets a model weigh the relationships between all words in a passage simultaneously, which made it possible to scale language models to billions of parameters. Large language models (LLMs) such as the GPT series and the open-source LLaMA family are transformers at enormous scale. They excel at text generation, question answering, summarization and code completion, and they are the engine behind chatbots, virtual assistants and document analysis tools. They are also expensive to train and to serve. No workload is more responsible for today's GPU demand.

Computer vision

Convolutional neural networks (CNNs) dominated image work after AlexNet's 2012 breakthrough, and they remain the workhorse for image classification, object detection and segmentation. Newer vision transformers apply the attention mechanism to image patches with comparable or better results. Business applications include reading medical scans, quality inspection on production lines, content moderation and the perception systems in autonomous vehicles.

Generative models

Generative models create new content rather than classify existing content. Generative adversarial networks (GANs) pioneered image synthesis, and diffusion models, which iteratively refine random noise into coherent output, now power the leading text-to-image systems such as Stable Diffusion. The same techniques are extending into video generation and text-to-speech. Commercial uses range from product mockups and concept art to marketing assets and voiceovers.

Speech recognition

Modern speech recognition models such as OpenAI's Whisper convert raw audio into text with steadily falling error rates. They enable meeting transcription, real-time captioning, call-center analytics and voice interfaces.

Reinforcement learning

Reinforcement learning (RL) trains an agent through trial and reward rather than labeled examples. It sits behind headline systems like AlphaGo and behind practical work in robotics, industrial control and logistics optimization. RL matters less to most first business deployments, but it benefits from the same GPU acceleration as the other families.

Multi-modal and specialized models

Multi-modal models combine text, images and audio in one system, so a single model can, for example, answer questions about a photograph. Specialized architectures also exist for time-series forecasting, tabular business data and edge devices, where techniques like quantization shrink models enough to run on phones and embedded hardware.

Why every family points back to GPUs

All of these architectures run on matrix and tensor operations that can be computed in parallel, and the modern ones carry parameter counts in the billions, which demands enormous memory bandwidth. Parallel math and heavy memory traffic are exactly what GPUs are built for, and that is the subject of the next part in this series.

A practical example

Suppose your team wants three capabilities: a chatbot that answers policy questions, automatic transcription of customer calls and generated product imagery for campaigns. That is three model families, not one project. The chatbot is an LLM workload with high memory requirements. Transcription is a speech model that runs efficiently on mid-range hardware. Image generation is a diffusion workload that performs well on GPUs a fraction of the price of the flagship chips. Teams that treat these as one undifferentiated "AI project" tend to overprovision, because they buy hardware sized for the largest workload and run everything on it.

Tip: classify before you shop

Write down each AI use case on your roadmap and label it with a model family before any infrastructure conversation. The label answers most sizing questions in advance. A useful shorthand: language models are memory-hungry, vision and speech models are compute-efficient and generative image models sit in the middle. Vendors will take the conversation to hardware immediately, and this one-page classification keeps the conversation anchored to your actual needs.

‍