Types of AI Models

There are many more types of AI models than the five categories we discussed earlier. Those five were just a starting point.

The important thing to understand is that AI models can be classified by what they do, what data they process, and how they learn. These classifications overlap: one model can belong to several categories at once.

Let's build a bigger map of the AI model ecosystem.

1. The major categories of AI models

1. Language models

Understand and generate text, answer questions, write code, and solve problems.

Examples: GPT, Claude, Llama, Qwen.

2. Computer vision models

Analyze images and video to identify objects, people, scenes, and visual patterns.

Examples: YOLO object detection, image segmentation models, image classifiers.

3. Image generation models

Create or edit images from text prompts, reference images, or other inputs.

Examples: FLUX, Stable Diffusion.

4. Audio and speech models

Recognize speech, transcribe audio, understand sounds, and generate synthetic voices.

Examples: Whisper, text-to-speech models, speech-to-speech models.

5. Video models

Generate video, predict motion between frames, or analyze existing footage.

Examples: video diffusion models and video understanding models.

6. 3D and spatial models

Reconstruct 3D objects or environments, estimate depth, and represent geometry or spatial relationships.

Examples: NeRF, 3D Gaussian Splatting, depth-estimation models.

7. Robotics and action models

Help robots or software-controlled systems choose actions and interact with an environment.

Examples: robot control policies, vision-language-action models.

8. Prediction and forecasting models

Predict future values from historical data, such as demand, sales, energy use, or equipment failure.

Examples: time-series forecasting models, regression models.

9. Recommendation models

Predict what a user may want to watch, buy, read, or listen to.

Examples: collaborative filtering, recommendation ranking models.

10. Scientific and simulation models

Help researchers model complex systems in biology, chemistry, physics, and the environment.

Examples: AlphaFold for protein structure prediction; specialized weather and molecular models.

11. Anomaly detection models

Identify unusual patterns that may indicate fraud, cyberattacks, faulty equipment, or suspicious activity.

Examples: isolation forests, autoencoders, specialized fraud-detection models.

12. Embedding models

Convert text, images, audio, or other data into numerical representations that capture useful similarities and relationships.

Used for semantic search, document retrieval, recommendations, and RAG.

13. Graph models

Learn from entities and their relationships, represented as nodes and edges.

Examples: graph neural networks for fraud detection, molecule analysis, and network prediction.

14. Document intelligence models

Extract and interpret information from PDFs, forms, invoices, tables, and scanned documents.

Examples: OCR models, document layout models, document understanding models.

These 14 categories are a useful practical map, not an official or exhaustive taxonomy. There are many more specialized model types, and some fit into several categories.

2. More specialized AI models

Beyond those 14, here are additional categories you'll encounter when building AI products.

Some of these are separate model families; others are specialized tasks performed by models from broader families.

3. There's another way to classify AI: how models learn

This is a different classification from what we covered above.

A. Supervised learning

The model learns from examples paired with known answers.

Example: Feed it thousands of emails labeled "spam" or "not spam" so it learns to classify new emails.

B. Self-supervised learning

The model learns from patterns within the data itself, without requiring a human-provided label for every training example.

Example: An LLM learns to predict missing or subsequent tokens from large text datasets.

C. Unsupervised learning

The model discovers structure in data without explicit target labels.

Example: Group customers into clusters based on purchasing patterns.

D. Reinforcement learning

A system learns a policy for selecting actions using rewards or other feedback.

Example: Train an agent to complete a task in a simulated environment.

Modern systems can combine several learning methods. For instance, an LLM may be pretrained using self-supervised learning and then post-trained using supervised examples and preference-based or reinforcement-learning techniques.

4. The biggest concept: these categories overlap

Imagine an AI system that takes a Telugu screenplay and produces a cinematic video.

Telugu screenplay

Text input

Language model

Understands the story and generates a shot breakdown

Image and video models

Generate visual concepts and moving scenes

Audio models

Generate dialogue, voice, music, or sound effects

Complete filmmaking application

Coordinates the models, assets, and editing workflow

A single product can use multiple models. Likewise, one multimodal model may handle several tasks itself.

5. How to organize all of this in your mind

Think of AI models through three independent questions:

• What does it do? Language, image generation, prediction, recommendation, speech recognition, and so on.
• What does it work with? Text, images, audio, video, numerical data, graphs, or combinations of these.
• How does it learn? Supervised learning, self-supervised learning, unsupervised learning, reinforcement learning, or a combination.