Understanding AI

I wanted to better understand how AI models are created. Not to become an expert, but to gain an appreciation for the abstractions I use every day.

This post will highlight what I’ve learned so far. It’s written for other engineers who are new to topics like neural networks, deep learning, and transformers.

Machine Learning

Software is deterministic. Given some input, if you run the program again, you will get the same output. A developer has explicitly written code to handle each case.

Most AI models¹ are not this way. They are probabilistic. Developers don’t have to explicitly program the instructions.

Machine learning teaches software to recognize patterns from data. Given some input, you might not get the same output². AI models like GPT (from OpenAI), Claude (from Anthropic), and Gemini (from Google) are "trained" on a large chunk of internet documents. These models learn patterns during training.

Then, there’s an API or chat interface where you can talk to the model. Based on some input, it can predict and generate sentences, images, or audio as output. You can think about machine learning as a subset of the broader AI category.

Neural Networks

AI models are built on neural networks. You can think of them as a giant web of decision-making pathways that learn from examples. Neural networks can be used for anything, but I'll focus on language models.

These networks consist of layers of interconnected neurons that process information:

1. An input layer where data enters the system. Input is converted into a numerical representation of words or tokens (more on tokens later).
2. Many hidden layers that create an understanding of patterns in the system. Neurons inside the layer apply weights to the input data and pass the result through an activation function³. This function outputs a value, often between 0 and 1, representing the neuron’s level of activation.
3. An output layer which produces the final result, such as predicting the next word in a sentence. The outputs at this stage are often referred to as logits, which are raw scores that get transformed into probabilities.

For example, if the input was "San", the model would assign a high probability to "Francisco" as the next word, while an unrelated word like "kitten" would have a near-zero probability.

A big takeaway for me is: it’s just math. You can build a neural network from first principles using linear algebra, calculus, and statistics. You likely won’t do this when there’s helpful abstractions like PyTorch, but it’s helpful for me to demystify what is happening under the hood.

Deep Learning

Deep learning is a subset of machine learning that involves neural networks with many layers, hence the term "deep." While a simple neural network may have only one or two hidden layers, deep learning models can have hundreds of layers.

These additional layers enable the network to learn complex patterns. For example, language models have been trained on multi-language datasets. This allows them to understand, generate, and translate text in multiple languages.

You can ask a question in English, and get a response back in Japanese.

Tokenization

Before a neural network can process text, the text must be converted into numerical data, a process known as tokenization.

1. Tokenization breaks down text into smaller units called tokens.
2. Each token is then mapped to a numerical value that the model can process.
3. Models learn to understand the statistical relationships between these tokens, which helps them predict and produce the next token in a given sequence.

To handle the complexity of human language, including rare words and misspellings, models use subword tokenization techniques like Byte Pair Encoding (BPE). BPE starts with individual characters and iteratively merges the most frequent pairs of symbols to form new tokens.

For instance, the word "unbelievable" might be tokenized into ["un", "believ","able"]. This allows the model to understand and generate words it has not explicitly seen during training. Newer models have also developed specialized tokenizers for processing math and code, as both have unique syntax and structure.

Pretraining

Large Language Models (LLMs) are trained on vast amounts of data.

Collecting and cleaning this data is much more complex than simply scraping websites. If you feed the model poor-quality data, you'll get poor-quality predictions in return. Garbage in, garbage out.

Training, also called pretraining, is the process of teaching the network to recognize patterns in the input data. Essentially, we're taking large chunks of internet text and compressing them into knobs the model can tune, known as weights and biases (collectively called parameters).

The number of parameters is often used as a measure of a model’s power⁷. For example, open-source models like Llama have versions ranging from 17 billion active parameters (Scout) to 288 billion active parameters (Behemoth).

You can think of training like a big for loop that runs many times, each time adjusting the weights and biases slightly. Each complete pass through the entire training dataset is called an epoch. But how does the network know how to adjust these values?

The training process involves several key steps:

1. Forward Pass: The network takes input data and passes it through its layers to produce an output. This is called the forward pass. During this step, the network makes a prediction based on its current weights and biases.
2. Loss Function: After obtaining the output, we need to measure how good or bad the prediction is. This is done using a loss function, which measures the prediction and gives us a value to try and minimize.
3. Backward Pass (Backpropagation): To improve the network’s performance, we need to adjust the weights and biases to reduce the loss. This involves computing the gradient of the loss function with respect to each weight and bias. The gradients indicate the direction and magnitude by which to adjust the parameters to minimize the loss.
4. Gradient Descent: Using the calculated gradients, we update the weights and biases. This process is called gradient descent, an optimization algorithm⁴ that iteratively adjusts the parameters to minimize the loss function.

weights, biases = init_random_parameters()

epochs = 10
learning_rate = 0.01 # See ⁴

for epoch in range(epochs):
  for input_data, target in training_dataset:
    predictions = forward_pass(input_data, weights, biases)
    loss = loss_function(predictions, target)
    gradients = backpropagation(loss, weights, biases)

    # Step 4: Gradient Descent (Parameter Update)
    weights -= learning_rate * gradients["weights"]
    biases -= learning_rate * gradients["biases"]

  print(f"Epoch {epoch + 1}/{epochs}, Loss: {loss}")

By repeatedly performing these steps over many epochs, the network learns to make better predictions. It uses the gradients calculated during backpropagation to adjust its internal parameters, effectively minimizing the loss function and improving its performance over time.

Transformers

"Attention Is All You Need" introduced the transformer architecture, a more efficient and powerful way of building language models.

Before processing, words are converted into numerical vectors called embeddings. These embeddings capture semantic information, meaning that similar words are represented by similar vectors in a high-dimensional space.

For example, the relationship between "king" and "queen" is similar to that of "man" and "woman". If you subtract the vector for "man" from "king" and add "woman", you get a result close to "queen".

Transformers use attention mechanisms, specifically self-attention. While attention had been used in previous papers and models, it only read sequences word by word. Transformers can consider the entire input sequence at once.

This allows the model to focus on different parts of the input when generating each part of the output, much like how a human understands language by considering the context of a sentence or paragraph.

For example, in the sentence "The cat sat on the mat because it was tired," the word "it" refers to "cat." Self-attention helps the model establish this connection by assigning higher weights to related words, enabling it to understand that "it" refers to "cat" and not "mat."

Using self-attention, transformers can process entire sequences of data in parallel rather than one element at a time, making training faster and more efficient, especially with GPUs.

Fine-tuning

After pretraining, the model may need to be adapted for specific tasks or aligned with human values. Fine-tuning is the process of adjusting a pretrained model to have a unique style or perform well in a specific domain.

Fine-tuning involves using a smaller, high-quality dataset relevant to the specific task. This data is often human-reviewed to ensure quality and alignment with the desired outcomes. During fine-tuning, the model’s parameters—the weights and biases we’ve discussed earlier—are adjusted slightly to improve performance on this new dataset.

Reinforcement Learning

To further improve the model’s usefulness and ensure it behaves in desirable ways, a technique called Reinforcement Learning from Human Feedback (RLHF) is used. RLHF involves a few steps:

1. The AI model generates several possible responses to a given input or question.
2. Human reviewers compare these responses and rank them from best to worst based on criteria like accuracy, helpfulness, and alignment.
3. Using the human rankings, a separate model called a "reward model" is trained to predict the quality of responses.
4. The original AI model is then fine-tuned using reinforcement learning techniques guided by the reward model.

RLHF can be effective with hundreds to thousands of high-quality human-rated examples. The emphasis is on quality over quantity, as each example is carefully judged by a human’s preferences.

For example, if you want a language model for customer support, you might fine-tune it on transcripts of customer service interactions and then use RLHF to ensure it responds politely and effectively resolves customer issues. Increasingly, AI is also used to generate initial content and humans help refine it through this feedback loop.

Inference

After training and fine-tuning, the model is ready for inference: the process of making predictions or generating outputs from new inputs. During inference, the model applies what it has learned to produce responses. The model may support multimodal input (text, images, audio, etc) and output.

One important parameter during inference is temperature, which controls how random or focused the model's outputs are. A lower temperature makes the model more deterministic, while a higher temperature increases creativity and randomness.

Modern language models can perform complex reasoning tasks using techniques like chain of thought. This involves the model generating intermediate reasoning steps before arriving at a final answer, much like how you might work through a math problem step by step⁵. CoT started as a prompting technique, but newer reasoning models have this behavior trained directly into the model.

Distillation

Sometimes you need a model that can run efficiently on smaller devices or handle millions of requests with lower inference costs. That’s where model distillation comes in, which is a technique for creating smaller, faster models that still perform well.

The process works by training a smaller "student" model to mimic a larger "teacher" model. Instead of learning from raw data, the student learns from the teacher’s outputs. Think of it like a compressed version of the knowledge. You lose some detail, but keep most of what matters.

For example, there are many distilled versions of Llama, which is an open-source model. The distilled versions run faster and have lower inference costs, while still being very good at certain tasks. The trade-off is usually a small decrease in quality for a significant gain in efficiency.

Evaluation

How do you know if your model is good? This is where evaluation (often shortened to "evals") becomes important. Just like you wouldn’t ship code without tests (right...) you shouldn’t deploy AI models without evals.

Evaluation involves testing your model against a set of tasks with known correct answers. These can be standardized benchmarks like Terminal-Bench for coding tasks, or custom evals specific to your use case. The key is to measure what actually matters for your app.

Good evals are comprehensive enough to catch edge cases, but also representative of real-world usage. Building custom evals for your domain is equally important as the model training itself. Otherwise, you won’t know if quality regresses.

Example: Coding Agent

Let’s say you want to build a coding agent⁶ like Cursor Composer:

1. You’d likely start with an existing model that already understands code, perhaps Kimi K2.5 or a similar open-source model. These models have been pretrained on massive amounts of public code and understand syntax, common patterns, and multiple programming languages.

2. Next, you’d fine-tune this model on data specific to the coding domain. This could include high-quality code, internal documentation, coding standards, or architectural patterns. The model learns the nuances of how real developers write code.

3. To ensure the assistant gives helpful suggestions, you’d likely use both RLHF and general reinforcement learning. RLHF works well with hundreds to thousands of carefully human-rated examples. But for coding, you can also use RL at a much larger scale by defining automated reward signals: does the generated code compile? Does it pass tests? Does the resulting diff actually solve the issue? This lets you run thousands of rollouts and continuously improve the model without needing a human to judge every single output.

4. Throughout training, you’d want the RL environment to closely match the real product environment. For example, if your agent has tools like file reading, code search, and terminal commands, you’d train the model using those exact same tools so it learns to use them effectively. This is how Cursor trained Composer: by running rollouts in an environment that mirrored their production IDE, the model learned to call tools in parallel, search codebases effectively, and read before editing.

5. Finally, you’d continuously evaluate the model using both automated benchmarks (like Terminal-Bench) and internal benchmarks built from real usage.

Then, you’d keep iterating. As developers use the model, their feedback provides more signal for training. RL is particularly powerful here because improvements compound: more rollouts, better rewards, and a tighter feedback loop between the model and the real coding environment it operates in.

Resources

Still interested? There’s so much more to learn. Hopefully I have piqued your interest. If you have other resources, please reach out and I'll add them here.

• Andrej Karpathy on Neural Nets
• Andrej Karpathy’s microGPT
• 3Blue1Brown on Neural Nets
• Machine Learning for Web Developers
• Build a Large Language Model (From Scratch)