How LLMs Work: Embeddings, Attention, and Feedforward Networks Explained

You’ve probably used a chatbot that feels eerily human. You type a question, it thinks for a split second, and then spits out an answer that makes sense. But what actually happens inside that black box? It’s not magic. It’s math. Specifically, it’s a specific architecture called the Transformer, introduced in 2017 by Google Brain researchers Vaswani et al. This paper, titled "Attention is All You Need," changed everything. Before this, models like Recurrent Neural Networks (RNNs) read text one word at a time, like you reading a book with your finger under each line. Transformers don’t do that. They look at the whole sentence at once. How? By relying on three core components: embeddings, attention mechanisms, and feedforward networks.

If you’re trying to understand why AI hallucinates, or why some models are better at coding than others, you need to grasp these three parts. They work together like a well-oiled machine. Miss one, and the whole thing falls apart. Let’s break down exactly how they function, without getting lost in endless equations.

The Foundation: What Are Embeddings?

Computers don’t understand words. They understand numbers. So, the first job of any Large Language Model (LLM) is to turn text into numbers. This process starts with tokenization. The model chops your sentence into small pieces called tokens. These aren’t always whole words. For example, "unbelievable" might be split into "un", "believ", and "able." Each token gets a unique ID from a massive dictionary, often containing 50,000+ entries in models like GPT-3.

But an ID number isn’t enough. The number 452 doesn’t tell the computer anything about the meaning of the word "king." That’s where embeddings come in. An embedding is a list of numbers-a vector-that represents the semantic meaning of a token. In BERT-base, these vectors have 768 dimensions. In GPT-3, they go up to 12,288. Think of this as placing every word in a giant, multi-dimensional map. Words with similar meanings end up close to each other on this map. "Cat" and "dog" are neighbors. "Car" and "truck" are nearby. "Banana" is far away from both.

This spatial relationship is powerful. If you take the vector for "King," subtract "Man," and add "Woman," the resulting point lands almost exactly on "Queen." This geometric property allows the model to understand analogies and relationships without being explicitly programmed with grammar rules. However, raw embeddings lose order information. Since transformers process all tokens in parallel, the model needs to know which word came first. To fix this, developers add positional embeddings-another set of numbers that encode the position of each token in the sequence. Without these, "I love you" and "You love I" would look identical to the model.

The Brain: Understanding Attention Mechanisms

If embeddings provide the raw data, attention provides the context. This is the heart of the transformer architecture. In traditional language models, understanding a pronoun like "it" was hard because the model had to remember the previous sentence step-by-step. Attention solves this by allowing every word to look at every other word simultaneously.

Here’s how it works. For each token, the model creates three vectors: Query (Q), Key (K), and Value (V). Imagine you’re searching for a document. Your search term is the Query. The titles of the documents are the Keys. The content of the documents is the Value. The model calculates how relevant each Key is to your Query. Mathematically, this involves multiplying Q and K, dividing by the square root of the dimension size to stabilize gradients, and applying a softmax function to get probabilities between 0 and 1. These probabilities determine how much "attention" one word should pay to another.

Consider the sentence: "The animal didn't cross the street because it was too tired." When the model processes "it," the attention mechanism looks back at "animal" and "street." It assigns a high attention score to "animal" and a low score to "street," effectively deciding that "it" refers to the animal, not the road. This capability is known as self-attention.

Modern LLMs use Multi-Head Attention. Instead of doing this calculation once, they do it multiple times in parallel using different "heads." GPT-2 uses 12 heads; GPT-3 uses up to 96. Why? Because different heads learn different types of relationships. One head might focus on syntactic structure (subject-verb agreement), while another focuses on semantic similarity (synonyms). This parallel processing allows the model to capture complex linguistic patterns that a single attention layer would miss. According to Databricks, attention mechanisms consume 60-70% of computational resources during inference, making them the most expensive part of the model to run.

The Processor: Feedforward Networks

After the attention layer has gathered context, the data moves to the feedforward network. This is essentially a simple neural network applied to each token independently. While attention mixes information across tokens, the feedforward network processes each token’s updated representation individually.

A standard feedforward block consists of two linear transformations separated by a non-linear activation function, typically GELU (Gaussian Error Linear Unit). The first transformation expands the vector size, usually to four times the original dimension. For BERT-base, this means expanding from 768 dimensions to 3,072. The second transformation shrinks it back down. This expansion-shrink pattern allows the model to create richer, more abstract representations of the input.

Think of it like this: Attention gathers all the relevant clues from the sentence. The feedforward network then takes those clues and decides what they mean. It acts as a knowledge base, storing factual information learned during training. Recent research suggests that specific neurons in these layers correspond to specific facts or concepts. For instance, a neuron might activate only when the model discusses French cities. This separation of duties is crucial. Attention handles the "who does what to whom" logic, while feedforward networks handle the "what does this concept mean" storage.

Putting It All Together: The Transformer Block

These three components don’t work in isolation. They are stacked in repeating units called transformer blocks. A typical block follows this sequence:

• Input Embedding + Positional Encoding: Converts tokens to vectors and adds position info.
• Layer Normalization: Stabilizes the inputs before the next layer.
• Multi-Head Self-Attention: Mixes information across tokens.
• Add & Norm: Adds the original input back (residual connection) and normalizes again.
• Feedforward Network: Processes each token’s representation.
• Add & Norm: Residual connection and normalization.

Models like GPT-3 stack 96 of these blocks sequentially. The output of one block becomes the input for the next. This deep stacking allows the model to build increasingly abstract representations. Early layers might detect basic grammar, middle layers identify phrases, and deeper layers understand intent and nuance. Residual connections are vital here-they allow gradients to flow backward through the network during training, preventing the vanishing gradient problem that plagued earlier deep networks.

Variations: Autoregressive vs. Autoencoding

Not all LLMs use these components in the same way. The direction of attention matters significantly. GPT-style models are autoregressive. They use causal masking, meaning a token can only attend to previous tokens, not future ones. This makes them perfect for generation-you predict the next word based on what came before. BERT-style models are autoencoders. They use bidirectional attention, allowing a token to see both left and right context. This makes BERT excellent for classification tasks like sentiment analysis but poor for generating text sequentially.

Newer architectures try to optimize these trade-offs. Meta’s Llama 2 introduced Grouped-Query Attention (GQA) to reduce memory usage by sharing key-value heads across multiple query heads. Google’s Gemini 1.5 expanded context windows to 1 million tokens, challenging the quadratic cost limit of standard attention. Despite these innovations, the core triad remains unchanged. As NVIDIA’s Chief Scientist Bill Dally noted, attention mechanisms will likely remain fundamental for the next decade.

Practical Challenges for Developers

If you’re building applications with LLMs, you’ll hit real-world limits. The biggest pain point is memory. Attention matrices grow quadratically with sequence length. Processing an 8,192-token sequence in a 7-billion parameter model can require over 75GB of VRAM. This is why techniques like FlashAttention-which optimizes GPU memory access-are so popular. They speed up computation by 3x without changing the math.

Another common issue is position embedding generalization. Standard absolute position embeddings fail if you ask the model to process sequences longer than those seen during training. Rotary Position Embeddings (RoPE), used in Llama models, solve this by encoding relative positions, allowing models to handle longer contexts gracefully. Developers also struggle with interpretability. Dr. Emily Bender points out that feedforward networks create a "black box" effect. We can see the attention scores, but tracing exactly how a feedforward neuron contributes to a final answer remains difficult.

Frequently Asked Questions

Next Steps for Learning

Understanding these components is just the start. If you want to dive deeper, experiment with visualizing attention maps using tools like BertViz. Try fine-tuning a small model like DistilBERT to see how changing hyperparameters affects performance. Keep an eye on emerging architectures like Mixture of Experts (MoE), which attempt to scale feedforward networks efficiently by activating only a subset of experts per token. The landscape changes fast, but the core principles of embeddings, attention, and feedforward networks remain the bedrock of modern AI.

• Sep, 6 2026
• Collin Pace
• 0
• Permalink

• Tags:
• Large Language Models
• Transformer Architecture
• Self-Attention Mechanism
• Word Embeddings
• Neural Networks

Written by Collin Pace

• 

  View all posts by: Collin Pace