TL;DR
Workbook answers and notes for Chapter 2 of Build a LLM from Scratch by Sebastian Raschka, focused on working with text data (tokenization, token IDs, embeddings, and context windows). But first, let’s reread Giles’s blog posts about Chapter 2 (part 2, part 3).
Giles’s blog posts
In part 2, Giles briefly explains byte-pair encoding with tiktoken. tiktoken starts with letters, numbers, and punctuation, then learns word patterns from the input and creates smarter tokens (in my blog post, I mentioned the pattern we can find in words like define and delfine, …). It seems that Giles read the byte-pair encoding paper that I skipped.
The second thing is Python tiktoken’s allowed_special option, which we use when passing special tokens like |endoftext|. If it encounters special tokens that are not in that list, it raises an error. Elixir tiktoken also accepts special tokens as input, but it does not raise an error. Here is a ChatGPT explanation of the problem in the Elixir tiktoken wrapper.
In part 3, Giles nicely explains the concept of batches and why we need them. Batches are just how we stack {input, target} pairs. One batch is a list of {input, target} pairs, where the number of elements in that list is batch_size. The second batch is the next such list. Doing LLM-related calculations on batches is faster than running calculations on each pair individually.
One-hot embeddings are also explained, and this is actually a pre-step to the embedding data explained in the book.
I learned something new from Giles’s understanding of Chapter 2. Let’s do the workbook questions!
Workbook multiple choice questions
What is the primary purpose of tokenization in the context of LLMs?
A. Tokenization is used to convert text into lowercase.
B. Tokenization splits text into individual words or special characters
C. Tokenization is used to identify the parts of speech in a sentence.
D. Tokenization is used to remove stop words from text.
My answer: B
What is the purpose of the <|unk|> token in a vocabulary used for LLMs?
A. The <|unk|> token is used to represent punctuation marks.
B. The <|unk|> token is used to mark the beginning of a sentence.
C. The <|unk|> token represents unknown words that were not present in the training data.
D. The <|unk|> token is used to mark the end of a sentence.
My answer: C
What is the primary task of an LLM during training?
A. LLMs are trained to translate text from one language to another.
B. LLMs are trained to summarize text.
C. LLMs are trained to answer questions based on a given text.
D. LLMs are trained to predict the next word in a sequence, given a preceding context.
My answer: D (all are LLM tasks, but D is the primary task)
What is the difference between absolute positional embeddings and relative positional embeddings?
A. Absolute positional embeddings encode the exact position of a token in a sequence, while relative positional embeddings encode the relative distance between tokens.
B. Absolute positional embeddings are only used for short sequences, while relative positional embeddings are used for longer sequences.
C. Absolute positional embeddings are more efficient than relative positional embeddings.
D. Relative positional embeddings are more accurate than absolute positional embeddings.
My answer is A (or, as it is stated in the book for relative positions, how far away a token is from other tokens).
The purpose of ____ in the context of LLMs is to provide information about the order and location of tokens within a sequence, helping the LLM understand the relationships between words.
A. attention mechanism
B. positional embeddings
C. tokenization
My answer: B
What is the final output of the input processing pipeline for an LLM, before it is fed into the main LLM layers?
A. The final output is a tensor of probabilities for each word in the vocabulary.
B. The final output is a tensor of text tokens.
C. The final output is a tensor of input embeddings, created by combining token embeddings and positional embeddings.
D. The final output is a tensor of token IDs.
My answer: C
HoHoHo! My answers are correct!
Understanding word embeddings
Why are word embeddings necessary for processing text data in deep learning models?
My answer:
Deep learning models are deep neural networks that work with tensors that have float numbers. We use float numbers because we need precision that we can not achive with integers. These data represent probabilites that particular words are related.
I failed on this one to mention that text is categorical and embeding vectors are compatible with mathematical operations used in deep neural networks.
Workbook answer:
Deep learning models operate on numerical data, while text is
categorical. Word embeddings convert words into continuous-valued
vectors, making them compatible with the mathematical operations used
in neural networks.
What is the main idea behind the Word2Vec approach to generating word embeddings?
My answer:
In Word2Vec approach, we translate words to embeding vectors that tell us how close of far away are words based on their meaning.
This was also incomplete answer.
Workbook answer:
Word2Vec trains a neural network to predict the context of a word given
the target word or vice versa. This approach assumes that words
appearing in similar contexts tend to have similar meanings, resulting in
clustered representations of related words in the embedding space.
As I do not understand what is context/target relationship, I asked for help ChatGPT 5.3. to explain workbook answer.
It trains a small neural network on a simple language task:
- `target -> context` (predict nearby words from a center word) = `Skip-gram`
- `context -> target` (predict the center word from surrounding words) = `CBOW`
Example:
- In “the cat sat on the mat”, if the target word is `cat`, context words might be `the`, `sat`.
During training, the model adjusts each word’s vector so it gets better at these predictions.
Why this creates meaning:
- Words used in similar situations (like `cat` and `dog`) tend to have similar neighboring words (`pet`, `food`, `runs`, `cute`).
- So the model learns similar vectors for them.
Result:
- Related words become clustered in embedding space.
- The vectors capture semantic relationships (similarity/relatedness) even though the model was only trained on prediction, not dictionary definitions.
Explain the trade-off involved in choosing the dimensionality of word embeddings.
My answer:
With bigger dimensionality we will have better word prediction, but we will need more process power to train such model, and more memory to use that LLM.
Workbook answer:
Higher dimensionality in word embeddings can capture more nuanced
relationships between words but comes at the cost of computational
efficiency. Lower dimensionality offers faster processing but may sacrifice
some semantic detail.
I was close enough with my answer.
How do LLMs typically handle word embeddings compared to using pretrained models like Word2Vec?
My answer:
I do not know how to compare word embeddings and Word2Vec.
Workbook answer:
LLMs often generate their own embeddings as part of the input layer and
optimize them during training. This allows for embeddings tailored to the
specific task and data, potentially leading to better performance than
using pre-trained embeddings.
Aha, so Word2Vec is pretraining phase that is not needed with embedding vectors as those are created during training phase.
What is the primary challenge associated with visualizing high-dimensional word embeddings?
My answer:
Primary challange is human imagination. Humans can visualize up to four dimensions, three space dimensions and time.
Close enough with workbook answer:
Our visual perception and common graphical representations are limited
to three dimensions or fewer. Visualizing high-dimensional embeddings
requires specialized techniques or dimensionality reduction methods.
Tokenizing text
What is the purpose of tokenizing text in the context of building a large language model?
My answer:
As deep neural networks does not understand human text, we must represent it with tensors with float numbers. To do that, we first need to break text into words (or subwords) and special characters. Then we create vocabulary where each token is represented with ID (integer). And in the end we map those IDs to embeding vectors that are ready for mathematical operations in deep neural networks.
workbook answer:
Tokenization is a crucial preprocessing step for creating embeddings for
an LLM. It involves splitting input text into individual tokens, which are
either words or special characters, to prepare the text for further
processing and embedding creation.
Partialy correct because I put under tokenization embeding step.
Describe the process of tokenizing text using Elixir regular expression module Regex
Regex.split(~r/([,.:;?_!"()\']|--|\s)/, text, include_captures: true, trim: true)
My answer:
Ok, when I see Regex, I always get chills at my neck. Here we go (with LLM help):
~r/regex_expression_goes_here/(capture_goes_here) we tell Regex to remember what was captured in regex expression| this separates there groups of regex expressions[list_of_equal_characters] so dear Regex, first try to split on characters between [ and ], and those are special characters from The Verdict novel.-- yes, we have in Verdict two hyphens in a row\s this means any space, as we know that space could be encoded in various ways.include_captures keep all those capture strings, which means everything that goes between ( and )trim do not return empty string, aka ""
Allegedly, somebody once solved one advent of code only using regex.
Workbook answer:
The ‘re.split’ function can be used to split text based on specific patterns.
By defining a regular expression that matches whitespace characters,
punctuation marks, and other special characters, we can separate the
text into individual tokens. The resulting list can then be further
processed to remove redundant whitespace characters.
I was correct, but I cheated, I asked llm about options :include_captures and trim
Why is it important to consider capitalization when tokenizing text for LLM training?
Didn’t know what to answer.
Workbook answer:
Capitalization helps LLMs distinguish between proper nouns and common
nouns, understand sentence structure, and learn to generate text with
proper capitalization. Therefore, preserving capitalization during
tokenization is beneficial for training effective language models.
Yes, now I remember, to distinguish noun types.
Explain the trade-off between removing whitespaces during tokenization and keeping them.
Didn’t know what to answer.
Workbook answer:
Removing whitespaces reduces memory and computing requirements.
However, keeping whitespaces can be useful for training models that are
sensitive to the exact structure of the text, such as Python code, which
relies on indentation and spacing.
Ah, famous Python idents! We are spoiled with Elixir syntax and mix format.
Match the term on the left to its description on the right:
| Term |
|
Definition |
| Word Embeddings |
|
The process of converting various data types, such as text, audio, or video, into a dense vector representation that deep learning models can understand. |
| Embedding |
|
The dimensionality of a word embedding, which determines the number of dimensions used to represent each word, influencing the complexity and computational efficiency of the model. |
| Word2Vec |
|
An algorithm that generates word embeddings by predicting the context of a word given the target word or vice versa, based on the idea that words appearing in similar contexts tend to have similar meanings. |
| Embedding Size |
|
A method of representing words as continuous-valued vectors, allowing deep learning models to process text data. |
My answer:
Word embeddings => row 4.
Embedding => row 1.
Word2Vec => row 3.
Embedding size => row 2.
Correct!
Converting tokens into token IDs
What is the purpose of converting tokens into token IDs?
My answer:
The purpose of converting tokens to IDs (integers) is to prepare them in a format that is suitable for mathematical operations in a deep neural network.
Workbook answer:
Converting tokens into token IDs is an intermediate step before
converting them into embedding vectors. This process allows for efficient
representation and processing of text data within a language model.
I would say my answer is correct and maybe even more concrete because it explains what this processing is in an LLM.
How is a vocabulary created for tokenization?
My answer:
We sort tokens, then remove duplicates, and then create a dictionary where each token gets a unique ID.
Workbook answer:
A vocabulary is created by tokenizing the entire training dataset, sorting
the unique tokens alphabetically, and assigning a unique integer to each
token. This mapping allows for efficient conversion between tokens and
their corresponding integer representations.
I did not mention that tokens are sorted alphabetically, but the workbook also misses that tokens are unique.
What is the purpose of the encode function in the SimpleTokenizerV1 module?
My answer:
The encode function converts tokens to vocabulary ID values.
Workbook answer:
The encode method takes text as input, splits it into tokens, and uses the
vocabulary to convert these tokens into their corresponding integer IDs.
This process allows for representing text data as a sequence of integers,
which can be processed by the language model.
I missed the tokenization part. Note that the workbook uses method, and I use function. If you use method in a conversation with Sasa Juric, he would definitely correct you!
What is the purpose of the decode function in the SimpleTokenizerV1 module?
My answer:
The decode function converts vocabulary IDs to token values.
Workbook answer:
The decode method takes a sequence of token IDs as input and uses the
inverse vocabulary to convert these IDs back into their corresponding
text tokens. This process allows for converting the output of the language
model, which is a sequence of integers, back into human-readable text.
Workbook answer is more extensive.
What is the limitation of using a vocabulary built from a small training set?
My answer:
The limitation is that we would have a lot of \|unk\| tokens, and our LLM would not be able to predict the next token reliably.
Workbook answer:
Using a vocabulary built from a small training set can lead to issues when
encountering new words or phrases not present in the training data. This
can result in errors during tokenization and decoding, highlighting the
importance of using large and diverse training sets for building robust
language models.
I did not mention that we hit \|unk\| because the input word is not part of the training dataset.
Match the term on the left to its description on the right:
| Term |
|
Description |
| Tokenization |
|
individual units of text that result from tokenization, representing words, punctuation, or other special characters. |
| Tokens |
|
used to define patterns in text, allowing for flexible and precise text manipulation, including tokenization. |
| Regular Expressions |
|
initial steps taken to prepare text data for further processing, such as tokenization, which makes the text suitable for use in language models. |
| Preprocessing |
|
splitting text into individual units, called tokens, which can be words, punctuation marks, or other special characters. |
My answer:
| Left Hand Column |
1 |
2 |
3 |
4 |
| Right Hand Column |
4 |
1 |
2 |
3 |
Correct answer.
Adding special context tokens
What are the two special tokens added to the vocabulary and what are their purposes?
My answer:
The purpose of \|unk\| is to mark a token that is not in the vocabulary.
The purpose of \|could_not_remember\| is to mark different input sources.
Workbook answer:
The two special tokens added are <|unk|> and <|endoftext|>. <|unk|>
represents unknown words not in the training data, while <|endoftext|>
separates unrelated text sources, helping the LLM understand their
distinct nature.
My answer is correct, aside from the fact that I could not remember endoftext.
How does the modified SimpleTokenizerV2 handle unknown words?
My answer:
It adds \|unk\| as an optional special_tokens attribute, so this token is used in place of all unknown tokens.
Workbook answer:
When encountering a word not in the vocabulary, SimpleTokenizerV2
replaces it with the <|unk|> token, ensuring that all words are
represented in the encoded text.
Correct answer.
Explain the purpose of the <|endoftext|> token when training on multiple independent documents.
My answer:
This special token separates different text inputs so we know when the input context changes.
Workbook answer:
The <|endoftext|> token acts as a marker between unrelated text
sources, signaling the start or end of a particular segment. This helps the
LLM understand that these texts, though concatenated for training, are
distinct entities.
Different segments is better phrasing than different text inputs.
A piece of the code has been removed from this listing. Which of these terms has been removed from position_1?
A unk
B \n
C <|unk|>
D |unk|
def encode(text, vocab) do
tokenize(text)
|> Enum.map(fn token ->
Enum.find(vocab, fn {vocab_token, _} -> token == vocab_token end)
|> case do
{_, id} -> id
nil -> find_unknown_token(vocab, token)
end
end)
end
defp find_unknown_token(vocab, token) do
Enum.find(vocab, fn {vocab_token, _} -> vocab_token == " position_1 " end)
|> case do
{_, id} -> id
nil -> raise "Token not found in vocab: #{token}"
end
end
My answer: C
Note: I used Elixir instead of the original Python.
What are the additional special tokens commonly used in LLMs, and what are their functions?
My answer:
We could use a token that aligns all input texts, separated with \|endoftext\|, to the same length.
Other common special tokens include [BOS] (beginning of sequence),
[EOS] (end of sequence), and [PAD] (padding). [BOS] marks the start of
a text, [EOS] indicates the end, and [PAD] is used to extend shorter texts
to match the length of the longest text in a batch for training.
Padding is what I could not remember.
Match the term on the left to its description on the right:
| Term |
|
Description |
| Vocabulary |
|
integer representations of tokens, used as an intermediate step before converting tokens into embedding vectors. |
| Token IDs |
|
the dataset used to build the vocabulary and train the language model. |
| Tokenizer |
|
a mapping from unique tokens to unique integer values, created by tokenizing the entire training dataset and sorting the tokens alphabetically. |
| Training Set |
|
a class that implements methods for encoding text into token IDs and decoding token IDs back into text. |
My answer:
| Left Hand Column |
1 |
2 |
3 |
4 |
| Right Hand Column |
3 |
1 |
4 |
2 |
Correct.
Byte pair encoding
What are the two stages in this diagram?
My answer:
Stage 1 is where the byte pair encoder splits input text into tokens.
Stage 2 is mapping tokens to their vocabulary ID values.
Correct answer.
What is the primary advantage of using Byte Pair Encoding (BPE) for tokenization, especially when dealing with unknown words?
My answer:
The byte pair encoder splits tokens into subwords, so it can map any word into tokens.
Workbook answer:
BPE tokenizers break down unknown words into smaller subword units or
even individual characters. This allows them to handle any word without
needing a special <|unk|> token, ensuring that the tokenizer and the LLM
can process any text, even if it contains words not present in the training
data.
I did not mention that a token could be a single character.
What is the total vocabulary size of the BPE tokenizer used in models like GPT-2, GPT-3, and the original ChatGPT?
My answer:
52345
Workbook answer:
The BPE tokenizer used in these models has a vocabulary size of 50,257,
with the <|endoftext|> token assigned the largest token ID.
Not correct.
How does the BPE tokenizer handle unknown words, such as someunknownPlace, without using <|unk|> tokens?
My answer:
Since the byte pair encoder uses subwords as tokens, such as so and me, it can split any word into byte tokens.
Workbook answer:
The BPE tokenizer breaks down unknown words into smaller subword
units or individual characters. This allows it to represent any word as a
sequence of known subword tokens or characters, enabling it to process
any text without needing a special token for unknown words.
I forgot to mention single characters.
What Python library is used to implement the BPE tokenizer in the provided code example?
My answer: tiktoken
Correct.
Data sampling with a sliding window
Explain the purpose of creating input-target pairs in the context of training a large language model (LLM).
My answer:
With input-target pairs, we encode in the training dataset what the next token is for a particular input.
Workbook answer:
the model with examples of text sequences and their corresponding next
words. This allows the LLM to learn the relationships between words and
predict the most likely next word in a given context.
Now I know the purpose of the sliding window.
My answer:
In the sliding window approach, we generate input-target pairs with a stride parameter that defines how many tokens we skip in the input to define the next input-target pair.
For example, with My name is Karlo Smid, stride 2, and input size 3, we have:
My name is => Karlo
is Karlo Smid => “”
Workbook answer:
The sliding window approach involves iterating through a text sequence
and extracting overlapping chunks of text as inputs. Each input chunk is
paired with the corresponding next word as the target. The window slides
across the text, creating multiple input-target pairs for training.
I correctly described the sliding window process.
Pieces of code have been removed from three places in this listing. Which of these terms have been removed, and where should they go?
A Nx.vector
B Tiktoken
C Tokenizer
D Nx.tensor
def chunk_dataset(txt, model, max_length, stride) do
{:ok, token_ids} = position_1.encode(model, txt, ["<|endoftext|>"])
if length(token_ids) < max_length + 1,
do: raise("Number of tokenized inputs must be at least max_length + 1")
num_chunks = length(token_ids) - max_length
chunks =
0..num_chunks//stride
|> Enum.reduce([input_chunks: [], target_chunks: []], fn i, acc ->
input_chunk = Enum.slice(token_ids, i..(i + max_length - 1))
target_chunk = Enum.slice(token_ids, (i + 1)..(i + max_length))
[
input_chunks: [position_2(input_chunk) | acc[:input_chunks]],
target_chunks: [position_2(target_chunk) | acc[:target_chunks]]
]
end)
[
input_chunks: Enum.reverse(chunks[:input_chunks]),
target_chunks: Enum.reverse(chunks[:target_chunks])
]
end
My answer:
position 1 => B
position 2 => D
position 3 => D
Correct as I created Elixir example.
My answer:
The stride parameter defines the size of the sliding window. This is the number of input tokens we skip when defining the next input-target pair.
Workbook answer:
The stride parameter determines the step size of the sliding window. A
smaller stride results in more overlapping input chunks, while a larger
stride creates less overlap. The choice of stride influences the amount of
data generated and the potential for capturing long-range dependencies
in the text.
I did not describe how it affects input-target pairs.
My answer:
The max_length parameter defines the number of tokens per input-target pair. Its purpose is to define how many chunks we want to split the list of input tokens into.
Workbook answer:
The max_length parameter defines the size of the input chunks extracted
from the text. It determines the number of tokens included in each input
sequence. A larger max_length allows the LLM to process longer
contexts, but it also increases the computational cost of training.
Again, I did not explain the effect. The effect is the context size, a famous LLM parameter!
Pieces of code have been removed from two places in this listing. Which of these terms have been removed, and where should they go?
A input_chunks
B Tiktoken
C dataset
D target_chunks
def create_dataloader_v1(opts) do
raw_text = Keyword.fetch!(opts, :raw_text)
batch_size = Keyword.get(opts, :batch_size, 4)
max_length = Keyword.get(opts, :max_length, 256)
stride = Keyword.get(opts, :stride, 128)
shuffle = Keyword.get(opts, :shuffle, true)
drop_last = Keyword.get(opts, :drop_last, true)
num_workers = Keyword.get(opts, :num_workers, 0)
# Create dataset
[input_chunks: input_chunks, target_chunks: target_chunks] =
LlmScratch.GptDatasetV1.chunk_dataset(raw_text, "code-davinci-002", max_length, stride)
# Zip input and target chunks together so each item is {input, target}
dataset = Enum.zip(input_chunks, position_1)
LlmScratch.DataLoader.new(position_2,
batch_size: batch_size,
shuffle: shuffle,
drop_last: drop_last,
num_workers: num_workers
)
end
My answer:
position_1 => target_chunks
position_2 => dataset
Correct because I created Elixir example.
What is the significance of using PyTorch’s Dataset and DataLoader classes for creating a data loader for LLM training?
My answer:
Dataset and DataLoader help us organize input training data into sliding windows and prepare it for the embedding stage.
Workbook answer:
PyTorch’s Dataset and DataLoader classes provide a convenient and
efficient way to manage and iterate over large datasets. They allow for
batching, shuffling, and parallel data loading, which are crucial for
optimizing the training process of LLMs.
Did not mention shuffle, concurrency and batching feature.
Match the term on the left to its description on the right:
| Term |
|
Description |
| Byte Pair Encoding (BPE) |
|
words that are not present in the tokenizer’s predefined vocabulary. |
| Subword Units |
|
total number of unique tokens that a tokenizer can recognize and represent. |
| Out-of-Vocabulary (OOV) Words |
|
smaller units of text that a BPE tokenizer breaks down words into, which can be individual characters or combinations of characters. |
| Vocabulary Size |
|
a tokenization scheme that breaks down words into smaller subword units or individual characters, allowing it to handle unknown words by representing them as sequences of subword tokens or characters. |
My answer:
| Left Hand Column |
1 |
2 |
3 |
4 |
| Right Hand Column |
4 |
3 |
1 |
2 |
Correct.
Creating token embeddings
Why are embedding vectors necessary for training GPT-like LLMs?
My answer:
They are necessary because, to encode how tokens relate to each other in an LLM, we need floating-point numbers instead of integer token IDs. With integers, we cannot encode these relationships because integers are discrete values.
Workbook answer:
Embedding vectors are essential for training GPT-like LLMs because these
models are deep neural networks that rely on the backpropagation
algorithm for learning. Backpropagation requires continuous vector
representations, which embedding vectors provide.
I missed mentioning the backpropagation algorithm.
How are embedding weights initialized in the beginning of LLM training?
My answer:
Embedding weights are initialized with random floating-point values at the beginning.
Workbook answer:
Embedding weights are initially assigned random values. These random
values serve as the starting point for the LLM’s learning process. During
training, the embedding weights are optimized through backpropagation
to improve the model’s performance.
This is correct, but the workbook also explains what happens to the initial weights during the training phase.
What is the missing stage from this diagram?
My answer:
Token IDs
Correct.
Describe the process of converting a token ID into an embedding vector using an embedding layer.
My answer:
We first create embedding weights with random values, for tensor with shape {vocabular_size, embedding_dimensions}. So each row in this tensor is for token ID, using zero based indexing. First row is for ID 0, second for ID 1, and so on. When we forward embeding weights on input IDs, we get new tensor of shape `{number_of_input_ids, embedding_dimensions}, and this is our embedding vector that we can use to train LLM.
Workbook answer:
The embedding layer acts as a lookup table. When given a token ID, it
retrieves the corresponding embedding vector from its weight matrix.
This embedding vector is a continuous representation of the token,
allowing the LLM to process it effectively.
Correct. But better explanation that this is lookup table for input IDs.
How does the embedding layer’s weight matrix relate to the vocabulary size and embedding dimension?
My answer:
Shape of embedding layer’s weight matrix is defines as {vocabulary_size, embedding_dimension}
Workbook answer:
The embedding layer’s weight matrix has a number of rows equal to the
vocabulary size, representing each unique token. The number of columns
corresponds to the embedding dimension, which determines the size of
the embedding vector for each token.
Correct.
Match the term on the left to its description on the right:
| Term |
|
Definition |
| Context Size |
|
number of positions the input window is shifted when creating the next batch of input–target pairs. |
| Input–Target Pairs |
|
a technique used to create input–target pairs from a text dataset by moving a window of tokens across the text. |
| Sliding Window |
|
number of tokens that the LLM uses as input to predict the next word. |
| Stride |
|
a set of data used to train an LLM, where the input is a sequence of tokens and the target is the next token in the sequence. |
My answers:
Context size => row 3
Input-Target Pairs => row 4
Sliding Window => row 2
Stride => row 1
Correct.
Encoding word positions
What is the main shortcoming of LLMs in terms of token order and how is it addressed?
My answer:
The main shortcoming of LLMs in terms of token order is that we always map token IDs to the same embedding vector weights because token IDs are indices of tokens in the vocabulary, and we do not have information about token position in the input text. Token position in the input text is important for next-token prediction. This problem is addressed by adding a vector with token absolute positions to the embedding vector.
Workbook answer:
LLMs’ self-attention mechanism lacks a notion of token order. To address
this, positional embeddings are introduced, which provide information
about the position of each token within a sequence.
Wrong. I missed that position indices are first forwarded to embedding weights and then added with token embeddings. I also did not mention that we are enhancing the self-attention mechanism.
Explain the difference between absolute and relative positional embeddings.
My answer:
In absolute positioning, we use the token index position in the input text. In relative positional embeddings, we use a value that represents how far tokens are from each other.
Workbook answer:
Absolute positional embeddings assign a unique embedding to each
position in a sequence, indicating its exact location. Relative positional
embeddings focus on the relative distance between tokens, allowing the
model to generalize better to sequences of varying lengths.
Correct. But exact location and index position could use a better explanation, which I provided in a later answer.
How are positional embeddings used in OpenAI’s GPT models?
My answer:
We add positional embeddings to token embeddings.
Workbook answer:
GPT models use absolute positional embeddings that are optimized during
training. These embeddings are not fixed or predefined but are learned
alongside the model’s other parameters.
Incorrect. I did not mention absolute positions or that those embeddings are not fixed but learned, because we also pass those positions together with token embeddings.
My answer:
In the case of absolute positions, we create a positional embedding vector that has, for each token, its index, which represents distance from the beginning of the input. Then we add this positional embedding vector to the token embedding vector.
Workbook answer:
Token embeddings are generated by mapping token IDs to vectors.
Positional embeddings are then added to these token embeddings,
resulting in input embeddings that incorporate both token identity and
positional information.
Incorrect. I forgot one step where positional IDs are also passed to obtain positional embeddings. Then we add token embeddings and positional embedding vectors.
What is the purpose of the token_embedding_layer and pos_embedding_layer in the code provided?
My answer:
These are token embedding and positional embedding layers.
Workbook answer:
The token_embedding_layer converts token IDs into embedding vectors,
while the pos_embedding_layer generates positional embeddings based
on the position of each token in the sequence.
Incorrect. I did not mention the conversion process.
Conclusion
That is it for the workbook questions. Open-ended questions are still hard for me, but I am satisfied with how I generally answered them in this chapter.
References
]]>
Matrix dot product: 
| Term | Description |
|——————-|————-|
| pretrained models | row 2 |
| fine-tuning | row 3 |
| base model | row 1 |
