I am recapping the blog post Fine-tuning to follow instructions by reviewing the chapter 7 workbook questions from Build an LLM from Scratch and reading Giles’ blog posts on chapter 7.
Giles blog posts
Giles wrote three blog posts: part24, part25, and part26.
In the part24 post, another sidestep, Giles tries to feed instruction prompts into the GPT-2 models mentioned in the book. The result is that none of those models could give a meaningful answer, but the base Qwen 3 model with 600M parameters gave meaningful instruction results.
In part25, Giles always thinks more deeply about topics presented in books. For example, in relation to the Alpaca prompt format, it was created because interaction with the model was one-shot, due to the limited size of the model context. Today’s models have context sizes in the hundreds of thousands of tokens, so they could accept lengthy conversation transcripts. Then he explains that padding tokens waste GPU cycles, so we have batches of different sizes. The question is why we do not put -100 instead of padding tokens in inputs, but only in targets. Giles explains that this is not valid token vocabulary, but targets are also vocabulary tokens, so I do not understand that reasoning.
part26 is the shortest blog post! It is about evaluating our instruction fine-tuned model using an Ollama 3 model.
Workbook questions
Quick questions
1 What is the primary challange that pretrained LLMs often face regarding instructions?
A Inability to complete sentences
B Difficulty in generating coherent text
C Limited vocabulary size.
D Struggling with specific instructions like grammar correction or voice conversion.
My answer: D
Correct answer: +
2 What is the key component in preparing da dataset for supervised instruction fine-tuning?
A Pre-trained language model.
B Optimization algorithm.
C Instruction-response pairs.
D Tokenization algorithm.
My answer: C
Correct answer: +
3 What data format is commonly used for instruction datasets due to its human and machine redability?
A JSON (JavaScript Object Notation)
B CSV (comma separated values)
C YAML (YAML Ain’t Markup Language)
D XML (Extensible Markup Language).
My answer: A
Correct answer: +
4 What is the purpose of a custom collate function in the context of instruction fine-tuning?
A To optimize the model’s architecture.
B To handle the specific formatting and requirements of the instruction fine-tuning dataset.
C To pre-process the input data.
D To evaluate the model’s performance
My answer: B
Correct answer: +
5 What is the purpose of using the ignore_index parameter (-100) in the custom collate function?
A To mark the end of a sequence.
B To indicate an unknown token.
C To exclude padding tokens from the loss calculation.
D To identify the start of a sequence.
My answer: C
Correct answer: +
6 What is the purpose of saving the fine-tuned model’s state dictionary?
A To compress the model for efficient storage.
B To visualize the model’s architecture.
C To save the model’s parameters for later use or reuse in other projects.
D To improve the model’s performance.
My answer: C
Correct answer: +
7.1 Introduction to instruction fine-tuning
1 What is the primary function of a pretrained LLM?
My answer: The primary function of a pretrained LLM is to provide model knoweledge of language in which was trained grammar patterns.
Correct answer: A pretrained LLM is primarily capable of text completion, meaning it can finish sentences or write text paragraphs given a fragment as input.
It seems that I still do not know to describe the primary task of pre-trained LLM.
2 What is the challange that pretrained LLMs often face?
My answer: The challange is that pretrained LLMs are not good at resolving instructions tasks.
Correct answer: Pretrained LLMs often struggle with specific instructions, such as grammar correction or voice conversion, requiring further fine-tuning.
This one is correct.
3 What is the purpose of instruction fine-tuning?
My answer: The purpose of instruction fine-tuning is to train model for instructions tasks solving using as fine-tuning data instruction set with response how to resolve this task.
Correct answer: Instruction fine-tuning aims to improve an LLM’s ability to follow specific instructions and generate desired responses based on those instructions.
My answer is correct.
4 What is a crucial aspect of instruction fine-tuning?
My answer: A crucail aspect of instruction fine-tuning is that we make LLM into chatbot.
Correct answer: Preparing a suitable dataset is a key aspect of instruction fine-tuning, providing the model with examples of instructions and desired responses.
Wrong answer.
7.2 Preparing a dataset for supervised instruction fine-tuning
1 What is the purpose of the instruction dataset used for fine-tuning a pretrained LLM?
My answer: Purpose of the instruction dataset is to provide training, validation and testing datasets so we could fine tune LLM to resolve instruction tasks.
Correct answer: The instruction dataset consists of instruction-response pairs that are used to train the LLM to follow instructions and generate appropriate responses based on given inputs.
My answer is wrong.
2 Describe the format of the instruction dataset used in this section.
My answer: Each record has three keys, instructions contains instruction task, input is empty and output contains result of instruction task.
Correct answer: The dataset is stored in a JSON file and contains entries that are Python dictionary objects. Each entry includes an ‘instruction’, ‘input’, and ‘output’ field, representing the task, input data, and desired response, respectively._
I forgot to answer that format is JSON.
3 What are the two prompt styles mentioned in the section, and how they differ?
My answer: We have alpaca format and phi3 format. In phi3 we are using user and assistant tags, and in alpaca we use instructions, input and output tags.
Correct answer: The two prompt styles are Alpaca and Phi-3. Alpaca uses a structured format with defined sections for instruction, input, and response, while Phi-3 employs a simpler format with designated <|user|> and <|assistant|> tokens
My answer is correct.
4 What is the purpose of format_input action and how does it work?
My answer: The purpose of format_input function is to format input dataset to alpaca or phi3 prompt format. It creates a chatbot prompt.
Correct answer: The format_input function converts the entries in the instruction dataset into the Alpaca-style input format. It constructs a formatted string that includes the instruction, input (if available), and a placeholder for the response.
My answer is wrong.
5 How is the instruction dataset divided into training, validation, and test sets?
My answer: We randomly divide instruction dataset, 85% is training data, 10% is testing set, and rest is validation set.
Correct answer: The dataset is divided into training, validation, and test sets using a specific ratio. The training set is used to train the model, the validation set is used to evaluate the model’s performance during training, and the test set is used to evaluate the model’s final performance.
My answer is correct.
6 Match the terms on the left to their description on the right:
My answer: Instruction-response pairs => 3, Prompt styles => 1, Alpaca prompt style => 4, Phi3 Prompt style => 2
Correct answer: +
7.3 Organizing data into training batches
1 What is the purpose of a custom collate function in instruction fine-tuning?
My answer: Purpose of a custom collate function is to prepare prompts in alpaca or phi3 formats from input data. It also pads input data to to longest batch value, and sets in target data padding tokens to -100 value, except for the one next to last token.
Correct answer: A custom collate function is used to handle the specific requirements and formatting of the instruction fine-tuning dataset. It ensures that training examples are padded to the same length within each batch, allowing for efficient processing by the model.
My answer is correct.
2 How does the custom collate function handle padding in instruction fine-tuning?
My answer: It padds inputs to the value of longest input in the batch. Targets padding tokens are replaced with -100 instead of one next to last token.
| Correct answer: _The custom collate function pads training examples to the length of the longest example in each batch, using the < | endoftext | > token ID (50256). This minimizes unnecessary padding by only extending sequences to match the longest one in each batch._ |
I forgot to answer value of padding token.
3 Explain the role of target token IDs in instruction fine-tuning and how they are generated.
My answer: Target token IDs are pair with the input so we know which token should come next. It is generated from input so that input is shiffted by one token to the left.
Correct answer: Target token IDs represent the desired output sequence that the model should generate. They are created by shifting the input token IDs one position to the right, omitting the first token and appending an end-of- text token. This setup allows the model to learn how to predict the next token in a sequence.
One token to the right and I did not answer that we are adding endoftext token when we reach end of input.
4 Why are padding tokens replaced with the -100 placeholder value in the target token IDs?
My answer: We are replacing padding tokens with -100 because -100 tokens are not considered in loss calculation.
Correct answer: Replacing padding tokens with -100 allows the cross entropy loss function to ignore them during training. This ensures that only meaningful data influences model learning and prevents padding tokens from contributing to the loss calculation
I did not answer that padding tokens should not contribute to loss function calculation.
5 What is the purpose of retaining one end-of-text token in the target sequence?
My answer: Keeping that token we let know llm model to know where is the end of input instruction.
Correct answer: Retaining one end-of-text token in the target sequence helps the LLM learn to generate end-of-text tokens, which act as an indicator that the generated response is complete.
Wrong answer, we are kepping endoftext token in target sequence, not input. That was the question.
6 Pieces of code have been removed from four places in this listing. Which of these terms have been removed and where should they go? Note that a term may appear more than once!
A true B false C format D encode_text! E full_text
defp encode_example(entry, tokenizer, opts, prompt_style, __1__) do
entry
# The model trains on the instruction/input prompt followed by the
# expected response target, matching the chapter 7 PyTorch dataset.
|> __3__(prompt_style)
|> __4__(tokenizer, opts)
end
defp encode_example(entry, tokenizer, opts, prompt_style, __2__) do
token_ids =
entry
|> __5__(prompt_style)
|> __6__(tokenizer, opts)
prompt_length =
entry
|> prompt_text(prompt_style)
|> encode_text!(tokenizer, opts)
|> Kernel.length()
%{token_ids: token_ids, prompt_length: prompt_length}
end
defp encode_example(
_entry,
_tokenizer,
_opts,
_prompt_style,
mask_out_instructions_in_target
) do
raise ArgumentError,
"expected :mask_out_instructions_in_target to be a boolean, got: #{inspect(mask_out_instructions_in_target)}"
end
My answer: 1 => B, 2 => A, 3 => E, 4 => D, 5 => E, 6 => D
Correct answer: +
7.4 Creating data loaders for an instruction dataset
1 What is the purpose of the custom_collate_fn function in the context of instruction fine-tuning?
My answer: The purpose is to prepare input data into batches with input/target pairs, to padd data with endoftext tokens and skip loss function calculation token.
Correct answer: The custom_collate_fn function is used to batch the instruction dataset, ensuring that the input and target tensors are moved to the specified device (CPU, GPU, or MPS) before being fed into the LLM for fine-tuning.
2 Explain the advantage of moving data to the target device within the custom_collate_fn instead of the main training loop.
My answer: Advantage is faster calculation for training data.
Correct answer: By performing the device transfer within the custom_collate_fn, the process becomes a background task, preventing it from blocking the GPU during model training and improving efficiency
3 How is the device setting determined and used in the custom_collate_fn?
My answer: Did not answer as Elixir implmentation is different.
Correct answer: The device setting is determined based on the availability of a GPU or MPS. The partial function from functools is used to create a new version of the custom_collate_fn with the device argument prefilled, ensuring that the function uses the correct device for data transfer.
4 What is the purpose of the allowed_max_length parameter in the customized_collate_fn?
My answer: We truncate inputs and targets to the value of this parameter. If value is nil, we keep original size of inputs/targets.
Correct answer: The allowed_max_length parameter is used to truncate the data to the maximum context length supported by the LLM model being fine-tuned, in this case, the GPT-2 model.
5 Describe the process of creating data loaders for the training, validation, and test sets using the DataLoader calss.
My answer: We randonly split input data set to the precentage of total input data set: training is 85%, validation is 10 % and rest is test set.
Correct answer: The DataLoader class is used to create data loaders for each set (training, validation, and test). The batch_size, collate_fn, shuffle, drop_last, and num_workers parameters are configured to control the batching process, shuffling, and data loading behavior_
6 Pieces of the code have been removed from four places in the listing. Which of these terms have been removed and where should they go?
A inputs B targets C maybe_truncate
def custom_collate(batch, pad_token_id, ignore_index, allowed_max_length)
when is_list(batch) do
examples = Enum.map(batch, &normalize_example/1)
# Find the longest sequence length after the one extra pad token that the
# Python collate function appends to every item.
batch_max_length =
examples
|> Enum.map(&(Kernel.length(&1.token_ids) + 1))
|> Enum.max(fn -> 0 end)
{inputs, targets} =
Enum.map(examples, fn %{token_ids: item, prompt_length: prompt_length} ->
# Elixir data is immutable, so this builds the equivalent of
# `new_item = item.copy(); new_item += [pad_token_id]`.
new_item = item ++ [pad_token_id]
# Pad the copied item to the longest sequence length in the batch.
padded = pad_to_length(new_item, batch_max_length, pad_token_id)
# Match `padded[:-1]` for inputs and `padded[1:]` for next-token
# targets.
inputs = Enum.slice(padded, 0, batch_max_length - 1)
targets =
padded
|> Enum.slice(1, batch_max_length - 1)
|> mask_extra_padding_targets(pad_token_id, ignore_index)
|> mask_prompt_targets(prompt_length, ignore_index)
# Optionally cap both rows to the model context length.
{
maybe_truncate(inputs, allowed_max_length),
__1__(targets, allowed_max_length)
}
end)
|> Enum.unzip()
{
Nx.tensor(__2__, type: {:s, 64}),
Nx.tensor(targets, type: {:s, 64})
}
end
My answer: 1 => C, 2 => A
Correct answer: +
7 Match the terms on the left with their description on the right:
| Term | Description | |
|---|---|---|
| Batching Process |
A function that defines how to combine individual data samples into a batch, specifically tailored for the requirements of instruction fine-tuning. | |
| Custom Collate Function |
The process of organizing training data into groups of samples, called batches, to improve training efficiency. | |
| Padding Tokens |
Special tokens added to the end of sequences to ensure all inputs in a batch have the same length, allowing for efficient processing by the model. | |
| Ignore Index |
A special value, typically -100, used to indicate padding tokens in the target sequence, preventing them from contributing to the loss calculation during training. |
My answer: 1 => 2, 2 => 1, 3 => 3, 4 => 4
Correct answer: +
7.5 Loading a pretrained LLM
1 Why is larger pretrained model, like gpt2-medium, prefered for instruction fine-tuning compared to the smaller gpt2-small model?
My answer: Because pretrained medium model shows better performace for instruction fine-tuning.
Correct answer: The smaller model lacks the capacity to learn and retain the complex patterns and nuanced behaviors required for high-quality instruction- following tasks. The larger model has more parameters, allowing it to handle more intricate instructions and generate more accurate responses.
Did not explain why.
2 What is the purpose of loading a pretrained LLM before starting instruction fine-tuning?
My answer: Purpose is that we reuse step of training the LLM model.
Correct answer: Loading a pretrained LLM provides a foundation for the fine-tuning process. It allows the model to leverage existing knowledge and patterns learned during pretraining, enabling it to learn new tasks more efficiently and effectively.
Did not explain why.
3 How does the code for loading a pretrained model differ from the code used for pretraining of classification fine-tuning?
My answer: ?
Correct answer: The code remains largely the same, but instead of specifying the ‘gpt2- small’ model, we now specify ‘gpt2-medium’ to load the larger model with 355 million parameters. This change reflects the choice of a more capable model for instruction fine-tuning.
Did not think of this simple answer.
4 What is the purpose of evaluating the pretrained LLM’s performance on a validation task before fine-tuning?
My answer: Purpose is to avoid oveffiting LLM model in pretrained phase.
Correct answer: Evaluating the pretrained LLM’s performance provides a baseline understanding of its capabilities before fine-tuning. This allows us to assess the impact of fine-tuning on the model’s ability to follow instructions and generate accurate responses.
Wrong answer.
5 How is the model’s generated response isolated from the input instruction in the provided code?
My answer: It is isolated with dedicated tag: generated_response.
Correct answer: The code subtracts the length of the input instruction from the start of the generated text, effectively removing the input text and leaving only the model’s generated response. The strip() function is then applied to remove any extra whitespace.
I did not answer that we need to remove instructions part from the response.
7.6 Fine-tuning the LLM on instruction data
1 What is the purpose of fine-tuning an LLM on instruction data?
My Answer: The purpose of fine-tuning an LLM on instruction data is to train LLM to understands the patterns for resolving instruction tasks.
Correct answer: Fine-tuning an LLM on instruction data aims to improve its ability to understand and follow instructions, leading to more accurate and relevant responses to user prompts.
2 Describe the process of fine-tuning an LLM on instruction data, highlighting the key steps involved.
My answer: We first format instructions input dataset in form of prompt. Then we split input data into batches where we padd each input to the longest input in particular batch. Every batch could have different max length. Padding token is |endoftext| token. Then we create input/target pairs where target is input shiffted for one token to the right. If we reached last input token, then we add endoftext token. We first padd targets then replace all padded tokens with -100, except for the first one that is closest to last token. -100 value are skipped while calculating the loss function. Then we train the model on that data using same algorithm as for pretraining step.
Correct answer: The process involves loading a pretrained LLM, preparing an instruction dataset, and training the model on this dataset using a suitable loss function and optimizer. The training process aims to minimize the loss, indicating improved performance in following instructions.
My answer is wrong because I did not mention loading pre-trained model.
3 What are some potential challanges encountered during fine-tuning an LLM on instruction data, and how can these challanges be addressed?
My answer: ?
Correct answer: Challenges include hardware limitations, such as memory constraints, which can be addressed by using a smaller model, reducing the batch size, or utilizing a GPU for faster training. Additionally, managing the length of input sequences can be crucial for efficient training.
4 How is the effectivness of fine-tuning evaluated during the training process?
My answer: Input set is randomly split in training, validation and test sets. We are calculating entropy loss function for those steps.
Correct answer: The effectiveness is evaluated by monitoring the training and validation losses. A decrease in these losses indicates that the model is learning to follow instructions better. Additionally, inspecting the generated responses during training provides qualitative insights into the model’s progress.
5 What is the significance of the Alpaca dataset in the context of fine-tuning LLMs?
My answer: Alpaca dataset is in form of prompt, setting first the context for instruction that should be resolved.
Correct answer: The Alpaca dataset is a valuable resource for fine-tuning LLMs on instruction data. It provides a large collection of diverse instructions and corresponding responses, enabling the model to learn a wider range of task-specific behaviors.
Its a larger instruction training dataset.
7.7 Extracting and saving responses
1 Describe the process of evaluating the performance of an instruction-fine-tuned Large Language Model (LLM) after training.
My answer: We have two options. First is that humans checks the quality of resolving instruction tasks. This technique does not scale, so we have second option, which is to use another llm for instruction tasks resloving. We asked for example ollama3 model to score the instruction task model resolution.
Correct answer: Evaluation involves extracting model-generated responses from a held- out test set, manually analyzing them, and then quantifying the response quality using various methods such as benchmarks, human comparison, or automated metrics.
2 What are the different methods for evaluating the performance of instruction-fine-tuned LLMs, and what are their relative strengths ans weaknesses?
My answer: Human evaluation and another llm score evaluation. Human evaluation does not scale well.
Correct answer: Methods include short-answer benchmarks (e.g., MMLU), human preference comparisons, and automated conversational benchmarks (e.g., AlpacaEval). Human evaluation provides valuable insights but is time-consuming, while automated methods are efficient but may lack the nuance of human judgment.
3 What is meant by conversational performance in the context of LLMs, and why is it important?
My answer: ?
Correct answer: Conversational performance refers to an LLM’s ability to engage in human-like communication, understanding context, nuance, and intent. It’s crucial for applications like chatbots where natural and coherent interaction is essential.
Understanding context, naunce and intent.
4 How can the response generated by an LLM be automatically evaluated using another LLM, and what are the advantages of this approach?
My answer: We construct a prompt that askes from another llm to score the response of our model. We send to model instructions and correct response.
Correct answer: An approach similar to AlpacaEval can be used, employing another LLM to evaluate the responses. This automated method is efficient, saving time and resources compared to manual human evaluation while still providing meaningful performance indicators
5 Explain the process of appending generated model responses to a test set and saving the updated data for later analysis.
My answer: ?
Correct answer: The generated model responses are added to a dictionary containing the test data. This updated data is then saved as a JSON file (e.g., ‘instruction-data-with-response.json’) for easy access and analysis in future sessions.
Saving model responses to json file.
7.8 Evaluating the fine-tuned LLM
1 What is the purpose of using a larger LLM to evaluate the responses of a fine-tuned model?
My answer: Purpose is to automate evaluation process of our fine-tuned llm.
Correct answer: A larger LLM, like Llama 3 or GPT-4, can be used to automatically assess the quality of responses generated by a fine-tuned model. This provides a more objective and scalable method for evaluating model performance compared to manually reviewing a few examples
2 Describe the process of using Ollama to evaluate the responses of a fine-tuned model.
My answer: We use testing instruction dataset. In json file we store instruction, model response, and correnct response (output). Then we send to ollama3 input, output and model response to evaluate model response. For that we create a prompt where we are asing ollama3 to score (0-100) model response.
Correct answer: Ollama, an open-source application, allows you to run LLMs locally. You can use the query_model function to send prompts to the LLM, such as asking it to score a model’s response on a scale of 0 to 100. The LLM’s evaluation can then be used to assess the overall performance of the fine-tuned model.
3 What aer some alternative LLMs that chan be used for evaluating model responses, besides the 8-billion-parameter Llama 3 model?
My answer: We could use Qwen 8B model.
Correct answer: Other LLMs, such as the 3.8-billion-parameter phi3 model or the larger 70-billion-parameter Llama 3 model, can be used with Ollama. The choice of model depends on the available computational resources and the desired level of performance.
4 How can the generate_model_score function be used to assess the performance of a fine-tuned model?
My answer: Each entry from test instructions dataset is sent to ollama3 evaluation in the form of prompt that request response score 1-100 based on input, expected output and output given by the model.
Correct answer: The generate_model_scores function iterates through a set of test data, sending prompts to the LLM to evaluate each model response. It then calculates the average score across all responses, providing a quantitative measure of the model’s performance.
5 What are some strategies for improving the performance of a fine-tuned model?
My answer: We could implement Lora for model linear layers. Or we could do training on broader instructions data set, like alpaca data set.
Correct answer: Strategies for improving model performance include adjusting hyperparameters during fine-tuning, increasing the size or diversity of the training dataset, experimenting with different prompts or instruction formats, and using a larger pretrained model.
6 Match the terms on the left with their description on the right:
| Test Set | A method of evaluating the conversational performance of a language model using another language model to assess the quality of responses. | |
| Conversational Performance |
The ability of a language model to engage in human-like communication, understanding context, nuance, and intent. | |
| Automated Conversational Benchmarks |
A portion of data that is held out from the training process and used to evaluate the performance of a trained model. |
My answer: 1 => 3, 2 => 2, 3 => 1
Correct answer: +
]]>
