blogs

Working with Data Tables with Playwright Java

2026-06-17T14:31:00Z

Master Playwright Java table automation: locate HTML tables, rows, columns, and cells with assertions and best practices.

Continue reading on Javarevisited »

Working with Data Tables with Playwright Java

2026-06-17T14:31:00Z

Master Playwright Java table automation: locate HTML tables, rows, columns, and cells with assertions and best practices.

Continue reading on Javarevisited »

Recap on fine-tuning to follow instructions

2026-06-17T09:00:00Z

TL;DR

I am recapping the blog post Fine-tuning to follow instructions by reviewing the chapter 7 workbook questions from Build an LLM from Scratch and reading Giles’ blog posts on chapter 7.

Giles blog posts

Giles wrote three blog posts: part24, part25, and part26.

In the part24 post, another sidestep, Giles tries to feed instruction prompts into the GPT-2 models mentioned in the book. The result is that none of those models could give a meaningful answer, but the base Qwen 3 model with 600M parameters gave meaningful instruction results.

In part25, Giles always thinks more deeply about topics presented in books. For example, in relation to the Alpaca prompt format, it was created because interaction with the model was one-shot, due to the limited size of the model context. Today’s models have context sizes in the hundreds of thousands of tokens, so they could accept lengthy conversation transcripts. Then he explains that padding tokens waste GPU cycles, so we have batches of different sizes. The question is why we do not put -100 instead of padding tokens in inputs, but only in targets. Giles explains that this is not valid token vocabulary, but targets are also vocabulary tokens, so I do not understand that reasoning.

part26 is the shortest blog post! It is about evaluating our instruction fine-tuned model using an Ollama 3 model.

Workbook questions

Quick questions

1 What is the primary challange that pretrained LLMs often face regarding instructions?

A Inability to complete sentences
B Difficulty in generating coherent text
C Limited vocabulary size.
D Struggling with specific instructions like grammar correction or voice conversion.

My answer: D

Correct answer: +

2 What is the key component in preparing da dataset for supervised instruction fine-tuning?

A Pre-trained language model.
B Optimization algorithm.
C Instruction-response pairs.
D Tokenization algorithm.

My answer: C

Correct answer: +

3 What data format is commonly used for instruction datasets due to its human and machine redability?

A JSON (JavaScript Object Notation)
B CSV (comma separated values)
C YAML (YAML Ain’t Markup Language)
D XML (Extensible Markup Language).

My answer: A

Correct answer: +

4 What is the purpose of a custom collate function in the context of instruction fine-tuning?

A To optimize the model’s architecture.
B To handle the specific formatting and requirements of the instruction fine-tuning dataset.
C To pre-process the input data.
D To evaluate the model’s performance

My answer: B

Correct answer: +

5 What is the purpose of using the ignore_index parameter (-100) in the custom collate function?

A To mark the end of a sequence.
B To indicate an unknown token.
C To exclude padding tokens from the loss calculation.
D To identify the start of a sequence.

My answer: C

Correct answer: +

6 What is the purpose of saving the fine-tuned model’s state dictionary?

A To compress the model for efficient storage.
B To visualize the model’s architecture.
C To save the model’s parameters for later use or reuse in other projects.
D To improve the model’s performance.

My answer: C

Correct answer: +

7.1 Introduction to instruction fine-tuning

1 What is the primary function of a pretrained LLM?

My answer: The primary function of a pretrained LLM is to provide model knoweledge of language in which was trained grammar patterns.

Correct answer: A pretrained LLM is primarily capable of text completion, meaning it can finish sentences or write text paragraphs given a fragment as input.

It seems that I still do not know to describe the primary task of pre-trained LLM.

2 What is the challange that pretrained LLMs often face?

My answer: The challange is that pretrained LLMs are not good at resolving instructions tasks.

Correct answer: Pretrained LLMs often struggle with specific instructions, such as grammar correction or voice conversion, requiring further fine-tuning.

This one is correct.

3 What is the purpose of instruction fine-tuning?

My answer: The purpose of instruction fine-tuning is to train model for instructions tasks solving using as fine-tuning data instruction set with response how to resolve this task.

Correct answer: Instruction fine-tuning aims to improve an LLM’s ability to follow specific instructions and generate desired responses based on those instructions.

My answer is correct.

4 What is a crucial aspect of instruction fine-tuning?

My answer: A crucail aspect of instruction fine-tuning is that we make LLM into chatbot.

Correct answer: Preparing a suitable dataset is a key aspect of instruction fine-tuning, providing the model with examples of instructions and desired responses.

Wrong answer.

7.2 Preparing a dataset for supervised instruction fine-tuning

1 What is the purpose of the instruction dataset used for fine-tuning a pretrained LLM?

My answer: Purpose of the instruction dataset is to provide training, validation and testing datasets so we could fine tune LLM to resolve instruction tasks.

Correct answer: The instruction dataset consists of instruction-response pairs that are used to train the LLM to follow instructions and generate appropriate responses based on given inputs.

My answer is wrong.

2 Describe the format of the instruction dataset used in this section.

My answer: Each record has three keys, instructions contains instruction task, input is empty and output contains result of instruction task.

Correct answer: The dataset is stored in a JSON file and contains entries that are Python dictionary objects. Each entry includes an ‘instruction’, ‘input’, and ‘output’ field, representing the task, input data, and desired response, respectively._

I forgot to answer that format is JSON.

3 What are the two prompt styles mentioned in the section, and how they differ?

My answer: We have alpaca format and phi3 format. In phi3 we are using user and assistant tags, and in alpaca we use instructions, input and output tags.

Correct answer: The two prompt styles are Alpaca and Phi-3. Alpaca uses a structured format with defined sections for instruction, input, and response, while Phi-3 employs a simpler format with designated <|user|> and <|assistant|> tokens

My answer is correct.

4 What is the purpose of format_input action and how does it work?

My answer: The purpose of format_input function is to format input dataset to alpaca or phi3 prompt format. It creates a chatbot prompt.

Correct answer: The format_input function converts the entries in the instruction dataset into the Alpaca-style input format. It constructs a formatted string that includes the instruction, input (if available), and a placeholder for the response.

My answer is wrong.

5 How is the instruction dataset divided into training, validation, and test sets?

My answer: We randomly divide instruction dataset, 85% is training data, 10% is testing set, and rest is validation set.

Correct answer: The dataset is divided into training, validation, and test sets using a specific ratio. The training set is used to train the model, the validation set is used to evaluate the model’s performance during training, and the test set is used to evaluate the model’s final performance.

My answer is correct.

6 Match the terms on the left to their description on the right:

My answer: Instruction-response pairs => 3, Prompt styles => 1, Alpaca prompt style => 4, Phi3 Prompt style => 2

Correct answer: +

7.3 Organizing data into training batches

1 What is the purpose of a custom collate function in instruction fine-tuning?

My answer: Purpose of a custom collate function is to prepare prompts in alpaca or phi3 formats from input data. It also pads input data to to longest batch value, and sets in target data padding tokens to -100 value, except for the one next to last token.

Correct answer: A custom collate function is used to handle the specific requirements and formatting of the instruction fine-tuning dataset. It ensures that training examples are padded to the same length within each batch, allowing for efficient processing by the model.

My answer is correct.

2 How does the custom collate function handle padding in instruction fine-tuning?

My answer: It padds inputs to the value of longest input in the batch. Targets padding tokens are replaced with -100 instead of one next to last token.

Correct answer: _The custom collate function pads training examples to the length of the longest example in each batch, using the <

endoftext

> token ID (50256). This minimizes unnecessary padding by only extending sequences to match the longest one in each batch._

I forgot to answer value of padding token.

3 Explain the role of target token IDs in instruction fine-tuning and how they are generated.

My answer: Target token IDs are pair with the input so we know which token should come next. It is generated from input so that input is shiffted by one token to the left.

Correct answer: Target token IDs represent the desired output sequence that the model should generate. They are created by shifting the input token IDs one position to the right, omitting the first token and appending an end-of- text token. This setup allows the model to learn how to predict the next token in a sequence.

One token to the right and I did not answer that we are adding endoftext token when we reach end of input.

4 Why are padding tokens replaced with the -100 placeholder value in the target token IDs?

My answer: We are replacing padding tokens with -100 because -100 tokens are not considered in loss calculation.

Correct answer: Replacing padding tokens with -100 allows the cross entropy loss function to ignore them during training. This ensures that only meaningful data influences model learning and prevents padding tokens from contributing to the loss calculation

I did not answer that padding tokens should not contribute to loss function calculation.

5 What is the purpose of retaining one end-of-text token in the target sequence?

My answer: Keeping that token we let know llm model to know where is the end of input instruction.

Correct answer: Retaining one end-of-text token in the target sequence helps the LLM learn to generate end-of-text tokens, which act as an indicator that the generated response is complete.

Wrong answer, we are kepping endoftext token in target sequence, not input. That was the question.

6 Pieces of code have been removed from four places in this listing. Which of these terms have been removed and where should they go? Note that a term may appear more than once!

A true B false C format D encode_text! E full_text

defp encode_example(entry, tokenizer, opts, prompt_style, __1__) do
    entry
    # The model trains on the instruction/input prompt followed by the
    # expected response target, matching the chapter 7 PyTorch dataset.
    |> __3__(prompt_style)
    |> __4__(tokenizer, opts)
  end

  defp encode_example(entry, tokenizer, opts, prompt_style, __2__) do
    token_ids =
      entry
      |> __5__(prompt_style)
      |> __6__(tokenizer, opts)

    prompt_length =
      entry
      |> prompt_text(prompt_style)
      |> encode_text!(tokenizer, opts)
      |> Kernel.length()

    %{token_ids: token_ids, prompt_length: prompt_length}
  end

  defp encode_example(
         _entry,
         _tokenizer,
         _opts,
         _prompt_style,
         mask_out_instructions_in_target
       ) do
    raise ArgumentError,
          "expected :mask_out_instructions_in_target to be a boolean, got: #{inspect(mask_out_instructions_in_target)}"
  end

My answer: 1 => B, 2 => A, 3 => E, 4 => D, 5 => E, 6 => D

Correct answer: +

7.4 Creating data loaders for an instruction dataset

1 What is the purpose of the custom_collate_fn function in the context of instruction fine-tuning?

My answer: The purpose is to prepare input data into batches with input/target pairs, to padd data with endoftext tokens and skip loss function calculation token.

Correct answer: The custom_collate_fn function is used to batch the instruction dataset, ensuring that the input and target tensors are moved to the specified device (CPU, GPU, or MPS) before being fed into the LLM for fine-tuning.

2 Explain the advantage of moving data to the target device within the custom_collate_fn instead of the main training loop.

My answer: Advantage is faster calculation for training data.

Correct answer: By performing the device transfer within the custom_collate_fn, the process becomes a background task, preventing it from blocking the GPU during model training and improving efficiency

3 How is the device setting determined and used in the custom_collate_fn?

My answer: Did not answer as Elixir implmentation is different.

Correct answer: The device setting is determined based on the availability of a GPU or MPS. The partial function from functools is used to create a new version of the custom_collate_fn with the device argument prefilled, ensuring that the function uses the correct device for data transfer.

4 What is the purpose of the allowed_max_length parameter in the customized_collate_fn?

My answer: We truncate inputs and targets to the value of this parameter. If value is nil, we keep original size of inputs/targets.

Correct answer: The allowed_max_length parameter is used to truncate the data to the maximum context length supported by the LLM model being fine-tuned, in this case, the GPT-2 model.

5 Describe the process of creating data loaders for the training, validation, and test sets using the DataLoader calss.

My answer: We randonly split input data set to the precentage of total input data set: training is 85%, validation is 10 % and rest is test set.

Correct answer: The DataLoader class is used to create data loaders for each set (training, validation, and test). The batch_size, collate_fn, shuffle, drop_last, and num_workers parameters are configured to control the batching process, shuffling, and data loading behavior_

6 Pieces of the code have been removed from four places in the listing. Which of these terms have been removed and where should they go?

A inputs B targets C maybe_truncate

def custom_collate(batch, pad_token_id, ignore_index, allowed_max_length)
      when is_list(batch) do
    examples = Enum.map(batch, &normalize_example/1)

    # Find the longest sequence length after the one extra pad token that the
    # Python collate function appends to every item.
    batch_max_length =
      examples
      |> Enum.map(&(Kernel.length(&1.token_ids) + 1))
      |> Enum.max(fn -> 0 end)

    {inputs, targets} =
      Enum.map(examples, fn %{token_ids: item, prompt_length: prompt_length} ->
        # Elixir data is immutable, so this builds the equivalent of
        # `new_item = item.copy(); new_item += [pad_token_id]`.
        new_item = item ++ [pad_token_id]

        # Pad the copied item to the longest sequence length in the batch.
        padded = pad_to_length(new_item, batch_max_length, pad_token_id)

        # Match `padded[:-1]` for inputs and `padded[1:]` for next-token
        # targets.
        inputs = Enum.slice(padded, 0, batch_max_length - 1)

        targets =
          padded
          |> Enum.slice(1, batch_max_length - 1)
          |> mask_extra_padding_targets(pad_token_id, ignore_index)
          |> mask_prompt_targets(prompt_length, ignore_index)

        # Optionally cap both rows to the model context length.
        {
          maybe_truncate(inputs, allowed_max_length),
          __1__(targets, allowed_max_length)
        }
      end)
      |> Enum.unzip()

    {
      Nx.tensor(__2__, type: {:s, 64}),
      Nx.tensor(targets, type: {:s, 64})
    }
  end

My answer: 1 => C, 2 => A

Correct answer: +

7 Match the terms on the left with their description on the right:

Term		Description
Batching Process		A function that defines how to combine individual data samples into a batch, specifically tailored for the requirements of instruction fine-tuning.
Custom Collate Function		The process of organizing training data into groups of samples, called batches, to improve training efficiency.
Padding Tokens		Special tokens added to the end of sequences to ensure all inputs in a batch have the same length, allowing for efficient processing by the model.
Ignore Index		A special value, typically -100, used to indicate padding tokens in the target sequence, preventing them from contributing to the loss calculation during training.

My answer: 1 => 2, 2 => 1, 3 => 3, 4 => 4

Correct answer: +

7.5 Loading a pretrained LLM

1 Why is larger pretrained model, like gpt2-medium, prefered for instruction fine-tuning compared to the smaller gpt2-small model?

My answer: Because pretrained medium model shows better performace for instruction fine-tuning.

Correct answer: The smaller model lacks the capacity to learn and retain the complex patterns and nuanced behaviors required for high-quality instruction- following tasks. The larger model has more parameters, allowing it to handle more intricate instructions and generate more accurate responses.

Did not explain why.

2 What is the purpose of loading a pretrained LLM before starting instruction fine-tuning?

My answer: Purpose is that we reuse step of training the LLM model.

Correct answer: Loading a pretrained LLM provides a foundation for the fine-tuning process. It allows the model to leverage existing knowledge and patterns learned during pretraining, enabling it to learn new tasks more efficiently and effectively.

Did not explain why.

3 How does the code for loading a pretrained model differ from the code used for pretraining of classification fine-tuning?

My answer: ?

Correct answer: The code remains largely the same, but instead of specifying the ‘gpt2- small’ model, we now specify ‘gpt2-medium’ to load the larger model with 355 million parameters. This change reflects the choice of a more capable model for instruction fine-tuning.

Did not think of this simple answer.

4 What is the purpose of evaluating the pretrained LLM’s performance on a validation task before fine-tuning?

My answer: Purpose is to avoid oveffiting LLM model in pretrained phase.

Correct answer: Evaluating the pretrained LLM’s performance provides a baseline understanding of its capabilities before fine-tuning. This allows us to assess the impact of fine-tuning on the model’s ability to follow instructions and generate accurate responses.

Wrong answer.

5 How is the model’s generated response isolated from the input instruction in the provided code?

My answer: It is isolated with dedicated tag: generated_response.

Correct answer: The code subtracts the length of the input instruction from the start of the generated text, effectively removing the input text and leaving only the model’s generated response. The strip() function is then applied to remove any extra whitespace.

I did not answer that we need to remove instructions part from the response.

7.6 Fine-tuning the LLM on instruction data

1 What is the purpose of fine-tuning an LLM on instruction data?

My Answer: The purpose of fine-tuning an LLM on instruction data is to train LLM to understands the patterns for resolving instruction tasks.

Correct answer: Fine-tuning an LLM on instruction data aims to improve its ability to understand and follow instructions, leading to more accurate and relevant responses to user prompts.

2 Describe the process of fine-tuning an LLM on instruction data, highlighting the key steps involved.

My answer: We first format instructions input dataset in form of prompt. Then we split input data into batches where we padd each input to the longest input in particular batch. Every batch could have different max length. Padding token is |endoftext| token. Then we create input/target pairs where target is input shiffted for one token to the right. If we reached last input token, then we add endoftext token. We first padd targets then replace all padded tokens with -100, except for the first one that is closest to last token. -100 value are skipped while calculating the loss function. Then we train the model on that data using same algorithm as for pretraining step.

Correct answer: The process involves loading a pretrained LLM, preparing an instruction dataset, and training the model on this dataset using a suitable loss function and optimizer. The training process aims to minimize the loss, indicating improved performance in following instructions.

My answer is wrong because I did not mention loading pre-trained model.

3 What are some potential challanges encountered during fine-tuning an LLM on instruction data, and how can these challanges be addressed?

My answer: ?

Correct answer: Challenges include hardware limitations, such as memory constraints, which can be addressed by using a smaller model, reducing the batch size, or utilizing a GPU for faster training. Additionally, managing the length of input sequences can be crucial for efficient training.

4 How is the effectivness of fine-tuning evaluated during the training process?

My answer: Input set is randomly split in training, validation and test sets. We are calculating entropy loss function for those steps.

Correct answer: The effectiveness is evaluated by monitoring the training and validation losses. A decrease in these losses indicates that the model is learning to follow instructions better. Additionally, inspecting the generated responses during training provides qualitative insights into the model’s progress.

5 What is the significance of the Alpaca dataset in the context of fine-tuning LLMs?

My answer: Alpaca dataset is in form of prompt, setting first the context for instruction that should be resolved.

Correct answer: The Alpaca dataset is a valuable resource for fine-tuning LLMs on instruction data. It provides a large collection of diverse instructions and corresponding responses, enabling the model to learn a wider range of task-specific behaviors.

Its a larger instruction training dataset.

7.7 Extracting and saving responses

1 Describe the process of evaluating the performance of an instruction-fine-tuned Large Language Model (LLM) after training.

My answer: We have two options. First is that humans checks the quality of resolving instruction tasks. This technique does not scale, so we have second option, which is to use another llm for instruction tasks resloving. We asked for example ollama3 model to score the instruction task model resolution.

Correct answer: Evaluation involves extracting model-generated responses from a held- out test set, manually analyzing them, and then quantifying the response quality using various methods such as benchmarks, human comparison, or automated metrics.

2 What are the different methods for evaluating the performance of instruction-fine-tuned LLMs, and what are their relative strengths ans weaknesses?

My answer: Human evaluation and another llm score evaluation. Human evaluation does not scale well.

Correct answer: Methods include short-answer benchmarks (e.g., MMLU), human preference comparisons, and automated conversational benchmarks (e.g., AlpacaEval). Human evaluation provides valuable insights but is time-consuming, while automated methods are efficient but may lack the nuance of human judgment.

3 What is meant by conversational performance in the context of LLMs, and why is it important?

My answer: ?

Correct answer: Conversational performance refers to an LLM’s ability to engage in human-like communication, understanding context, nuance, and intent. It’s crucial for applications like chatbots where natural and coherent interaction is essential.

Understanding context, naunce and intent.

4 How can the response generated by an LLM be automatically evaluated using another LLM, and what are the advantages of this approach?

My answer: We construct a prompt that askes from another llm to score the response of our model. We send to model instructions and correct response.

Correct answer: An approach similar to AlpacaEval can be used, employing another LLM to evaluate the responses. This automated method is efficient, saving time and resources compared to manual human evaluation while still providing meaningful performance indicators

5 Explain the process of appending generated model responses to a test set and saving the updated data for later analysis.

My answer: ?

Correct answer: The generated model responses are added to a dictionary containing the test data. This updated data is then saved as a JSON file (e.g., ‘instruction-data-with-response.json’) for easy access and analysis in future sessions.

Saving model responses to json file.

7.8 Evaluating the fine-tuned LLM

1 What is the purpose of using a larger LLM to evaluate the responses of a fine-tuned model?

My answer: Purpose is to automate evaluation process of our fine-tuned llm.

Correct answer: A larger LLM, like Llama 3 or GPT-4, can be used to automatically assess the quality of responses generated by a fine-tuned model. This provides a more objective and scalable method for evaluating model performance compared to manually reviewing a few examples

2 Describe the process of using Ollama to evaluate the responses of a fine-tuned model.

My answer: We use testing instruction dataset. In json file we store instruction, model response, and correnct response (output). Then we send to ollama3 input, output and model response to evaluate model response. For that we create a prompt where we are asing ollama3 to score (0-100) model response.

Correct answer: Ollama, an open-source application, allows you to run LLMs locally. You can use the query_model function to send prompts to the LLM, such as asking it to score a model’s response on a scale of 0 to 100. The LLM’s evaluation can then be used to assess the overall performance of the fine-tuned model.

3 What aer some alternative LLMs that chan be used for evaluating model responses, besides the 8-billion-parameter Llama 3 model?

My answer: We could use Qwen 8B model.

Correct answer: Other LLMs, such as the 3.8-billion-parameter phi3 model or the larger 70-billion-parameter Llama 3 model, can be used with Ollama. The choice of model depends on the available computational resources and the desired level of performance.

4 How can the generate_model_score function be used to assess the performance of a fine-tuned model?

My answer: Each entry from test instructions dataset is sent to ollama3 evaluation in the form of prompt that request response score 1-100 based on input, expected output and output given by the model.

Correct answer: The generate_model_scores function iterates through a set of test data, sending prompts to the LLM to evaluate each model response. It then calculates the average score across all responses, providing a quantitative measure of the model’s performance.

5 What are some strategies for improving the performance of a fine-tuned model?

My answer: We could implement Lora for model linear layers. Or we could do training on broader instructions data set, like alpaca data set.

Correct answer: Strategies for improving model performance include adjusting hyperparameters during fine-tuning, increasing the size or diversity of the training dataset, experimenting with different prompts or instruction formats, and using a larger pretrained model.

6 Match the terms on the left with their description on the right:


Test Set		A method of evaluating the conversational performance of a language model using another language model to assess the quality of responses.
Conversational Performance		The ability of a language model to engage in human-like communication, understanding context, nuance, and intent.
Automated Conversational Benchmarks		A portion of data that is held out from the training process and used to evaluate the performance of a trained model.

My answer: 1 => 3, 2 => 2, 3 => 1

Correct answer: +

Continuous Everything, Except Thinking

2026-06-17T08:37:22Z

My friend Joep Schuurkes wrote a great response to my chapter in Taking Testing Seriously last year, but thanks to LinkedIn’s fantastic algorithm, it didn’t come into my feed and I only saw it when someone sent it to me via email – about a week ago!

His review focuses on the line where I said modern delivery seems obsessed with “continuous everything” except continuous thinking.

“So much today is about “continuous everything” – continuous deployment, continuous integration – and speed. The pace at which people are doing things is really, really disruptive to thinking deeply about problems. Then the goal gets displaced from getting good products and services to customers quickly, and turns into, “How do we back something out quickly when we inevitably screw up?” It’s continuous everything – except continuous thinking.” – Taking Testing Seriously

Joep’s main point is that continuous integration, delivery, and deployment are not really about speed, they are about risk.

Specifically, they are meant to address certain kinds of technical risk: integration risk, deployment risk, release risk. He also makes the distinction between being able to deploy and deciding to release.

On that, we agree.

Being able to deploy as safely as possible should be a business decision, and being able to get that information quickly and respond is very valuable.

And I also agree with Joep that risk appetite matters and changes/adapts to your project and business context. A startup changing website copy is not doing the same job as a bank changing their payments infrastructure.

So yes, technical capability should not determine release cadence. Risk should.

But here’s where I disagree with Joep about how continuous integration/delivery/deployment is positioned or delivered by the Agile industrial complex.

The continuous integration and deployment folks do a little more than “imply speed”, and that has a direct result in how it’s implemented.

In the purest textbook version, CI/CD might be about reducing delivery risk, but that’s not how it’s sold. It’s sold as “faster time to market,” “code releases happen faster,” “ship better quality code faster,” “run pipelines in minutes, not hours,” and “get code to production faster.”

And in my experience, that is exactly how I’ve encountered it in the wild.

In enterprise tech, that gets translated into velocity targets, throughput measurement, increased deployment frequency, dopamine dashboards, and pressure to ship if everything’s green.

That’s what I’m talking about.

As well, CI/CD evangelists just like XP bros or most agilistas, seem to love gaslighting people trying to use this stuff with the same “my process isn’t broken, you’re just doing it wrong”, which only makes cutting corners more likely.

So the best version of “continuous anything” may be sensible, but the common version I see teams struggling to implement (or living with the consequences of test automation elevated to a quality strategy) has the business (and testers) losing the time and agency required to think – continually.

Anyway, thanks for the feedback Joep!

The post Continuous Everything, Except Thinking first appeared on Quality Remarks.

How to safely build trust in AI generated code

2026-06-17T06:30:00Z

You’re rolling out AI-assisted engineering wrong, and it's going to slow you down! Learn to buidl trust safely and quickly.

The post How to safely build trust in AI generated code first appeared on The Quality Duck.

Selenium Nightly Releases: A QA Discipline Habit

2026-06-17T03:35:42Z

Selenium nightly releases are not random noise. They are a cheap weekly drill for release-note reading, browser compatibility checks, and safer automation upgrades.

The post Selenium Nightly Releases: A QA Discipline Habit appeared first on Software Testing & Automation.

Playwright Visual Testing: Day 9 Tutorial

2026-06-17T03:34:29Z

Learn Playwright visual testing with TypeScript: screenshots, baselines, masks, ARIA snapshots, CI diff review, and common pitfalls.

The post Playwright Visual Testing: Day 9 Tutorial appeared first on Software Testing & Automation.

AI-Augmented QA: The Complete Workflow for Using LLMs to 10x Your Testing Productivity

2026-06-16T22:00:00Z

Before: 2 hours writing tests. After: 10 minutes with AI + 20 minutes refining. The 5-step AI QA workflow for test generation, debugging, risk analysis, and bug reports.

The post AI-Augmented QA: The Complete Workflow for Using LLMs to 10x Your Testing Productivity appeared first on Software Testing & Automation.

A holistic approach to "spec-driven development"

2026-06-16T16:11:27Z

A common phrase we hear in the context of AI-assisted development is "spec-driven development" or SDD. There are many definitions of SDD floating around. Here is a good one from Birgitta Böckeler:

Spec-driven development means writing a “spec” before writing code with AI (“documentation first”). The spec becomes the source of truth for the human and the AI.

So what's a "spec"? Birgitta offers this definition:

A spec is a structured, behavior-oriented artifact - or a set of related artifacts - written in natural language that expresses software functionality and serves as guidance to AI coding agents.

There are many AI-based tools available for teams who want to practice SDD. They can build code exactly to the spec given to them. What we miss in all the talk about SDD is - where does the spec come from? How do we know it really specifies the capabilities that the customers want and need?

For those that missed the world of waterfall, spec stands for specification. The specs that were written then are not what we mean now. Let’s dive into that a bit more.

We still need the good practices and principles

The way teams build production and test code is clearly changing. The principles and practices we have followed to create examples, executable tests, prototypes and other artifacts that show how each new feature behaves have not changed. It's even more important that we build the shared understanding of each new software change.

When you go to learn about SDD, it sounds like a new waterfall process - create a big batch of specs, feed it to the agents, and generate a huge amount of code. We're hearing more every day about "cognitive debt", also called "comprehension debt", on the part of teams trying to review and understand these huge batches of generated code. It's not humanly possible.

Applying the Holistic Testing Model to SDD

Looking at the Holistic Testing Model, would we change anything when using AI to create code and tests from specs using AI agentic development? Whether or not we adopt AI-assisted development, working incrementally and iteratively, one small change at a time, is the road to success.

We still need to test the business ideas and make sure they will solve customers' problems. We still need to analyze and manage risks. It's still important to prioritize the top few quality attributes for each new change. Sure, AI can help us process analytics about customer needs, so maybe it can speed up some of these activities.

Build quality in throughout the SDLC

Using examples and tests to guide development is even more critical if we're going to trust AI tools to build the code for a new capability. Behavior-driven / acceptance test-driven development and Specification by Example (remember that?) help make sure we deliver what our customers want and need. And yes, AI can speed up building prototypes that help to ensure we know how a new change should behave.

Automating the tests based on examples that emerge from conversations among delivery and business team members goes more quickly with AI assistance. However, we still need those human conversations to ensure shared understanding of what to build.

Some of the testing activities we plan based on the right-hand side of the model will also be quicker and easier if we employ AI tools wisely. They still need to happen because we don't want to take our human brains out of the equation.

If you take nothing else away from this post, just remember, "work in small batches". This principle for successful software development, happy teams and happy customers dates from the 1970s. It's time more teams learned and followed it.

More learning

If you'd like more details about how using a specification-by-example approach, making one tiny change at a time, we recommend Antony Marcano's work on this topic. His article, "What (Almost) Everyone Gets Wrong About Spec-Driven Development with AI", is a great place to start. We also enjoyed the video Antony generated from it using Google NotebookLM.

Test the Feature, Not the Endpoint

2026-06-16T06:00:00Z

They say, an API is a window into a whole new world. Well, I said it.

But really, it’s a peep hole. We need a bunch of them to really get a good picture.

An API might get you something, or do something for you. But it rarely comes alone. In order to be effective, it needs other friends.

For example, we can create a user with a POST API. But unless we call a GET on it, that user is lost in the database.

So when we’re testing, we should look at the feature, not just the API.

Which kinds of conflicts with how we “test APIs”. Because they are testable on their own. They are a piece of functionality, that fits into a workflow, that fits into the feature.

But in this context, testing an API is like a unit test.

Lay down the pitchfork, I’m not done. What does a unit test do? It proves that piece of code does what we expect from it.

What does a test for that POST do? THE SAME.

But – I hear you say through the pitchfork – but that API is a real functional piece, that can live on its own, with a possible applicative outcome.

Yes – I say, backing away – so is a function, or a class.

But – you lower the pitchfork, with a confused look on your face – but a unit is something small. An API is really code in different places – server, logic, database. It can be big.

Ok, you got me there. There’s a bunch of differences between API tests and unit tests. But I want to get back to the important one for our discussion today.

Like a passing unit test doesn’t tell you if the whole system works, a single API test passing doesn’t tell you if the feature is working.

In the case of a unit test, you need more integrated tests, maybe API or an end-to-end test to create the confidence it works. An API test? The same.

So, the next time somebody throws you an API spec, and tells you to test it, sure, do that. But if you really want to know the whole feature is working? Add a couple of workflow, multi-API, end-to-end tests.

You’ll sleep better.

Want to go deeper into feature-level test design and multi-API workflows?

→ The Tactician — hands-on test design, assertions, debugging, and AI-powered testing. Coming soon.

→ The Strategist — planning, prioritization, and deciding what to test before you open Postman.

The post Test the Feature, Not the Endpoint first appeared on TestinGil.

AI QA Agents: From Prompts to Runnable Checks

2026-06-16T04:35:17Z

AI QA agents should not stop at test-case text. Day 8 shows a practical workflow that turns prompts into runnable checks, evidence, and eval gates.

The post AI QA Agents: From Prompts to Runnable Checks appeared first on Software Testing & Automation.

Playwright API Testing: Day 8 Tutorial

2026-06-16T03:35:58Z

Playwright API Testing tutorial for Day 8: write REST checks, seed UI state, mock network responses, and debug API failures in TypeScript.

The post Playwright API Testing: Day 8 Tutorial appeared first on Software Testing & Automation.

Playwright MCP vs Traditional Test Scripts

2026-06-16T03:35:25Z

Playwright MCP vs traditional test scripts is not a replacement debate. Compare repeatability, observability, CI fit, and debugging cost before agents touch your release gate.

The post Playwright MCP vs Traditional Test Scripts appeared first on Software Testing & Automation.

Book Review: Software Testing: A Craftsman`s Approach

2026-06-16T00:00:00Z

My insights from "Software Testing: A Craftsman`s Approach" by Paul C Jorgensen

How to Communicate QA Value to Non-Technical Stakeholders: The Language That Gets Budget

2026-06-15T22:00:00Z

Zero incidents, clean releases — and nobody noticed. How to translate invisible QA work into language executives understand: money saved, time gained, risk reduced.

The post How to Communicate QA Value to Non-Technical Stakeholders: The Language That Gets Budget appeared first on Software Testing & Automation.

Your test isn’t flaky: Page 2 just stole data from page 1

2026-06-15T17:00:00Z

The problem nobody spots right away

You run your end-to-end tests. All green. You run them again. One of them fails, in a spot where you changed nothing. Run them once more and that test passes again, but a different one starts complaining.

Frustrating? Yes, it is! And your first reflex is logical. You blame the test, or the test framework that must be “flaky”. It looks like pure bad luck after all. Red today, green tomorrow, without a single change to the code.

Yet that picture is wrong again. The test is exposing something real. There is a small bug in the production code. The sorting tells the database which field to sort on, but not what to do when two records share the same value.

The database then fills that gap itself. And not always in the same way. The result is that the order of records is not fixed. In production you rarely notice it. Your tests put their finger right on it.

Where it really goes wrong: the paged table

We rarely show a list all at once. Displaying thousands of orders onto a single screen is unreadable and very slow. So we cut them up into pages. Page 1 shows the first ten records, page 2 the next ten, and so on.

It is important to realise that every page is a separate question to the database. The database does not remember what it gave you on page 1. Behind each page sits the very same query again. The only difference is an offset that says “skip the first so many records” and a limit that says “then give me ten”.

Picture a screen with orders, sorted by creation time. Per page the database receives something like this:

-- page 1
SELECT * FROM orders ORDER BY created_at LIMIT 10 OFFSET 0;
-- page 2
SELECT * FROM orders ORDER BY created_at LIMIT 10 OFFSET 10;

As long as every record has a unique timestamp, this goes fine. In our database, we sort always on date, even if another sorting parameter is used. We then sort second on date. The order is fixed, so “the first ten” on page 1 really are the first ten. The database counts off ten, and page 2 starts neatly at the eleventh. No overlap, no gaps.

But what if several records share the same timestamp? Then the database has no idea which of them is “first”. To it, those records are completely equal. And because each page is a separate query, it may put those equal records in one order on page 1 and in a different order on page 2.

The dangerous consequence is easy to guess. A record you just saw on page 1 can suddenly show up again in the query for page 2. And another record then falls exactly between the two pages and disappears from view entirely. Same data, same query, and yet a different result every time.

For a test that checks page 1 first and then page 2, that is fatal. The content shifts under your feet, without anything in the data having changed.

Why test data triggers this so quickly

In production this bug rarely occurs. Most of the time people work at a human pace: one record in the morning, the next half an hour later. Timestamps stay unique, the sort order holds, and nobody notices anything is off. But it is not impossible. When a team is working at full speed and multiple colleagues save records at nearly the same moment, the sort order can become unstable. The bug is there, it just does not get triggered often enough to raise an alarm.

The automated tests told a different story and they did so before a single user ever touched the application. Because tests create records at machine speed, a dozen records land in the database within the same second. They all share the same sort value and the instability shows up immediately. The tests go red. Not every run, not reliably, just now and then. And that irregular pattern is exactly what makes you think of flaky tests first, rather than a real defect in the production code.

In a test everything runs at top speed. No waiting user, no thinking time. In a loop you create ten orders, and they all sit within the same second:

// Cypress: quickly set up ten records
for (let i = 0; i < 10; i++) {
 cy.request('POST', '/api/orders', { customer: `customer-${i}` });
}

In MariaDB a DATETIME column without fractional seconds stores time accurate to the second. If ten records are created within the same second, they all get the exact same value. Ten records, one sort value. The database has to guess who comes first, and that guess may turn out differently per page.

But there is a scenario that is even worse: a DATE column. MariaDB’s DATE type stores only the year, month, and day, without any hours, minutes, or seconds. Every record created on the same calendar day gets the exact same sort value. In a test that creates ten records in a loop, they all land on today’s date. And this time the problem is not limited to your test environment. In production too, dozens of records can share the same date. The instability is then always there, not just when the test runs fast.

That is why this is such a nasty bug. By hand you almost never click it loose, because you are too slow. But your automated test is lightning fast and provokes the problem on purpose. So your test is not the cause, it is the first thing fast enough to see the bug.

The solution: a tie-breaker

A tie-breaker is a second sort field. It steps in the moment the main field has equal values. That way the database always keeps a fixed way to cut the knot.

The best tie-breaker field is the id. Every record has a unique id, so two records with the same timestamp still get a fixed order.

-- without tie-breaker: order uncertain on equal timestamp
ORDER BY created_at

-- with tie-breaker: always the same order
ORDER BY created_at, id

For the Java developers on your team. In Spring Data this is a small addition. The code was already sorting on createdAt, but missed the second field. You just stick id on the end:

// Sort by createdAt (ASC), with id as tie-breaker
Sort sort = Sort.by(Sort.Direction.ASC, "createdAt")
 .and(Sort.by(Sort.Direction.ASC, "id"));

List<Order> orders = orderRepository.findAll(sort);

Working with pagination? Then you put that same sorting into a Pageable. The tie-breaker then carries neatly across all pages:

Pageable pageable = PageRequest.of(
 0, // page number (0-based)
 20, // page size
 Sort.by("createdAt").ascending()
 .and(Sort.by("id").ascending())
);

Page<Order> page = orderRepository.findAll(pageable);

With that second field added the rule is: equal timestamp? Then the lowest id wins. Always the same record, in every query, on every page, in every test run.

What you can do as a tester

Do you see unstable tests around the order or the content of a list? These are strong questions for your development team:

“Do we also sort on id as a second field?” The key question. Is the answer no, and does the sorting happen on a timestamp in seconds? Then you have probably found your cause.

“Is the column a DATE, a DATETIME, or a TIMESTAMP?” A DATE column in MariaDB stores only the calendar day, so every record created on the same day is completely equal for sorting purposes. The instability then shows up in production too, not just in tests. A DATETIME or TIMESTAMP in seconds is already much better, but milliseconds give you the most breathing room.

“Can we create the test data with a fixed order?” Sometimes it helps to deliberately keep control over the order in your test, instead of relying on the timestamp.

Why you should not fix your tests if you think it it flaky

When a test goes red now and then, the temptation is great to loosen it until it goes green again. For instance by no longer checking the exact order, but only whether a record sits somewhere on the page:

// strict: relies on the exact order
cy.get('[data-test=row]').first().should('contain', 'customer-0');

// looser: only checks the record is present somewhere
cy.get('[data-test=row]').should('contain', 'customer-0');

It looks like a quick fix, but think through what is really happening here. Suppose the bug strikes while your test is on page 2. A record that actually belongs on page 1 now shows up again on page 2. And a record that did belong on page 2 has disappeared entirely. So your test sees the wrong content, and it fails rightly.

That is not noise. That is exactly what a real user gets in front of them too. They click to the next page and see an order they just saw, while another order shows up nowhere. The data seems to jump around.

In other words: your test is not unstable, it is doing its job perfectly. It catches a real bug. If you loosen your assertions to get rid of the red, you hide a problem that genuinely affects your users. The bug is not in your test, and so it should not be fixed there.

The right reflex is therefore not “how do I get my test green”, but “how do I prove this is a real bug”. Keep your strict check. Try to reproduce the pattern deliberately by creating records in quick succession and then paging through them. And pass it on to your team, with the missing tie-breaker as the concrete solution.

A small change in the code, with a big impact on the stability of your test suite.

Reading the Buzz: Getting Past a Quiet No

2026-06-15T16:02:18Z

This week’s post is authored by Colette Molteni. Colette and I connected after I wrote about what six months on the PCT taught me about presence and clarity. Colette writes Empathy Elevated, a publication about the skills that matter more as technical work accelerates - judgment, emotional intelligence, and clear communication.

This piece is about something I’ve watched happen in hundreds of meetings: the silent ‘no’ that everyone in the room can feel and nobody names. I think you’ll find it as useful as I did.

Subscribe to Empathy Elevated here.

You are sitting in sprint 2, and this Agile project is beginning to look like a monstrosity; instead of a nice, orderly project you can close out in JIRA at the end of the quarter.

The consultant has their screen up, with the user stories. The stories should incrementally radically change the system they have been using for a decade. It’s not a repeat of what we have been doing manually in an online tool, but rather a rethink of how it is approached. It is a reimagining of the total build-out for this phase 1 deployment.

One of your team members in the upper-right corner of your Zoom panel is clearly squirming, even though their face is expressionless.

Another in the lower center of your screen has wandering eyes. On camera, but not present.

You don’t let it distract you. The consultant clearly does not either. They end the review lightning-fast, then abruptly ask, “Any questions?”

There is a long 5-second silence, followed by some light head nods and one “yes” from one of the senior leaders, said so softly it sounds like it’s at the end of a tunnel.

Your manager, a senior leader who is more technically oriented, remarks, “I will need to follow up with you later.”

The consultant is perky and exclaims, “Okay, great!”

Team Buzz Is the Room Talking Without Talking

The consultant heard a “yes,” but it was really a “no” camouflaged as one. They bulldozed through the presentation of user stories without a pause. Your side failed to stop the charge forward.

The metrics were fine. The stories on the screen were written cleanly, and nothing was technically wrong. You could see how it would mostly flow right.

But most of the room was not technically minded.

Most of the room had their fight-or-flight response activated at the prospect of a change they did not fully understand.

Without the option to fight, the fight became a whimper with soft-spoken words, and an almost out-of-body experience of trying to squirm away or divert focus elsewhere. It was a charade of pretense, mistaken for honest agreement.

The room said no, just not out loud.

The room was buzzing, with not much to be audible.

I have lived this, as the technically minded stakeholder observing the room, but missing the signal.

I could have put a pause. I could have set the time to actually follow up and avoid a repeat. But no, I missed it, so there was nothing to act on.

I want you to spot the signal and have the moves to get past it.

The Hesitant Yes Costs More Than an Honest No

The hesitant yes is costly, stumbling, and easy to miss

It looks like eyes wandering or another excuse to be off camera. It can be the silence after “any questions?” that carries on, not out of shyness but rather a recalibration of the minds of the observers. The “I’ll get back to you” with no timing or deadlines intact appears like progress, but it’s a dead end.

At its face, an innovation that is better than the current system, more efficient, user-friendly, and easily distributed, as was the case of the end product discussed by the consultant, should be welcomed.

But the mind does not latch on to facts; it often gravitates towards feelings, what is familiar.

If buy-in is skipped, it does not disappear.

It festers.

It resurfaces downstream, louder, like thunder that breaks the silence on a summer evening.

You cannot ignore it then. It is directly in your earshot and in your line of vision.

It is faster now to resolve than to postpone.

Later, it is reworked, and the follow-up continues in perpetuity.

But it can be mitigated early.

The hesitant yes is a pass, and the honest no is the way.

Unpacking the No: A Sequence That Works

You know it, that the soft Yes that’s really a No is not ideal. But how do you get past it?

You can make the moves when the rest of the room remains quiet and accepts the status quo, even though the buzz is contrary. You can dig into why that is, get to the roots, and pull them out.

That’s the only way a project like this one flourishes. Letting honest no’s stay silent is comfortable now, but costly later. Keeps the garden, drops the rhetorical question stacked on top of it.

Four moves - where we go:

Pause - do not read silence as consent. It is easy to do this, but it is a trap. Stop the charge forward if this happens. It is usually the first signal that the room’s buzz is not in consensus, and blockers will pop up like whack-a-mole later. Name it. Yes, out loud. Ask plainly whether there are questions or feedback, and mean it as a real question. Don’t try to fill the silence yourself, as that puts your words in the mouth of someone else. The discipline here is not filling the silence with a yes you conjure up and force.

Ask - it’s not about asking, “Why are you hesitant?” That demands a defense, even for the most emotionally regulated person. A gentler way, as usual, can be as simple as “Can you unpack that a little for me?” It is neutral and asking for a description, not naming a negatively associated emotion. Hand over the floor. Let the other person or people collect their thoughts.

Listen & make them feel heard - is about real listening, not waiting to respond. Lean in your posture a bit. Keep your eyes directed towards them. Don’t worry, you do not need to stare into their eyes like you are reading their soul. No, just make sure you are not distracted, keeping your eyes on your Slack notification on your left monitor. Listen to learn, and let them know, at a deep subconscious level, that they are being heard.

Reflect it back and decide - it’s about naming the real objection out loud. They hear you got it right. After listening intently, you recap for them what you think you heard. It helps you, but it is also a simple acknowledgment. Now that you have it out in the open, you do not have to solve it in the next 59 seconds before your next meeting at the top of the hour. Actually, allowing a little bit of breathing room, even if it is later that day, is better. It settles the nerves. Then set dedicated time to solve it together with them, not at them. This makes move 4 actually deliver the collaboration promise it’s named for.

A Hesitant Yes Is Just a Slower No

A hesitant yes feels like progress. It lets the meeting end on time and the deck advance to the next slide. But it doesn’t close anything — it defers it. The objection you didn’t surface in sprint 2 doesn’t disappear. It waits. And it comes back later as rework, as a stalled rollout, as a follow-up thread that never resolves.

The honest no is uncomfortable in the moment. The hesitant yes is uncomfortable for a quarter.

Pause when the room goes quiet. Ask in a way that invites a description, not a defense. Listen to learn, not to reply. Reflect the real objection back, and set dedicated time to resolve it together. None of these moves requires you to solve the problem in the room. They only require you to name it before it goes underground.

The buzz is the room talking. Your job is to give it permission to be heard.

The First Winter: Clearing Weeds and Planting Trees

2026-06-15T14:08:17Z

In October 2020, we got our new house. Built 2013, it’s not only new to us, and this is a major upside. While I definitely can appreciate many aspects of old houses, the comfort of a modern house fits us first-time parents very well.

The site is pretty large for a regular house, at about 10 500 m². Eight years ago when the house was built, all the existing old trees were cut down. The weeds took over, as they often do after deforestation if nothing else is planted. The two dominant species were blackberries (I’m not sure which kind) and “Cytisus scoparius.” Birch trees also managed to cover some ground.

It didn’t take long before we made other plans for this place. Specifically, I decided to try restoring the forest, in a way that makes sense in this area. That means a majority of oak and beech, some sections of birch, and a sprinkle of cherry, pine, and spruce.

Of course, I’ll be long gone before the grand result of this project. Future generations can hopefully bask in the glory of its success. In the meantime, I’ll enjoy watching these trees grow up.

Please note that I’m a clumsy hobbyist. I do get solid advice from my sister who is a trained gardener, and from various other sources, but you shouldn’t trust anything I say or do.

Autumn Colors, All Around

When we arrived, nature was showing its autumn palette of yellow, red, and brown. The following photos are from walks in the area surrounding our house.

Time To Work

Alright, enough about the surrounding nature. Around mid November I started executing on our plan. At first, I had no power tools or protective gear. I think that lasted about a day, after which I came in covered in scars.

The blackberries were everywhere. I mean it. They often grew up to a height of two meters by growing inside and across the Cytisus scoparius branches, creating what I can only describe as a jungle of barbed wire.

These two have become my mortal enemies, the main targets of this operation. Blackberries is weed that is notoriusly hard to get rid of, and Cytisus scoparius is classified as invasive in some countries, though not in Sweden.

I cleared out a very small portion, including some very promising oaklings. While scarred, my motivation was now on top. I ordered a clearing saw and a full set of protective gear.

While waiting for the delivery, I collected acorns down by the road and planted them in old planting boxes. Finally, I covered them in thick layer of oak leaves, which sadly didn’t stop birds from the enjoying the smörgåsbord. We’ll soon see how many of the 200-300 acorns that make it. My hopes are low on this one.

When the saw arrived, I did a first pass on the area closest to the house. It took a while until I found a good technique and which blade to use.

I soon realized that a more efficient workflow would be to do it in two separate phases:

Get rid of as much blackberry as possible
Cut down all Cytisus scoparius and other excess trees

And so it went.

Finally, the north side of the road was cleared. There remains a bunch of birch trees to be cut, but that can wait a bit.

At this point, I had some space to work with. There’s a road construction planned at the corner of our lot, right through a little pond of self-seeded oaklings. Instead of them getting covered by gravel, I decided to move them to the newly cleared grounds. The first batch was around 25-30 trees, I think.

The Other Side

Awkwardly, the south side of the road remained uncleared. There, the Cytisus scoparius grew even taller and more dense.

Speaking of the multi-sport court – it’s perhaps not the most attractive part of this place, in my opinion. But I’m guessing our toddler might have different opinions on this matter, and so it could come in handy.

An audiobook and some sweat later, the south side was cleared.

A panorama taken from the middle of the scene, just as I was done clearing.

We got help from relatives to pile up the remaining branches. There’s a little bit left to do but the main chunk of work is done.

Planting Oak and Beech

In a handful spots there grew small oak and beech plants that needed moving. Either they grew too densely, below larger trees where they’d only get shade, or in areas that needed to be kept clear (sides of the road, for example.) I started moving them into newly cleared areas to get a nicer spread of trees across the entire area.

I’ve replanted these trees with marker sticks. This makes it easy to spot them. Also, I can support the plants by tying them to the stick, and use them to fix the net cages.

Pruning

Even with the fierce competition going on after the deforestation, a bunch of oak trees managed to break through. There are some growing 4-5 m tall, already. Unfortunately they’ve grown very close to other plants and got intertwined, so I’ve spent some time pruning.

Probably the largest of the post-deforestation oak trees, this one got liberated from a crowd of four. I’m very happy with this result!

Even messier, this bunch counted almost ten separate trees. I picked out one and cut down the others. It’s shaped strangely, but it might find its way.

Pruning trees like these is usually done between July and September, but I wanted to give these a full season of unhindered growth. Hopefully I won’t regret that decision.

Springtime!

Last week, spring was declared in southern Sweden. Finally! I spent the weekend caging in small trees to (hopefully) protect them from wildlife, and to give them a more focused upwards growth. I’ve used cable ties to straighten and support the smaller trees, but I’m not sure if that’s a bad idea. Maybe I need to swap them for some softer rope.

Some of the larger trees had grown a bit sideways due to competition, so I bought heavy piles and used them for support.

And that’s where I am now, eagearly awaiting the warmth of spring and summer, and the blossoming of trees and flowers. We really don’t know what might show up!

On the left, a net cage fitted onto a marker stick. On the right, a larger tree now supported by a pile, hopefully making it grow straight and tall.

If you’ve read this far, I hope you have enjoyed the first part of our journey. If so, let me know, and I might post more pictures and words on this subject. You can also follow me on Twitter, where I nowadays torture my followers with both tech and gardening tweets.

Fine-tuning to follow instructions

2026-06-15T09:00:00Z

TL;DR

This post fine-tunes a pretrained GPT-2 355M model on instruction data in Elixir. It prepares an Alpaca-style dataset, builds custom batching and target masking, trains the model for two epochs, saves checkpoints and loss metrics, extracts test responses, and evaluates them with a local Ollama model. The fine-tuned Alpaca-style model reaches an average Ollama score of about 50, while the Phi-3 prompt style and instruction/input masking variants perform slightly worse in these experiments.

This post follows Fine-tuning for classification and uses the source code from my GitHub repository.

7.1 Introduction to instruction fine-tuning

What does instruction fine-tuning mean? Pretraining teaches an LLM how to continue text from the input by predicting the most probable next token. With that objective, an LLM can finish sentences or create paragraphs. After instruction fine-tuning, the LLM can learn to follow commands, such as translating text into Croatian or correcting English grammar in a given input. As with classification, we need to go through several steps: download and prepare the dataset, organize it into batches, create data loaders, load a pretrained model, instruction fine-tune that model, inspect the modeling loss, extract responses, perform qualitative evaluation, and score the responses.

7.2 Preparing a dataset for supervised instruction fine-tuning

The first step is to get the training data and prepare it for instruction training with our pretrained OpenAI GPT-2 model. We are extending the FineTuneDataLoader module with a function that downloads and saves instruction training data in JSON format. The instruction training data was prepared by Sebastian, the book author:

@instruction_data_url "https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch07/01_main-chapter-code/instruction-data.json"

@type instruction_record :: %{
          String.t() => String.t()
        }

  @doc """
  Downloads and loads the instruction fine-tuning dataset.

  If `file_path` already exists, the local file is reused. Otherwise the JSON
  file is downloaded from `url`, saved to `file_path`, and then decoded.

  ## Input Parameters

    * `file_path` - local JSON cache path. Defaults to
      `"instruction-data.json"`.
    * `url` - URL of the instruction dataset JSON. Defaults to the chapter 7
      instruction-data file from the LLMs-from-scratch repository.

  ## Output

  Returns the decoded JSON as a list of maps with string keys. Each decoded
  instruction map contains:

    * `"instruction"` - the task description or question.
    * `"input"` - optional context for the task. This is an empty string when
      no extra input is provided.
    * `"output"` - the expected response used as the supervised fine-tuning
      target.
  """
  @spec download_and_load_instructions_file(Path.t(), String.t()) :: [instruction_record()]
  def download_and_load_instructions_file(
        file_path \\ @instruction_data_path,
        url \\ @instruction_data_url
      ) do
    unless File.exists?(file_path) do
      File.mkdir_p!(Path.dirname(file_path))
      download_file!(url, file_path)
    end

    file_path
    |> File.read!()
    |> Jason.decode!()
  end

The next function we need formats the training data into the Alpaca prompt format that we feed into our LLM as the training dataset:

  @doc """
  Formats an instruction dataset entry as an Alpaca-style model input.

  This mirrors the chapter 7 prompt template:

      Below is an instruction that describes a task. Write a response that
      appropriately completes the request.

      ### Instruction:
      ...

      ### Input:
      ...

  The input section is included only when the entry's `"input"` field is not an
  empty string.

  ## Input Parameters

    * `entry` - map containing string keys `"instruction"` and `"input"`, such
      as records returned by `download_and_load_instructions_file/2`.

  ## Output

  Returns the formatted prompt string without the desired response.
  """
  @spec format_input(instruction_record()) :: String.t()
  def format_input(%{"instruction" => instruction, "input" => input})
      when is_binary(instruction) and is_binary(input) do
    instruction_text =
      "Below is an instruction that describes a task. " <>
        "Write a response that appropriately completes the request." <>
        "\n\n### Instruction:\n#{instruction}"

    input_text =
      if input == "" do
        ""
      else
        "\n\n### Input:\n#{input}"
      end

    instruction_text <> input_text
  end

We split the input data into training, validation, and test sets for the same reason as in classification training. Here is the test that uses those two new functions:

  defmodule LlmFromScratch7Test do
  use ExUnit.Case

  alias LlmScratch.FineTuneDataLoader

  @instruction_data_url "https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch07/01_main-chapter-code/instruction-data.json"

  @tag :download
  test "7.1 downloads and loads instruction data" do
    file_path =
      System.tmp_dir!()
      |> Path.join("llm_scratch_instruction_data")
      |> Path.join("instruction-data.json")

    data =
      FineTuneDataLoader.download_and_load_instructions_file(
        file_path,
        @instruction_data_url
      )

    assert length(data) == 1_100

    assert Enum.at(data, 50) == %{
             "instruction" => "Identify the correct spelling of the following word.",
             "input" => "Ocassion",
             "output" => "The correct spelling is 'Occasion.'"
           }

    assert Enum.at(data, 999) == %{
             "instruction" => "What is an antonym of 'complicated'?",
             "input" => "",
             "output" => "An antonym of 'complicated' is 'simple'."
           }

    entry_50 = Enum.at(data, 50)
    model_input_50 = FineTuneDataLoader.format_input(entry_50)
    desired_response_50 = "\n\n### Response:\n#{entry_50["output"]}"

    assert model_input_50 <> desired_response_50 ==
             "Below is an instruction that describes a task. " <>
               "Write a response that appropriately completes the request." <>
               "\n\n### Instruction:\nIdentify the correct spelling of the following word." <>
               "\n\n### Input:\nOcassion" <>
               "\n\n### Response:\nThe correct spelling is 'Occasion.'"

    entry_999 = Enum.at(data, 999)
    model_input_999 = FineTuneDataLoader.format_input(entry_999)
    desired_response_999 = "\n\n### Response:\n#{entry_999["output"]}"

    assert model_input_999 <> desired_response_999 ==
             "Below is an instruction that describes a task. " <>
               "Write a response that appropriately completes the request." <>
               "\n\n### Instruction:\nWhat is an antonym of 'complicated'?" <>
               "\n\n### Response:\nAn antonym of 'complicated' is 'simple'."

    train_portion = trunc(length(data) * 0.85)
    test_portion = trunc(length(data) * 0.1)
    val_portion = length(data) - train_portion - test_portion

    train_data = Enum.slice(data, 0, train_portion)
    test_data = Enum.slice(data, train_portion, test_portion)
    val_data = Enum.slice(data, train_portion + test_portion, val_portion)

    assert length(train_data) == 935
    assert length(val_data) == 55
    assert length(test_data) == 110
  end
end

7.3 Organizing data into training batches

The process of organizing instruction data into training batches is more complicated than organizing classification data or pretraining data into batches. The reason is that instruction data uses a prompt format. Sebastian breaks this process into five steps:

use prompt template
tokenize prompt templates
pad inputs to the same length
create target token IDs
replace padding tokens with placeholders

Here are the first two steps implemented in the InstructionDataset module, along with a test for this new functionality:

 defmodule LlmScratch.InstructionDataset do
  @moduledoc """
  Pre-tokenized instruction fine-tuning dataset.

  This mirrors the PyTorch `InstructionDataset` example from chapter 7. A
  dataset is built from decoded instruction maps, formats each map as an
  Alpaca-style prompt plus response, and pre-tokenizes the full text during
  construction.

  ## Examples

      iex> data = [
      ...>   %{
      ...>     "instruction" => "Name the capital of France.",
      ...>     "input" => "",
      ...>     "output" => "Paris."
      ...>   }
      ...> ]
      iex> dataset = LlmScratch.InstructionDataset.new(data, "code-davinci-002")
      iex> LlmScratch.InstructionDataset.length(dataset)
      1
      iex> LlmScratch.InstructionDataset.get(dataset, 0)
      [21106, 318, 281, 12064, 326, 8477, 257, 4876, 13, 19430, 257,
       2882, 326, 20431, 32543, 262, 2581, 13, 198, 198, 21017, 46486,
       25, 198, 5376, 262, 3139, 286, 4881, 13, 198, 198, 21017, 18261,
       25, 198, 40313, 13]
  """

  @enforce_keys [:data, :encoded_texts]
  defstruct [:data, :encoded_texts]

  @end_of_text "<|endoftext|>"

  @type instruction_record :: LlmScratch.FineTuneDataLoader.instruction_record()
  @type tokenizer :: String.t()
  @type t :: %__MODULE__{
          data: [instruction_record()],
          encoded_texts: [[integer()]]
        }

  @doc """
  Creates an instruction dataset from decoded instruction records.

  Each record is formatted with `LlmScratch.FineTuneDataLoader.format_input/1`,
  then the target response is appended as:

      \\n\\n### Response:\\n...

  The resulting full text is encoded immediately and stored in
  `dataset.encoded_texts`.

  ## Input Parameters

    * `data` - list of decoded instruction maps. Each map must contain string
      keys `"instruction"`, `"input"`, and `"output"`.
    * `tokenizer` - Tiktoken model name accepted by `Tiktoken.encode/3`.
    * `opts` - optional keyword list controlling tokenization.

  ## Options

    * `:allowed_special` - special tokens allowed when `tokenizer` is a
      Tiktoken model name. Defaults to `["<|endoftext|>"]`.

  ## Output

  Returns a `%LlmScratch.InstructionDataset{}`. The `:data` field contains the
  original records, and `:encoded_texts` contains one pre-tokenized token-id
  list per record.

  ## Examples

      iex> data = [
      ...>   %{
      ...>     "instruction" => "Classify the sentiment.",
      ...>     "input" => "I loved it.",
      ...>     "output" => "Positive."
      ...>   }
      ...> ]
      iex> dataset = LlmScratch.InstructionDataset.new(data, "code-davinci-002")
      iex> dataset.data
      [
        %{
          "input" => "I loved it.",
          "instruction" => "Classify the sentiment.",
          "output" => "Positive."
        }
      ]
      iex> dataset.encoded_texts
      [
        [21106, 318, 281, 12064, 326, 8477, 257, 4876, 13, 19430, 257,
         2882, 326, 20431, 32543, 262, 2581, 13, 198, 198, 21017, 46486,
         25, 198, 9487, 1958, 262, 15598, 13, 198, 198, 21017, 23412, 25,
         198, 40, 6151, 340, 13, 198, 198, 21017, 18261, 25, 198, 21604,
         1800, 13]
      ]
  """
  @spec new([instruction_record()], tokenizer(), keyword()) :: t()
  def new(data, tokenizer, opts \\ []) when is_list(data) do
    # Pre-tokenize each complete training example once so repeated dataset
    # access does not rebuild the prompt or call the tokenizer again.
    encoded_texts =
      Enum.map(data, fn entry ->
        entry
        # The model trains on the instruction/input prompt followed by the
        # expected response target, matching the chapter 7 PyTorch dataset.
        |> full_text()
        |> encode_text!(tokenizer, opts)
      end)

    # Keep the decoded records for inspection while serving encoded examples
    # from `encoded_texts`, which mirrors the PyTorch dataset fields.
    %__MODULE__{
      data: data,
      encoded_texts: encoded_texts
    }
  end

  @doc """
  Returns one encoded instruction example by index.

  ## Input Parameters

    * `dataset` - `%LlmScratch.InstructionDataset{}` returned by `new/3`.
    * `index` - zero-based row index.

  ## Output

  Returns the pre-tokenized token-id list for the requested record.
  """
  @spec get(t(), non_neg_integer()) :: [integer()]
  def get(%__MODULE__{} = dataset, index) when is_integer(index) and index >= 0 do
    Enum.fetch!(dataset.encoded_texts, index)
  end

  @doc """
  Returns the number of records in the dataset.

  ## Input Parameters

    * `dataset` - `%LlmScratch.InstructionDataset{}` returned by `new/3`.

  ## Output

  Returns the number of instruction records as a non-negative integer.
  """
  @spec length(t()) :: non_neg_integer()
  def length(%__MODULE__{} = dataset) do
    Kernel.length(dataset.data)
  end

  defp full_text(%{"output" => output} = entry) when is_binary(output) do
    instruction_plus_input = LlmScratch.FineTuneDataLoader.format_input(entry)
    response_text = "\n\n### Response:\n#{output}"

    instruction_plus_input <> response_text
  end

  defp encode_text!(text, tokenizer, opts) when is_binary(tokenizer) do
    allowed_special = Keyword.get(opts, :allowed_special, [@end_of_text])
    {:ok, token_ids} = Tiktoken.encode(tokenizer, text, allowed_special)
    token_ids
  end
end

The next step is to pad inputs to the same length. Note that each batch can have a different length. We pad to the length of the longest input in the batch.

@pad_token_id 50_256
@doc """
  Pads a batch of token-id sequences and stacks them into an input tensor.

  This mirrors the first draft of the chapter 7 Python collate function:

      inputs_1 = [0, 1, 2, 3, 4]
      inputs_2 = [5, 6]
      inputs_3 = [7, 8, 9]
      batch = (inputs_1, inputs_2, inputs_3)
      print(custom_collate_draft_1(batch))

  The function finds the longest sequence length in the batch after adding one
  extra pad token, pads every item to that length, then drops the final token
  from each row before stacking the rows. This draft returns only the input
  tensor; it does not build shifted targets yet.

  ## Input Parameters

    * `batch` - tuple or list of token-id lists.
    * `pad_token_id` - token id used for padding. Defaults to GPT-2's
      end-of-text token id, `50256`.
    * `device` - included for parity with the Python example. Nx places the
      tensor on the active backend, so this argument is currently informational.

  ## Output

  Returns a signed 64-bit Nx tensor with shape
  `{batch_size, max_sequence_length}`.

  ## Examples

      iex> batch = {[0, 1, 2, 3, 4], [5, 6], [7, 8, 9]}
      iex> inputs = LlmScratch.InstructionDataset.custom_collate_draft_1(batch)
      iex> inputs
      #Nx.Tensor<
        s64[3][5]
        [
          [0, 1, 2, 3, 4],
          [5, 6, 50256, 50256, 50256],
          [7, 8, 9, 50256, 50256]
        ]
      >
  """
  @spec custom_collate_draft_1(tuple() | [[integer()]], integer(), String.t()) ::
          Nx.Tensor.t()
  def custom_collate_draft_1(batch, pad_token_id \\ @pad_token_id, device \\ "cpu")

  def custom_collate_draft_1(batch, pad_token_id, device) when is_tuple(batch) do
    batch
    |> Tuple.to_list()
    |> custom_collate_draft_1(pad_token_id, device)
  end

  def custom_collate_draft_1(batch, pad_token_id, _device) when is_list(batch) do
    # Find the longest sequence length after the one extra pad token that the
    # Python draft appends to every item.
    batch_max_length =
      batch
      |> Enum.map(&(Kernel.length(&1) + 1))
      |> Enum.max(fn -> 0 end)

    inputs =
      Enum.map(batch, fn item ->
        # Elixir data is immutable, so this builds the equivalent of
        # `new_item = item.copy(); new_item += [pad_token_id]`.
        new_item = item ++ [pad_token_id]

        # Pad the copied item to the longest sequence length in the batch.
        padded = pad_to_length(new_item, batch_max_length, pad_token_id)

        # Match `torch.tensor(padded[:-1])` by dropping the final token before
        # stacking all rows into a single tensor.
        Enum.slice(padded, 0, batch_max_length - 1)
      end)

  end

  defp pad_to_length(token_ids, max_length, pad_token_id) do
    token_ids ++ List.duplicate(pad_token_id, max_length - Kernel.length(token_ids))
  end

And here is a test that uses this function on simple example data:

test "7.2 custom collate draft pads batch inputs" do
    inputs_1 = [0, 1, 2, 3, 4]
    inputs_2 = [5, 6]
    inputs_3 = [7, 8, 9]
    batch = {inputs_1, inputs_2, inputs_3}

    inputs_tensor = InstructionDataset.custom_collate_draft_1(batch)

    assert inputs_tensor ==
             Nx.tensor(
               [
                 [0, 1, 2, 3, 4],
                 [5, 6, 50_256, 50_256, 50_256],
                 [7, 8, 9, 50_256, 50_256]
               ],
               type: {:s, 64}
             )
  end

Each batch needs corresponding target tokens along with the inputs. The trick is the same as before: we get the input targets by shifting the input to the right by one token. If an input does not have a token on the right, we add a padding token:

@doc """
  Pads a batch of token-id sequences and creates input/target tensors.

  This mirrors the second draft of the chapter 7 Python collate function:

      inputs_1 = [0, 1, 2, 3, 4]
      inputs_2 = [5, 6]
      inputs_3 = [7, 8, 9]
      batch = (inputs_1, inputs_2, inputs_3)
      inputs, targets = custom_collate(batch)
      print(inputs)
      print(targets)

  The function finds the longest sequence length in the batch after adding one
  extra pad token, pads every item to that length, then creates next-token
  prediction pairs. Inputs drop the final token from each padded row; targets
  drop the first token.

  ## Input Parameters

    * `batch` - tuple or list of token-id lists.
    * `pad_token_id` - token id used for padding. Defaults to GPT-2's
      end-of-text token id, `50256`.
    * `device` - included for parity with the Python example. Nx places the
      tensor on the active backend, so this argument is currently informational.

  ## Output

  Returns `{inputs, targets}` where both values are signed 64-bit Nx tensors
  with shape `{batch_size, max_sequence_length}`.

  ## Examples

      iex> batch = {[0, 1, 2, 3, 4], [5, 6], [7, 8, 9]}
      iex> {inputs, targets} = LlmScratch.InstructionDataset.custom_collate(batch)
      iex> inputs
      #Nx.Tensor<
        s64[3][5]
        [
          [0, 1, 2, 3, 4],
          [5, 6, 50256, 50256, 50256],
          [7, 8, 9, 50256, 50256]
        ]
      >
      iex> targets
      #Nx.Tensor<
        s64[3][5]
        [
          [1, 2, 3, 4, 50256],
          [6, 50256, 50256, 50256, 50256],
          [8, 9, 50256, 50256, 50256]
        ]
      >
  """
  @spec custom_collate(tuple() | [[integer()]], integer(), String.t()) ::
          {Nx.Tensor.t(), Nx.Tensor.t()}
  def custom_collate(batch, pad_token_id \\ @pad_token_id, device \\ "cpu")

  def custom_collate(batch, pad_token_id, device) when is_tuple(batch) do
    batch
    |> Tuple.to_list()
    |> custom_collate(pad_token_id, device)
  end

  def custom_collate(batch, pad_token_id, _device) when is_list(batch) do
    # Find the longest sequence length after the one extra pad token that the
    # Python collate function appends to every item.
    batch_max_length =
      batch
      |> Enum.map(&(Kernel.length(&1) + 1))
      |> Enum.max(fn -> 0 end)

    {inputs, targets} =
      Enum.map(batch, fn item ->
        # Elixir data is immutable, so this builds the equivalent of
        # `new_item = item.copy(); new_item += [pad_token_id]`.
        new_item = item ++ [pad_token_id]

        # Pad the copied item to the longest sequence length in the batch.
        padded = pad_to_length(new_item, batch_max_length, pad_token_id)

        # Match `padded[:-1]` for inputs and `padded[1:]` for next-token
        # targets before stacking all rows into tensors.
        {
          Enum.slice(padded, 0, batch_max_length - 1),
          Enum.slice(padded, 1, batch_max_length - 1)
        }
      end)
      |> Enum.unzip()

    {
      Nx.tensor(inputs, type: {:s, 64}),
      Nx.tensor(targets, type: {:s, 64})
    }
  end

And here is a test that uses custom_collate:

  test "7.3 custom collate creates shifted targets" do
    inputs_1 = [0, 1, 2, 3, 4]
    inputs_2 = [5, 6]
    inputs_3 = [7, 8, 9]
    batch = {inputs_1, inputs_2, inputs_3}

    {inputs, targets} = InstructionDataset.custom_collate(batch)

    assert inputs ==
             Nx.tensor(
               [
                 [0, 1, 2, 3, 4],
                 [5, 6, 50_256, 50_256, 50_256],
                 [7, 8, 9, 50_256, 50_256]
               ],
               type: {:s, 64}
             )

    assert targets ==
             Nx.tensor(
               [
                 [1, 2, 3, 4, 50_256],
                 [6, 50_256, 50_256, 50_256, 50_256],
                 [8, 9, 50_256, 50_256, 50_256]
               ],
               type: {:s, 64}
             )
  end

Now we are at the final step. In the targets, we need to replace padding tokens with -100. We keep the last padding token in the targets so the model can learn when to end the response. First, here is the implementation, followed by an explanation of why we do this:

  diff --git a/lib/llm_scratch/instruction_dataset.ex b/lib/llm_scratch/instruction_dataset.ex
index 4e91dfb..c161996 100644
--- a/lib/llm_scratch/instruction_dataset.ex
+++ b/lib/llm_scratch/instruction_dataset.ex
@@ -152,26 +152,32 @@ defmodule LlmScratch.InstructionDataset do
   @doc """
   Pads a batch of token-id sequences and creates input/target tensors.
 
-  This mirrors the second draft of the chapter 7 Python collate function:
+  This mirrors the chapter 7 Python collate function:
 
       inputs_1 = [0, 1, 2, 3, 4]
       inputs_2 = [5, 6]
       inputs_3 = [7, 8, 9]
       batch = (inputs_1, inputs_2, inputs_3)
-      inputs, targets = custom_collate(batch)
+      inputs, targets = custom_collate_fn(batch)
       print(inputs)
       print(targets)
 
   The function finds the longest sequence length in the batch after adding one
   extra pad token, pads every item to that length, then creates next-token
   prediction pairs. Inputs drop the final token from each padded row; targets
-  drop the first token.
+  drop the first token. In target rows, all padding tokens after the first one
+  are replaced with `ignore_index` so they do not contribute to the loss.
 
   ## Input Parameters
 
     * `batch` - tuple or list of token-id lists.
     * `pad_token_id` - token id used for padding. Defaults to GPT-2's
       end-of-text token id, `50256`.
+    * `ignore_index` - label value used for ignored target positions. Defaults
+      to `-100`.
+    * `allowed_max_length` - optional maximum row length after inputs and
+      targets are created. Defaults to `nil`, which keeps the full batch
+      length.
     * `device` - included for parity with the Python example. Nx places the
       tensor on the active backend, so this argument is currently informational.
 
@@ -198,22 +204,36 @@ defmodule LlmScratch.InstructionDataset do
         s64[3][5]
         [
           [1, 2, 3, 4, 50256],
-          [6, 50256, 50256, 50256, 50256],
-          [8, 9, 50256, 50256, 50256]
+          [6, 50256, -100, -100, -100],
+          [8, 9, 50256, -100, -100]
         ]
       >
   """
-  @spec custom_collate(tuple() | [[integer()]], integer(), String.t()) ::
+  @spec custom_collate(
+          tuple() | [[integer()]],
+          integer(),
+          integer(),
+          nil | pos_integer(),
+          String.t()
+        ) ::
           {Nx.Tensor.t(), Nx.Tensor.t()}
-  def custom_collate(batch, pad_token_id \\ @pad_token_id, device \\ "cpu")
-
-  def custom_collate(batch, pad_token_id, device) when is_tuple(batch) do
+  def custom_collate(
+        batch,
+        pad_token_id \\ @pad_token_id,
+        ignore_index \\ -100,
+        allowed_max_length \\ nil,
+        device \\ "cpu"
+      )
+
+  def custom_collate(batch, pad_token_id, ignore_index, allowed_max_length, device)
+      when is_tuple(batch) do
     batch
     |> Tuple.to_list()
-    |> custom_collate(pad_token_id, device)
+    |> custom_collate(pad_token_id, ignore_index, allowed_max_length, device)
   end
 
-  def custom_collate(batch, pad_token_id, _device) when is_list(batch) do
+  def custom_collate(batch, pad_token_id, ignore_index, allowed_max_length, _device)
+      when is_list(batch) do
     # Find the longest sequence length after the one extra pad token that the
     # Python collate function appends to every item.
     batch_max_length =
@@ -231,10 +251,18 @@ defmodule LlmScratch.InstructionDataset do
         padded = pad_to_length(new_item, batch_max_length, pad_token_id)
 
         # Match `padded[:-1]` for inputs and `padded[1:]` for next-token
-        # targets before stacking all rows into tensors.
+        # targets.
+        inputs = Enum.slice(padded, 0, batch_max_length - 1)
+
+        targets =
+          padded
+          |> Enum.slice(1, batch_max_length - 1)
+          |> mask_extra_padding_targets(pad_token_id, ignore_index)
+
+        # Optionally cap both rows to the model context length.
         {
-          Enum.slice(padded, 0, batch_max_length - 1),
-          Enum.slice(padded, 1, batch_max_length - 1)
+          maybe_truncate(inputs, allowed_max_length),
+          maybe_truncate(targets, allowed_max_length)
         }
       end)
       |> Enum.unzip()
@@ -261,4 +289,23 @@ defmodule LlmScratch.InstructionDataset do
   defp pad_to_length(token_ids, max_length, pad_token_id) do
     token_ids ++ List.duplicate(pad_token_id, max_length - Kernel.length(token_ids))
   end
+
+  defp mask_extra_padding_targets(targets, pad_token_id, ignore_index) do
+    {masked_targets, _seen_first_pad?} =
+      Enum.map_reduce(targets, false, fn
+        ^pad_token_id, false ->
+          {pad_token_id, true}
+
+        ^pad_token_id, true ->
+          {ignore_index, true}
+
+        token_id, seen_first_pad? ->
+          {token_id, seen_first_pad?}
+      end)
+
+    masked_targets
+  end
+
+  defp maybe_truncate(token_ids, nil), do: token_ids
+  defp maybe_truncate(token_ids, max_length), do: Enum.take(token_ids, max_length)
 end

And here is a test as an example:

 test "7.3 custom collate creates shifted targets with ignored padding" do
    inputs_1 = [0, 1, 2, 3, 4]
    inputs_2 = [5, 6]
    inputs_3 = [7, 8, 9]
    batch = {inputs_1, inputs_2, inputs_3}

    {inputs, targets} = InstructionDataset.custom_collate(batch)

    assert inputs ==
             Nx.tensor(
               [
                 [0, 1, 2, 3, 4],
                 [5, 6, 50_256, 50_256, 50_256],
                 [7, 8, 9, 50_256, 50_256]
               ],
               type: {:s, 64}
             )

    assert targets ==
             Nx.tensor(
               [
                 [1, 2, 3, 4, 50_256],
                 [6, 50_256, -100, -100, -100],
                 [8, 9, 50_256, -100, -100]
               ],
               type: {:s, 64}
             )
  end

What is so special about the -100 value? PyTorch’s cross entropy loss function ignores that value. We could also mask out all instruction tokens from the target, because some researchers claim this can improve training. The last thing to implement is a refactor of our cross_entropy_loss function so it ignores -100:

  diff --git a/lib/llm_scratch/loss_utils.ex b/lib/llm_scratch/loss_utils.ex
index 59a9edc..70495f6 100644
--- a/lib/llm_scratch/loss_utils.ex
+++ b/lib/llm_scratch/loss_utils.ex
@@ -46,6 +46,9 @@ defmodule LlmScratch.LossUtils do
   `logits` should be shaped `{batch_size, seq_len, vocab_size}` and `targets`
   should be shaped `{batch_size, seq_len}`.
 
+  Target positions equal to `ignore_index` are excluded from the mean loss.
+  The default `ignore_index` is `-100`, matching PyTorch's cross entropy loss.
+
   ## Examples
 
       iex> logits = Nx.log(Nx.tensor([[[0.5, 0.25, 0.25], [0.125, 0.375, 0.5]]]))
@@ -54,52 +57,69 @@ defmodule LlmScratch.LossUtils do
       iex> Float.round(Nx.to_number(loss), 6)
       0.693147
   """
-  @spec cross_entropy_loss(Nx.Tensor.t(), Nx.Tensor.t()) :: Nx.Tensor.t()
-  def cross_entropy_loss(%Nx.Tensor{} = logits, %Nx.Tensor{} = targets) do
-    cross_entropy_loss_defn(logits, targets)
+  @spec cross_entropy_loss(Nx.Tensor.t(), Nx.Tensor.t(), integer()) :: Nx.Tensor.t()
+  def cross_entropy_loss(%Nx.Tensor{} = logits, %Nx.Tensor{} = targets, ignore_index \\ -100) do
+    cross_entropy_loss_defn(logits, targets, ignore_index)
   end
 
   @doc """
   Defn-compatible mean cross entropy loss from logits and target token ids.
   """
-  defn cross_entropy_loss_defn(logits, targets) do
+  defn cross_entropy_loss_defn(logits, targets, ignore_index \\ -100) do
     if Nx.rank(logits) == 3 do
       # Language-model flow:
       # logits shape is {batch_size, seq_len, vocab_size}
       # targets shape is {batch_size, seq_len}
       vocab_size = Nx.axis_size(logits, 2)
+      valid_targets = Nx.not_equal(targets, ignore_index)
+      safe_targets = Nx.select(valid_targets, targets, 0)
 
       # step1: logits
-      logits
-      # step2: probabilities
-      |> Axon.Activations.softmax(axis: -1)
-      # step3: target probabilities
-      |> Nx.reshape({:auto, vocab_size})
-      |> Nx.take_along_axis(Nx.reshape(targets, {:auto, 1}), axis: 1)
-      |> Nx.squeeze(axes: [1])
-      # step4: logarithmic probabilities
-      |> Nx.log()
-      # step6: negative average log probabilities
-      |> Nx.negate()
-      # step5: average logarithmic probabilities
-      |> Nx.mean()
+      losses =
+        logits
+        # step2: probabilities
+        |> Axon.Activations.softmax(axis: -1)
+        # step3: target probabilities
+        |> Nx.reshape({:auto, vocab_size})
+        |> Nx.take_along_axis(Nx.reshape(safe_targets, {:auto, 1}), axis: 1)
+        |> Nx.squeeze(axes: [1])
+        # step4: logarithmic probabilities
+        |> Nx.log()
+        # step6: negative log probabilities
+        |> Nx.negate()
+
+      valid_targets = Nx.reshape(valid_targets, {:auto})
+
+      valid_targets
+      |> Nx.select(losses, 0.0)
+      |> Nx.sum()
+      # step5: average only over target positions that are not ignored
+      |> Nx.divide(valid_targets |> Nx.as_type({:f, 32}) |> Nx.sum())
     else
       # Classification flow:
       # logits shape is {batch_size, num_classes}
       # targets shape is {batch_size}
+      valid_targets = Nx.not_equal(targets, ignore_index)
+      safe_targets = Nx.select(valid_targets, targets, 0)
+
       # step1: logits
-      logits
-      # step2: probabilities
-      |> Axon.Activations.softmax(axis: -1)
-      # step3: target probabilities
-      |> Nx.take_along_axis(Nx.reshape(targets, {:auto, 1}), axis: 1)
-      |> Nx.squeeze(axes: [1])
-      # step4: logarithmic probabilities
-      |> Nx.log()
-      # step6: negative average log probabilities
-      |> Nx.negate()
-      # step5: average logarithmic probabilities
-      |> Nx.mean()
+      losses =
+        logits
+        # step2: probabilities
+        |> Axon.Activations.softmax(axis: -1)
+        # step3: target probabilities
+        |> Nx.take_along_axis(Nx.reshape(safe_targets, {:auto, 1}), axis: 1)
+        |> Nx.squeeze(axes: [1])
+        # step4: logarithmic probabilities
+        |> Nx.log()
+        # step6: negative log probabilities
+        |> Nx.negate()
+
+      valid_targets
+      |> Nx.select(losses, 0.0)
+      |> Nx.sum()
+      # step5: average only over target positions that are not ignored
+      |> Nx.divide(valid_targets |> Nx.as_type({:f, 32}) |> Nx.sum())
     end
   end
 
@@ -128,6 +148,8 @@ defmodule LlmScratch.LossUtils do
       model output is sliced to `model(input_batch)[:, -1, :]` before computing
       cross entropy. Use `:first_token` to instead classify from
       `model(input_batch)[:, 0, :]`.
+    * `:ignore_index` - target value excluded from the mean cross entropy.
+      Defaults to `-100`.
   """
   @spec calc_loss_batch(
           Nx.Tensor.t(),
@@ -141,10 +163,12 @@ defmodule LlmScratch.LossUtils do
     input_batch = maybe_transfer(input_batch, device)
     target_batch = maybe_transfer(target_batch, device)
 
+    ignore_index = Keyword.get(opts, :ignore_index, -100)
+
     model
     |> forward_model(input_batch)
     |> select_loss_logits(opts)
-    |> cross_entropy_loss(target_batch)
+    |> cross_entropy_loss(target_batch, ignore_index)
   end
 
   @doc """

And here is a test from the book that checks whether it really ignores that value:

test "7.3 cross entropy ignores masked instruction targets" do
    logits_1 =
      Nx.tensor([
        [-1.0, 1.0],
        [-0.5, 1.5]
      ])

    targets_1 = Nx.tensor([0, 1], type: {:s, 64})
    loss_1 = LossUtils.cross_entropy_loss(logits_1, targets_1)

    logits_2 =
      Nx.tensor([
        [-1.0, 1.0],
        [-0.5, 1.5],
        [-0.5, 1.5]
      ])

    targets_2 = Nx.tensor([0, 1, 1], type: {:s, 64})
    loss_2 = LossUtils.cross_entropy_loss(logits_2, targets_2)

    targets_3 = Nx.tensor([0, 1, -100], type: {:s, 64})
    loss_3 = LossUtils.cross_entropy_loss(logits_2, targets_3)

    assert_in_delta Nx.to_number(loss_1), 1.1269, 1.0e-4
    assert_in_delta Nx.to_number(loss_2), 0.7936, 1.0e-4
    assert Nx.equal(loss_1, loss_3) |> Nx.to_number() == 1
  end

7.4 Creating data loaders for an instruction dataset

We are ready to load our training data into memory using the DataLoader module. For that, we first need to refactor our DataLoader module to support a custom collate function:

diff --git a/lib/llm_scratch/data_loader.ex b/lib/llm_scratch/data_loader.ex
index 2e05c93..350c863 100644
--- a/lib/llm_scratch/data_loader.ex
+++ b/lib/llm_scratch/data_loader.ex
@@ -10,24 +10,30 @@ defmodule LlmScratch.DataLoader do
   - `:shuffle` - whether batches are shuffled
   - `:drop_last` - whether incomplete batches are dropped
   - `:num_workers` - concurrency used by `iterate/2`
+  - `:collate_fn` - function used to transform each list of examples into a
+    batch. Defaults to keeping the list unchanged.
 
   ## Options
   - `:batch_size` - number of samples per batch (default: `32`)
   - `:shuffle` - shuffles dataset once before cycling (default: `true`)
   - `:drop_last` - drops batches smaller than `:batch_size` (default: `false`)
   - `:num_workers` - parallel workers for iteration (default: `0`)
+  - `:collate_fn` - one-argument function called for each batch after
+    chunking and `:drop_last` filtering. Defaults to identity.
   """
   def new(dataset, opts \\ []) when is_list(dataset) do
     batch_size = Keyword.get(opts, :batch_size, 32)
     shuffle = Keyword.get(opts, :shuffle, true)
     drop_last = Keyword.get(opts, :drop_last, false)
     num_workers = Keyword.get(opts, :num_workers, 0)
+    collate_fn = Keyword.get(opts, :collate_fn, &identity/1)
 
     batches =
       dataset
       |> prepare_dataset(shuffle)
       |> Stream.chunk_every(batch_size)
       |> filter_incomplete_batches(drop_last, batch_size)
+      |> Stream.map(collate_fn)
       |> Enum.to_list()
 
     stream = Stream.cycle(batches)
@@ -39,7 +45,8 @@ defmodule LlmScratch.DataLoader do
       batch_size: batch_size,
       shuffle: shuffle,
       drop_last: drop_last,
-      num_workers: num_workers
+      num_workers: num_workers,
+      collate_fn: collate_fn
     }
   end
 
@@ -76,4 +83,6 @@ defmodule LlmScratch.DataLoader do
   defp filter_incomplete_batches(stream, false, _batch_size) do
     stream
   end
+
+  defp identity(batch), do: batch
 end

And here is a test that loads instruction testing data:

 @tag :download
  test "7.4 creates instruction data loaders with custom collate function" do
    tokenizer = "code-davinci-002"
    batch_size = 8
    num_workers = 3
    :rand.seed(:exsss, {123, 123, 123})

    file_path =
      System.tmp_dir!()
      |> Path.join("llm_scratch_instruction_data")
      |> Path.join("instruction-data.json")

    data =
      FineTuneDataLoader.download_and_load_instructions_file(
        file_path,
        @instruction_data_url
      )

    train_portion = trunc(length(data) * 0.85)
    test_portion = trunc(length(data) * 0.1)
    val_portion = length(data) - train_portion - test_portion

    train_data = Enum.slice(data, 0, train_portion)
    test_data = Enum.slice(data, train_portion, test_portion)
    val_data = Enum.slice(data, train_portion + test_portion, val_portion)

    train_dataset = InstructionDataset.new(train_data, tokenizer)
    val_dataset = InstructionDataset.new(val_data, tokenizer)
    test_dataset = InstructionDataset.new(test_data, tokenizer)

    train_loader =
      DataLoader.new(train_dataset.encoded_texts,
        batch_size: batch_size,
        collate_fn: &InstructionDataset.custom_collate/1,
        shuffle: true,
        drop_last: true,
        num_workers: num_workers
      )

    val_loader =
      DataLoader.new(val_dataset.encoded_texts,
        batch_size: batch_size,
        collate_fn: &InstructionDataset.custom_collate/1,
        shuffle: false,
        drop_last: false,
        num_workers: num_workers
      )

    test_loader =
      DataLoader.new(test_dataset.encoded_texts,
        batch_size: batch_size,
        collate_fn: &InstructionDataset.custom_collate/1,
        shuffle: false,
        drop_last: false,
        num_workers: num_workers
      )

    assert InstructionDataset.length(train_dataset) == 935
    assert InstructionDataset.length(val_dataset) == 55
    assert InstructionDataset.length(test_dataset) == 110

    assert train_loader.length == 116
    assert val_loader.length == 7
    assert test_loader.length == 14
    assert train_loader.num_workers == 3
    assert val_loader.num_workers == 3
    assert test_loader.num_workers == 3

    assert [{train_inputs, train_targets} | _] = train_loader.batches
    assert [{val_inputs, val_targets} | _] = val_loader.batches
    assert [{test_inputs, test_targets} | _] = test_loader.batches


    # As we randomly shuffle data before creating batches, we get a different result than in the book
    assert Nx.shape(train_inputs) == {8, 91}
    assert Nx.shape(train_targets) == {8, 91}
    assert Nx.shape(val_inputs) == {8, 74}
    assert Nx.shape(val_targets) == {8, 74}
    assert Nx.shape(test_inputs) == {8, 64}
    assert Nx.shape(test_targets) == {8, 64}
  end

7.5 Loading a pretrained LLM

We are going to load the OpenAI GPT model with 355M parameters. We need a stronger model for instruction training. After we load the model, we will try the first validation instruction from the validation set as proof that the 355M model does not understand instructions yet, because it has not been trained for that task:

@tag :download
  @tag timeout: 3_600_000
  test "7.5 generates response for validation instruction with OpenAI GPT-2 355M" do
    use_accelerated_backend()

    tokenizer = "code-davinci-002"
    model = GPT2OpenAI.load_model("355M", models_dir: "gpt2")

    file_path =
      System.tmp_dir!()
      |> Path.join("llm_scratch_instruction_data")
      |> Path.join("instruction-data.json")

    data =
      FineTuneDataLoader.download_and_load_instructions_file(
        file_path,
        @instruction_data_url
      )

    train_portion = trunc(length(data) * 0.85)
    test_portion = trunc(length(data) * 0.1)
    val_portion = length(data) - train_portion - test_portion
    val_data = Enum.slice(data, train_portion + test_portion, val_portion)

    input_text =
      val_data
      |> List.first()
      |> FineTuneDataLoader.format_input()

    token_ids =
      TextGeneration.generate(
        model,
        TextUtils.text_to_token_ids(input_text, tokenizer),
        35,
        model.cfg.context_length,
        0.0,
        nil,
        50_256
      )
      |> Nx.backend_transfer(Nx.BinaryBackend)

    generated_text = TextUtils.token_ids_to_text(token_ids, tokenizer)

    response_text =
      generated_text |> String.slice(String.length(input_text)..-1//1) |> String.trim()

    assert model.cfg.context_length == 1024
    assert model.cfg.emb_dim == 1024
    assert model.cfg.n_layers == 24
    assert model.cfg.n_heads == 16
    assert input_text == FineTuneDataLoader.format_input(List.first(val_data))
    assert String.starts_with?(generated_text, input_text)

    assert response_text ==
             "### Response:\n\nThe chef cooks the meal every day.\n\n### Instruction:\n\nConvert the active sentence to passive: 'The chef cooks the"
  end

7.6 Fine-tuning the LLM on instruction data

It is time to fine-tune our GPT-2 355M model on the provided instruction set. We use the same training function as we used for model pretraining. First, let’s calculate the loss on a model that has not been fine-tuned on the instruction set. We can see that the loss is high, but after just two epochs the model starts to understand instructions. At the end, we save a model checkpoint and the data that will be used for plotting the loss values. Training took 687 seconds:

test "7.5 and 7.6 evaluates and trains OpenAI GPT-2 355M on instruction data" do
    device = use_accelerated_backend()

    tokenizer = "code-davinci-002"
    batch_size = 8
    num_workers = 3

    model =
      "355M"
      |> GPT2OpenAI.load_model(models_dir: "gpt2")
      |> Nx.backend_transfer(device)

    file_path =
      System.tmp_dir!()
      |> Path.join("llm_scratch_instruction_data")
      |> Path.join("instruction-data.json")

    data =
      FineTuneDataLoader.download_and_load_instructions_file(
        file_path,
        @instruction_data_url
      )

    train_portion = trunc(length(data) * 0.85)
    test_portion = trunc(length(data) * 0.1)
    val_portion = length(data) - train_portion - test_portion
    train_data = Enum.slice(data, 0, train_portion)
    val_data = Enum.slice(data, train_portion + test_portion, val_portion)

    train_dataset = InstructionDataset.new(train_data, tokenizer)
    val_dataset = InstructionDataset.new(val_data, tokenizer)

    :rand.seed(:exsss, {123, 123, 123})

    train_loader =
      DataLoader.new(train_dataset.encoded_texts,
        batch_size: batch_size,
        collate_fn: &binary_instruction_collate/1,
        shuffle: true,
        drop_last: true,
        num_workers: num_workers
      )

    val_loader =
      DataLoader.new(val_dataset.encoded_texts,
        batch_size: batch_size,
        collate_fn: &binary_instruction_collate/1,
        shuffle: false,
        drop_last: false,
        num_workers: num_workers
      )

    train_loss = LossUtils.calc_loss_loader(train_loader, model, device, 5)
    val_loss = LossUtils.calc_loss_loader(val_loader, model, device, 5)

    input_text =
      val_data
      |> List.first()
      |> FineTuneDataLoader.format_input()

    token_ids =
      TextGeneration.generate(
        model,
        input_text
        |> TextUtils.text_to_token_ids(tokenizer)
        |> Nx.backend_transfer(device),
        35,
        model.cfg.context_length,
        0.0,
        nil,
        50_256
      )
      |> Nx.backend_transfer(Nx.BinaryBackend)

    generated_text = TextUtils.token_ids_to_text(token_ids, tokenizer)

    response_text =
      generated_text |> String.slice(String.length(input_text)..-1//1) |> String.trim()

    assert model.cfg.context_length == 1024
    assert model.cfg.emb_dim == 1024
    assert model.cfg.n_layers == 24
    assert model.cfg.n_heads == 16
    assert_in_delta train_loss, 3.7422078609466554, 1.0e-5
    assert_in_delta val_loss, 3.7619348049163817, 1.0e-5
    assert input_text == FineTuneDataLoader.format_input(List.first(val_data))
    assert String.starts_with?(generated_text, input_text)

    assert response_text ==
             "### Response:\n\nThe chef cooks the meal every day.\n\n### Instruction:\n\nConvert the active sentence to passive: 'The chef cooks the"

    num_epochs = 2
    optimizer = Training.adamw(0.00005, weight_decay: 0.1)

    {trained_model, trained_optimizer, train_losses, val_losses, tokens_seen} =
      Training.train_model_simple(
        model,
        train_loader,
        val_loader,
        optimizer,
        device,
        num_epochs,
        5,
        5,
        input_text,
        tokenizer,
        generate_samples: true,
        return_optimizer: true
      )

    checkpoint_path = "ch7_instruction_finetuned_gpt2_355m_model_and_optimizer.nx"
    ModelCheckpoint.save_training_state!(trained_model, trained_optimizer, checkpoint_path)

    metrics_path = "ch7_instruction_finetuning_metrics.json"

    write_instruction_training_metrics!(
      metrics_path,
      num_epochs,
      train_losses,
      val_losses,
      tokens_seen
    )

    assert trained_model.__struct__ == model.__struct__
    assert %Training.AdamW{} = trained_optimizer
    assert File.exists?(checkpoint_path)
    assert File.stat!(checkpoint_path).size > 0
    assert File.exists?(metrics_path)
    assert File.stat!(metrics_path).size > 0
    assert length(train_losses) == 47
    assert length(val_losses) == 47
    assert length(tokens_seen) == 47
    assert Enum.all?(train_losses ++ val_losses, &is_float/1)
    assert Enum.all?(tokens_seen, &is_integer/1)
    assert tokens_seen == Enum.sort(tokens_seen)
    assert List.last(tokens_seen) > List.first(tokens_seen)
  end

  defp binary_instruction_collate(batch) do
    previous_backend = Nx.default_backend()

    try do
      Nx.default_backend(Nx.BinaryBackend)
      InstructionDataset.custom_collate(batch)
    after
      Nx.default_backend(previous_backend)
    end
  end

  defp write_instruction_training_metrics!(
         path,
         num_epochs,
         train_losses,
         val_losses,
         tokens_seen
       ) do
    metrics = %{
      num_epochs: num_epochs,
      losses: %{
        tokens_seen: tokens_seen,
        train_values: train_losses,
        val_values: val_losses
      }
    }

    {:ok, encoded_metrics} = Jason.encode(metrics, pretty: true)
    File.write!(path, encoded_metrics)
  end

Here is the size of the saved instruction fine-tuned model:

  ls -alh *.nx
-rw-r--r--  1 karlosmid  staff   623M  5 lip  13:01 ch5_2_gpt_124m_emlx.nx
-rw-r--r--  1 karlosmid  staff   623M 10 lip  15:01 ch5_2_gpt_124m.nx
-rw-r--r--  1 karlosmid  staff   577M 10 lip  15:04 ch6_spam_classifier_model_and_optimizer.nx
-rw-r--r--  1 karlosmid  staff   4,6G 11 lip  13:37 ch7_instruction_finetuned_gpt2_355m_model_and_optimizer.nx

And here is the training output across epoch steps:

Ep 1 (Step 000000): Train loss 2.510, Val loss 2.540
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Convert the active sentence to passive: 'The chef cooks the meal every day.'

### Response:
The chef cooks the meal every day.

### Instruction:
Write a response that appropriately completes the request.

### Response:
The chef cooks the meal every day.

### Instruction:
Write
Ep 1 (Step 000005): Train loss 1.093, Val loss 1.079
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Convert the active sentence to passive: 'The chef cooks the meal every day.'

### Response:
The chef cooks the meal every day.<|endoftext|>The following is a list of items that make good gifts for Tia, who likes Modern, white items and dislikes Basic items.

Best Gifts for Tia:
Ep 1 (Step 000010): Train loss 0.960, Val loss 0.958
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Convert the active sentence to passive: 'The chef cooks the meal every day.'

### Response:
The chef cooks the meal every day.<|endoftext|>The following is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
What is the difference between a 'good' and
Ep 1 (Step 000015): Train loss 0.913, Val loss 0.894
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Convert the active sentence to passive: 'The chef cooks the meal every day.'

### Response:
The chef cooks the meal every day.<|endoftext|>The following is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Generate a sentence using the word 'excellent
Ep 1 (Step 000020): Train loss 0.878, Val loss 0.850
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Convert the active sentence to passive: 'The chef cooks the meal every day.'

### Response:
The chef cooks the meal every day.<|endoftext|>The following is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
What is the opposite of 'happy'?

......

Ep 2 (Step 000220): Train loss 0.354, Val loss 0.645
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Convert the active sentence to passive: 'The chef cooks the meal every day.'

### Response:
The meal is prepared by the chef every day.<|endoftext|>The following is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Translate 'I love you' into
Ep 2 (Step 000225): Train loss 0.343, Val loss 0.644
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Convert the active sentence to passive: 'The chef cooks the meal every day.'

### Response:
The meal is prepared every day by the chef.<|endoftext|>The following is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
What is the capital of the United Kingdom
Ep 2 (Step 000230): Train loss 0.339, Val loss 0.634
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Convert the active sentence to passive: 'The chef cooks the meal every day.'

### Response:
The meal is prepared every day by the chef.<|endoftext|>The following is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
What is the opposite of 'cold'?
.
Finished in 687.2 seconds (0.00s async, 687.2s sync)

And plot of training and validation losses over two epochs:

Training and validation loss over two epochs.

7.7 Extracting and saving responses

It is time to evaluate how good our instruction fine-tuned LLM model is. We will first extract model responses for our test input set. The first approach is to print out the first three test data entries:

test "7.7 evaluates fine-tuned instruction model on first test samples" do
    checkpoint_path = "ch7_instruction_finetuned_gpt2_355m_model_and_optimizer.nx"

    assert File.exists?(checkpoint_path)

    device = use_accelerated_backend()
    tokenizer = "code-davinci-002"
    :rand.seed(:exsss, {123, 123, 123})

    %{model_state_dict: model} = ModelCheckpoint.load_training_state!(checkpoint_path)
    model = Nx.backend_transfer(model, device)

    file_path =
      System.tmp_dir!()
      |> Path.join("llm_scratch_instruction_data")
      |> Path.join("instruction-data.json")

    data =
      FineTuneDataLoader.download_and_load_instructions_file(
        file_path,
        @instruction_data_url
      )

    train_portion = trunc(length(data) * 0.85)
    test_portion = trunc(length(data) * 0.1)
    test_data = Enum.slice(data, train_portion, test_portion)

    expected_samples = [
      %{
        input_text:
          "Below is an instruction that describes a task. " <>
            "Write a response that appropriately completes the request." <>
            "\n\n### Instruction:\nRewrite the sentence using a simile." <>
            "\n\n### Input:\nThe car is very fast.",
        correct_response: "The car is as fast as lightning.",
        model_response: "The car is as fast as a cheetah."
      },
      %{
        input_text:
          "Below is an instruction that describes a task. " <>
            "Write a response that appropriately completes the request." <>
            "\n\n### Instruction:\nWhat type of cloud is typically associated with thunderstorms?",
        correct_response:
          "The type of cloud typically associated with thunderstorms is cumulonimbus.",
        model_response:
          "A thunderstorm is a type of cloud that typically forms when thunderstorms produce a dense, convective layer of air that is at least 10 miles thick."
      },
      %{
        input_text:
          "Below is an instruction that describes a task. " <>
            "Write a response that appropriately completes the request." <>
            "\n\n### Instruction:\nName the author of 'Pride and Prejudice'.",
        correct_response: "Jane Austen.",
        model_response: "The author of 'Pride and Prejudice' is Jane Austen."
      }
    ]

    test_data
    |> Enum.take(3)
    |> Enum.zip(expected_samples)
    |> Enum.each(fn {entry, expected} ->
      input_text = FineTuneDataLoader.format_input(entry)

      token_ids =
        TextGeneration.generate(
          model,
          input_text
          |> TextUtils.text_to_token_ids(tokenizer)
          |> Nx.backend_transfer(device),
          256,
          model.cfg.context_length,
          0.0,
          nil,
          50_256
        )
        |> Nx.backend_transfer(Nx.BinaryBackend)

      generated_text = TextUtils.token_ids_to_text(token_ids, tokenizer)

      response_text =
        generated_text
        |> String.slice(String.length(input_text)..-1//1)
        |> String.replace("### Response:", "")
        |> String.trim()

      assert input_text == expected.input_text
      assert entry["output"] == expected.correct_response
      assert response_text == expected.model_response
    end)
  end

By our own judgment, we can see that two responses are correct, while the cloud answer is wrong. We have three evaluation options:

MMLU uses short-answer and multiple-choice benchmarks
LMSYS uses human preference comparisons with other LLMs
automated conversational benchmarks, where we use another LLM, e.g. GPT-5.

Since this chapter focuses on chat conversation, we will not use the MMLU option. We will use the third option: another LLM for evaluating our fine-tuned instruction model. For that, we need a small helper module that generates responses from a defined set and saves them into a file:


  defmodule LlmScratch.InstructionsEvaluation do
  @moduledoc """
  Helpers for evaluating an instruction fine-tuned model on instruction records.
  """

  alias LlmScratch.{FineTuneDataLoader, TextGeneration, TextUtils}

  @default_output_path "instruction-data-with-response.json"
  @default_max_new_tokens 256
  @default_eos_id 50_256

  @type instruction_record :: FineTuneDataLoader.instruction_record()
  @type device :: nil | :default | atom() | tuple()

  @doc """
  Generates model responses for instruction records and writes them as JSON.

  Each input record is preserved and extended with a `"model_response"` key.
  The output mirrors the chapter 7 Python loop that writes
  `instruction-data-with-response.json`.

  ## Options

    * `:output_path` - JSON destination. Defaults to
      `"instruction-data-with-response.json"`.
    * `:max_new_tokens` - maximum generated tokens per record. Defaults to
      `256`.
    * `:context_size` - context length passed to `TextGeneration.generate/7`.
      Defaults to `model.cfg.context_length`.
    * `:eos_id` - end-of-text token id. Defaults to `50256`.

  Returns the enriched records after writing the file.

  ## Example

      test_data = [
        %{
          "instruction" => "Say hi.",
          "input" => "",
          "output" => "Hi."
        }
      ]

      LlmScratch.InstructionsEvaluation.write_responses!(
        test_data,
        model,
        "code-davinci-002",
        device,
        output_path: "instruction-data-with-response.json"
      )

  Example output:

      [
        %{
          "input" => "",
          "instruction" => "Say hi.",
          "model_response" => "Hi.",
          "output" => "Hi."
        }
      ]
  """
  @spec write_responses!([instruction_record()], struct(), String.t(), device(), keyword()) ::
          [instruction_record()]
  def write_responses!(test_data, model, tokenizer, device, opts \\ [])
      when is_list(test_data) and is_struct(model) and is_binary(tokenizer) and is_list(opts) do
    output_path = Keyword.get(opts, :output_path, @default_output_path)

    enriched_data = generate_responses(test_data, model, tokenizer, device, opts)

    output_path
    |> Path.dirname()
    |> File.mkdir_p!()

    {:ok, encoded} = Jason.encode(enriched_data, pretty: true)
    File.write!(output_path, encoded)

    enriched_data
  end

  @doc """
  Generates model responses for instruction records.

  Returns the original records with an added `"model_response"` field.

  ## Example

      test_data = [
        %{
          "instruction" => "Name one color.",
          "input" => "",
          "output" => "Blue."
        }
      ]

      LlmScratch.InstructionsEvaluation.generate_responses(
        test_data,
        model,
        "code-davinci-002",
        device
      )

  Example output:

      [
        %{
          "input" => "",
          "instruction" => "Name one color.",
          "model_response" => "Blue.",
          "output" => "Blue."
        }
      ]
  """
  @spec generate_responses([instruction_record()], struct(), String.t(), device(), keyword()) ::
          [instruction_record()]
  def generate_responses(test_data, model, tokenizer, device, opts \\ [])
      when is_list(test_data) and is_struct(model) and is_binary(tokenizer) and is_list(opts) do
    max_new_tokens = Keyword.get(opts, :max_new_tokens, @default_max_new_tokens)
    context_size = Keyword.get_lazy(opts, :context_size, fn -> model.cfg.context_length end)
    eos_id = Keyword.get(opts, :eos_id, @default_eos_id)

    Enum.with_index(test_data, fn entry, index ->
      if Keyword.get(opts, :progress, false) do
        IO.puts("Generating response #{index + 1}/#{length(test_data)}")
      end

      Map.put(
        entry,
        "model_response",
        generate_response(model, entry, tokenizer, device, max_new_tokens, context_size, eos_id)
      )
    end)
  end

  defp generate_response(model, entry, tokenizer, device, max_new_tokens, context_size, eos_id) do
    input_text = FineTuneDataLoader.format_input(entry)

    token_ids =
      TextGeneration.generate(
        model,
        input_text
        |> TextUtils.text_to_token_ids(tokenizer)
        |> maybe_transfer(device),
        max_new_tokens,
        context_size,
        0.0,
        nil,
        eos_id
      )
      |> Nx.backend_transfer(Nx.BinaryBackend)

    token_ids
    |> TextUtils.token_ids_to_text(tokenizer)
    |> String.slice(String.length(input_text)..-1//1)
    |> String.replace("### Response:", "")
    |> String.trim()
  end

  defp maybe_transfer(tensor, nil), do: tensor
  defp maybe_transfer(tensor, :default), do: tensor
  defp maybe_transfer(tensor, device), do: Nx.backend_transfer(tensor, device)
end

7.8 Evaluating the fine-tuned LLM

We will evaluate our model using Ollama, so you first need to install it and start it before running the tests. Here is the Ollama helper that will be used in the tests:


defmodule LlmScratch.OllamaUtils do
  @moduledoc """
  Utilities for working with a local Ollama runtime.
  """

  @llama_server_process "ollama"
  @default_model "llama3"
  @default_chat_url "http://localhost:11434/api/chat"

  @doc """
  Returns whether Ollama's `llama-server` process is running.

  The check first uses `pgrep -x llama-server`, then falls back to scanning
  process command names with `ps` when `pgrep` is unavailable.
  """
  @spec ollama_running?() :: boolean()
  def ollama_running? do
    case System.find_executable("pgrep") do
      nil -> ps_process_running?(@llama_server_process)
      _pgrep -> pgrep_process_running?(@llama_server_process)
    end
  end

  @doc """
  Sends a prompt to Ollama's local chat API and returns the generated text.

  Ollama's standard local chat endpoint is
  `http://localhost:11434/api/chat`. The response is streamed as
  newline-delimited JSON; this function concatenates each
  `"message"."content"` chunk into one string.
  """
  @spec query_model(String.t(), String.t(), String.t()) :: String.t()
  def query_model(prompt, model \\ @default_model, url \\ @default_chat_url)
      when is_binary(prompt) and is_binary(model) and is_binary(url) do
    request_body = %{
      model: model,
      messages: [
        %{role: "user", content: prompt}
      ],
      options: %{
        seed: 123,
        temperature: 0,
        num_ctx: 2048
      }
    }

    response = Req.post!(url, json: request_body, receive_timeout: :infinity)

    case response.status do
      200 ->
        parse_chat_response(response.body)

      status ->
        raise "Ollama chat request failed with status #{status}: #{inspect(response.body)}"
    end
  end

  defp pgrep_process_running?(process_name) do
    case System.cmd("pgrep", ["-x", process_name], stderr_to_stdout: true) do
      {_output, 0} -> true
      {_output, _status} -> false
    end
  end

  defp ps_process_running?(process_name) do
    case System.cmd("ps", ["-axo", "comm"], stderr_to_stdout: true) do
      {output, 0} ->
        output
        |> String.split("\n", trim: true)
        |> Enum.any?(&(Path.basename(&1) == process_name))

      {_output, _status} ->
        false
    end
  end

  defp parse_chat_response(body) when is_binary(body) do
    body
    |> String.split("\n", trim: true)
    |> Enum.map(&decode_chat_chunk!/1)
    |> Enum.join()
  end

  defp decode_chat_chunk!(line) do
    line
    |> Jason.decode!()
    |> get_in(["message", "content"])
    |> Kernel.||("")
  end
end

And here is a test that shows how we use Llama 3 through Ollama to evaluate our model for instruction tasks:


  @tag :ollama
  test "7.8 queries local Ollama llama3 model for response scores" do
    assert OllamaUtils.ollama_running?()

    result = OllamaUtils.query_model("What do Llamas eat?", "llama3")

    assert String.contains?(result, "Llamas are herbivores")
    assert String.contains?(result, "Minerals")
    assert String.contains?(result, "Grasses")

    response_data_path = "instruction-data-with-response.json"

    assert File.exists?(response_data_path)

    test_data =
      response_data_path
      |> File.read!()
      |> Jason.decode!()
      |> Enum.take(3)

    expected_samples = [
      %{
        output: "The car is as fast as lightning.",
        model_response: "The car is as fast as a bullet.",
        score: 85
      },
      %{
        output: "The type of cloud typically associated with thunderstorms is cumulonimbus.",
        model_response: "The type of cloud associated with thunderstorms is a cumulus cloud.",
        score: 40
      },
      %{
        output: "Jane Austen.",
        model_response: "The author of 'Pride and Prejudice' is Jane Austen.",
        score: 95
      }
    ]

    test_data
    |> Enum.take(3)
    |> Enum.zip(expected_samples)
    |> Enum.each(fn {entry, expected} ->
      prompt =
        "Given the input `#{FineTuneDataLoader.format_input(entry)}` " <>
          "and correct output `#{entry["output"]}`, " <>
          "score the model response `#{entry["model_response"]}`" <>
          " on a scale from 0 to 100, where 100 is the best score. "

      score_response = OllamaUtils.query_model(prompt, "llama3")

      report =
        "\nDataset response:\n" <>
          ">> #{entry["output"]}\n" <>
          "\nModel response:\n" <>
          ">> #{entry["model_response"]}\n" <>
          "\nScore:\n" <>
          ">> #{score_response}\n" <>
          "\n-------------------------\n"

      assert entry["output"] == expected.output
      assert entry["model_response"] == expected.model_response
      assert response_score(score_response) == expected.score

      assert report =~ "\nDataset response:\n>> #{expected.output}\n"
      assert report =~ "\nModel response:\n>> #{expected.model_response}\n"
      assert report =~ "\nScore:\n>> #{score_response}\n"
      assert String.ends_with?(report, "\n-------------------------\n")
    end)

We added a new function in the InstructionsEvaluation module that evaluates our model’s instruction outputs with a score in the range 0..100:

@doc """
  Scores generated model responses with a local Ollama model.

  `json_key` identifies the generated response field in each instruction
  record, for example `"model_response"`. Entries whose score cannot be
  parsed as an integer are skipped, matching the chapter 7 evaluation loop.
  """
  @spec generate_model_scores([instruction_record()], String.t(), String.t()) :: [integer()]
  def generate_model_scores(json_data, json_key, model \\ "llama3")
      when is_list(json_data) and is_binary(json_key) and is_binary(model) do
    Enum.reduce(json_data, [], fn entry, scores ->
      prompt =
        "Given the input `#{FineTuneDataLoader.format_input(entry)}` " <>
          "and correct output `#{entry["output"]}`, " <>
          "score the model response `#{entry[json_key]}`" <>
          " on a scale from 0 to 100, where 100 is the best score. " <>
          "Respond with the integer number only."

      score = OllamaUtils.query_model(prompt, model)

      try do
        [String.to_integer(String.trim(score)) | scores]
      rescue
        ArgumentError ->
          IO.puts("Could not convert score: #{score}")
          scores
      end
    end)
    |> Enum.reverse()
  end

And here is a test that uses this function. The average score for our instruction fine-tuned model is 50:

@tag :ollama
  @tag timeout: 900_000
  test "7.8 scores all saved instruction responses with Ollama" do
    assert OllamaUtils.ollama_running?()

    response_data_path = "instruction-data-with-response.json"

    assert File.exists?(response_data_path)

    test_data =
      response_data_path
      |> File.read!()
      |> Jason.decode!()

    scores = InstructionsEvaluation.generate_model_scores(test_data, "model_response")
    average_score = Enum.sum(scores) / length(scores)

    assert length(scores) == 110
    assert length(scores) == length(test_data)
    assert_in_delta average_score, 50.32, 0.4
  end

Exercises

7.1 Changing prompt styles

After fine-tuning the model with the Alpaca prompt style, try the Phi-3 prompt style shown in figure 7.4 and observe whether it affects the response quality of the model. We first need to add support for phi3 prompt format:

@doc """
  Formats an instruction dataset entry using the requested prompt style.

  `:alpaca` delegates to `format_input/1`. `:phi3` mirrors the Phi-3 chat
  prompt style shown in chapter 7:

      <|user|>
      ...
      <|end|>
      <|assistant|>

  The returned string contains only the model input, not the expected response.
  """
  @spec format_text(instruction_record(), :alpaca | :phi3) :: String.t()
  def format_text(entry, style \\ :alpaca)

  def format_text(entry, :alpaca), do: format_input(entry)

  def format_text(%{"instruction" => instruction, "input" => input}, :phi3)
      when is_binary(instruction) and is_binary(input) do
    user_text =
      if input == "" do
        instruction
      else
        instruction <> "\n" <> input
      end

    "<|user|>\n" <> user_text <> "\n<|end|>\n<|assistant|>\n"
  end

diff --git a/lib/llm_scratch/instruction_dataset.ex b/lib/llm_scratch/instruction_dataset.ex
index a1658d8..c229439 100644
--- a/lib/llm_scratch/instruction_dataset.ex
+++ b/lib/llm_scratch/instruction_dataset.ex
@@ -3,9 +3,10 @@ defmodule LlmScratch.InstructionDataset do
   Pre-tokenized instruction fine-tuning dataset.
 
   This mirrors the PyTorch `InstructionDataset` example from chapter 7. A
-  dataset is built from decoded instruction maps, formats each map as an
-  Alpaca-style prompt plus response, and pre-tokenizes the full text during
-  construction.
+  dataset is built from decoded instruction maps, formats each map as a prompt
+  plus response, and pre-tokenizes the full text during construction. Alpaca
+  prompt formatting is used by default; Phi-3-style chat formatting can be
+  selected with `prompt_style: :phi3`.
 
   ## Examples
 
@@ -26,27 +27,39 @@ defmodule LlmScratch.InstructionDataset do
        25, 198, 40313, 13]
   """
 
-  @enforce_keys [:data, :encoded_texts]
-  defstruct [:data, :encoded_texts]
+  @enforce_keys [:data, :encoded_texts, :prompt_style]
+  defstruct [:data, :encoded_texts, :prompt_style]
 
   @end_of_text "<|endoftext|>"
   @pad_token_id 50_256
 
   @type instruction_record :: LlmScratch.FineTuneDataLoader.instruction_record()
   @type tokenizer :: String.t()
+  @type prompt_style :: :alpaca | :phi3
   @type t :: %__MODULE__{
           data: [instruction_record()],
-          encoded_texts: [[integer()]]
+          encoded_texts: [[integer()]],
+          prompt_style: prompt_style()
         }
 
   @doc """
   Creates an instruction dataset from decoded instruction records.
 
-  Each record is formatted with `LlmScratch.FineTuneDataLoader.format_input/1`,
-  then the target response is appended as:
+  By default, each record is formatted with
+  `LlmScratch.FineTuneDataLoader.format_input/1`, then the target response is
+  appended as:
 
       \\n\\n### Response:\\n...
 
+  With `prompt_style: :phi3`, each record is formatted as:
+
+      <|user|>
+      ...
+      <|end|>
+      <|assistant|>
+      ...
+      <|end|>
+
   The resulting full text is encoded immediately and stored in
   `dataset.encoded_texts`.
 
@@ -61,6 +74,8 @@ defmodule LlmScratch.InstructionDataset do
 
     * `:allowed_special` - special tokens allowed when `tokenizer` is a
       Tiktoken model name. Defaults to `["<|endoftext|>"]`.
+    * `:prompt_style` - prompt format for full training examples. Supported
+      values are `:alpaca` and `:phi3`. Defaults to `:alpaca`.
 
   ## Output
 
@@ -97,6 +112,8 @@ defmodule LlmScratch.InstructionDataset do
   """
   @spec new([instruction_record()], tokenizer(), keyword()) :: t()
   def new(data, tokenizer, opts \\ []) when is_list(data) do
+    prompt_style = Keyword.get(opts, :prompt_style, :alpaca)
+
     # Pre-tokenize each complete training example once so repeated dataset
     # access does not rebuild the prompt or call the tokenizer again.
     encoded_texts =
@@ -104,7 +121,7 @@ defmodule LlmScratch.InstructionDataset do
         entry
         # The model trains on the instruction/input prompt followed by the
         # expected response target, matching the chapter 7 PyTorch dataset.
-        |> full_text()
+        |> full_text(prompt_style)
         |> encode_text!(tokenizer, opts)
       end)
 
@@ -112,7 +129,8 @@ defmodule LlmScratch.InstructionDataset do
     # from `encoded_texts`, which mirrors the PyTorch dataset fields.
     %__MODULE__{
       data: data,
-      encoded_texts: encoded_texts
+      encoded_texts: encoded_texts,
+      prompt_style: prompt_style
     }
   end
 
@@ -269,13 +287,24 @@ defmodule LlmScratch.InstructionDataset do
     }
   end
 
-  defp full_text(%{"output" => output} = entry) when is_binary(output) do
+  defp full_text(%{"output" => output} = entry, :alpaca) when is_binary(output) do
     instruction_plus_input = LlmScratch.FineTuneDataLoader.format_input(entry)
     response_text = "\n\n### Response:\n#{output}"
 
     instruction_plus_input <> response_text
   end
 
+  defp full_text(%{"output" => output} = entry, :phi3) when is_binary(output) do
+    entry
+    |> LlmScratch.FineTuneDataLoader.format_text(:phi3)
+    |> Kernel.<>(output)
+    |> Kernel.<>("\n<|end|>")
+  end
+
+  defp full_text(_entry, prompt_style) do
+    raise ArgumentError, "unsupported prompt style: #{inspect(prompt_style)}"
+  end
+
   defp encode_text!(text, tokenizer, opts) when is_binary(tokenizer) do
     allowed_special = Keyword.get(opts, :allowed_special, [@end_of_text])
     {:ok, token_ids} = Tiktoken.encode(tokenizer, text, allowed_special)

Here is a test for instruction fine-tuning using the Phi-3 format:

@tag :download
  @tag :train_long
  @tag timeout: 3_600_000
  test "exercise 7.1 trains instruction model with Phi-3 prompt style" do
    device = use_accelerated_backend()

    tokenizer = "code-davinci-002"
    batch_size = 8
    num_workers = 3

    model =
      "355M"
      |> GPT2OpenAI.load_model(models_dir: "gpt2")
      |> Nx.backend_transfer(device)

    file_path =
      System.tmp_dir!()
      |> Path.join("llm_scratch_instruction_data")
      |> Path.join("instruction-data.json")

    data =
      FineTuneDataLoader.download_and_load_instructions_file(
        file_path,
        @instruction_data_url
      )

    train_portion = trunc(length(data) * 0.85)
    test_portion = trunc(length(data) * 0.1)
    val_portion = length(data) - train_portion - test_portion
    train_data = Enum.slice(data, 0, train_portion)
    val_data = Enum.slice(data, train_portion + test_portion, val_portion)

    train_dataset = InstructionDataset.new(train_data, tokenizer, prompt_style: :phi3)
    val_dataset = InstructionDataset.new(val_data, tokenizer, prompt_style: :phi3)

    :rand.seed(:exsss, {123, 123, 123})

    train_loader =
      DataLoader.new(train_dataset.encoded_texts,
        batch_size: batch_size,
        collate_fn: &binary_instruction_collate/1,
        shuffle: true,
        drop_last: true,
        num_workers: num_workers
      )

    val_loader =
      DataLoader.new(val_dataset.encoded_texts,
        batch_size: batch_size,
        collate_fn: &binary_instruction_collate/1,
        shuffle: false,
        drop_last: false,
        num_workers: num_workers
      )

    train_loss = LossUtils.calc_loss_loader(train_loader, model, device, 5)
    val_loss = LossUtils.calc_loss_loader(val_loader, model, device, 5)

    input_text =
      val_data
      |> List.first()
      |> FineTuneDataLoader.format_text(:phi3)

    token_ids =
      TextGeneration.generate(
        model,
        input_text
        |> TextUtils.text_to_token_ids(tokenizer)
        |> Nx.backend_transfer(device),
        35,
        model.cfg.context_length,
        0.0,
        nil,
        50_256
      )
      |> Nx.backend_transfer(Nx.BinaryBackend)

    generated_text = TextUtils.token_ids_to_text(token_ids, tokenizer)

    num_epochs = 2
    optimizer = Training.adamw(0.00005, weight_decay: 0.1)

    {trained_model, trained_optimizer, train_losses, val_losses, tokens_seen} =
      Training.train_model_simple(
        model,
        train_loader,
        val_loader,
        optimizer,
        device,
        num_epochs,
        5,
        5,
        input_text,
        tokenizer,
        generate_samples: true,
        return_optimizer: true
      )

    checkpoint_path = "ch7_instruction_finetuned_gpt2_355m_phi3_model_and_optimizer.nx"
    ModelCheckpoint.save_training_state!(trained_model, trained_optimizer, checkpoint_path)

    metrics_path = "ch7_instruction_finetuning_phi3_metrics.json"

    write_instruction_training_metrics!(
      metrics_path,
      num_epochs,
      train_losses,
      val_losses,
      tokens_seen
    )

    assert train_dataset.prompt_style == :phi3
    assert val_dataset.prompt_style == :phi3
    assert String.starts_with?(input_text, "<|user|>\n")
    assert String.ends_with?(input_text, "\n<|end|>\n<|assistant|>\n")
    assert String.starts_with?(generated_text, input_text)
    assert is_float(train_loss)
    assert is_float(val_loss)
    assert trained_model.__struct__ == model.__struct__
    assert %Training.AdamW{} = trained_optimizer
    assert File.exists?(checkpoint_path)
    assert File.stat!(checkpoint_path).size > 0
    assert File.exists?(metrics_path)
    assert File.stat!(metrics_path).size > 0
    assert length(train_losses) == 47
    assert length(val_losses) == 47
    assert length(tokens_seen) == 47
    assert Enum.all?(train_losses ++ val_losses, &is_float/1)
    assert Enum.all?(tokens_seen, &is_integer/1)
    assert tokens_seen == Enum.sort(tokens_seen)
    assert List.last(tokens_seen) > List.first(tokens_seen)
  end

And here is the training output:

  Ep 1 (Step 000000): Train loss 2.150, Val loss 2.147
<|user|>
Convert the active sentence to passive: 'The chef cooks the meal every day.'
<|end|>
<|assistant|>
<|user|>
<|end|>
<|assistant|>
<|end|>
<|assistant|>
<|end|>
<|assistant|>
<|end|>
Ep 1 (Step 000005): Train loss 1.200, Val loss 1.174
<|user|>
Convert the active sentence to passive: 'The chef cooks the meal every day.'
<|end|>
<|assistant|>
The chef cooks the meal every day.
<|end|><|endoftext|>The following is a list of all the Pokémon in the Pokémon Trading Card Game.
For a list of all Pokémon in the Pokémon Trading Card Game, see Pokémon.
If
Ep 1 (Step 000010): Train loss 1.061, Val loss 1.044
<|user|>
Convert the active sentence to passive: 'The chef cooks the meal every day.'
<|end|>
<|assistant|>
The chef cooks the meal every day.
<|end|><|endoftext|>The following is a guest post by Dr. David J. Karp, a professor of psychiatry at the University of California, San Francisco.
The term 'psychosis'
Ep 1 (Step 000015): Train loss 1.005, Val loss 0.975
<|user|>
Convert the active sentence to passive: 'The chef cooks the meal every day.'
<|end|>
<|assistant|>
The chef cooks the meal every day.
<|end|><|endoftext|>The following is a guest post by Dr. David B. Smith.
The following is a guest post by Dr. David B. Smith.
The following is a guest

...

Ep 2 (Step 000215): Train loss 0.411, Val loss 0.716
<|user|>
Convert the active sentence to passive: 'The chef cooks the meal every day.'
<|end|>
<|assistant|>
The meal is prepared by the chef every day.
<|end|><|endoftext|>The following is a list of all the animals that are native to the United States.
1. Squirrel
2. Eagle
3. Bat
4. Eagle
Ep 2 (Step 000220): Train loss 0.385, Val loss 0.716
<|user|>
Convert the active sentence to passive: 'The chef cooks the meal every day.'
<|end|>
<|assistant|>
The meal is prepared every day by the chef.
<|end|><|endoftext|>The following is a list of all the animals that are native to the United States.
1. Squirrel
2. Eagle
3. Bat
4. Eagle
Ep 2 (Step 000225): Train loss 0.372, Val loss 0.714
<|user|>
Convert the active sentence to passive: 'The chef cooks the meal every day.'
<|end|>
<|assistant|>
The meal is prepared every day by the chef.
<|end|><|endoftext|>The following is a list of all the animals that are native to the United States.
1. Squirrel
2. Eagle
3. Bat
4. Cro
Ep 2 (Step 000230): Train loss 0.367, Val loss 0.701
<|user|>
Convert the active sentence to passive: 'The chef cooks the meal every day.'
<|end|>
<|assistant|>
The meal is cooked every day by the chef.
<|end|><|endoftext|>The following is a list of all the animals that are classified as birds.
Birds
Cetacea
Cetacea is a type of bird.

.
Finished in 669.8 seconds (0.00s async, 669.8s sync)

This test calculates the Llama 3 score through Ollama. The score is worse than with the Alpaca prompt format:

@tag :download
  @tag :train
  @tag :ollama
  @tag timeout: 1_800_000
  test "exercise 7.1 scores Phi-3 prompt style instruction model with Ollama" do
    assert OllamaUtils.ollama_running?()

    checkpoint_path = "ch7_instruction_finetuned_gpt2_355m_phi3_model_and_optimizer.nx"

    assert File.exists?(checkpoint_path)

    device = use_accelerated_backend()
    tokenizer = "code-davinci-002"

    %{model_state_dict: model} = ModelCheckpoint.load_training_state!(checkpoint_path)
    model = Nx.backend_transfer(model, device)

    file_path =
      System.tmp_dir!()
      |> Path.join("llm_scratch_instruction_data")
      |> Path.join("instruction-data.json")

    data =
      FineTuneDataLoader.download_and_load_instructions_file(
        file_path,
        @instruction_data_url
      )

    train_portion = trunc(length(data) * 0.85)
    test_portion = trunc(length(data) * 0.1)
    test_data = Enum.slice(data, train_portion, test_portion)

    output_path = "instruction-data-with-response-phi3.json"

    enriched_data =
      test_data
      |> InstructionsEvaluation.write_responses!(model, tokenizer, device,
        output_path: output_path,
        prompt_style: :phi3
      )

    scores = InstructionsEvaluation.generate_model_scores(enriched_data, "model_response")
    average_score = Enum.sum(scores) / length(scores)
    metrics_path = "ch7_instruction_finetuning_phi3_ollama_scores.json"

    write_ollama_score_metrics!(metrics_path, scores, average_score)

    assert File.exists?(output_path)
    assert File.exists?(metrics_path)
    assert length(enriched_data) == 110
    assert length(scores) == 110
    assert length(scores) == length(enriched_data)
    assert_in_delta average_score, 46.67, 0.4
  end

Exercise 7.2 Instruction and input masking

After completing the chapter and fine-tuning the model with InstructionDataset, replace the instruction and input tokens with the -100 mask to use the instruction masking method illustrated in figure 7.13. Then evaluate whether this has a positive effect on model performance. First, we need to support masking input prompts in targets:

diff --git a/lib/llm_scratch/instruction_dataset.ex b/lib/llm_scratch/instruction_dataset.ex
index c229439..342a03b 100644
--- a/lib/llm_scratch/instruction_dataset.ex
+++ b/lib/llm_scratch/instruction_dataset.ex
@@ -27,19 +27,22 @@ defmodule LlmScratch.InstructionDataset do
        25, 198, 40313, 13]
   """
 
-  @enforce_keys [:data, :encoded_texts, :prompt_style]
-  defstruct [:data, :encoded_texts, :prompt_style]
+  @enforce_keys [:data, :encoded_texts, :prompt_style, :mask_out_instructions_in_target]
+  defstruct [:data, :encoded_texts, :prompt_style, :mask_out_instructions_in_target]
 
   @end_of_text "<|endoftext|>"
   @pad_token_id 50_256
 
   @type instruction_record :: LlmScratch.FineTuneDataLoader.instruction_record()
+  @type encoded_example ::
+          [integer()] | %{token_ids: [integer()], prompt_length: non_neg_integer()}
   @type tokenizer :: String.t()
   @type prompt_style :: :alpaca | :phi3
   @type t :: %__MODULE__{
           data: [instruction_record()],
-          encoded_texts: [[integer()]],
-          prompt_style: prompt_style()
+          encoded_texts: [encoded_example()],
+          prompt_style: prompt_style(),
+          mask_out_instructions_in_target: boolean()
         }
 
   @doc """
@@ -76,12 +79,18 @@ defmodule LlmScratch.InstructionDataset do
       Tiktoken model name. Defaults to `["<|endoftext|>"]`.
     * `:prompt_style` - prompt format for full training examples. Supported
       values are `:alpaca` and `:phi3`. Defaults to `:alpaca`.
+    * `:mask_out_instructions_in_target` - when `true`, instruction/input
+      prompt targets are masked with `-100` so the loss is computed only after
+      the prompt. Padding targets are still masked independently by
+      `custom_collate/4`. Defaults to `false`.
 
   ## Output
 
   Returns a `%LlmScratch.InstructionDataset{}`. The `:data` field contains the
-  original records, and `:encoded_texts` contains one pre-tokenized token-id
-  list per record.
+  original records, and `:encoded_texts` contains one pre-tokenized example per
+  record. With `mask_out_instructions_in_target: false`, each example is a
+  token-id list. With `mask_out_instructions_in_target: true`, each example
+  also carries prompt-length metadata used by `custom_collate/4`.
 
   ## Examples
 
@@ -113,16 +122,13 @@ defmodule LlmScratch.InstructionDataset do
   @spec new([instruction_record()], tokenizer(), keyword()) :: t()
   def new(data, tokenizer, opts \\ []) when is_list(data) do
     prompt_style = Keyword.get(opts, :prompt_style, :alpaca)
+    mask_out_instructions_in_target = Keyword.get(opts, :mask_out_instructions_in_target, false)
 
     # Pre-tokenize each complete training example once so repeated dataset
     # access does not rebuild the prompt or call the tokenizer again.
     encoded_texts =
       Enum.map(data, fn entry ->
-        entry
-        # The model trains on the instruction/input prompt followed by the
-        # expected response target, matching the chapter 7 PyTorch dataset.
-        |> full_text(prompt_style)
-        |> encode_text!(tokenizer, opts)
+        encode_example(entry, tokenizer, opts, prompt_style, mask_out_instructions_in_target)
       end)
 
     # Keep the decoded records for inspection while serving encoded examples
@@ -130,7 +136,8 @@ defmodule LlmScratch.InstructionDataset do
     %__MODULE__{
       data: data,
       encoded_texts: encoded_texts,
-      prompt_style: prompt_style
+      prompt_style: prompt_style,
+      mask_out_instructions_in_target: mask_out_instructions_in_target
     }
   end
 
@@ -226,7 +233,7 @@ defmodule LlmScratch.InstructionDataset do
       >
   """
   @spec custom_collate(
-          tuple() | [[integer()]],
+          tuple() | [encoded_example()],
           integer(),
           integer(),
           nil | pos_integer()
@@ -248,15 +255,17 @@ defmodule LlmScratch.InstructionDataset do
 
   def custom_collate(batch, pad_token_id, ignore_index, allowed_max_length)
       when is_list(batch) do
+    examples = Enum.map(batch, &normalize_example/1)
+
     # Find the longest sequence length after the one extra pad token that the
     # Python collate function appends to every item.
     batch_max_length =
-      batch
-      |> Enum.map(&(Kernel.length(&1) + 1))
+      examples
+      |> Enum.map(&(Kernel.length(&1.token_ids) + 1))
       |> Enum.max(fn -> 0 end)
 
     {inputs, targets} =
-      Enum.map(batch, fn item ->
+      Enum.map(examples, fn %{token_ids: item, prompt_length: prompt_length} ->
         # Elixir data is immutable, so this builds the equivalent of
         # `new_item = item.copy(); new_item += [pad_token_id]`.
         new_item = item ++ [pad_token_id]
@@ -272,6 +281,7 @@ defmodule LlmScratch.InstructionDataset do
           padded
           |> Enum.slice(1, batch_max_length - 1)
           |> mask_extra_padding_targets(pad_token_id, ignore_index)
+          |> mask_prompt_targets(prompt_length, ignore_index)
 
         # Optionally cap both rows to the model context length.
         {
@@ -287,8 +297,51 @@ defmodule LlmScratch.InstructionDataset do
     }
   end
 
+  defp encode_example(entry, tokenizer, opts, prompt_style, false) do
+    entry
+    # The model trains on the instruction/input prompt followed by the
+    # expected response target, matching the chapter 7 PyTorch dataset.
+    |> full_text(prompt_style)
+    |> encode_text!(tokenizer, opts)
+  end
+
+  defp encode_example(entry, tokenizer, opts, prompt_style, true) do
+    token_ids =
+      entry
+      |> full_text(prompt_style)
+      |> encode_text!(tokenizer, opts)
+
+    prompt_length =
+      entry
+      |> prompt_text(prompt_style)
+      |> encode_text!(tokenizer, opts)
+      |> Kernel.length()
+
+    %{token_ids: token_ids, prompt_length: prompt_length}
+  end
+
+  defp encode_example(
+         _entry,
+         _tokenizer,
+         _opts,
+         _prompt_style,
+         mask_out_instructions_in_target
+       ) do
+    raise ArgumentError,
+          "expected :mask_out_instructions_in_target to be a boolean, got: #{inspect(mask_out_instructions_in_target)}"
+  end
+
+  defp normalize_example(token_ids) when is_list(token_ids) do
+    %{token_ids: token_ids, prompt_length: nil}
+  end
+
+  defp normalize_example(%{token_ids: token_ids, prompt_length: prompt_length})
+       when is_list(token_ids) and is_integer(prompt_length) and prompt_length >= 0 do
+    %{token_ids: token_ids, prompt_length: prompt_length}
+  end
+
   defp full_text(%{"output" => output} = entry, :alpaca) when is_binary(output) do
-    instruction_plus_input = LlmScratch.FineTuneDataLoader.format_input(entry)
+    instruction_plus_input = prompt_text(entry, :alpaca)
     response_text = "\n\n### Response:\n#{output}"
 
     instruction_plus_input <> response_text
@@ -305,6 +358,10 @@ defmodule LlmScratch.InstructionDataset do
     raise ArgumentError, "unsupported prompt style: #{inspect(prompt_style)}"
   end
 
+  defp prompt_text(entry, prompt_style) do
+    LlmScratch.FineTuneDataLoader.format_text(entry, prompt_style)
+  end
+
   defp encode_text!(text, tokenizer, opts) when is_binary(tokenizer) do
     allowed_special = Keyword.get(opts, :allowed_special, [@end_of_text])
     {:ok, token_ids} = Tiktoken.encode(tokenizer, text, allowed_special)
@@ -331,6 +388,19 @@ defmodule LlmScratch.InstructionDataset do
     masked_targets
   end
 
+  defp mask_prompt_targets(targets, nil, _ignore_index), do: targets
+
+  defp mask_prompt_targets(targets, prompt_length, ignore_index) do
+    mask_count = max(prompt_length - 1, 0)
+
+    targets
+    |> Enum.with_index()
+    |> Enum.map(fn
+      {_target, index} when index < mask_count -> ignore_index
+      {target, _index} -> target
+    end)
+  end
+
   defp maybe_truncate(token_ids, nil), do: token_ids
   defp maybe_truncate(token_ids, max_length), do: Enum.take(token_ids, max_length)
 end

Here is the only change that we need to make in the test:

train_dataset =
      InstructionDataset.new(train_data, tokenizer, mask_out_instructions_in_target: true)

    val_dataset =
      InstructionDataset.new(val_data, tokenizer, mask_out_instructions_in_target: true)

Score is slightly worse: 49.78

Exercise 7.3 Fine-tuning on the original Alpaca dataset

The Alpaca dataset, by researchers at Stanford, is one of the earliest and most popular openly shared instruction datasets, consisting of 52,002 entries. As an alternative to the instruction-data.json file we use here, consider fine-tuning an LLM on this dataset. The dataset is available at https://mng.bz/NBnE. This dataset contains 52,002 entries, which is approximately 50 times more than the dataset we used here, and most entries are longer. Thus, I highly recommend using a GPU for the training, which will accelerate the fine-tuning process. If you encounter out-of-memory errors, consider reducing the batch_size from 8 to 4, 2, or even 1. Lowering the allowed_max_length from 1,024 to 512 or 256 can also help manage memory problems.

Per epoch, we have 5,525 steps, which is 20x more than Sebastian’s training set. With one epoch, training took 24,000 seconds and the average Llama 3 score through Ollama was 43.15. With two epochs, training took 41,060.8 seconds and improved the average score to 50.54.

Exercise 7.4 Parameter-efficient fine-tuning with LoRA

To instruction fine-tune an LLM more efficiently, modify the code in this chapter to use the low-rank adaptation method (LoRA) from appendix E. Compare the training run time and model performance before and after the modification.

With LoRA, training on Sebastian’s instruction set took 648 seconds, compared with 687 seconds without LoRA, with a slightly worse accuracy score of 49.9, compared with 50.32 without LoRA. Here are the commits that implemented the LoRA algorithm: feat: 🎸 lora support, feat: 🎸 lora support in model inspect, and test: 💍 lora test.

The Price of Kindness

2026-06-15T07:43:36Z

A humorous and emotional story of friendship, betrayal, family, faith, and life’s lessons.

Continue reading on ILLUMINATION »

blogs

Working with Data Tables with Playwright Java

Working with Data Tables with Playwright Java

Recap on fine-tuning to follow instructions

TL;DR

Giles blog posts

Workbook questions

Quick questions

7.1 Introduction to instruction fine-tuning

7.2 Preparing a dataset for supervised instruction fine-tuning

7.3 Organizing data into training batches

7.4 Creating data loaders for an instruction dataset

7.5 Loading a pretrained LLM

7.6 Fine-tuning the LLM on instruction data

7.7 Extracting and saving responses

7.8 Evaluating the fine-tuned LLM

Continuous Everything, Except Thinking

How to safely build trust in AI generated code

Selenium Nightly Releases: A QA Discipline Habit

Playwright Visual Testing: Day 9 Tutorial

AI-Augmented QA: The Complete Workflow for Using LLMs to 10x Your Testing Productivity

A holistic approach to "spec-driven development"

Build quality in throughout the SDLC

Test the Feature, Not the Endpoint

AI QA Agents: From Prompts to Runnable Checks

Playwright API Testing: Day 8 Tutorial

Playwright MCP vs Traditional Test Scripts

Book Review: Software Testing: A Craftsman`s Approach

How to Communicate QA Value to Non-Technical Stakeholders: The Language That Gets Budget

Your test isn’t flaky: Page 2 just stole data from page 1

The problem nobody spots right away

Where it really goes wrong: the paged table

Why test data triggers this so quickly

The solution: a tie-breaker

What you can do as a tester

Why you should not fix your tests if you think it it flaky

Reading the Buzz: Getting Past a Quiet No

​Team Buzz Is the Room Talking Without Talking

​The Hesitant Yes Costs More Than an Honest No

​Unpacking the No: A Sequence That Works

A Hesitant Yes Is Just a Slower No

The First Winter: Clearing Weeds and Planting Trees

Autumn Colors, All Around

Time To Work

The Other Side

Planting Oak and Beech

Pruning

Springtime!

Fine-tuning to follow instructions

TL;DR

7.1 Introduction to instruction fine-tuning

7.2 Preparing a dataset for supervised instruction fine-tuning

7.3 Organizing data into training batches

7.4 Creating data loaders for an instruction dataset

7.5 Loading a pretrained LLM

7.6 Fine-tuning the LLM on instruction data

7.7 Extracting and saving responses

7.8 Evaluating the fine-tuned LLM

Exercises

7.1 Changing prompt styles

Exercise 7.2 Instruction and input masking

Exercise 7.3 Fine-tuning on the original Alpaca dataset

Exercise 7.4 Parameter-efficient fine-tuning with LoRA

The Price of Kindness

Team Buzz Is the Room Talking Without Talking

The Hesitant Yes Costs More Than an Honest No

Unpacking the No: A Sequence That Works