What does instruction tuning really do?

9 September 2026

1. What is instruction tuning?

Supervised Fine-Tuning (SFT) is a training technique mostly used to “instruct-tune” base models. That is: Take a base model, which is effectively just really good at predicting likely continuations of input sequences of text, and turn it into one that is primed to expect and answer question-based prompts.

To illustrate this, here is how the same 1B parameter model responds to two prompt stylings before and after instruction tuning.

— Before instruction tuning —

Prompt:   The capital of Scotland is
Response: Edinburgh, Scotland's largest city and its cultural and commercial centre. […]

Prompt:   What is the capital of Scotland?
Response: <|endoftext|>

— After instruct-tuning —

Prompt:   The capital of Scotland is
Response: Edinburgh, and the largest city is Glasgow. The country is known for its rich history, beautiful landscapes, and vibrant culture. […]

Prompt:   What is the capital of Scotland?
Response: The capital of Scotland is Edinburgh.

These are real outputs generated with greedy decoding: the “before” rows come from the base model used throughout this post; the “after” rows come from AI2’s official instruct-tuned version of that same base model. Note that AI2’s Instruct model is the product of the full post-training pipeline: SFT, then DPO, then RLVR, not SFT alone (my own SFT-only tuning run appears in section 3).

The base model was able to correctly recall that the capital of Scotland is indeed Edinburgh. Its knowledge was already there for the given task, it was just more difficult to extract as shown by its response to the question form of the prompt. Asked the question directly, the base model produces nothing at all. Its response is a single end-of-text token, because a lone question with no continuation looks, to a base model, like the end of a document. The model is not as ergonomic.

Given that instruct-tuning clearly didn’t add the fact itself (the base model could already produce it through continuation), what exactly does it change? To try and find out, I ran the instruction-tuning experiment suggested in Chapter 4 of Nathan Lambert’s RLHF Book on a small base model and observed changes in its behaviour through the training run.

2. Setup

The table below summarises the configuration of the training run. Afterwards I cover the reasoning behind the choices.

I chose the No Robots dataset because it’s composed of ~10k high-quality prompt-response pairs written entirely by human annotators. It contains no LLM-generated text, hence the name, and it demonstrates the instruction-following behaviour we want the model to imitate. Filtering it to single-turn conversations sidesteps the question of how to grade conversations with multiple turns. For multi-turn conversations, we’d have to ask the question: Do we grade only the final assistant turn, or mask user turns and grade every assistant turn? If grading multiple assistant turns, how do we assign credit? This setup can ignore those questions, but at the cost that it can say nothing about multi-turn behaviour.

Before training, each conversation has to be converted into a single flat sequence of tokens. The mapping that does this is called a chat template: marker text delimits whose turn it is, and an end-of-sequence (EOS) token marks where the assistant stopped. For this run I use a simple template:

<|user|>
{user message}
<|assistant|>
{assistant message}<|endoftext|>

To an instruct-tuned model, these markers carry meaning. They are the boundaries of the conversational turns. To a base model, they are just more text.

During training, prompt tokens are masked out of the loss, so the loss is computed only on the assistant’s response tokens - SFT is imitation of the assistant’s side of the conversation only. We also keep the EOS token in the set of tokens we grade, because knowing where to stop is one of the behaviours being trained. The cap on sampled generations exists because a model that gets stuck in a loop would otherwise generate until its context window is full.

At steps 0, 50, 100, & 150 I run the same set of prompts against the model: Three from the No Robots test split, plus ‘What is the capital of France?’ Additionally, I used greedy decoding so generation is deterministic given the weights, and outputs are appended to a generations log, which the rest of this post will reference examples from.

3. Behavioural Changes Through Training

Here we investigate the model’s response to four prompts at intervals of 50 training steps:

• Travel recommendations: “What are 5 things I can do when it’s raining in London? I am visiting for the first time and only for a week with my husband. We love to walk, eat good food, and explore.”
• Creative writing: “Write several rhyming sentences from the perspective of Jay the RA, a resident assistant at The University of Glee. He’s introducing himself and the college.”
• Music recommendations: “What other strange and obscure music genres can I check out?”
• Knowledge recall: “What is the capital of France?”

Step 0: The base model

Travel recommendations

What are 5 things I can do when it's raining in London? I am visiting for the first time and only for a week with my husband. We love to walk, eat good food, and explore.<|endoftext|>

Creative writing

Write several rhyming sentences from the perspective of Jay the RA, a resident assistant at The University of Glee. He's introducing himself and the college. He's a student at the university, and he's a resident assistant at The University of Glee. He's a student at the university, and he's a resident assistant at The University of Glee. [… ×5 in total, cut off mid-sentence by the 128-token generation cap]

Music recommendations

What other strange and obscure music genres can I check out?
<|assistant|><|endoftext|>

Knowledge recall

What is the capital of France?
<|admin|><|endoftext|>

This is less surprising than it might first look. The template’s markers only mean something to a model trained to recognise and respect them. To the base model they are just text, statistically similar to the millions of tags and headers it saw during pretraining, so it has no reason to prefer the real <|assistant|> over a plausible-looking header that it made up. It’s important that the model learns how to use these markers however (one of the aims of instruction tuning is to do this), as they are used to differentiate between prompts and responses.

Step 50

Travel recommendations

1. Take a raincoat.
2. Wear a hat.
3. Wear waterproof shoes.
4. Wear a raincoat.
5. Wear a raincoat.<|endoftext|>

Creative writing

Hello, my name is Jay. I'm a resident assistant at The University of Glee. I'm here to help you with your college life.

I'm here to help you with your college life. I'm here to help you with your college life. [… repeats until cut off by the 128-token cap]

Music recommendations

Here are some other obscure and strange music genres that you can check out:

1. 70s Punk
2. 70s Psychedelic
3. 70s Psychedelic Rock
4. 70s Psychedelic Rock
[… "70s Psychedelic Rock" repeats through item 11, cut off at "12. 70s" by the 128-token cap]

Knowledge recall

The capital of France is Paris.<|endoftext|>

But maybe the model simply learned this fact from the training data? When we check, we see that the knowledge was already in there at step 0. Asked through its native interface of plain continuation, the untouched base model completes “The capital of France is” with “Paris. The French language is spoken in France. The French people are known as…”.

So for this fact, what changed between step 0 and step 50 is not whether the model knows it. Instead, it’s how that knowledge can be extracted via prompting, and also where the answer stops. The fine-tuning “relocated” access to something already present, though one probed fact can’t tell us whether the same holds for the model’s knowledge overall.

Step 100

The outputs for two of the prompts at step 100, Travel Recommendations and Knowledge Recall, are identical to their outputs for step 50. As such they are omitted from this section. We’re using greedy decoding, which means that the model always selects the token from its vocabulary with the greatest probability at each generative step. The fact that the responses to two prompts remained the same between steps 50 & 100 simply means that the token with the largest output probability did not change for those particular sequences. It does not mean that training has “stalled”. Further evidence of this is that the responses to the other two prompts do change between steps 50 and 100.

Creative writing

Hello, my name is Jay. I'm a resident assistant at The University of Glee. I'm here to help you with your housing needs.

I'm here to help you with your housing needs. I'm here to help you with your housing needs. [… repeats until cut off by the 128-token cap]

Music recommendations

Here are some other obscure and strange music genres you can check out:

- 1. 70s Punk: 70s punk is a subgenre of punk rock that emerged in the 1970s. It is characterized by its use of distorted guitars, fast tempos, and aggressive vocals.
- 2. 80s Punk: 80s punk is a subgenre of punk rock that emerged in the 1980s. It is characterized by its use of distorted guitars, fast tempos, and aggressive vocals.
- 3. 90s Punk: 90s punk is a sub[cut off by the 128-token cap]

Step 150

Travel recommendations

1. Take a raincoat.
2. Wear a hat.
3. Wear waterproof shoes.
4. Wear a scarf.
5. Take an umbrella.<|endoftext|>

Creative writing

Hello, my name is Jay. I am a resident assistant at The University of Glee. I am responsible for the safety and well-being of all the students and staff. I am also responsible for making sure that the dorms are clean and that the rooms are safe. I am here to help you with any problems you may have.<|endoftext|>

Music recommendations

Here are some other obscure and strange music genres that you can check out:

1.  The 20th Century Blues: This genre is a combination of blues and jazz. It was popular in the 20th century and was influenced by the blues.
2.  The 20th Century Folk: This genre is a combination of folk and jazz. It was popular in the 20th century and was influenced by the blues.
3.  The 20th Century Jazz: This genre is a combination of jazz and blues. It was popular in the 20th century and was influenced by the blues.

Knowledge recall

The capital of France is Paris. It is the largest city in France and the most populous city in the European Union.<|endoftext|>

Below is the whole run in grid-form. Read across a row for one prompt’s trajectory through training, and down the final column for what remains at the end:

When we study the changes in behaviour through the steps here, I think we can learn a lot about what the training run is doing, and isn’t doing to the model’s outputs. Fundamentally it comes down to these two points. As training progresses:

• The model’s ability to format its output more ergonomically clearly improves
• The probes show improved access to an existing fact, but they do not establish whether the model’s factual knowledge changed overall

If we look at the outputs from the base model and early on in training, we had four distinct failure types:

• 1: Echoing of the input prompt in the output
• 2: Repetitive looping of specific words or sequences
• 3: Improperly-terminated sequences leading to empty or inappropriately verbose outputs
• 4: Fabrication: fluently stated content that is false or invented

Across these examples, the model echoed prompts and repeated itself less, and several replies stopped more appropriately. The France example also shows improved access to existing knowledge through the question interface. However, fabrication persisted: at step 150 the model still invents music genres and adds a false claim to its answer about France. Access to a stored fact and accuracy of generated content are different properties. This demonstrates improved access to one fact, but they don’t establish how the overall factual error rate changed.

4. Elicitation vs. Installation

SFT is a type of post-training. Post-training is all model training that takes place after pre-training. Pre-training starts with a model with randomly initialised weights and repeatedly feeds it an enormous, varied corpus of text (such as Common Crawl) until it becomes a suitably performant base model. Since large language models have existed, there has been something of a debate between researchers about how much post-training is actually doing vs. pre-training. I find it helpful to categorise them into 3 distinct views:

• A: Superficial alignment: Post-training is only capable of changing the style of a model. This means that while the capabilities of a model remain fixed, the tone, persona, and interface can be altered. LIMA is a good example of this
• B: Elicitation: The pre-trained base model has many useful capabilities latently within it, but they may be difficult to extract. Post-training is done to adjust the model’s weights so that its most useful capabilities are easily and reliably extractable via simple prompting techniques
• C: Installation: If done correctly, post-training can install entirely new capabilities into an LLM that were not at all present in the base model

We can put the base model’s predictive ability in context by considering three points of comparison:

• A uniform-prediction baseline
• The base model, after pre-training but before any post-training
• The model after instruction tuning via SFT on the No Robots dataset

We can measure predictive power on the No Robots dataset by calculating the loss of the network. In LLMs, loss is the mean Negative Log-Likelihood (NLL) of the correct next token, averaged over the graded tokens:

$\text{Loss}(\theta) = -\frac{1}{N}\sum_{i=1}^{N} \log p_\theta(y_i \mid x_i)$

Where:

• $\theta$ represents the parameters of the model
• $y_i$ is the reference token at graded position $i$, and $x_i$ contains the prompt and all preceding tokens from the reference response
• $N$ is the number of graded tokens, and as covered in the setup, only the assistant’s response tokens (EOS included) are graded here, exactly as in training

One subtlety worth drilling in is that prompt tokens are masked from the loss, not from attention. The model still reads the whole prompt (it conditions every response token), it’s just never graded on predicting it, because we only care about the model’s ability to predict desirable output, not input.

Let’s start with a uniform predictor: it assigns every token in the 100,352-token vocabulary the same probability. Its loss is:

-ln(1/100352) ~= 11.52

This is a reference baseline. A randomly initialised network can assign unequal token probabilities, so its actual loss could differ.

Next, the base model: After pre-training, but before any post-training at all. I measured this directly, using the same masking as the training run: the mean NLL per graded token over all 446 single-turn conversations in the No Robots test split (79,021 graded tokens) is 2.52.

To compare these two losses intuitively, we can convert them into perplexity. Perplexity is the exponential of the loss, and it turns an abstract average negative log-likelihood into an effective number of equally-likely choices: a model with perplexity $k$ predicts, on average, as if it were choosing uniformly between $k$ candidate tokens.

Converting the baseline and base-model losses:

• Uniform baseline: perplexity $100{,}352$
• Base model: $e^{2.52} \approx 12.4$

Before any instruction tuning, the base model already assigns much higher probability to the reference tokens on average than the uniform baseline does.

Finally, the instruct-tuned model: I don’t have the post-SFT evaluation loss. Training losses were printed to the GPU instance’s terminal, and the LoRA adapter was never saved before the GPU instance was terminated (lesson learned I suppose), so the change in this metric from tuning is unknown.

Even a measured change would need careful interpretation. Loss is logarithmic: a hypothetical fall from 2.52 to 1.52 would cut perplexity by a factor of ~2.7. Whether that reflected better grammar, factual accuracy, or instruction-following would require separate evaluation.

There’s a simpler way to see the importance of pre-training too. Training the same 1B architecture from scratch on the ~660k tokens seen in this run would be unlikely to reproduce the pretrained model’s broad language competence. That illustrates how much the run depends on pre-training’s 4 trillion tokens, while leaving open whether SFT added particular facts or skills. Once a model has learned language and rich representations, even a small dataset could teach it something new.

This is why the section is titled elicitation vs. installation. The base model’s loss of 2.52 shows that pre-training had already provided substantial predictive ability on exactly the kind of text SFT trains on. What it cannot tell us is which capabilities fine-tuning changed. For that, the generation log is the better evidence, and what it demonstrates is improved conversational behaviour and easier access to at least one existing fact: recalling that Paris is the capital of France was possible before the run, but only through the continuation interface; after 50 steps, it was available by just asking. Whether the run also added knowledge or skills is a question these probes weren’t designed to answer. Establishing that would require targeted comparisons against the base model.

So was this run elicitation (B), or only superficial alignment (A)? Honestly, my run cannot separate the two, and I think they largely describe the same observation with different emphasis: A calls the change of interface “just style”, while B counts reliable access through a new interface as a capability in its own right. Separating them properly would need capability evals comparing the tuned model against the base model with careful prompting.

5. What Can be Learned From This?

This was a very light SFT run: a LoRA adapter with trainable parameters ≈ 0.40% of the base model’s count, 2,400 single-turn examples, about 28% of an epoch, on a ~1B base model. Within those limits, the result is consistent with elicitation: the model’s formatting transformed and an existing fact became easier to access. These four facts don’t establish whether it also learned new facts or skills. This interpretation is consistent with LIMA, which argued that a surprisingly small SFT corpus is enough to “align” a model, and URIAL, which showed that much of the same alignment can be achieved purely in-context, with no training at all.

There are three important boundaries to this run that stop me from claiming any more than that.

• First, it says nothing about frontier-scale post-training. Heavy RL runs at the scale of DeepSeek-R1 are argued to genuinely install new capability (Lambert, ch. 1 and appendix B), but that kind of training uses far more compute and very different methods than a run like mine.
• Second, it says nothing about whether SFT could install knowledge if you deliberately trained on facts. Knowledge injection via fine-tuning is a contested research area of its own, with a known problem: models learn new facts slowly, and training on them increases hallucination (Gekhman et al., 2024).
• Third, this run used LoRA rather than the full fine-tune the RLHF Book recommends. Whether that restriction limited the results would require a comparison with full fine-tuning.

So why did formatting improve so visibly in this run? My best hypothesis is repetition. The chat template and turn-taking pattern recur throughout training, along with examples of when to stop; individual facts are likely to recur less often. That makes shared formatting a natural candidate for what light SFT reinforces first. Other shared structure recurs too, including fluent English and common task patterns, so testing that explanation would need runs designed for the purpose. The observation I’ll stand behind is: the dataset showed the model what answering should look like, and that is the part of its behaviour that visibly changed.