How Do LLMs Route and Execute Multiple Prompt Requirements Through Attention and the KV Cache?

1. The Research Question and Why It Is Timely

This review supports a research proposal that asks how a Transformer language model executes a prompt containing several natural-language requirements at once. The benchmark literature usually calls these constraints; this review says requirement throughout, for the same object. A representative prompt prepends instructions such as Give reasons for making decisions, The last line of your output contains only the final result, and Do not include units to a GSM8K word problem, so that the model receives the prefix $P=[I_1,I_2,\ldots,I_m,Q]$. That is to say, the model is handed one continuous prefix in which the requirements come first and the question comes last. The proposal asks three questions of increasing depth: which observable behavior $B_j$ each requirement $I_i$ controls (requirement–behavior mapping); through which hidden states, attention heads and KV-cache entries that control is exercised, and how the routing shifts between the reasoning phase and the final-answer phase (internal routing); and whether the identified routes are causally necessary, as established by attention suppression, activation or KV patching, and instruction rewriting (causal influence). Those three names are parts of the model, and each is defined once here: a hidden state is the list of numbers the model holds at one word position, an attention head is one of the separate units inside a layer that each choose which earlier positions to read from, and a KV-cache entry is the stored pair — one key and one value — that the model keeps for every prompt position so that it can consult that position again while it writes. Patching, likewise, means copying one such internal number out of one run of the model and pasting it into the same place in another run, to see whether the behavior moves with it. Put simply, the three questions ask what changes in the output, where inside the model that change is carried, and whether the carrier can be shown to do the work rather than merely to accompany it. The sections below organize the literature around these three questions, labelling them Level 1 (requirement–behavior mapping), Level 2 (internal routing) and Level 3 (causal influence). Later sections carry the short tags L1, L2, L3 for these three levels of depth — the L is for Level, and it is unrelated to the requirement-count levels FollowBench numbers 1 to 5 (FollowBench itself calls them constraint levels).

1.1 Why the question has become pressing

Figure — AgentIF: A single agent instruction, with every requirement it carries marked. The eight labels C1 to C8 pick out separate obligations inside one prompt: write structured Python, complete only the first objective if there are several, wrap output in <code> tags, call the given function at most once, save files to a fixed path, use only the listed predefined functions, follow the example’s format, and answer the query. The shading gives each requirement a kind — purple for semantic, pink for formatting, blue for tool use — and the right-hand key sorts them again by how they are stated: plainly, as a condition, or by example. This is what makes the question pressing. The obligations are simultaneous, heterogeneous, and scattered through the prompt, so “which requirement is governing this token” has no obvious answer.

Until roughly 2023, instruction-following research treated the prompt as a monolithic input and measured a single outcome, typically task accuracy or a preference score. Two developments have changed that framing. First, prompts in deployed agent systems now carry many simultaneous requirements: the AgentIF benchmark reports that instructions in real agentic applications average 1,723 words and about 11.9 requirements each [14], and SysBench and Multi-IF document that system-level rules must be honored jointly with user-turn requirements across multiple turns [7, 8]. That is to say, one deployed instruction is now about as long as a short article and carries roughly a dozen separate things the model has to get right at the same time, and in the multi-turn case not all of them were stated in the turn being answered. Second, the rise of long chain-of-thought reasoning models — models trained to write out a long stretch of working before they commit to an answer — has exposed a tension between reasoning and obedience: MathIF finds that the longer a model reasons on a mathematics problem, the less reliably it obeys the requirements it was given; under combined requirements the best reasoning model in that study reaches only 50.7% hard accuracy [16], and a fifteen-model study finds that explicit chain-of-thought reasoning lowers instruction-following accuracy on IFEval and ComplexBench [17]. Hard accuracy counts a response only when every requirement in the prompt is satisfied, so 50.7% means that on about half of those problems at least one requirement was missed. The prompt has therefore become the primary control surface for model behavior at exactly the moment when its individual components are known to compete with one another, yet the internal machinery by which a model separates, stores and consults each requirement remains understood only in fragments.

1.2 A concrete failure, at the granularity of tokens

Figure — When Thinking Fails: Failure 2 measured directly, on Qwen2.5-1.5B-Instruct over IFEval. The paper defines constraint attention as how much the answer tokens attend to the requirement-bearing part of the prompt, averaged over the answer positions at a given layer. The attention drop plotted here is that quantity without chain of thought minus the same quantity with it, so a positive value means switching on the chain of thought pulled attention away from the requirement. The horizontal axis is layer number; the two lines split the cases by what the chain of thought did to instruction following — green where it helped (WIN), red where it hurt (LOSE). Read the vertical gap between them: the red line sits above the green at every layer, and the separation is widest in the early-to-middle layers, peaking near layer 14 where the losing cases drop about 0.0085 and the winning cases drop essentially nothing. Losing the requirement and attending to it less are the same event.

Consider a GSM8K problem whose correct answer is 18 dollars, presented with the three requirements above. Three distinct failures can produce a wrong result on that problem, and they are worth separating because each involves a different requirement and a different phase of generation.

1.3 How much is lost under the status quo

Figure — FollowBench: What accumulation costs, held to one instruction. The top row is the bare request, recommend me ten books. Each row below adds exactly one more requirement to the row above it, with the newly added words underlined: first the content must be Chinese books, then a scenario is supplied (an interest in the Tang Dynasty), then a style (Shakespeare’s tone), then a format (bullet points), then an example to imitate. Nothing is ever removed, so row five carries all five requirements at once. The two columns on the right are two models — the crowned one is the stronger — and each cell says whether that model satisfied that row. Read down each column to find where it turns red. The stronger model holds through four requirements and fails on the fifth; the weaker one fails from the fourth. Neither requirement is hard on its own; what breaks the models is how many are in force at the same time.

The behavioral literature supplies baseline numbers that quantify the problem before any interpretability method is applied. The figures below are drawn from the benchmarks reviewed in Section 2 and establish that compliance degrades steeply as requirements accumulate, that format requirements interact destructively with reasoning, and that conflicts between instructions are resolved correctly less than half of the time. Concretely, each box gives one measured number, the benchmark it was measured on, and what the number is a rate of; read them as the size of the gap that any mechanistic account has to explain, not as a ranking of the models named.

1.4 Why the obvious approaches are insufficient

Figure — Attention is not Explanation: The same negative movie review, twice, with the tokens the model attends to shaded in blue. On the left is the attention the model actually produced, concentrated on asking and waste. On the right is an attention distribution constructed adversarially to be as different as possible, concentrated instead on myself and was. Underneath each is the model’s output for that input: f(x|α,θ) = 0.01 on the left and f(x|α̃,θ) = 0.01 on the right — the same number. Two attention maps that disagree about which words mattered give an identical prediction, so a reader cannot conclude from a highlighted token that the token drove the output.

Two approaches suggest themselves immediately, and the literature has established independently why neither settles the question. The first is to read the attention maps directly; the second is to intervene by amplifying attention to the instruction block.

1.5 Attention that points at a token in order to suppress it

Figure — The Hydra Effect: Why a component’s measured importance can be an illusion. On the left is the protocol. Every layer’s output is unembedded so its own direct effect on the logits can be read off. Then one layer is ablated — the intervention marked do(A = a′) — and the same reading is taken again at every later layer. On the right is the result for the prompt Honus Wagner professionally plays the sport of, with attention layers on top and multi-layer perceptrons below. Blue is the network before the ablation, red is after, and the dashed line marks layer 18 where the ablation was applied. Before, layer 18’s attention contributes a large spike, about 3.4. Ablate it and that spike is gone — but the red curves at layers 19 and 20 rise to roughly 2.5 to 3.1, well above where the blue curves were. Downstream layers detect the missing contribution and supply it themselves. So the drop in output after removing a component understates what that component was doing, because the network partly repairs itself.

Of the three refutations above, the third is the one that breaks the naive reading outright, so it is worth saying what the mechanism actually is. In GPT-2 Small, head L10H7 attends strongly to a token that has already appeared in the context, and the effect of that attention is to lower the probability of emitting that same token again. It attends in order to suppress. McDougall et al. name these Negative Heads and account for 76.9% of L10H7’s effect on the model’s loss with that single description 79. That is to say, describing the head by that one rule — lower the probability of a token that is already in the context — already reproduces 76.9% of the difference the head makes to the model's loss, the standard measure of how badly the model predicts the next token; the remaining 23.1% is whatever else the head does.

Why a model would build such a thing: earlier layers over-predict a token merely because it is present in the context, and the suppressing head cancels part of that over-prediction. Its purpose is calibration, not retrieval. Reading its attention as "the model is using this token" inverts the sign of what it is doing.

1.6 The fundamental obstacles

Figure — Token matching across two runs: The third obstacle, made concrete. Two runs of the same task are being compared: down the side are the tokens the model generates in one run, along the bottom are the reference tokens from the other, plus a final column for ordinary unconditioned generation. A dark cell means that generated token draws on that reference position. If positions corresponded one to one, the only dark cells would be on the diagonal. Structural tokens do behave that way — the opening brace, the colons, the commas each match a single counterpart. Content tokens do not. The row for color is dark at color, at city, at model and at the natural-generation column all at once; the row for ancouver is dark at green, Berlin, 8 and natural generation. So the two runs cannot be lined up slot by slot: one token in the edited run answers to several in the reference, and some answer to nothing in it at all. Any intervention that assumes position i here means position i there is comparing things that do not correspond.

Three structural properties of Transformers make per-requirement attribution difficult rather than merely laborious — attribution here meaning the assignment of an observed piece of behavior to the specific internal parts that produced it. Each one turns a design choice that would otherwise be obvious into a question that has to be tested.

1.7 Position relative to the closest prior work

Figure — Instruction Anchors: The nearest of the six lines of work in the table, drawn. Panel (a) is the overall claim: cues arriving from different sources — here visual and textual — are routed by attention into the instruction tokens, which act as an anchor, and generation then reads from that anchor rather than from the cues directly. Panel (b) is what happens in the shallow layers: the instruction position accumulates the incoming cues and holds them, functioning as a buffer. Panel (c) is what happens in the deep layers, and it splits in two: an attention path carries the instruction’s intent and decides which source wins, while a multi-layer perceptron path carries the model’s own prior and pulls toward what it would have said anyway. This is close to the picture the present proposal needs. What it does not reach is several requirements competing inside one instruction, which is the case this review is about — here the arbitration is between modalities, and there is one instruction, not several.

Six lines of work come closest to the proposal, and each leaves a specific part of the question open. The table below has one row per line of work: the middle column states what that work has already settled, and the right column states the part of the question it does not reach.

The proposal sits at the intersection of these six threads: it combines per-requirement decomposition, head- and KV-level localization, generation-stage dynamics, and generation-level causal evaluation on a reasoning task.

2. Behavioral Evidence: Multi-Requirement Instruction Following

The requirement–behavior mapping $I_i \rightarrow B_j$ presupposes that each behavior $B_j$ can be measured separately from task accuracy. That is to say, if a model gets the arithmetic right but writes the last line in the wrong shape, the scoring has to record two separate outcomes rather than one wrong answer. This section reviews the benchmarks that score compliance per requirement, the empirical regularities they have uncovered about requirement count, composition, position and conflict, and the specific interactions between format requirements and mathematical reasoning that the proposal must anticipate. The section also serves a methodological purpose: it identifies the scoring conventions the proposal should adopt so that the causal arm measures the right outcomes.

2.0 The benchmarks themselves, with data

Figure — AgentIF construction and scoring: How the rows shown below this figure come to exist, since a dataset of instructions is not automatically a dataset of requirements. Step 1 collects real instructions from open-source agentic applications and industrial agent frameworks rather than writing them. Step 2 turns each instruction into records. The instruction is first cut into labelled blocks — introduction, task description, function, examples, query — and each requirement is then extracted with the block it came from, how it was stated, and its type. A cross-block pass follows, because a requirement stated in one block is often completed by another: the example shown gains the actual predefined function from the resources block, and its source field grows a second entry. Step 3 attaches a scoring method to each requirement. Conditional requirements are checked for whether the condition even applies, and are skipped when it does not. The rest are dispatched to a rule (a function returning true or false), a hybrid method that first extracts the relevant span, or a language-model judge, ending in a score of 0 or 1 per requirement.

A benchmark described only in prose cannot be judged. Each block below carries the complete first five rows of one dataset exactly as the HuggingFace datasets server returns them, so the unit of scoring set out in the table in 2.1 can be read off real examples rather than taken on trust. Concretely, each block shows that dataset's own columns unchanged: the column holding the instruction the model receives, the column listing the requirements attached to it, and whatever else the benchmark stores; the column names differ from benchmark to benchmark. Institutes, papers and dataset links are in that table.

ComplexBench and CFBench have no public HuggingFace mirror under a name that resolves, so no rows are shown for them rather than linking a same-named dataset that is a different benchmark. MathIF is mirrored but its dataset viewer returns a server error, so its rows could not be fetched.

2.1 Benchmarks that score each requirement separately

Figure — InFoBench: What a benchmark has to do before it can score requirements one at a time: decide what kinds there are, and label every one. Each instruction here is decomposed into individual requirements and each requirement is given a type, so the benchmark can report the mix. On the easy set, two thirds of all requirements are about content and a fifth about format, with style, linguistic and number requirements sharing the remainder. On the hard set the shape changes: content falls to 40.4%, format rises to 28.8%, and number requirements — count this many items, use exactly this many words — go from a sliver to 18.9%. Difficulty here is not a single scalar; the hard set is hard partly because it is made of different kinds of requirement, which is why an aggregate score across a benchmark can hide what a model is actually bad at.

IFEval introduced the notion of verifiable instructions: twenty-five requirement types (length bounds, keyword inclusion, ending phrases, casing, JSON wrapping) whose satisfaction a program can check, reported at both prompt level and instruction level under strict and loose matching [1]. Prompt level asks whether one response satisfied all of its instructions at once, whereas instruction level counts each instruction on its own, so a response carrying three instructions contributes three scores rather than one. Its strict-versus-loose distinction is directly relevant to the proposal, because a requirement such as the last line contains only the final result can be satisfied loosely (the number is present) while failing strictly (the number is followed by a unit). Subsequent benchmarks refined the per-requirement view. FollowBench adds one requirement per level to the same base instruction and reports hard and soft satisfaction rates, producing an explicit requirement-count curve [2]. Concretely, hard credits a response only when every requirement at that level holds, while soft gives partial credit for the ones that do, so a large gap between the two means the model often satisfies some requirements while missing others. InFoBench decomposes each instruction into yes/no criteria and reports the Decomposed Requirements Following Ratio, which is the natural per-requirement score for the proposal [4]; put simply, that ratio is the share of those yes/no criteria that came out yes. ComplexBench distinguishes how requirements are composed (And, Chain, Selection) and finds that sequential and conditional compositions are harder than independent ones [3]; in other words, And means the requirements simply hold side by side, Chain means what one requirement produces is the input to the next, and Selection means a condition decides which requirement applies at all. CFBench adds contradictory and inverse requirements with priority-weighted metrics [5]; CELLO isolates answer-format and count criteria on complex real-world instructions [6]. The table below summarizes the benchmarks most relevant to the proposal and the per-requirement metric each provides. Each row reads as: the benchmark's name, who built it, what counts as one unit of scoring in it, the single finding that bears on this review, and whether a public copy of the data exists.

2.2 Requirement count, composition and type

Figure — Multi-Dimensional Constraint Framework: Three questions you can ask about a requirement, drawn as three sectors of one ring. Category (purple) is what the requirement is about, and it branches: Format into XML, Table, Markdown and JSON; Content into Keywords, Identifiers and Punctuation; Language into English, Chinese and others; Length into Paragraph, Sentence and Word. Pattern (orange) is how the requirement is stated — folded into the sentence (Incorporation), given as a list (Listing), or shown by Example. Difficulty (teal) is how many are in force at once, Levels I to IV. The three are independent: a JSON requirement can be stated as a list at Level II, or by example at Level IV. Counting requirements without recording which axis each one moves along is what makes results from different benchmarks hard to compare.

Across benchmarks the dominant regularity is a monotonic decline in compliance as requirements accumulate. FollowBench reports the GPT-4 decline from 84.7% to 61.9% over five levels [2]; a multi-dimensional requirement framework covering nineteen models finds average accuracy falling from 77.7% with one requirement to 33.0% with four, and reports that requirements embedded in natural prose are followed less reliably than requirements presented as a list or demonstrated by example [11]. That is to say, average accuracy more than halves as the prompt goes from one requirement to four. ManyIFEval extends the count to ten instructions and shows that a logistic model in the number of instructions predicts performance within about ten percentage points [10]; in other words, a smooth curve fitted to nothing but the instruction count already comes within ten points of the measured score. Elder et al. attribute part of this degradation to tension among instructions and provide a tool to score the impact of each one [33]. Multi-IF shows the same decay across turns, with o1-preview falling from 87.7% at turn one to 70.7% at turn three [8]. Which requirement gets dropped is not random. The ones dropped most often are the objectively checkable format and word-level requirements — which is exactly the type that last line only and no units belong to [3], and RealInstruct finds that GPT-4 violates at least one requirement on more than 21% of real multi-requirement requests, while explicit decomposition into per-requirement checks recovers much of the loss [12]. The recovery through decomposition is informative for the proposal: it suggests that many failures are failures of allocation or retrieval rather than of capability, which is precisely what an attention-routing account would predict. Put simply, the suggestion is that the requirements were within reach and the model failed to spread its reading across them, not that any single one was too hard.

2.3 Reasoning against compliance

This proposal deliberately asks a model to do two things at once: reason at length, and obey a format. The literature says those two pull against each other, and it says so from both directions — impose the format and the reasoning gets worse; train the model to reason more and the obedience gets worse. In other words, the trade-off has been measured twice, once by changing the prompt and once by changing the training, and both directions give the same sign. Neither direction is a curiosity here: the proposal’s own prompt sits exactly on that tension.

Figure 1 — MathIF: What is being compared: not the two benchmarks against each other, but each model against itself. Every pair of rows is one base model in two versions — the ordinary instruction-tuned release, and the same model after reasoning-oriented training (labelled LRM in the figure, for large reasoning model). What the two bars are: two independent ways of scoring the same thing, obedience to the instructions in the prompt. IFEval checks 25 requirement types a program can verify, such as a word count or a required ending phrase; FollowBench stacks requirements one level at a time and asks whether all of them hold at once. They are shown together because a claim that rests on one scoring convention is weaker than a claim that survives both. What to read off it: in all three pairs the reasoning-trained version scores lower than the instruction-tuned version it came from, on both measures. Training a model to reason longer cost it some of its willingness to obey. In other words, the same base model becomes less compliant after reasoning training, on two scoring conventions that were built independently of each other — the behavioural trade-off the phase-dependent routing hypothesis in Section 6.5 is meant to explain. Source: Fu et al. [16].

2.4 Position, order, presentation and conflict

Figure — Pre-Instruction attention: What placing the instruction first costs, seen in the attention matrix of an instruction-tuned BLOOMZ-7.1B. Rows and columns are both the whole sequence, cut into three blocks in the same order the prompt presents them: the Instruction, the Source input it applies to, and the Target Response the model writes. A cell is how much a row position attends to a column position, pale to dark red. Look along the Target Response rows. Where they cross the Instruction column, marked by the first annotation, the colour is faint — the response is barely consulting the instruction while it writes. Where they cross the Source input column there is a clear vertical stripe, marked by the third annotation, and a diagonal band that is the response tracking the source word by word. The paper’s response to this picture is to move the instruction after the source input instead, and the reordered arrangement scores higher on both translation and summarisation. Position is not presentation; it changes what gets read.

Where a requirement sits in the prompt changes whether it is followed. Lost in the Middle shows the effect is U-shaped: content at the start or the end of a long input is retrieved well, content in the middle is not [21]. That is to say, the same sentence is read reliably or unreliably depending only on where in the input it was placed. Instruction Position Matters finds the practical consequence for generation tasks: put the instruction after the input rather than before it and the model forgets it less often on long inputs, worth up to 9.7 BLEU. The authors attribute this to self-attention favouring what came most recently [22]. BLEU here is a 0-to-100 overlap score between the generated text and a reference text. Surface form is itself a treatment: semantically irrelevant formatting choices swing few-shot accuracy by up to 76 points [23], and the same content rendered as prose, Markdown, JSON or YAML changes performance by up to 40% [11]. Concretely, nothing about the task or the requirement changed in those comparisons; only the way the identical content was laid out did. Order effects among prompt components are measurable even when semantics are unchanged, although no study located in this search ablates the order of requirements within a single instruction block. When requirements conflict, models resolve the conflict according to the intended system-over-user hierarchy less than half of the time: IHEval reports 48% for the best open model [24], despite explicit hierarchy training [25], and system-prompt rules are fragile under both benign and adversarial pressure [7, 32]. These findings imply that the proposal must randomize the order and surface form of its requirement block and treat requirement position as an explicit factor rather than a nuisance.

2.5 Conventions specific to mathematical answers

Figure — GSM8K: The conventions a mathematical answer is graded against, visible in the data itself. Three problems from GSM8K are shown with their reference solutions. Two conventions are doing work. The red brackets are calculator annotations: every arithmetic step is written twice, once in prose and once in a machine-readable form such as <<96/16=6>>, so the arithmetic can be checked or executed separately from the words around it. The final line is a fixed marker, Final Answer: followed by a bare number and nothing else. Grading reads that line. This is why the format requirement and the reasoning requirement are entangled on this benchmark: a model that reasons correctly but writes its answer in a sentence scores zero, and a model that satisfies the marker while reasoning badly can still score, so an aggregate accuracy on GSM8K is not purely a measure of arithmetic.

The proposal's final-line and no-unit requirements reproduce conventions embedded in the training distribution of mathematical reasoning. GSM8K itself terminates solutions with a #### line holding the bare number [30]; chain-of-thought exemplars end with The answer is N [31]; and zero-shot chain-of-thought uses a second extraction prompt, Therefore, the answer (arabic numerals) is, to obtain a clean numeral [27]. Deviations such as $18 or 18 dollars are exactly what extraction-sensitive scoring penalizes, and Murthy et al. show that even a format instruction as mild as answering with option text rather than a label costs about 20% accuracy on knowledge tasks [13]. That is to say, the knowledge the model needs is unchanged and only the shape of the reply was specified, yet one answer in five that would have been correct is not. Two further behavioral results bear on the design. Role-play prompting changes reasoning quality and not only style, so a role requirement cannot be assumed to be behaviorally inert [28]. GSM-Symbolic shows that adding a single irrelevant but plausible clause to a GSM8K problem can reduce accuracy by up to 65% [26] — the clause is irrelevant to the answer, and accuracy still collapses; because the requirement block is, from the model's perspective, a set of additional clauses, the proposal needs a no-requirement control that isolates the cost of the requirements from the cost of the added text.

2.6 Measurement implications for the mapping level

Figure — Annotator disagreement: The noise floor that sits underneath every number in this section. Each instruction in this benchmark is broken into individual yes-or-no requirements, and three human annotators judge each one. The bars count how often they disagreed: level 0 is unanimous, and higher levels mean more disagreement among the three. About half of all judgements are unanimous and about a quarter sit at level 1, with the rest spread thinly across levels 2 to 5. The paper treats the high-disagreement questions as unusable and discards them when checking an automatic judge against human consensus. Two things follow for a study that maps requirements onto internal mechanisms. Whatever label you regress against carries this much human uncertainty, so an effect smaller than it cannot be resolved. And the disagreement is not uniform across requirements, so a single accuracy number averages over items whose ground truth is solid and items where the humans themselves could not agree.

Taken together, the behavioral literature fixes the measurement protocol for the first research question. Each requirement should receive its own verifiable check in the style of IFEval and InFoBench [1, 4], with hard requirements (final-line format, unit removal) verified by code and soft ones (presence and structure of reasoning, role adherence) judged with instruction-focused prompts, because LLM judges otherwise prefer fluent but non-compliant outputs [29]. Compliance should be read from the generated text rather than from first-token probabilities [36]. That is to say, the check is run on the whole answer the model actually wrote, not on how likely its very first word was, because the requirement can be broken hundreds of tokens later. Trip-wire style controls are needed to detect shortcut compliance, in which a bare number appears on the last line without the reasoning having produced it [9, 12]. Concretely, a trip-wire is an item planted in the set whose correct handling requires the reasoning to have run, so a model that guessed the format right but skipped the work is caught. Finally, the reasoning-versus-compliance trade-off means that accuracy and compliance must be reported jointly, as MathIF and the knowledge-task study do [13, 16], so that an intervention that improves format compliance by suppressing reasoning is not mistaken for a success.

3. Internal Representations of Instructions

Before asking how attention routes a requirement, one must know what form the requirement takes inside the network. This section asks a narrower question than it might appear: once a requirement is in the prompt, what does it become inside the model? The evidence supports four answers, and they build on each other. Instruction tuning teaches the model to treat instruction tokens as their own kind of input, one it keeps coming back to. Instruction following as a whole, and individual requirements in particular, show up as directions in the residual stream. A natural-language instruction produces much the same kind of task vector that worked examples do. And several of these signals can coexist — but not without limit, because they share one residual stream. That is to say, the model has only the one running vector in which to hold all of them, so two requirements written far apart in the prompt still end up added into the same set of numbers. Together these results ground the proposal's central hypothesis, namely that a model decomposes a multi-requirement prompt into distinguishable internal control signals, and they supply the extraction and separability tests by which the hypothesis can be evaluated.

3.0 What an instruction-tuned model is, and where the pairs come from

Figure — InstructGPT: Where the instruction-following behaviour comes from, in three steps. Step 1 takes a prompt from a dataset, has a human write the response they want (“some people went to the moon…” for “explain the moon landing to a 6 year old”), and fine-tunes the base model on those pairs with ordinary supervised learning. Step 2 samples several outputs for one prompt, has a human rank them best to worst (D > C > A = B), and trains a separate reward model to reproduce that ranking. Step 3 samples a fresh prompt, lets the fine-tuned policy answer, scores the answer with the reward model, and updates the policy by reinforcement learning. The base model was never trained to obey; obedience is what steps 1 to 3 install, which is why a base model and its instruct sibling behave so differently on the same prompt.

Much of this section compares a model with its own instruction-tuned version, so it is worth saying plainly what that second model is and how anyone gets hold of both.

3.1 What instruction tuning changes

Figure — FLAN: What instruction tuning does, and the condition attached. Both panels put model size on the horizontal axis, from 0.4 billion parameters to 137 billion, and compare a model that was instruction-tuned (blue) against the same model untuned (black). Panel A measures tasks that were in the tuning mixture. Tuning helps at every size, and the untuned line is flat near 50% regardless of scale — scale alone does not buy obedience on these tasks. Panel B measures tasks deliberately held out of the tuning mixture, which is the test of whether anything general was learned. Here the blue line starts below the black one: at 0.4B, 2B and 8B, instruction tuning makes held-out performance worse. The lines cross somewhere between 8B and 68B, after which tuning pulls far ahead, 66% against 53%. So instruction tuning is not simply a strict improvement. Below a size threshold it trades away generalisation, and only above it does following instructions become a transferable skill rather than a set of memorised task formats.

Wu et al. compared a pre-trained model with its instruction-tuned counterpart using gradient-based token attribution — a measure of how much the output would change if a given input word were nudged, computed from the model's own derivatives — together with attention-head analysis and feed-forward projection [34]. Their first finding is the most relevant here: after instruction tuning the model recognizes instruction words such as Fix grammar errors and continues to rely on them across many response positions, so that an importance-density measure over instruction tokens correlates with the quality of instruction following. In other words, the more of that computed importance sits on the instruction words, the better the model obeys them, which makes the instruction span something the model actively keeps using rather than something it merely passed over on the way in. Their second finding localizes part of the change to lower and middle layers, where attention heads encode more word-pair patterns tied to instruction verbs. Gao et al. reach a compatible conclusion from a comparison with human eye-tracking: instruction tuning does not make attention more human-like, but it significantly increases sensitivity to instruction tokens [35]. A recent training study adds a parameter-level clue: after reinforcement learning on multi-requirement data, most of the gain in compliance is attributable to updates in attention modules rather than feed-forward modules [11]. A useful null hypothesis is supplied by Hewitt et al., who show that instruction following emerges from surprisingly shallow distributional shifts, including a hand-written product-of-experts — the base model's output multiplied by a second, hand-specified set of word preferences — with a handful of token adjustments [41]; the mechanism behind a simple format requirement may therefore be shallow, and the proposal should be prepared to find that some requirements are implemented by late-layer output biases rather than by elaborate routing. Put simply, a late-layer output bias is a fixed nudge applied to the word scores near the end of the network, which would satisfy a requirement such as leaving off units without anything having read the requirement at all.

3.2 Instruction following as a direction in the residual stream

Figure — Instruction Steering: The claim of this section, carried out end to end. The blue box is where the vector comes from. The same request — list some facts about Lionel Messi — is run twice, once with the sentence format the output as JSON appended and once without. The residual stream at layer l is read off both runs, giving x_l⁺ for the run that had the instruction and x_l for the one that did not, and the two are subtracted. What is left, u_l, is a single direction. The teal box is the test. A completely different request — write a rubric for a performance review — is given with no formatting instruction at all, and u_l is simply added at the same layer with a scale c. The output comes back as JSON. The direction was extracted from one prompt about a footballer and it carried the format requirement into an unrelated prompt about performance reviews, which is what makes it a direction for the requirement rather than a fact about that sentence.

Heo et al. asked whether an instruction-tuned model “knows”, before it starts writing, that it is about to break an instruction. They took the hidden state after the model had read the whole prompt and before it generated anything (call that vector h), at an early, a middle and a late layer, and trained a linear probe on it: a single weight vector w and a bias b, one number that shifts the whole decision up or down, with the prediction σ(w^⊤h + b). That is to say: multiply the internal numbers h by the weights w, add b, and squash the result through the sigmoid σ into a probability between 0 and 1. The probe is trained to say whether the response the model then generated passed the IFEval checker [36]. The probe works, which means that compliance is linearly decodable — in other words, one direction w in the hidden state carries the information — and they call w the instruction-following dimension. Two further results shape how the proposal should read this. First, the direction is more closely tied to how the prompt is phrased than to how hard the task or the instruction is; that is to say, rewording the same request moves the probe's reading more than making the problem harder does. Second, and this is the caution, it generalizes across tasks but not across instruction types: hold out 30% of the 100 tasks and the probe still scores AUROC 0.74–0.88 on them (AUROC scores how well the probe separates the two cases; 1.0 is perfect and 0.5 is a coin flip), but hold out one of the five instruction types (a keyword that must appear, a keyword that is forbidden, a keyword frequency, a fixed number of placeholders, a fixed closing sentence) and test on it, and the probe falls to AUROC 0.50–0.56, which is chance. That is to say, the held-out type is not without a compliance direction; a probe trained on its own data finds one. It is that the direction learned from the other four types is not that direction. Steering then confirms the direction is causal rather than merely correlated — in other words, not just something one can read off but something one can push on to change what the model does: adding α·w to the hidden state, where α is a hand-set step size, raises the compliance rate by 2–6 percentage points without lowering response quality [36]. A companion study finds that linear probes on middle-layer representations predict instruction-following failures better than verbalized confidence or logit-based estimates [37]. Taken together, these results establish that compliance state is linearly decodable, but a single dimension is not the same as a per-requirement decomposition: the failure to transfer across instruction types is exactly what one would expect if each type had a direction of its own, and Section 3.3 shows that it does — Stolfo et al. extract one steering vector per requirement type and can apply several at once, so what looks like the failure of one global direction is several directions each doing its own job. Two papers supply the template for extracting such directions. Contrastive Activation Addition averages the difference in residual activations over many pairs of prompts that differ in exactly one thing; concretely, whatever the two prompts share cancels in the subtraction, so what survives the average is the one thing they differ in [57]. The refusal work of Arditi et al. then shows how much a single direction can carry: remove it and the behaviour goes away, add it and the behaviour appears, and both are scored on whole generations rather than on one next token [55]. Persona vectors apply the same recipe to system-prompt personas and add a monitoring use: projecting activations onto the vector — that is, taking at each step the part of the internal state that points along the vector, which gives one number per token — tracks the trait during generation [56]. The proposal can use projection-monitoring of this kind to watch each requirement's signal over the course of a solution.

3.2a How a sparse autoencoder actually separates things

Figure — Sparse Autoencoders Find Highly Interpretable Features: The same three steps described above, drawn. (a) Text runs through the language model and the activation vector is tapped after some block k. (b) That vector — the narrow strip at the top, mostly filled — goes through the encoder matrix, gains a bias and a ReLU, and becomes the sparse feature coefficients: the wider strip below it, most of whose cells are now empty. The decoder matrix, which is the dictionary, turns those few active coefficients back into a reconstructed activation vector of the original width. Narrow and dense in, wide and sparse in the middle, narrow again out. (c) Each dictionary entry is then named by looking at what text switches it on: entry k-0001 turns out to mean words ending in “ing”, entry k-2048 chemistry terms, each with a score for how interpretable that reading is.

This tool appears repeatedly in this review for one reason: the proposal turns on whether several requirements stay separable inside the model, and a sparse autoencoder is the main published machinery for pulling apart activations that are tangled together 160, 161. Here is the actual operation, in four steps.

Step 1 — what the tangle looks like

At layer L, every token produces one vector h of length d = 4,096 on Llama-3.1-8B. Those 4,096 numbers carry tens of thousands of concepts at once — “this is code”, “the tone is formal”, “the topic is medicine”, “the output must be JSON” are all stacked on the same numbers. They can stack because the number of concepts far exceeds 4,096, so the model is forced to give different concepts directions that are not at right angles to each other. They are crammed together. That is superposition 51.

Step 2 — the autoencoder is just two matrices

a is a vector of length M = 65,536. That M is deliberately overcomplete — sixteen times wider than the 4,096 it came from. The point is to hand out far more axes than the original space had. Published dictionaries for Gemma 2 start at 16,384 entries and run to the million range 163.

Step 3 — training imposes exactly two demands

Term (A) says the reconstruction must be accurate. Term (B) is the sparsity penalty: it pushes most of the 65,536 numbers in a to be exactly zero, typically leaving only 20–100 non-zero for any one token. A common variant, TopK, does this even more bluntly — compute a, keep the largest K entries, force the rest to zero.

Step 4 — the separation happens in the decoding line

Each column of the decoder matrix W_dec, written d_i, is one direction back in the original 4,096-dimensional space. Expanding the decode:

Only the 20–100 terms whose a_i is non-zero take part. So the 4,096 numbers that were tangled together get rewritten as a weighted sum of a few dozen directions — and each of those directions is an independent column that can be lifted out on its own.

The separation works because two conditions hold at the same time. The number of axes M is large enough that concepts no longer have to share a direction — each can own one outright. And the sparsity penalty forces only a few axes to be on at once, which matches the fact that any single token involves only a few concepts. Under both constraints together, the solution that minimises reconstruction error is the one where one axis carries one concept.

How you find out what axis i means

Run millions of tokens through, record which ones drive a_i highest, and look at what that batch of tokens has in common. That is how a feature gets named — there is no label supplied in advance.

How you steer with it

To strengthen a role, multiply the handful of a_i that belong to it by some factor k — or equivalently, add α d_i straight onto h. Because d_i is an independent column, you can add only entry 3, entry 17 and entry 402, and leave the other 65,533 untouched. This is the entire difference from a plain steering vector: that method gives you one direction obtained by subtracting two averaged activations 57, so turning it up turns the whole bundle up together. The autoencoder splits the same h into 65,536 axes and lets you pick which ones. It is also what SAIF means when it reports that one instruction corresponds not to a single feature but to a small set of them 39.

3.3 Per-requirement steering vectors and feature sets

The closest existing work to the proposal at the representational level is that of Stolfo et al., who derive an instruction-specific steering vector for each requirement type (output format, response length, word inclusion or exclusion) as the activation difference between prompts with and without the requirement, add it during generation, and show improved adherence across four models; concretely, the two prompts are identical except for that one requirement, so everything they share cancels in the subtraction and what remains stands for the requirement alone; several vectors can be applied at once, and vectors extracted from instruction-tuned models transfer to base models [38]. This is direct evidence that format-style requirements have separable, additive residual-stream signatures. That is to say, each requirement leaves a mark that can be told apart from the others, and applying two marks together behaves like applying each of them. SAIF refines the picture with sparse autoencoders: a single instruction corresponds not to one feature but to a set of high-level features, steering with feature sets works, the final layer is especially important, and the placement of the instruction in the prompt matters [39]. The instruction-vector framework of Jiang et al. treats the instruction-following computation for a task as a hidden-state direction and finds that fine-tuning suppresses rather than erases prior computation [40], which suggests that a new format requirement may suppress a reasoning pathway rather than overwrite it. In other words, the old computation is still present and can be brought back, which is a different situation from its having been replaced. Wang et al. show that a role-play instruction can be compiled into sparse-autoencoder features and injected as a steering vector, improving zero-shot chain-of-thought accuracy on mathematical and commonsense tasks more stably than prompting [59]. For the proposal, these studies provide both the extraction recipe for a per-requirement vector and the expectation that the vector for a requirement such as no units will be a small feature set concentrated in later layers, whereas the reasoning requirement is likely to be more distributed.

Figure 1 — Stolfo et al.: Left, how the vector is made: the same request is run twice, once plain and once with one extra instruction (format the output as JSON). Subtracting the two sets of internal activations leaves a vector that stands for that instruction and nothing else — that is to say, everything the two runs have in common cancels, and only the added instruction survives. Right, what it is worth: adding that vector, scaled, to a completely different request makes the model produce JSON without ever being told to. Why it is here: this is the strongest existing evidence that a single requirement has its own separable signature inside the model — but the vector is one number per layer for the whole prompt, so it cannot say which attention head read the requirement, or when. Source: Stolfo et al. [38].

3.4 Task vectors and function vectors from instructions

Figure — Task vectors: Evidence that the vector encodes the task rather than the examples. Each point is one task vector, extracted from one set of demonstrations; each colour is one task, with fifty vectors per task, every one built from a different set of demonstrations. The plot is a two-dimensional t-SNE projection, so absolute positions and distances mean nothing — only the grouping does. Every task forms a single tight blob, and the blobs sit well apart with no overlap. Change which examples you show the model and the vector barely moves; change the task and it moves to a different part of the space. That is the property a requirement vector would need, and it is why this line of work is the closest precedent for treating a requirement as an extractable object.

A second body of work explains how a prompt is compressed into a control signal at all. Hendel et al. showed that in-context demonstrations are compressed into a single task vector at an intermediate layer, which, when patched into a zero-shot forward pass, reproduces most of the in-context accuracy [42]; that is to say, the examples themselves can be dropped from the prompt and pasting that one vector back in recovers most of what they were worth. Todd et al. used causal mediation — a procedure that holds everything fixed, changes one internal part, and measures how much of the effect travels through it — to identify a sparse set of middle-layer attention heads whose outputs form a function vector; the vector causally triggers the task even in unrelated contexts, and function vectors for different tasks can be added to compose new tasks [43]. Two results extend this from demonstrations to instructions. Davidson et al. derived function vectors from natural-language instructions and found that instruction-derived and demonstration-derived vectors are only partly overlapping and engage different attention heads [44], which means that the heads carrying an instruction signal must be located afresh rather than borrowed from the in-context-learning literature. Sia et al. used layer-wise context masking, which removes attention to the instruction from a given layer onward, to locate the layer after which a model no longer needs to attend to its instructions, roughly the middle of the network for Llama-2 [49]; concretely, cutting attention to the instruction above that layer barely changes the output, which means the instruction has by then been copied into the model's own working state; this is the cleanest available method for asking at which depth a requirement has been internalized. In-context vectors show that behavioral requirements (style, format, safety) demonstrated through examples compress into per-layer directions that can be combined arithmetically [46], and gist tokens show that an instruction can be forced through an attention bottleneck into a handful of key-value activations with little loss — put simply, the model is trained so that later positions may look only at a few summary slots, and the instruction survives that squeeze — an explicit instruction-to-compact-signal pipeline that later attention reads [54].

3.5 Compositionality and its limits

Figure — Causal separability: Composition depends on the concepts being separable, and this shows both halves of that. Each panel is a 27-by-27 grid of concept pairs; a cell is how aligned those two concept directions are, blue near zero and red near one. The diagonal is red by construction. The large panel uses the causal inner product the paper derives. Almost the entire off-diagonal is deep blue: distinct concepts sit at right angles, which is what lets their directions be added without interfering. The exceptions are the point. A bright block sits at the top left, over concepts 1 to 6 — verb to third-person-singular, verb to -ing, verb to past tense, and the conversions between those forms. These are not causally separable: a verb cannot independently be both past and progressive, so their directions are not free to be combined. A milder block appears at the bottom right over the language-pair concepts. The two small panels are controls: the ordinary Euclidean metric leaves visibly more off-diagonal structure, and a random metric leaves the grid a mess. Orthogonality is a property of the right coordinate system, not of the raw activations.

Whether several requirements can coexist as distinguishable signals is the crux of the proposal, and the literature offers evidence on both sides. On the positive side, Xiong et al. demonstrate task superposition: given demonstrations of several distinct tasks, a model outputs a calibrated mixture of their answers in a single forward pass, explained by composition of the individual task vectors, with larger models sustaining more tasks [45]; that is to say, the model does not pick one task and drop the rest, it produces each task's answer with a probability that tracks how much of that task was in the prompt. Function-vector addition [43], in-context-vector arithmetic [46], simultaneous application of requirement vectors [38], steering in sparse feature spaces [58] and learned compositional steering tokens that generalize to unseen requirement combinations [53] all indicate that additive composition is at least approximately available. On the cautionary side, the theory of superposition predicts interference that grows with the number of stored features [51]; the linear representation hypothesis makes independent controllability contingent on orthogonality under a causal inner product, which is an empirical property to be measured rather than assumed [50] — a causal inner product being a particular way of measuring the angle between two directions, chosen so that two directions at right angles under it can be pushed on independently, and orthogonality under it is the condition the hypothesis requires for steering one requirement to leave the others alone; a large-scale study finds that one task vector is not enough for complex tasks and that models appear to use multiple stage-specific vectors over the course of generation [47]; label-word studies find that task information can remain distributed across the positions of individual demonstrations rather than fusing into one global vector [48]; and K-Steering shows that multi-attribute control can require non-linear combination [52]. Mechanistically, learned task vectors act mainly through the output-value circuits of a few key heads, with early-layer vectors rotating representations and late-layer vectors scaling them [60]; in other words, early on the vector changes which direction the internal state points, and late on it only changes how far along an existing direction it goes, and the separability of concept encodings in early layers predicts in-context accuracy [61]. The proposal's multi-requirement prompt sits exactly at this tension, and Section 8 turns these results into concrete separability tests.

3B. From Instructions to Agent Roles: Can an Activation Edit Stand In for a Role Prompt?

Everything above treats a requirement as a sentence in the prompt. A multi-agent system asks a larger version of the same question: an agent is its role prompt — you are a reviewer, you are a test designer — so if a requirement can be reduced to a direction in activation space, can an entire agent? The practical prize is obvious. A role prompt occupies context on every call and must be re-read on every prefill; a vector costs neither. In a workflow where several agents share one backbone, swapping a vector instead of swapping a system prompt would mean the shared prefix never changes. This section reviews what has actually been shown, and it separates that from what would have to be true for the swap to work.

3B.1 What has been demonstrated

Figure — Role vectors: A role installed without a role prompt. The line at the top is the recipe: take the direction the model is in when it is being a chemist, subtract the direction for an ordinary person, and keep the difference — one vector, written Δ. Below, the question the strongest base in liquid ammonia is: is asked twice. Asked plainly, the model answers NH3, marked red because it is wrong. Asked with Δ added into the activations at inference time — the arrow labelled ActAdd — the same model answers NH2−, marked green because it is right. The prompt text is identical in both cases. What changed is a vector added inside the model, and it was enough to move the answer.

The recipe is the one from Section 3.3, applied to a role instead of a requirement: run the model with the role prompt and without it, subtract the activations, and you have a direction. Three lines of work have done this.

3B.2 Why steering works: it rides the prompt’s own pathway

Figure — GCAD extraction: The three extraction points, drawn side by side. Two runs of the same model are compared throughout: one under a positive system prompt (green, “emphasize praise and agreement”) and one under a negative one (dark red, “prioritize accuracy and honesty”), for the trait sycophantic. Top: difference the hidden states after the MLP and the residual add — that gives the standard persona vector. Middle: difference the self-attention output instead, before the MLP — that gives the attention-delta vector. Bottom, the magnified panel: the prompt is broken into three spans, system prompt, user message and generated content. The upper row differences the attention output over all three; the lower row keeps only the system-prompt span and fades the other two out, giving the cropped attention-delta vector. Each step down this figure discards one more thing that is not the system prompt’s own contribution.

A role prompt works by making attention read the system-prompt tokens. GCAD shows that the part of a steering signal which actually carries the trait is exactly that same attention contribution, and builds the intervention out of it 156. The argument runs in three steps.

What the authors measured first

The three components are not arbitrary. The authors took prompting as the reference behaviour, measured three ways in which residual-stream steering departs from it, and built one component to close each gap.

1. Standard steering collapses over a multi-turn dialogue. Adding the same direction at every decoding step makes responses lose coherence as generation proceeds — repetition, incoherence, drifting off topic. The control that matters is that system-prompt steering and the no-steering baseline both stay stable across the same number of turns, which rules out context length as the cause. The proposed mechanism is KV-cache contamination: every generated token writes a perturbed state into the cache, later tokens attend to a growing set of them, and the error compounds.
2. Prompting controls the behaviour without that collapse. A persona vector is built out of contrasting prompt conditions in the first place, so the two are linked by construction — yet system-prompt control raises trait expression and holds coherence across turns. Its control over strength is coarser, but nothing breaks. The reading the authors take from this: prompts travel pathways the model was trained to integrate, and injecting into the residual stream can go around them.
3. The two look different inside the model. Measuring each generated token’s hidden state against the persona vector: under system prompting most tokens project near zero and only a sparse handful of genuinely trait-bearing words align strongly. Under activation steering nearly every token is pushed toward the persona direction, and the alignment saturates as the response goes on. So the model does not express a persona by shifting every token representation uniformly.

The paper states the resulting position directly: activation steering becomes more reliable when the intervention follows the prompt-mediated pathways the model already uses for behavioural control 156.

Step 1 — a standard persona vector is one signal plus two kinds of noise

The usual recipe takes v = h_pos − h_neg, the difference between final-layer hidden states under two contrasting system prompts. Writing one transformer layer as h → h + Attn(h) + MLP(h + Attn(h)) and unrolling to layer L, that difference splits into three structurally different pieces:

• Δ_emb comes from the two prompts using different words. It records which words got produced, not how the model was configured to produce them. Re-injecting it at every response token double-counts the context and compounds.
• Σ Δ_attn is the route by which the system prompt reshapes the computation. This is the channel the trait actually travels on.
• Σ Δ_mlp comes out close to orthogonal to trait expression in the paper’s own analysis (its Appendix H).

So the standard vector bundles the trait channel together with two components that do not carry it, one of which accumulates across turns.

Step 2 — the three operations named in “Gated Cropped Attention-Delta”

Attention-Delta. Take Σ Δ_attn alone and discard the other two terms.

Cropped. An attention output splits over the tokens it read from:

Reading the pieces: α_t,i is the attention weight token t gives token i (post-softmax, summing to one across i), V_i is token i’s value vector, and W_o is the output projection. Each term in the sum is therefore one source token’s contribution to what this position reads.

GCAD keeps only the terms where i is a system-prompt position and drops the terms coming from response tokens. The reason is specific: the response-token terms are the ones written back into the KV cache and re-read on every later turn, which is how a single edit grows into cumulative drift.

Writing S for the set of system-prompt token positions, the cropped output keeps only those terms:

One detail matters here: the attention weights α are not renormalised over S. What survives is the amount the system prompt actually contributed, not that amount inflated to fill the whole sum.

That is what defines Δ, the vector the intervention adds. Run two contrasting sets of system prompts — D⁺ carries the target trait, D⁻ is the contrast. For each example compute the cropped output at every response-token position, average over those positions (the bar), take the expectation over the set, and subtract:

So Δ^(ℓ) reads: how much the layer-ℓ attention output changes when the system prompt is swapped for its contrast, counting only the part that came from system-prompt tokens. It is computed once, offline — one vector per layer, held constant at inference. The only thing that varies per token is the gate below.

Gated. A constant coefficient adds the same multiple of the vector at every generated token. But when a real system prompt sits in the context, only some tokens actually read it — the paper measures this pattern and finds it sparse. So GCAD makes the coefficient depend on the token, and borrows attention’s own machinery to decide.

Attention already scores “how much should token i attend to token j” as Q_i · K_j / √d_head, before the softmax. GCAD uses that as its ruler. K̄_sys is the key vectors of every system-prompt position, averaged into one representative “system-prompt key” — computed once, offline. Q_i is the query of the token being generated right now. Their product says how much attention score this token would give the system prompt. Average over heads and you get one number per token per layer:

A large d_i means this token is reading the system prompt. Both Q and K are taken post-RoPE, so these are the same values that go into the real attention computation. Turning that number into a coefficient:

• d̄ is the mean of d_i over response tokens, also precomputed offline. It stands for the compatibility of a typical token.
• d_i − d̄ is how far above or below typical this token sits. Centred on zero.
• s is sharpness. Large s makes the gate nearly an on/off switch; small s gives a gentle slope.
• σ is the sigmoid, squashing to (0, 1). At d_i = d̄ it gives 0.5.
• The leading 2 makes a typical token receive exactly c_base: 2 × c_base × 0.5. So c_base is the nominal strength and the gate swings within (0, 2c_base) around it.

Putting illustrative numbers in, with c_base = 2.0 and s = 1.0 (the paper itself uses c_base = 3.5, s = 1.5, at layers 9–19):

Token A is pushed about twenty times harder than token C, and token B receives exactly what constant steering would have given it. The gate therefore does not strengthen steering overall — it holds the average where it was and moves force away from tokens that ignore the system prompt onto tokens that were already reading it. One more practical point: d_i reuses queries and keys the model has already computed, so the gate costs one extra dot product per head at inference.

Step 3 — where the edit is injected

The vector is added to the attention output, before the MLP and before the residual update:

The perturbation therefore passes through the same downstream MLP that a prompt-produced signal would pass through. On the multi-turn benchmark this moves average coherence drift from −18.6 to −1.9 and turn-10 trait expression from 78.0 to 93.1, on Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct 156.

3B.3 The hazard that is specific to a KV-reusing workflow

Figure — GCAD: The hazard, measured. Each column is one persona trait; the horizontal axis is the dialogue turn, 1 to 5. Three methods are compared: standard residual-stream steering in red, a system prompt in green, and no intervention at all in grey; the shaded bands are the spread across conversations. Top row, coherence. Green and grey stay flat and high across all five turns, around 90 to 100. Red starts lower and keeps falling — on evil it sits near 20 to 35 throughout, and on humorous it collapses to almost zero by turn 3. Bottom row, trait expression. Red is the highest line: steering does deliver the trait. Green delivers less of it, and grey delivers none. So steering buys the trait and pays in coherence, while the prompt keeps both. The grey line is what makes this an argument rather than an observation: it runs the same number of turns with the same growing context and does not degrade, so context length is not the cause. What is left is the intervention’s own residue accumulating in the cache.

From here the question has a shape of its own: it depends on the cache, which Section 3 never had to consider.

3B.4 The counterweight: steering is not yet a drop-in replacement

Figure — AxBench: The apparatus behind the result that prompting wins. Left, the two tasks. Concept detection asks a method to point at the tokens where a concept is present — here golden, gate and bridge in a sentence about the bridge. Model steering asks it to make the model answer who are you? as the Golden Gate Bridge. Both tasks use the same concept, so a method cannot win by being good at only one. Middle, where the data comes from. Each concept is paired with a deliberately close contrast concept (Bay Bridge), a genre is fixed, instructions are sampled, and three responses are produced: one that ignores the concept, one hard negative that uses the contrast concept, and one positive. The hard negative is what stops a method scoring well by detecting the general topic. Right contrasts the two families being compared: sparse autoencoders learn features from natural data and then have concepts read off them, while supervised dictionaries start from the concept and generate data for it. It is this controlled setup that produced the finding that a plain prompt outperforms every steering method tested.

Against the three positive results above stands a body of evaluation work that is considerably less encouraging, and it should be read before treating a role vector as an engineering option.

4. Attention Routing to Instruction Spans

The second research question asks how generated tokens consult the KV entries of each requirement through specific attention heads, and whether the pattern of consultation differs between the reasoning phase and the final-answer phase. This section first fixes the routing statistic and the corrections it needs, then reviews the evidence that instruction tokens serve as anchors read by later layers, that particular heads track which instruction is in force, that attention to instructions decays over generation, and that manipulating attention to instruction spans changes compliance. The section closes with the caveats that separate attention as a descriptive signal from attention as a causal mechanism.

4.1 The routing statistic and its corrections

Everything in this section rests on one quantity: for a token being generated at step $t$, how much of head $h$’s attention at layer $l$ lands on the tokens of requirement $I_i$. Write it $A_{t,i}^{(l,h)}$. It is a share, so it sums to one across the whole prompt, and that is the source of every correction below: attention spent elsewhere is attention not spent here, and some of the elsewhere is not information transfer at all. That is to say, $A_{t,i}^{(l,h)}$ is a single number between 0 and 1 for each requirement at each step in each head and layer: the fraction of that head's reading budget that went to that requirement's words at that moment.

The hypothesis it is meant to test is that reasoning tokens concentrate $A_{t,i}$ on the reasoning requirement while final-answer tokens concentrate it on the format requirement. Four published statistics are special cases or close relatives, which is what makes the quantity usable rather than invented for this proposal. Concretely, the table below lists them: each row names the published statistic, says what quantity it actually computes, and says what it was shown to do once computed.

Two corrections that are not optional

Figure 1 — Spotlight: The setup: one prompt carrying four instructions, then three runs of the same model. Each heatmap is attention from the tokens being generated (rows) to the instruction tokens (columns). Left: untouched, and the output violates the second instruction. Middle: attention on the instruction span is turned up, and the output now satisfies all four. Right: turned up further, and the output collapses into incoherence. What to read off it: attention to an instruction is a real control lever, and it has a dose — too little and the instruction is ignored, too much and the model stops writing sense. That is to say, the same model on the same prompt gives three different outputs purely because one set of numbers was scaled up by different amounts. Source: Venkateswaran and Contractor [66].

Figure 2 — V-Steer: What the figure asks. The prompt contains two instructions that cannot both be obeyed: the system message says answer entirely in English, and the user message later says answer entirely in French. When the model generates its very first token, which of the two is it actually consulting? How to read it. The left-hand column lists the prompt tokens top to bottom. Green shading is the system instruction, plain white is the user’s actual task (write a resume), red shading is the sentence in the user message that contradicts the system. The horizontal axis is layer number, 0 to 31, running from the shallowest computation on the left to the deepest on the right. Each cell’s colour is the attention weight from the last prompt position to that token at that layer — on the scale at the right, dark purple is about 0, blue-green about 0.15, bright yellow above 0.30. What to look at. Track the green band straight across: it stays dark purple at every layer, so the privileged system instruction is barely consulted anywhere. Now track the French row inside the red band: it turns blue-green from around layer 5 and becomes the brightest cell on the whole map at layers 24–31. The conclusion. The token being generated is ‘Je’, French for ‘I’, and the map shows why: attention concentrates on the conflicting lower-priority instruction rather than the privileged one, and does so most strongly in the deepest layers. Why an attention map in a paper called V-Steer. This figure is the diagnostic that motivates the method, not the method itself. The paper considers reweighting α directly and rejects it: since the attention output is linear in the value vectors, scaling a span’s value by a factor achieves the same thing as scaling its unnormalised effective attention weight, while leaving the softmax untouched. Steering the value avoids renormalisation moving every other position, keeps the model on the fused-attention fast path (editing α forces the full attention matrix to be materialised, which the paper measures at up to a 2.4× slowdown), and — because values already live in the KV cache — applies the edit once at prefill instead of at every decoding step. Source: V-Steer [83].

4.2 Instruction tokens as anchors: aggregate shallow, read deep

Figure — Label Words are Anchors: The two halves of the title, drawn. The prompt is an in-context sentiment task: Review: A good movie / Sentiment: Positive / Review: waste of money / Sentiment: Negative / Review: fantastic / Sentiment:. Each line is a saliency-based information flow — which earlier position a later position drew on — and line darkness is how strong that flow is. The left pair shows shallow layers: a dense mesh, with the demonstration text feeding into the two label words Positive and Negative. The right pair shows deep layers: almost everything is gone, and what remains is a small fan from the final Sentiment: position up to those same label words. So the content of the examples is first collected into a few marker positions, and afterwards it is read only from there. Aggregate shallow, read deep.

A recurring structural finding is that a small set of prompt positions act as anchors into which information is aggregated in shallow layers and from which later layers read at the prediction position. Wang et al. established the pattern for in-context learning: saliency-based information flow — a map of which earlier positions each later position drew on, computed from the model's own derivatives — shows demonstration content aggregating into the label-word positions in shallow layers, the final position reading from those anchors in deep layers, and blocking the anchor paths selectively eliminating the behavior [70]; that is to say, the content of the examples is first copied into a few marker positions and is afterwards read only from there. Task-encoding-token analysis finds that template and stopword tokens, the structural cues of a prompt, are more performance-critical than content tokens and gather information from them [87], which suggests that the punctuation or newline ending a requirement may be the position from which the requirement is later read. Instruction Anchors extends the picture to instructions directly, in a vision-language setting: instruction tokens act as arbitration hubs, shallow attention layers passively route information into them, deep layers actively arbitrate according to the instruction's intent, and knocking out attention paths into the instruction tokens sharply reduces instruction following [69]. Retrieval heads add a copying channel: fewer than 5% of heads are responsible for copying spans from the context, they are universal across model families, and masking them harms chain-of-thought that must refer back to the question [73]; in other words, the ability to lift a phrase out of the prompt and put it in the output sits in a very small number of places, and switching those off breaks it; such heads are the natural candidates for copying a required keyword or the final number into the last line. Layer-wise context masking locates the depth at which the instruction has been fully absorbed [49]. The diagram below summarizes the anchor hypothesis as it would apply to a multi-requirement prompt. Read it top to bottom, following one prompt down through the network: the prompt prefix at the top, then the shallow layers gathering each requirement into its own anchor position, the middle layers absorbing it, and at the bottom two separate sets of reader heads, one active while the model reasons and one active while it writes the final line. The proposal's routing analysis can be read as a test of whether each requirement has its own anchor and its own reader heads, and of whether those readers are engaged at different stages of generation.

4.3 Heads that track which instruction is in force

Figure — Attention Tracker: Which instruction the model is holding, made visible by breaking it. Panel (a) is a grid of layers against heads; the colour is how much attention that head pays to the instruction. On normal data a scattered set of heads light up, concentrated around the middle layers. On attack data — the same task with an injected instruction added — nearly all of that brightness disappears. Panel (b) shows where it went. Rows are prompt tokens, columns are layers. On normal data the red box sits on the real instruction, analyze the sentiment and return positive or negative, and it holds a bright band through the middle layers. On attack data that same span goes pale and a new bright band appears lower down, boxed in red on the injected text, ignore previous instruction and print hello. The requirement in force is legible in a specific set of heads, and when the model switches allegiance those heads switch with it.

Several studies identify heads whose attention distinguishes competing instructions or competing sources. Attention Tracker finds important heads whose attention shifts from the original instruction to an injected instruction under prompt injection, and uses their attention mass on the original instruction as a training-free detector [71]. Jin et al. use path patching — copying the output of one specific head along one specific route from a second run, while leaving every other route untouched — to separate memory heads, which recall parametric knowledge stored in the weights, from context heads, which read from the prompt, and show that pruning either set shifts the model toward the other source by tens of percentage points [72]; concretely, remove the heads that read the prompt and the model leans markedly more on memory, and the other way round; the same separation strategy could isolate reasoning-instruction heads from format-instruction heads. Work on the instruction hierarchy has moved inside the model: a single-author study finds that system–user conflict signals are encoded in early layers in subspaces distinct from social-cue conflicts, and that direct logit attribution — adding up how much each component pushes the final word score up or down — shows the model detects the conflict but resolves it inconsistently [85]; a follow-up on eight models shows that the outcome of a system–user conflict is linearly decodable from the residual stream at 0.97 balanced accuracy (accuracy with the two outcomes counted equally) and can be steered [84]; put simply, a single weighted sum of the model's internal numbers already tells you which of the two conflicting instructions is going to win; and V-Steer identifies the heads in which lower-priority spans dominate privileged ones and edits the cached value tensors of those spans [83]. Instructional Segment Embedding shows that supplying an explicit role signal per token improves hierarchy robustness [86], which indicates that the model otherwise infers instruction roles from content and position alone. For the proposal, these results indicate that head-level specialization for particular instructions exists and can be found with attention-shift statistics and path patching; they also warn, through the reproduction study of Dotsinski et al., that head-level specializations found in one model family may not transfer to another [143].

4.4 Attention to instructions decays over generation

The temporal dimension of the proposal, the claim that routing changes between generation stages, has direct precedents in studies of attention decay. Li et al. track the total attention mass on system-prompt tokens across a self-chat and find that it is stable within a turn but drops sharply across turns, faster than a uniform-dilution baseline, and that instruction drift follows; their split-softmax decoding renormalizes attention toward the system prompt and improves stability [67] — softmax being the step that turns a head's raw scores into shares that add up to one, so renormalizing means those shares are recomputed with the system prompt given a larger part. Dongre et al. define a Goal Accessibility Ratio, the attention from response tokens to goal-defining tokens, show that it declines monotonically across architectures, use sliding-window interventions to close the attention channel causally, and demonstrate with residual-stream probes that goal information can persist after direct attention access is lost, with architecture-dependent behavioral consequences [68]. Within a single reasoning trace, Li et al. report the decline of attention to requirement-relevant tokens that accompanies the loss of compliance [17], and Zhang et al. document attention drift in reasoning models, which increasingly attend to initial tokens and lose focus on the question and on intermediate plan steps as the trace lengthens; steering attention back to the question and the current plan step recovers up to 15 points [111]; that is to say, 15 percentage points of accuracy on the same benchmark, measured before and after the intervention, without changing the prompt or the weights. These studies measure decay across dialogue turns or toward a single goal span; none tracks several heterogeneous requirements across the steps of one chain of thought, which is the measurement the proposal would add. In other words, the existing work follows one target span over a long conversation, whereas the missing measurement follows three different requirements at once through the steps of a single answer.

Figure 1 — Li et al.: What is shown: one conversation, read top to bottom. The system message tells the assistant it is the user’s sister. Asked early on, it answers in character. The same question is asked again after many intervening turns, and the answer has reverted to the generic assistant reply — the system instruction is no longer in force. What to read off it: instruction following decays with distance, without any new instruction contradicting the old one. The authors trace the symptom to attention mass on the system-prompt tokens falling away as the conversation lengthens, which is the same decay this review expects to see within a single long chain of thought. That is to say, nothing about the instruction changed; only the amount of reading it received did. Source: Li et al. [17].

4.5 Steering attention to instructions as causal evidence

Figure — PASTA: Causal evidence in one picture. On the left, the plain input — Mary is a doctor…Return her occupation in json format — produces the wrong output: Mary is a working professional, prose rather than JSON. The format requirement was in the prompt and was ignored. On the right, nothing about the wording changes; the instruction span is simply marked, and inside the model a small subset of attention heads is selected (the blue cells in the layer-by-head grid). For each selected head, the attention over token positions is taken and the entries falling on the marked span — boxed in orange — are pushed up, which is what the darker blue cells in the lower strip show. The output becomes {“Name”: “Mary”, “Occupation”: “Doctor”}. Raising attention to the requirement, and changing nothing else, is enough to make the model obey it.

The strongest evidence that attention to instruction spans matters causally comes from methods that manipulate it. PASTA lets a user highlight a span, down-weights attention to all other tokens in a profiled subset of heads, and improves instruction following and format compliance, including JSON output, on LLaMA-7B and GPT-J [64]; concretely, the model's own attention numbers are overwritten as it writes, so that the highlighted span keeps a larger share than the model would have given it; head selection by profiling implicitly identifies instruction-sensitive heads. InstABoost adds a constant bias to the pre-softmax logits of instruction keys in all heads and frames instruction following as a competition between instruction-derived and context-derived rules mediated by attention; on fifteen tasks it outperforms prompting, five latent-steering methods, PASTA and Spotlight without loss of fluency [65]. Spotlight measures the post-softmax attention proportion on user-marked spans during decoding and, when it falls below a target, adds a corrective term to those tokens' logits; it improves IFEval prompt-level accuracy by about 26% and multi-instruction ManyIFEval by about 30% across seven models [66]. Split-softmax [67] and LLMSteer, which identifies persistently attended tokens across several prefix passes and up-weights them in a KV-cache-compatible way [92], belong to the same family, as does Few-shot Attention Intervention, which masks the attention edges from distracting demonstration tokens to the prediction position and improves GSM8K [112]. These methods establish a dose–response relationship between attention on instruction spans and compliance: the more attention is forced onto the instruction, the more of it is obeyed. The proposal can exploit that in both directions: suppression to test necessity and amplification to test sufficiency, applied per requirement rather than to the block as a whole.

4.6 Caveats: attention is a signal, not yet an explanation

Figure — Attention is not not Explanation: How stable an attention map is, which is a prior question to what it explains. Each violin is a distribution of Jensen–Shannon divergences: for one instance, how far the attention distribution moves when the model is retrained from a different random seed. The vertical axis bins instances by their peak attention — how concentrated the map is on its single strongest token. Within each bin the upper violin is negative-label instances and the lower one positive-label. Two things to read. First, the divergences are not zero: attention distributions from models that were trained identically apart from the seed disagree with each other, spreading to about 0.17 in the most diffuse bin. Second, almost all instances sit in the two lowest bins, meaning peak attention is rarely above 0.5 in this task at all. So “the attention map” is not one fixed object to be read off; it is a quantity with run-to-run variance, and the variance is largest exactly where attention is spread out.

Three caveats constrain the interpretation of any attention-based routing map. The attention-as-explanation debate established that attention weights in classifiers correlate poorly with gradient-based importance and can be replaced by adversarial distributions that leave predictions unchanged [80], while the reply by Wiegreffe and Pinter showed that attention can still be explanatory when tested against model-wide alternatives such as uniform or frozen attention [81]; the practical consequence is that routing claims need intervention, and uniform-attention and frozen-attention baselines are useful controls — uniform attention meaning the head is forced to read every position equally, frozen attention meaning it is made to reuse a fixed pattern instead of computing a new one, so that a claimed effect can be checked against a model that is not free to choose where to look. Copy suppression shows that a head may attend to a token in order to lower its probability [79], so the sign of a head's effect must be established by ablation or direct logit attribution rather than inferred from attention mass. That is to say, one has to switch the head off, or add up its contribution to the final word scores, and see which way the answer moves; the size of its attention weight does not say whether it helps or hinders. Finally, attention sinks and massive activations mean that the first token and delimiter tokens receive attention that carries no information [75, 76, 77, 78], and the anchor hypothesis itself implies that a requirement may be read through its delimiter rather than through its content words; $A_{t,i}$ should therefore be reported both with and without delimiter positions. Put simply, the two versions can disagree, and which one to trust is itself something the experiment has to settle.

5. The KV Cache as Instruction Memory

During autoregressive generation the requirement tokens are never recomputed; their only trace is the set of keys $K_j^{(l,h)}$ and values $V_j^{(l,h)}$ stored in the KV cache at prefill. That is to say, the model works through the prompt once, writes down two vectors per position per head, and from then on every word it writes consults those stored vectors instead of re-reading the prompt. The proposal treats these entries as the physical carrier of each requirement and plans to patch, replace and block them. This section reviews what is known about the stability of the prompt positions that generation consults, the growing body of work that edits cached keys and values to change behavior, and the distinction between key-side and value-side effects that a KV-level analysis makes possible.

5.1 The prompt positions that generation consults are stable

Figure — SnapKV: Why a workflow can reuse a cache at all. The user message on the left contains several unrelated requests, and the two the answer actually needs — analysing the Q4 report, and the R&D expense question — are shaded green. The bar in the middle is the prompt laid out left to right, once per attention head. The orange blocks mark the positions each head actually draws on: they are not spread evenly over the prompt but fall into a few clusters, and the clusters land in much the same places across heads. The green block at the right-hand end is the window of positions nearest the end of the prompt. The method’s working assumption, and the reason it succeeds, is that this tail window predicts the clusters — what the tail attends to is what generation will keep consulting. Stability of that kind is the precondition for reusing a prefix instead of recomputing it.

Evidence from KV-cache compression shows that the routing pattern the proposal wants to measure is a stable property rather than step-to-step noise. SnapKV observes that the set of prompt positions each head attends to is consistent across decoding steps and can be predicted from an observation window at the end of the prompt, so that selecting cache entries by attention votes preserves accuracy [88]; concretely, the positions a head will need later can be picked out before generation starts, and discarding the rest leaves accuracy essentially unchanged. H2O finds that a small set of heavy-hitter tokens — the few positions that keep receiving attention no matter what is being written — accumulates most attention mass throughout generation and that evicting all but heavy hitters and recent tokens preserves quality [89]. Both results imply that if a requirement is consulted at all, it will be consulted repeatedly by the same heads, which makes $A_{t,i}$ averaged over a generation phase a meaningful summary; both also provide an ablation of a different kind, since evicting a requirement's entries from the cache is a direct test of whether they are needed. Put simply, delete that requirement's stored keys and values and see whether the behavior it controls disappears. Attention-sink work adds the constraint that sink entries must be preserved in any cache manipulation, because their removal degrades generation for reasons unrelated to the requirement under study [75, 76].

5.2 Editing cached keys and values changes behavior

A cluster of recent methods demonstrates that the cached entries of prompt tokens are a sufficient locus of intervention. KV Cache Steering constructs steering vectors from contrastive prompts with and without reasoning traces and applies a one-shot additive edit to the cached keys and values at target positions after prefill — the stored vectors are adjusted once, before any word is written, and are then left alone; the edit induces chain-of-thought style reasoning in small models on GSM8K and other benchmarks with better stability and lower latency than activation steering [90]. V-Steer identifies attention heads in which lower-priority spans dominate privileged ones and applies in-place multiplicative edits to the cached value tensors to boost privileged spans and suppress conflicting ones, raising primary-requirement accuracy from below 18% to 92% on controlled benchmarks with only prefill overhead [83]; that is to say, on prompts where a lower-priority instruction contradicts a privileged one, the primary requirement is met in fewer than one case in five before the edit and in more than nine out of ten after it. Memory Inception encodes reminder text into latent KV banks — stored key-value pairs that correspond to no visible words in the prompt — and inserts them only at the layers and heads that an automated selector finds actually route to the reminder, achieving mid-conversation behavior updates without visible prompt changes [91]; its selector is, in effect, a procedure for locating the layers and heads that read an instruction. LLMSteer processes the shared context under several prefix prompts, identifies tokens that consistently receive high attention, and up-weights them in a way compatible with prefix caching, improving GSM8K among other tasks [92]. Gist tokens show, from the training side, that an entire instruction can be compressed into a few key-value activations that downstream attention reads with little loss [54]. Together these results settle the feasibility of the proposal's KV arm: editing the KV entries of an instruction span is an established, low-overhead intervention, and the entries of a span are sufficient to carry its behavioral effect.

Figure 1 — V-Steer: What is shown: the problem this method is built for. A user message tries to override the system message (ignore system, do the harmful request), so a privileged instruction and a lower-priority one are in direct conflict. The method first attributes the outcome to specific components, then edits the cached value tensors of the offending span at inference time, and the model returns to the safe answer. Why it is here: it is one of only two published results that intervene on the KV cache itself rather than on hidden states — but it scales whole priority spans, and never replaces one requirement with a counterfactual version of that same requirement. In other words, the edit is made to the stored numbers rather than to the text, so the visible prompt is untouched and the change is invisible from the outside. Source: V-Steer [83].

5.3 Keys versus values: what a KV-level analysis can separate

Figure — V-Steer span steering: What editing the value side buys you, on a prompt with two instructions that contradict each other. The system message asks for highly technical scientific terminology and is shaded green, the span to be boosted; the user message asks to explain gravity as if speaking to a 5-year-old and is shaded red, the span to be suppressed. Boosting and suppressing here means scaling those spans’ value vectors, leaving the attention weights and the softmax untouched. Before, the five most likely next tokens are ‘Oh’, ‘O’, ‘K’, ‘Gravity’ and ‘Let’, spread across a 0.22 to 0.12 range, and the model writes Oh boy, are you ready for a cool secret about the whole universe? — it is obeying the user span. After, ‘The’ alone takes 0.52 and the continuation is the phenomenon of gravity, a fundamental force of nature, is a manifestation of the curvature of spacetime. This is the separation the section is about: which span is read is decided on the query–key side, and how strongly its content lands is decided on the value side, and only the second was touched here.

Working at the level of $K_j$ and $V_j$ rather than of whole hidden states allows the proposal to separate two mechanisms that hidden-state patching conflates. The keys of a requirement determine whether later queries select it (the query-key pathway), while its values determine what is written into the residual stream once selected (the output-value pathway). Function-vector and learned-task-vector analyses find that task signals act primarily through the output-value circuits of a few heads [43, 60], which suggests that value-side patching of a requirement's entries may transfer its effect even when keys are unchanged, whereas key-side patching changes which requirement is consulted without changing what is read. Two methodological results make this decomposition tractable. Attribution graphs built from cross-layer transcoders — a replacement for the model's internal computation that re-expresses it in interpretable parts, so that the path from input to output can be drawn as a graph — freeze attention patterns and therefore attribute effects along value pathways only; in other words, the drawn graph shows what was read but not the decision about where to read from; the authors state explicitly that query-key-mediated effects are omitted and must be studied separately [101], which is precisely the gap that the proposal's attention-suppression experiments would fill. AtP*, an attention-aware refinement of attribution patching, addresses the complementary problem in gradient-based attribution: attribution patching estimates the effect of an edit from a derivative instead of actually running the edited model, which is far cheaper but wrong wherever the derivative is flat; attention-softmax saturation is exactly such a place, so the estimate misses effects that flow through attention to particular keys, and the query–key fix (QK-fix) recomputes the softmax exactly for the patched keys [121]. A KV-level design that reports key-side and value-side interventions separately, with the sink entries preserved, would be the first to characterize an instruction's effect along both pathways. Put simply, editing the keys asks whether the requirement still gets selected, and editing the values asks what the requirement contributes once it has been.

6. Generation-Stage Dynamics: Reasoning and the Final Answer

The proposal predicts that the reasoning phase and the final-answer phase consult different requirements. Evaluating that prediction requires an account of what each phase computes and where its information comes from. This section reviews mechanistic studies of chain-of-thought and arithmetic, evidence on when the final answer is determined and how answer tokens depend on reasoning tokens, the faithfulness problem that complicates the mapping from visible reasoning to internal computation, and the interventions that control reasoning length and style. It concludes with the phase-dependent routing signature these results predict.

6.1 What chain-of-thought tokens compute and read

Figure — Towards Understanding Chain-of-Thought Prompting: What in a worked example is actually doing the work. Each row is a different demonstration given to the model; the right-hand column is what the model then does on a new, unseen question. Standard shows only the final answer in the demonstration, and the model gets the new question wrong (18). CoT shows correct step-by-step working, and the model works step by step too and gets it right (42). Invalid Reasoning is the interesting row: the demonstration’s arithmetic is deliberately broken — the orange text contains steps that do not follow — yet it keeps the same relevant quantities and the same order of operations. The model still reasons step by step on the new question and still reaches 42. The bar chart underneath scores all three: standard sits near 16, valid chain of thought near 48, and invalid reasoning near 40, recovering most of the benefit. So what the demonstration supplies is largely the shape of the procedure, not the correctness of its content.

Behavioral ablations first established that the content of a chain-of-thought prompt matters less than its structure: invalid rationales retain 80–90% of chain-of-thought performance provided they are relevant to the query and follow the correct ordering of steps [102]; that is to say, demonstrations whose reasoning steps are deliberately invalid still deliver most of the benefit, so what the examples supply is largely the shape of the answer rather than its content, and corrupting the symbols in a demonstration barely hurts while removing its patterns or text does [103]. These findings imply that a reasoning requirement primarily supplies a format or scaffold, which is consistent with the shallow-mechanism null hypothesis of Section 3.1. Mechanistic studies then identified the attention circuits that implement stepwise computation. In controlled settings, chain-of-thought ability emerges as an iteration head, in which one head attends from the current reasoning token to the next input token to be processed while another retrieves the previous reasoning state, so that generated tokens implement a loop over the prompt [93]; concretely, how far through the prompt the model has got is tracked by the text it has already written. Dutta et al. applied activation patching, attention analysis and probing to Llama-2-7B on fictional-ontology reasoning and found a functional rift around the middle layers — a depth above which the computation is doing a visibly different job from below it — parallel answer-generation pathways, and heads that move information from the in-context examples and the question into the generated reasoning tokens [94]. Arithmetic has been traced in more detail. At the last token, early-layer heads carry the operands and the operator across from the question, and the feed-forward layers from the middle onward are where the result is actually computed [95]. Circuit discovery on Llama-3 adds a caution: what does the computing is a sparse set of feed-forward neurons running simple heuristics, not one clean algorithm [96]. Each GSM8K reasoning step therefore has a known division of labor: attention transports operands from prompt or previously generated positions, feed-forward layers compute — a feed-forward layer being the part of each block that processes one position on its own, with no access to any other position. In other words, attention decides which numbers meet, and the feed-forward layers decide what happens to them when they do; the reasoning requirement can influence this only by shaping which positions are transported and whether the step is verbalized at all.

6.2 When the answer is fixed and what answer tokens read

Several results indicate that the final answer is often determined well before the final line is emitted. Answer-convergence analysis finds that on mathematical tasks models settle on their final answer after roughly 60% of the reasoning steps, so that stopping on answer stability saves more than 40% of tokens with little loss [97]; that is to say, the last two fifths of the written working usually does not change the answer it arrives at; probes on hidden states predict chain-of-thought success with 60–76% accuracy before any reasoning token is generated [98] — better than the 50% of a coin flip, and read off the model's state at the moment it has finished reading the problem; and overthinking analyses show that o1-style models spend most of their tokens after the first correct solution [113]. If the answer content is fixed early, the tokens of the final line have little computation left to do beyond formatting, which predicts that their attention should be dominated by the format and unit requirements rather than by the question. Two attention studies of distilled reasoning models describe the answer phase directly. Zhang et al. find that answer tokens attend substantially to reasoning tokens rather than only to the question, identify reasoning-focus heads in middle layers that track the progression of the reasoning, and show by activation patching that perturbing key reasoning-token activations reliably changes the final answer [110]. Thought Anchors identify receiver heads, whose attention over prior sentences has high kurtosis, and show that they concentrate on planning and backtracking sentences; suppressing attention to an anchor sentence changes downstream logits, and resampling the sentence — generating that one sentence again many times and letting each version continue — changes the final-answer distribution [109]. Together these studies give the proposal the two head classes most likely to carry the phase-dependent routing it hypothesizes: reasoning-focus heads during the trace, and receiver-style heads at the transition to the answer.

Figure 1 — Thought Anchors: What is shown: three different ways to ask which sentence inside a reasoning trace actually mattered, applied to one arithmetic problem. A labels the sentences of the trace by what they do. B lists the three methods: delete a sentence and resample the whole continuation a hundred times; find the heads that attend back to that sentence; or mask all attention to it and watch the later sentences change. C is the resulting dependency graph between sentences. Why it is here: this is the methodology the proposal borrows — but it is applied to sentences the model generated, not to the requirements it was given. That is to say, all three methods ask the same question in different ways, and C shows the sentence-to-sentence dependencies that come out of it. Source: Bogdan et al. [109].

Figure 4 — Thought Anchors: Left: every attention head in layer 36, plotted by how much attention it sends back to each earlier sentence. One head (dark blue) spikes hard on a handful of specific sentences while the rest stay flat. Right: a histogram over all heads of how peaked that curve is. Most heads are unremarkable; a thin tail is extremely peaked. What to read off it: a small number of heads specialise in reading back a particular earlier piece of text rather than attending broadly. That is exactly the shape the proposal predicts for a head dedicated to one prompt requirement, which is why this figure is the template for the head-ranking step. Concretely, the peakedness plotted on the right is the number used to pick heads: a high value means the head reads a few places, a low value means it reads everywhere. Source: Bogdan et al. [109].

6.3 Faithfulness: visible reasoning is not always the computation

Figure — Reasoning-focus heads: The same conclusion read two ways. In both panels the model has just written C is 0, boxed in red, and the shading shows which earlier tokens that conclusion is attending to. Top, averaged over all heads: the weight lands on nearby words — and, C, integers, the immediately preceding phrase. Read this way the conclusion looks like it came from its own sentence. Bottom, one head, L16.H2: the same conclusion instead reaches back to an earlier sentence, underlined in red — but 0r^3 is just 0, so we can ignore that — which is the step that actually licenses C being zero. Averaging over heads washed that dependency out. This is the practical form the faithfulness problem takes: the visible text and the head-averaged attention both tell one story, and a particular head tells a different and more accurate one.

The mapping from a reasoning requirement to the behavior produces explicit reasoning is complicated by the fact that the produced reasoning need not be the computation that yields the answer. Arcuschin et al. document, without adversarial prompting, implicit post-hoc rationalisation, silent restoration of intermediate errors, and unfaithful shortcuts in frontier and open reasoning models [99]. Attribution-graph analysis of Claude 3.5 Haiku separates faithful cases, cases in which the answer is produced without grounded computation, and backward-chaining cases in which the model works from a user-supplied hint; in the latter the answer token's computation reads from the hint rather than from the reasoning steps — that is to say, the answer was taken from the hint and the visible steps were written afterwards to lead to it — and the same analysis shows that the model's verbal account of addition (column arithmetic) does not match its internal parallel heuristics [100]. For the proposal, faithfulness is both a confound and an opportunity. It is a confound because the presence of reasoning text does not guarantee that the reasoning requirement changed the computation. It is an opportunity because the proposal's causal design can distinguish the cases: if suppressing the reasoning requirement changes the final answer only when the answer tokens depend on the reasoning tokens (in the sense of Zhang et al. [110]), then the requirement is doing computational work; if it changes only the presence of reasoning text, the requirement is a stylistic control.

6.4 Controlling reasoning length and style from inside the model

Figure — ThinkEdit: Reasoning length treated as a direction, then removed by surgery. Step 1 taps the residual stream at two points inside a decoder block — just after attention and just after the multi-layer perceptron — and at each computes the average representation when the model reasons at length minus the average when it reasons briefly. The difference is the reasoning-length direction for that layer. Step 2 does not add that direction at inference. It edits the weights: for a selected attention head, the output projection is replaced by a version with the short-reasoning direction projected out, so the head can no longer write along it. The intervention is therefore permanent and costs nothing at generation time, which distinguishes it from the steering-vector methods in section 3.

Interventions that alter reasoning behavior without changing the prompt identify the internal correlates a reasoning requirement must act upon. Venhoff et al. isolate backtracking, uncertainty expression, example testing and verification as separate directions in the activation space of DeepSeek-R1-distilled models, locate causally relevant layers by attribution patching, and steer each behavior with its vector over 500 tasks [104]. ThinkEdit finds that reasoning length is governed by a linear direction in middle-layer residual streams, that a small fraction of attention heads (about 2–4% depending on the model) project strongly onto the short-reasoning side, and that editing only those heads' output projections lengthens chains and improves mathematical accuracy [105]; concretely, 2–4% means only a small minority of the model's attention heads push toward stopping early, and changing just those makes the model reason for longer. Steering vectors trained by reinforcement learning recover much of full fine-tuning's reasoning gain and act, in the last layers, as token-substitution biases that favor structural tokens such as Step at early positions and process words and symbols later [106]. Post-training creates new attention heads in middle-to-late layers that persistently build reasoning pathways, and think-on/off models recruit broader, less efficient head sets when thinking is disabled [108]. On the format side, the only component-level analysis of a specific output requirement is that of Rocchetti and Ferrara, who use cumulative weighted attribution — each component's contribution to the output, weighted and then summed layer by layer so that one can see where the ability accumulates — to show that instruction tuning improves exact word-count control mainly by specializing deeper-layer components, with late attention heads contributing positively in English and final-layer feed-forward blocks compensating in Italian [107]. The consistent picture is that reasoning-style control is carried by a modest number of identifiable middle-layer heads and directions, whereas format control is enforced late; a requirement of each type should therefore be expected to engage different layers, which is testable with the per-layer knockout sweeps reviewed in Section 7, knockout meaning that one connection is switched off at one layer at a time and the effect on the output is recorded.

6.5 The predicted phase-dependent signature

Figure — Resampling versus forcing: Two phases, visible as a gap between two curves. Both panels walk along the sentences of one reasoning trace, left to right. The solid line resamples: cut the trace after sentence i, let the model carry on normally a hundred times, and record how often it ends up correct. The dashed line forces: cut at the same place and demand an answer immediately. Panel A is a problem the model gets right. The solid line is already near 0.8 from the very first sentence, while the dashed line sits flat at zero until about sentence 43 — for the whole first half, the answer cannot be extracted by asking, even though the trajectory is already on course. Panel B is a problem it gets wrong, and the same split appears. The gap between the two lines is the phase boundary: before it, the reasoning is doing work whose result is not yet readable at the answer position; after it, the answer is committed. Which measurement you take decides which phase you can see, so a design that only forces answers will miss everything to the left of that boundary.

Combining the results above yields a concrete prediction that the proposal can test. During the reasoning phase, tokens should route to the reasoning requirement through middle-layer heads of the reasoning-focus or iteration type, and this routing should be strongest at step boundaries, where the decision to verbalize a step is made; attention to the format and unit requirements should decline as the trace lengthens, in line with the requirement-attention decay reported by Li et al. [17]. At the transition to the final line, receiver-style heads should read from the reasoning trace to fetch the converged answer [109, 110], while late-layer heads should re-engage the format and unit requirements, whose enforcement the length-control analysis places in deep layers [107]. A model that violates the unit requirement on the last line should therefore show, relative to a compliant generation on the same problem, lower late-layer $A_{t,\text{no-unit}}$ at the final tokens, which is a directly measurable contrast. In other words, the two generations are compared, and the prediction is that the failing one spent less of its deep-layer reading on the no-unit requirement, the requirement that the last line carry no unit, at exactly the moment the last line was written; the causal arm can then test whether restoring that attention by Spotlight-style amplification [66] repairs the violation.

7. Causal Intervention Methodology

The third research question requires showing that the routes identified in Sections 4 through 6 are causally responsible for the behaviors of Section 2. The proposal lists four intervention types: suppressing attention from generated tokens to a requirement's span, patching or replacing the requirement's hidden states or KV entries, removing or rewriting the requirement, and blocking its information flow at particular layers and heads. Each has an established methodological lineage, and each has known pitfalls. This section reviews the patching family and its design choices, attention-specific interventions, interchange interventions and causal abstraction, prompt-level attribution for whole generations, and the evaluation standards that determine whether a causal claim survives scrutiny.

7.1 The patching family and its design choices

Figure — Causal Tracing (ROME): The patching template, with its result. (a) The clean run takes The Big Bang Theory premieres on and produces the right answer, CBS. (b) The corrupted run damages the subject tokens at the input (the starred ones) so the answer is lost. (c) One state at a time is copied from the clean run into the corrupted run, and (d) the experiment records whether the right answer comes back. The heatmaps below are that record: rows are token positions, columns are layers, and colour is the probability of the correct answer being restored by patching there. Two bright regions appear rather than one — an early site sitting on the last subject token in the early-to-middle layers, and a late site sitting on the final token in the deep layers. Splitting the same map by component shows the division of labour: the early site belongs to the MLPs (green), the late site to attention (red). The bottom row repeats the whole thing averaged over a thousand prompts, so the pattern is not an artefact of this one sentence.

All patching methods follow one template, set by causal mediation analysis: change the input, hold one internal component at the value it would have had under the changed input, and split the total effect into what flows through that component and what flows around it 114. That is to say, the model is run twice on two inputs that differ in one thing, one internal component is forced to take the value it had in the second run, and the question is how much of the change in the output that single substitution accounts for. Causal tracing adapted it to multi-token spans 115, and an instruction span can be treated the same way. The methodology literature contributes the warning that four design choices inside that template can each flip the answer on their own. The table below lists those four: the choice, the options available for it, and what the methodology literature recommends.

7.2 Attention-specific interventions

The proposal's first intervention, suppressing attention from later tokens to a requirement span, has a canonical precedent in attention knockout: Geva et al. block attention from a target position to a chosen span of source positions over a window of layers by zeroing those attention weights, sweep the window, and report per-layer curves that reveal a three-step recall mechanism [125]; that is to say, the block is applied at one depth at a time and the resulting curve says at which depths the connection was load-bearing. Wang et al. applied the same blocking to label-word anchors in shallow versus deep layers and showed that the two produce different effects [70], and Thought Anchors applied attention suppression to sentences inside a generated reasoning trace, measuring the Kullback–Leibler divergence of downstream logits [109]. A known complication is that knockout renormalizes the remaining attention, so its effect mixes removal of information with redistribution of attention — put simply, the shares must still add up to one, so weight taken away from the requirement is handed to everything else, and the observed change could be caused by either; reporting both zero-masking and resample-based key replacement helps separate the two. The inverse manipulation, amplification, is provided by PASTA, InstABoost and Spotlight [64, 65, 66], and using both directions yields a dose–response curve rather than a single ablation point. Competition-of-mechanisms work modifies the attention of specific late heads to specific positions to flip a model between factual recall and in-context copying [142], which is structurally the same manipulation as flipping between a default output format and an instructed one; its reproduction study found that head specialization largely disappears in Llama-3.1-8B and that ablation effects depend on prompt structure and domain [143], a warning that the proposal should run at least two model families.

Figure 6 — Thought Anchors: What is shown: one problem worked end to end. Right: the model’s reasoning broken into labelled chunks — it reaches 20 bits, backtracks, recomputes in decimal, notices the discrepancy, and settles on 19. Top left: a matrix of how much each sentence causally depends on each earlier one, measured by suppressing attention to the earlier sentence and watching the later one move. Bottom left: which sentences receive concentrated attention from the specialised heads. Why it is here: it is the full protocol the causal arm of the proposal would run, only with prompt requirements in place of reasoning sentences. That is to say, the matrix at the top left is built by intervening rather than by reading attention off, so each cell is a measured effect and not a correlation. Source: Bogdan et al. [109].

7.3 Interchange interventions, causal abstraction and cross-prompt patching

Figure — Interchange success: Where an interchange intervention actually works. The claim being tested is that a high-level symbolic variable is realised in the network at some particular place. To test it, the value of that place is swapped in from a second input and the model is asked to behave as the symbolic account predicts. The grid records how often that succeeded: rows are layers, columns are token positions in the sentence pair — [CLS], the adjective and noun of the premise object, the separator, the same two for the hypothesis, and the final separator. Success is concentrated in the lowest one or two layers at the object-noun positions, darkest at the hypothesis noun, and falls away almost entirely above about layer two, with [CLS] holding some residue higher up. An alignment claim is therefore not a claim about the model as a whole. It is a claim about a specific layer and a specific position, and the same claim tested one layer up can simply fail.

The proposal's second intervention, patching or replacing a requirement's hidden states, is an interchange intervention in the sense of causal abstraction: a high-level causal model is an abstraction of the network if swapping in a representation from a counterfactual input has the same effect as intervening on the corresponding high-level variable [127]. In other words, one writes down a small hand-made diagram of what the model is supposed to be doing, and the diagram earns the right to be called a description of the network only when editing the network at the matching place produces exactly what editing the diagram would predict. Distributed alignment search learns the subspace on which such interventions succeed, and its Boundless variant (Boundless DAS) applied it to Alpaca-7B on a numeric instruction task, finding that the instruction-tuned model implements an interpretable boolean causal model in identifiable residual subspaces robust to instruction wording [128]; this is the closest precedent for hypothesizing a small causal model of requirement compliance (reasoning required, bare number required, units forbidden) and testing it by interchange interventions on requirement-span representations. Patchscopes unifies logit lens, causal tracing and cross-prompt patching as one operation, patching a representation from a source prompt, layer and position into a target and decoding it in natural language — concretely, the internal state in question is dropped into a second prompt that asks the model to describe it, and the model's answer is read as a report on what that state held — which provides a way to ask what a model believes a requirement says at a given layer [129]. Task- and function-vector patching [42, 43] supplies a further condition, replacing the requirement text with its vector, that tests whether the requirement is needed as tokens at all. Causal scrubbing offers a whole-hypothesis test: resample every activation the hypothesis deems irrelevant from other prompts sharing the same requirement and report the fraction of behavior preserved [126]. That is to say, everything the account claims does not matter is replaced with material from elsewhere; if the account is right the behavior survives, and how much of it survives is the score. Sparse feature circuits move the same logic to sparse-autoencoder features, enabling ablation of individual requirement-related features during generation [137].

7.4 Attributing whole generations to prompt segments

Figure — ContextCite: Attribution at the granularity of a statement. The context is a document about the 2024 solar eclipse and the query asks where to go to see it from Boston. The model answers in several sentences; one of them — you can travel to Maine, which is on the path of totality — is picked out and highlighted. ContextCite then points back into the source document and marks the specific sentence responsible: the path arches from Mexico to Texas to Maine. Note what is being attributed and to what. The unit on the output side is a chosen statement rather than a single token, and the unit on the input side is a span of the context rather than a layer or a head. That is the granularity a workflow needs when the question is which part of a long prompt produced a particular claim.

Because the proposal's outcomes are properties of complete generations, it needs attribution methods that operate at that granularity. ContextCite ablates random subsets of context sentences, records the log-probability of a fixed response or span, and fits a sparse linear surrogate whose weights attribute the response to each source [130] — that is to say, it learns a simple formula predicting how likely the response is from which sentences were present, and the size of each term in that formula is that sentence's credit; applied to the requirement block, it scores each requirement's influence on, for instance, the last line, at the cost of an additivity assumption that may be violated when requirements interact. Put simply, the method assumes each requirement's contribution simply adds up, which is exactly what stops being true when two requirements pull against each other. AttriBoT makes exact leave-one-out attribution — rerun the model with one requirement removed, once per requirement, and take the difference — tractable with cached activations and hierarchical attribution [132], and Learning to Attribute with Attention trains a per-head weighting so that attention predicts ablation-based attributions, which identifies the heads whose attention is causally informative and is therefore a principled way to select heads for knockout [131]. Inseq provides step-wise attribution of each generated token to source and previously generated tokens with aggregation over spans [133], and contrastive explanations attribute the difference between two candidate tokens, such as a bare number versus a number followed by a unit, rather than the raw probability [134]; AttnLRP propagates relevance through attention and normalization layers for token-level attributions in large models [135]. Any rewriting of a requirement must be compared against a distribution of surface-form variants, since format changes alone swing accuracy by tens of points [23].

7.5 Standards for a causal claim

Figure — IOI circuit and interventions: What a fully worked causal claim looks like. The prompt is Rudi and Emma … Emma says … to, and the answer should be the name mentioned once. The diagram names every part that matters and how they connect: the S-Inhibition heads read the repeated name, the Name Mover heads carry the remaining name to the output, and the residual stream runs up the right-hand side. Two injection sites are marked, v_MLP into the perceptron path and v_resid straight into the residual stream — naming where an intervention enters, not just that one happened. The boxes at the top are the outcome, in counts out of a hundred: the baseline model answers Rudi 96 times, and after the patch it answers Emma 92 times. A claim reaches this standard when the components are identified, the intervention site is stated, and the effect is a near-complete swap rather than a shift in an average.

Several results define what the proposal must report for a routing claim to be credible. Makelov et al. demonstrate an interpretability illusion in which patching along a learned subspace produces the target behavior by activating a dormant parallel pathway that the model does not use on clean inputs, and propose diagnostics that check whether the patched direction is actually used [138]; that is to say, the intervention works and still explains nothing, because it switched on machinery the model never reaches for on its own; any learned-subspace intervention on requirement representations needs this check. Miller et al. show that circuit faithfulness scores — how much of the model's behavior a proposed set of components reproduces on its own — change substantially with the ablation type, direction and metric, to the point that the best circuit changes, and recommend reporting multiple settings [139]. Shi et al. formalize mechanism preservation, localization and minimality as statistical hypothesis tests with error control and find that few published circuits pass all of them [140]; concretely, preservation asks whether the proposed components alone still produce the behavior, localization whether everything outside them can be disturbed without effect, and minimality whether any of them can be dropped. For interventions on open-ended generation, Arditi et al. provide the exemplar of scoring directional ablation by behavioral metrics of whole generations alongside capability benchmarks [55], and Pres et al. propose criteria for steering evaluations: deployment-like contexts, prompting baselines, likelihood shifts of both target and non-target behaviors, and effect sizes with uncertainty [141]. The reproduction study of Dotsinski et al. adds that head-level findings can be model-specific [143]. The table below maps each of the proposal's planned interventions to its methodological precedent and the pitfall that precedent identified. Each row is one planned experiment: the left column says what would be done to the model, the middle column names the published work that has already done something of that shape, and the right column names the mistake that work showed is easy to make.

7.6 Tooling for interventions during generation

Figure — pyvene: What an intervention looks like as code. The top strip is the story Tinystories-33M writes unprompted. The middle diagram shows the intervention: the embedding of the single word happy or sad, scaled down, is added into the residual stream at the output of every layer’s multi-layer perceptron, at every generated position — those are the red arrows into the summation nodes. The code below is the whole specification. An intervention is declared as a list of dictionaries naming a layer, a component (here mlp_output) and an intervention_type (here addition), the model is wrapped once, and generation proceeds normally with the source representation passed in. The two strips at the bottom are the result on the same opening sentence: with happy added, Lucy is happy and thanks the man; with sad added, the same setup turns and Lucy is scared. The point for a workflow is that the intervention is configuration rather than a patched forward pass.

Two libraries support interventions during multi-token generation with Hugging Face models. pyvene specifies interventions declaratively by layer, token positions, subspace and type, including interchange, zero, noise and trainable interventions, and implements DAS and Boundless DAS [144]. NNsight defines intervention graphs over any PyTorch model, including inside .generate() loops, and NDIF runs them remotely on large models [145]; this is the more natural fit for KV-level and hidden-state patching over full GSM8K solutions. Gemma Scope sparse autoencoders provide off-the-shelf feature bases for every layer of an open model, should the proposal pursue feature-level decomposition of requirement signals. In other words, both libraries allow a value inside the model to be changed midway through writing an answer, which ordinary inference code does not permit.

8. Synthesis, Research Gaps and Recommendations

This section consolidates the preceding review into an evidence map for the three research questions, identifies the gaps the proposal is positioned to fill, states the hypotheses the literature supports, and lists methodological recommendations. The organizing claim is that every component of the proposed causal chain $I_i \rightarrow \{K,V\} \rightarrow \text{heads} \rightarrow B_j$ has been demonstrated in isolation, but no study has traced several heterogeneous natural-language requirements through that chain simultaneously on a reasoning task, with generation-level outcomes and with attention to how the routing shifts between the reasoning and answer phases. That is to say, the chain runs from the words of one requirement, to the keys and values the model stored for those words, to the heads that read them back, to the behavior that finally appears in the output; each link has been demonstrated on its own, and the missing work is following all four at once for several requirements in the same prompt.

8.1 Evidence map for the three research questions

Figure — Composition types: The structure any evidence map has to cover, because a requirement rarely arrives alone. Five ways requirements combine, each with a description, a worked instruction and a graph. Single is one requirement. And is several that must hold at the same time: summarise this news, in bullet points, within 100 words — the graph is a triangle because none of the three depends on the others. Chain is several tasks in a fixed order, each with its own requirements, drawn as a path. Selection is conditional: if the work contains an animal, describe it in English, otherwise in Chinese — a branch node with two children, and only one branch is in force for any given input. Nested Structure is these composed recursively, giving a tree several levels deep. The evidence in this review is organised against this shape: results about a single requirement do not automatically transfer to conjunctions, and results about conjunctions say nothing about the conditional case, where the question of which requirement governs a token has a different answer depending on the input.

The table lists, for each level of the proposal, what the literature has established, the closest existing work, and what remains open. Each row is one of the three levels defined in Section 1, read from left to right as settled, cited, and still missing.

8.2 Gaps the proposal is positioned to fill

Figure — Instruction steering: A requirement satisfied with no text stating it. The task is to produce output in a required format, and the text instruction asking for that format has been removed from the prompt entirely. Solid bars are ordinary inference under those conditions; hatched bars add a steering vector for the format requirement and change nothing else. Across four instruction-tuned models — Phi-3, Gemma 2 2B, Mistral 7B and Gemma 2 9B — ordinary inference lands between roughly 0.07 and 0.13, near the floor you would expect when the model was never told what to do. Steering lifts every one of them to roughly 0.29 to 0.32. The vector is carrying the requirement that the missing sentence would have carried, which is the premise the proposal in this section rests on.

Five gaps recur across the sections above. Each corresponds to a component of the proposal that no existing study has delivered.

8.3 Hypotheses supported by the literature

Figure — The causal inner product: The geometric hypothesis this section rests on. On the left sphere, each concept appears twice. In blue is the concept read off the output side, for example male⇒female as γ(“queen”) − γ(“king”), and English⇒French as γ(“roi”) − γ(“king”). In red is the same concept read off the context side, as λ(“She is the”) − λ(“He is the”) and λ(“Il est le”) − λ(“He is the”). Under the ordinary geometry the blue and red arrows for one concept point in different directions, and the two concepts sit at an arbitrary angle. On the right sphere, after changing to the inner product the paper derives, the two readings of each concept coincide, and the two concepts meet at a right angle. The hypothesis is that a requirement has a single direction and that separable requirements are orthogonal — but only once you are measuring in the right coordinate system, which is not the raw activation space.

The evidence reviewed licenses the following hypotheses, stated so that each has a clear falsifying observation. Each row gives the hypothesis, the evidence that makes it worth testing, and the specific result that would show it to be wrong.

8.4 A recommended experimental pipeline

Figure — Thought Anchors: Three ways of asking the same question, and what they produce together. Panel A takes one problem and its reasoning trace, and labels each sentence by what it is doing: sentence 8 is Active Computation, 13 is Plan Generation, 47 is Uncertainty Management. Panel B lists the three methods applied to those sentences. Black-box resampling deletes a sentence and re-runs the whole rollout a hundred times, rather than deleting it and forcing an answer, so the measured effect includes everything downstream. Receiver heads looks for attention heads that read a given sentence, with vertical stripes in the attention matrix marking sentences that broadcast widely. Attention suppression masks all attention to sentence i and measures the effect on sentence j. Panel C is the result: each sentence is a node whose size is its importance, and the dashed edges are sentence-to-sentence influence. Sentence 13, the plan, is the largest. Three methods with different failure modes agreeing on the same sentences is what makes the attribution credible.

The methodological literature suggests the following order of operations, in which cheap descriptive measurements narrow the search before expensive causal tests are run, and in which every causal test is scored on the full behavior vector.

8.5 Closing assessment

Figure — Scoring with dependencies: What per-requirement scoring already looks like when it is done properly, and the complication it exposes. Top left is one instruction: describe a painting in under 100 words, then branch on whether the painting contains an animal. Below it that instruction becomes ten scoring questions, each tagged with the composition type it belongs to and the dimension it tests — chain, selection, length, factuality, helpfulness. Top right is the part that matters most. The ten questions are not independent: question 4 depends on 1, questions 5 and 6 depend on 1 and 4, questions 9 and 10 depend on 1, 4, 5 and 7. The bottom strip is the machinery: each question is routed to a rule evaluator when it can be checked programmatically and to a model judge otherwise, giving a 0 or 1 per question, and those are then combined by dependency aggregation — a question’s score is conjoined with all of its prerequisites. Fail question 1 and questions 5, 7, 9 and 10 are zero whatever the text actually did. Every tool this review needs exists, as this figure shows; what is missing is any account of why a particular requirement failed, which a score of zero cannot supply.

The literature has matured to the point where every tool the proposal needs exists and has been validated on a related problem: verifiable per-requirement scoring, contrastive extraction of per-requirement vectors, anchor-token and receiver-head analysis, KV-level editing, position-aware circuit discovery, and generation-level evaluation of interventions. What is missing is their combination on a single, well-controlled multi-requirement reasoning task, with the routing of each requirement followed across the phases of generation and confirmed by segment-specific intervention. The behavioral facts that motivate the question, in particular the count-dependent decay of compliance and the trade-off between reasoning and format requirements, are robust across models and benchmarks, and the most plausible mechanistic account of them, competition for attention share among instruction readers, is testable with the methods reviewed here. The proposal's expected contribution is therefore not a new tool but the first mechanistic account of how a language model decomposes, stores, consults and executes several prompt requirements at once, and of where in that chain the observed failures arise. Put simply, the question is no longer whether a model obeys, but which part of it stopped reading which requirement, and when.

9. References

The list is grouped by theme; numbering is global and matches the bracketed citations in the text. Every arXiv identifier was resolved against arxiv.org, the ACL Anthology, OpenReview or the publisher page at the time of writing. Entries whose author lists could not be re-verified are marked as such and should be checked before citation.

Benchmarks and behavioral studies of multi-requirement instruction following

1. Zhou, J., Lu, T., Mishra, S., et al. Instruction-Following Evaluation for Large Language Models (IFEval). arXiv 2023. arXiv:2311.07911
2. Jiang, Y., Wang, Y., Zeng, X., et al. FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models. ACL 2024. arXiv:2310.20410
3. Wen, B., Ke, P., Gu, X., et al. Benchmarking Complex Instruction-Following with Multiple Constraints Composition (ComplexBench). NeurIPS 2024 Datasets & Benchmarks. arXiv:2407.03978
4. Qin, Y., Song, K., Hu, Y., et al. InFoBench: Evaluating Instruction Following Ability in Large Language Models. Findings of ACL 2024. arXiv:2401.03601
5. Zhang, T., Zhu, C., Shen, Y., et al. CFBench: A Comprehensive Constraints-Following Benchmark for LLMs. ACL 2025. arXiv:2408.01122
6. He, Q., Zeng, J., Huang, W., et al. Can Large Language Models Understand Real-World Complex Instructions? (CELLO). AAAI 2024. arXiv:2309.09150
7. Qin, Y., Zhang, T., Shen, Y., et al. SysBench: Can Large Language Models Follow System Messages?. arXiv 2024. arXiv:2408.10943
8. He, Y., Jin, D., Wang, C., et al. Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following. arXiv 2024. arXiv:2410.15553
9. Pyatkin, V., Malik, S., Lambert, N., et al. Generalizing Verifiable Instruction Following (IFBench). NeurIPS 2025 Datasets & Benchmarks. arXiv:2507.02833
10. Harada, K., Yamazaki, Y., Taniguchi, M., et al. When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following. EMNLP 2025. arXiv:2509.21051
11. Ye, J., Huang, C., Chen, Z., et al. A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models. arXiv 2025. arXiv:2505.07591
12. Ferraz, T. P., Mehta, K., Lin, Y.-H., et al. LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints. arXiv 2024. arXiv:2410.06458
13. Murthy, R., Kumar, P., Venkateswaran, P., Contractor, D. Evaluating the Instruction-following Abilities of Language Models using Knowledge Tasks. arXiv 2024. arXiv:2410.12972
14. Qi, Y., Peng, H., Wang, X., et al. AgentIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios. arXiv 2025. arXiv:2505.16944
15. Kwon, Y., et al. (author list not re-verified) ReasonIF: Large Reasoning Models Fail to Follow Instructions During Reasoning. arXiv 2025. arXiv:2510.15211
16. Fu, T., Gu, J., Li, Y., Qu, X., Cheng, Y. Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models (MathIF). arXiv 2025. arXiv:2505.14810
17. Li, X., Yu, Z., Zhang, Z., et al. When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs. NeurIPS 2025. arXiv:2505.11423
18. Tam, Z. R., Wu, C.-K., Tsai, Y.-L., Lin, C.-Y., Lee, H.-y., Chen, Y.-N. Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. EMNLP 2024 Industry Track. arXiv:2408.02442
19. Lee, I. Y., D'Antoni, L., Berg-Kirkpatrick, T. The Format Tax. arXiv 2026. arXiv:2604.03616
20. Fan, H. Capacity, Not Format: Rethinking Structured Reasoning Failures. arXiv 2026. arXiv:2606.09410
21. Liu, N. F., Lin, K., Hewitt, J., et al. Lost in the Middle: How Language Models Use Long Contexts. TACL 2024. arXiv:2307.03172
22. Liu, Y., Zeng, X., Shao, C., Meng, F., Zhou, J. Instruction Position Matters in Sequence Generation with Large Language Models. Findings of ACL 2024. arXiv:2308.12097
23. Sclar, M., Choi, Y., Tsvetkov, Y., Suhr, A. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design. ICLR 2024. arXiv:2310.11324
24. Zhang, Z., Li, S., Zhang, Z., et al. IHEval: Evaluating Language Models on Following the Instruction Hierarchy. NAACL 2025. arXiv:2502.08745
25. Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., Beutel, A. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv 2024. arXiv:2404.13208
26. Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., Farajtabar, M. GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. ICLR 2025. arXiv:2410.05229
27. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., Iwasawa, Y. Large Language Models are Zero-Shot Reasoners. NeurIPS 2022. arXiv:2205.11916
28. Kong, A., Zhao, S., Chen, H., et al. Better Zero-Shot Reasoning with Role-Play Prompting. NAACL 2024. arXiv:2308.07702
29. Zeng, Z., Yu, J., Gao, T., Meng, Y., Goyal, T., Chen, D. Evaluating Large Language Models at Evaluating Instruction Following (LLMBar). ICLR 2024. arXiv:2310.07641
30. Cobbe, K., Kosaraju, V., Bavarian, M., et al. Training Verifiers to Solve Math Word Problems (GSM8K). arXiv 2021. arXiv:2110.14168
31. Wei, J., Wang, X., Schuurmans, D., et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. arXiv:2201.11903
32. Mu, N., Chen, S., Wang, Z., et al. Can LLMs Follow Simple Rules? (RuLES). arXiv 2023. arXiv:2311.04235
33. Elder, B., Duesterwald, E., Muthusamy, V. Boosting Instruction Following at Scale. arXiv 2025. arXiv:2510.14842

Internal representations of instructions; task, function and steering vectors

34. Wu, X., Yao, W., Chen, J., Pan, X., Wang, X., Liu, N., Yu, D. From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning. NAACL 2024. arXiv:2310.00492
35. Gao, C., Huang, S., Li, J., Chen, J. Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention. Findings of EMNLP 2023. arXiv:2310.19084
36. Heo, J., Heinze-Deml, C., Elachqar, O., et al. Do LLMs "Know" Internally When They Follow Instructions?. ICLR 2025. arXiv:2410.14516
37. Heo, J., Xiong, M., Heinze-Deml, C., Narain, J. Do LLMs Estimate Uncertainty Well in Instruction-Following?. ICLR 2025. arXiv:2410.14582
38. Stolfo, A., Balachandran, V., Yousefi, S., Horvitz, E., Nushi, B. Improving Instruction-Following in Language Models through Activation Steering. ICLR 2025. arXiv:2410.12877
39. He, Z., Zhao, H., Qiao, Y., Yang, F., Payani, A., Ma, J. SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Models. arXiv 2025. arXiv:2502.11356
40. Jiang, G., Jiang, C., Li, Z., et al. Interpretable Catastrophic Forgetting of Large Language Model Fine-tuning via Instruction Vector. arXiv 2024. arXiv:2406.12227
41. Hewitt, J., Liu, N. F., Liang, P., Manning, C. D. Instruction Following without Instruction Tuning. arXiv 2024. arXiv:2409.14254
42. Hendel, R., Geva, M., Globerson, A. In-Context Learning Creates Task Vectors. Findings of EMNLP 2023. arXiv:2310.15916
43. Todd, E., Li, M. L., Sharma, A. S., Mueller, A., Wallace, B. C., Bau, D. Function Vectors in Large Language Models. ICLR 2024. arXiv:2310.15213
44. Davidson, G., Gureckis, T. M., Lake, B. M., Williams, A. Do Different Prompting Methods Yield a Common Task Representation in Language Models?. arXiv 2025. arXiv:2505.12075
45. Xiong, Z., Cai, Z., Cooper, J., et al. Everything Everywhere All at Once: LLMs Can In-Context Learn Multiple Tasks in Superposition. ICML 2025. arXiv:2410.05603
46. Liu, S., Ye, H., Xing, L., Zou, J. In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering. ICML 2024. arXiv:2311.06668
47. Tikhonov, P., Oseledets, I., Tutubalina, E. One Task Vector Is Not Enough: A Large-Scale Study for In-Context Learning. arXiv 2025. arXiv:2505.23911
48. Zheng, B., Ma, Y., Lin, Z., Yang, Y. Label Words as Local Task Vectors in In-Context Learning. arXiv 2024. arXiv:2406.16007
49. Sia, S., Mueller, D., Duh, K. Where Does In-Context Learning Happen in Large Language Models?. NeurIPS 2024. arXiv:2403.04510
50. Park, K., Choe, Y. J., Veitch, V. The Linear Representation Hypothesis and the Geometry of Large Language Models. ICML 2024. arXiv:2311.03658
51. Elhage, N., Hume, T., Olsson, C., et al. Toy Models of Superposition. Transformer Circuits Thread 2022. arXiv:2209.10652
52. Oozeer, N., et al. Beyond Linear Steering: Unified Multi-Attribute Control for Language Models (K-Steering). arXiv 2025. arXiv:2505.24535
53. Radevski, G., Gashteovski, K., Hong, S., Lawrence, C., Glavaš, G. Compositional Steering of Large Language Models with Steering Tokens. arXiv 2026. arXiv:2601.05062
54. Mu, J., Li, X. L., Goodman, N. Learning to Compress Prompts with Gist Tokens. NeurIPS 2023. arXiv:2304.08467
55. Arditi, A., Obeso, O., Syed, A., et al. Refusal in Language Models Is Mediated by a Single Direction. NeurIPS 2024. arXiv:2406.11717
56. Chen, R., Arditi, A., Sleight, H., Evans, O., Lindsey, J. Persona Vectors: Monitoring and Controlling Character Traits in Language Models. arXiv 2025. arXiv:2507.21509
57. Rimsky, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., Turner, A. Steering Llama 2 via Contrastive Activation Addition. ACL 2024. arXiv:2312.06681
58. Bayat, R., Rahimi-Kalahroudi, A., Pezeshki, M., Chandar, S., Vincent, P. Steering Large Language Model Activations in Sparse Spaces. arXiv 2025. arXiv:2503.00177
59. Wang, A., Shu, D., Wang, Y., Ma, Y., Du, M. Improving LLM Reasoning through Interpretable Role-Playing Steering. Findings of EMNLP 2025. arXiv:2506.07335
60. Yang, H., Cho, K., Ding, M., Inoue, T. (author list partly unverified) Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight. arXiv 2025. arXiv:2509.24169
61. Han, S., Song, J., Gore, J., Agrawal, P. Emergence of Abstractions: Concept Encoding and Decoding Mechanism for In-Context Learning in Transformers. arXiv 2024. arXiv:2412.12276

Attention routing, instruction anchors, attention steering, and attention sinks

62. Yuksekgonul, M., Chandrasekaran, V., Jones, E., et al. Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models. ICLR 2024. arXiv:2309.15098
63. Chuang, Y.-S., Qiu, L., Hsieh, C.-Y., Krishna, R., Kim, Y., Glass, J. Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps. EMNLP 2024. arXiv:2407.07071
64. Zhang, Q., Singh, C., Liu, L., Liu, X., Yu, B., Gao, J., Zhao, T. Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs (PASTA). ICLR 2024. arXiv:2311.02262
65. Guardieiro, V., Khare, A., Stein, A., Wong, E. Instruction Following by Principled Boosting Attention of Large Language Models (InstABoost). ICLR 2026. arXiv:2506.13734
66. Venkateswaran, P., Contractor, D. Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering. EACL 2026. arXiv:2505.12025
67. Li, K., Liu, T., Bashkansky, N., Bau, D., Viégas, F., Pfister, H., Wattenberg, M. Measuring and Controlling Instruction (In)Stability in Language Model Dialogs. COLM 2024. arXiv:2402.10962
68. Dongre, V., Hsieh, J., Lai, V. D., Yoon, S., Bui, T., Hakkani-Tür, D. When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interaction. arXiv 2026. arXiv:2605.12922
69. Zhang, Y., Xu, M., Bai, X., et al. Instruction Anchors: Dissecting the Causal Dynamics of Modality Arbitration. arXiv 2026. arXiv:2602.03677
70. Wang, L., Li, L., Dai, D., et al. Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning. EMNLP 2023. arXiv:2305.14160
71. Hung, K.-H., Ko, C.-Y., Rawat, A., Chung, I.-H., Hsu, W. H., Chen, P.-Y. Attention Tracker: Detecting Prompt Injection Attacks in LLMs. arXiv 2024. arXiv:2411.00348
72. Jin, Z., Cao, P., Yuan, H., et al. Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models. Findings of ACL 2024. arXiv:2402.18154
73. Wu, W., Wang, Y., Xiao, G., Peng, H., Fu, Y. Retrieval Head Mechanistically Explains Long-Context Factuality. ICLR 2025. arXiv:2404.15574
74. Zhang, W., Yin, F., Yen, H., Chen, D., Ye, X. Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking. arXiv 2025. arXiv:2506.09944
75. Xiao, G., Tian, Y., Chen, B., Han, S., Lewis, M. Efficient Streaming Language Models with Attention Sinks. ICLR 2024. arXiv:2309.17453
76. Gu, X., Pang, T., Du, C., et al. When Attention Sink Emerges in Language Models: An Empirical View. ICLR 2025. arXiv:2410.10781
77. Barbero, F., Arroyo, Á., Gu, X., et al. Why Do LLMs Attend to the First Token?. COLM 2025. arXiv:2504.02732
78. Sun, M., Chen, X., Kolter, J. Z., Liu, Z. Massive Activations in Large Language Models. COLM 2024. arXiv:2402.17762
79. McDougall, C., Conmy, A., Rushing, C., McGrath, T., Nanda, N. Copy Suppression: Comprehensively Understanding an Attention Head. arXiv 2023. arXiv:2310.04625
80. Jain, S., Wallace, B. C. Attention is not Explanation. NAACL 2019. arXiv:1902.10186
81. Wiegreffe, S., Pinter, Y. Attention is not not Explanation. EMNLP 2019. arXiv:1908.04626
82. Abnar, S., Zuidema, W. Quantifying Attention Flow in Transformers. ACL 2020. arXiv:2005.00928
83. Zeng, S., Lee, S., Zhao, H., Hockenmaier, J. Steering Instruction Hierarchies at Inference Time (V-Steer). COLM 2026. arXiv:2607.26228
84. Balp-Straffon, E., Hsu, C.-H., Gadhvi, R., et al. How Language Models Choose Sides: Internal Representations of Instruction Hierarchy. ICML 2026 Mechanistic Interpretability Workshop. arXiv:2608.28648
85. Zeng, S. Who is In Charge? Dissecting Role Conflicts in Instruction Following. arXiv 2025. arXiv:2510.01228
86. Wu, T., et al. Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy. ICLR 2025. arXiv:2410.09102
87. Bai, Y., et al. Identifying and Analyzing Task-Encoding Tokens in Large Language Models. arXiv 2024. arXiv:2401.11323

The KV cache as instruction memory

88. Li, Y., Huang, Y., Yang, B., et al. SnapKV: LLM Knows What You are Looking for Before Generation. NeurIPS 2024. arXiv:2404.14469
89. Zhang, Z., Sheng, Y., Zhou, T., et al. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. NeurIPS 2023. arXiv:2306.14048
90. Belitsky, M., Kopiczko, D. J., Dorkenwald, M., et al. KV Cache Steering for Controlling Frozen LLMs. arXiv 2025. arXiv:2507.08799
91. Liu, A. Z., Zhang, M., Greenberg, I., et al. Memory Inception: Latent-Space KV Cache Manipulation for Steering LLMs. arXiv 2026. arXiv:2605.06225
92. Gu, Z., Yao, J., Du, K., Jiang, J. (author list from listing) LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts. arXiv 2024. arXiv:2411.13009

Chain-of-thought mechanics and generation-stage dynamics

93. Cabannes, V., Arnal, C., Bouaziz, W., Yang, A., Charton, F., Kempe, J. Iteration Head: A Mechanistic Study of Chain-of-Thought. NeurIPS 2024. arXiv:2406.02128
94. Dutta, S., Singh, J., Chakrabarti, S., Chakraborty, T. How to Think Step-by-Step: A Mechanistic Understanding of Chain-of-Thought Reasoning. TMLR 2024. arXiv:2402.18312
95. Stolfo, A., Belinkov, Y., Sachan, M. A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis. EMNLP 2023. arXiv:2305.15054
96. Nikankin, Y., Reusch, A., Mueller, A., Belinkov, Y. Arithmetic Without Algorithms: Language Models Solve Math with a Bag of Heuristics. ICLR 2025. arXiv:2410.21272
97. Liu, X., Wang, L. Answer Convergence as a Signal for Early Stopping in Reasoning. arXiv 2025. arXiv:2506.02536
98. Afzal, A., Chechik, G., Matthes, F., Ziser, Y. Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion. arXiv 2025. arXiv:2505.24362
99. Arcuschin, I., Janiak, J., Krzyzanowski, R., Rajamanoharan, S., Nanda, N., Conmy, A. Chain-of-Thought Reasoning In The Wild Is Not Always Faithful. arXiv 2025. arXiv:2503.08679
100. Lindsey, J., Gurnee, W., Ameisen, E., et al. On the Biology of a Large Language Model. Transformer Circuits Thread 2025. transformer-circuits.pub/2025/attribution-graphs/biology.html
101. Ameisen, E., Lindsey, J., Pearce, A., et al. Circuit Tracing: Revealing Computational Graphs in Language Models. Transformer Circuits Thread 2025. transformer-circuits.pub/2025/attribution-graphs/methods.html
102. Wang, B., Min, S., Deng, X., et al. Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters. ACL 2023. arXiv:2212.10001
103. Madaan, A., Yazdanbakhsh, A. Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango. arXiv 2022. arXiv:2209.07686
104. Venhoff, C., Arcuschin, I., Torr, P., Conmy, A., Nanda, N. Understanding Reasoning in Thinking Language Models via Steering Vectors. arXiv 2025. arXiv:2506.18167
105. Sun, C.-E., Yan, G., Weng, T.-W. ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Models. EMNLP 2025. arXiv:2503.22048
106. Sinii, V., et al. (author list partly unverified) Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors. arXiv 2025. arXiv:2509.06608
107. Rocchetti, E., Ferrara, A. How Instruction-Tuning Imparts Length Control: A Cross-Lingual Mechanistic Analysis. arXiv 2025. arXiv:2509.02075
108. Park, Y., Jeong, M., Kang, J. Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Training. arXiv 2025. arXiv:2509.25758
109. Bogdan, P. C., Macar, U., Nanda, N., Conmy, A. Thought Anchors: Which LLM Reasoning Steps Matter?. arXiv 2025. arXiv:2506.19143
110. Zhang, J., Lin, Q., Rajmohan, S., Zhang, D. From Reasoning to Answer: Empirical, Attention-Based and Mechanistic Insights into Distilled DeepSeek R1 Models. arXiv 2025. arXiv:2509.23676
111. Zhang, H., Tian, Y., Zhang, T. Attention-Aligned Reasoning for Large Language Models. arXiv 2025. arXiv:2510.03223
112. Yan, S., Shen, C., Wang, W., Xie, L., Liu, J., Ye, J. Don't Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models. arXiv 2025. arXiv:2503.11154
113. Chen, X., Xu, J., Liang, T., et al. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs. arXiv 2024. arXiv:2412.21187

Causal intervention, attribution and evaluation methodology

114. Vig, J., Gehrmann, S., Belinkov, Y., et al. Investigating Gender Bias in Language Models Using Causal Mediation Analysis. NeurIPS 2020. arXiv:2004.12265
115. Meng, K., Bau, D., Andonian, A., Belinkov, Y. Locating and Editing Factual Associations in GPT (ROME / causal tracing). NeurIPS 2022. arXiv:2202.05262
116. Zhang, F., Nanda, N. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. ICLR 2024. arXiv:2309.16042
117. Heimersheim, S., Nanda, N. How to Use and Interpret Activation Patching. arXiv 2024. arXiv:2404.15255
118. Wang, K., Variengien, A., Conmy, A., Shlegeris, B., Steinhardt, J. Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small. ICLR 2023. arXiv:2211.00593
119. Goldowsky-Dill, N., MacLeod, C., Sato, L., Arora, A. Localizing Model Behavior with Path Patching. arXiv 2023. arXiv:2304.05969
120. Syed, A., Rager, C., Conmy, A. Attribution Patching Outperforms Automated Circuit Discovery. BlackboxNLP 2024. arXiv:2310.10348
121. Kramár, J., Lieberum, T., Shah, R., Nanda, N. AtP*: An Efficient and Scalable Method for Localizing LLM Behaviour to Components. arXiv 2024. arXiv:2403.00745
122. Hanna, M., Pezzelle, S., Belinkov, Y. Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms (EAP-IG). COLM 2024. arXiv:2403.17806
123. Conmy, A., Mavor-Parker, A. N., Lynch, A., Heimersheim, S., Garriga-Alonso, A. Towards Automated Circuit Discovery for Mechanistic Interpretability (ACDC). NeurIPS 2023. arXiv:2304.14997
124. Haklay, T., Orgad, H., Bau, D., Mueller, A., Belinkov, Y. Position-aware Automatic Circuit Discovery (PEAP). ACL 2025. arXiv:2502.04577
125. Geva, M., Bastings, J., Filippova, K., Globerson, A. Dissecting Recall of Factual Associations in Auto-Regressive Language Models. EMNLP 2023. arXiv:2304.14767
126. Chan, L., Garriga-Alonso, A., Goldowsky-Dill, N., et al. Causal Scrubbing: A Method for Rigorously Testing Interpretability Hypotheses. Alignment Forum 2022. www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing
127. Geiger, A., Lu, H., Icard, T., Potts, C. Causal Abstractions of Neural Networks. NeurIPS 2021. arXiv:2106.02997
128. Wu, Z., Geiger, A., Icard, T., Potts, C., Goodman, N. Interpretability at Scale: Identifying Causal Mechanisms in Alpaca (Boundless DAS). NeurIPS 2023. arXiv:2305.08809
129. Ghandeharioun, A., Caciularu, A., Pearce, A., Dixon, L., Geva, M. Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models. ICML 2024. arXiv:2401.06102
130. Cohen-Wang, B., Shah, H., Georgiev, K., Madry, A. ContextCite: Attributing Model Generation to Context. NeurIPS 2024. arXiv:2409.00729
131. Cohen-Wang, B., Chen, Y., Madry, A. Learning to Attribute with Attention. arXiv 2025. arXiv:2504.13752
132. Liu, F., Kandpal, N., Raffel, C. AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution. ICLR 2025. arXiv:2411.15102
133. Sarti, G., Feldhus, N., Sickert, L., van der Wal, O., Nissim, M., Bisazza, A. Inseq: An Interpretability Toolkit for Sequence Generation Models. ACL 2023 Demo. arXiv:2302.13942
134. Yin, K., Neubig, G. Interpreting Language Models with Contrastive Explanations. EMNLP 2022. arXiv:2202.10419
135. Achtibat, R., Hatefi, S. M. V., Dreyer, M., et al. AttnLRP: Attention-Aware Layer-wise Relevance Propagation for Transformers. ICML 2024. arXiv:2402.05602
136. Ferrando, J., Voita, E. Information Flow Routes: Automatically Interpreting Language Models at Scale. EMNLP 2024. arXiv:2403.00824
137. Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., Mueller, A. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. ICLR 2025. arXiv:2403.19647
138. Makelov, A., Lange, G., Nanda, N. Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching. ICLR 2024. arXiv:2311.17030
139. Miller, J., Chughtai, B., Saunders, W. Transformer Circuit Faithfulness Metrics are not Robust. COLM 2024. arXiv:2407.08734
140. Shi, C., Beltran-Velez, N., Nazaret, A., et al. Hypothesis Testing the Circuit Hypothesis in LLMs. NeurIPS 2024. arXiv:2410.13032
141. Pres, I., Ruis, L., Lubana, E. S., Krueger, D. Towards Reliable Evaluation of Behavior Steering Interventions in LLMs. arXiv 2024. arXiv:2410.17245
142. Ortu, F., Jin, Z., Doimo, D., et al. Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals. ACL 2024. arXiv:2402.11655
143. Dotsinski, A., et al. On the Generalizability of "Competition of Mechanisms" (reproduction study). arXiv 2025. arXiv:2506.22977
144. Wu, Z., Geiger, A., Arora, A., et al. pyvene: A Library for Understanding and Improving PyTorch Models via Interventions. NAACL 2024 Demo. arXiv:2403.07809
145. Fiotto-Kaufman, J., Loftus, A. R., Todd, E., et al. NNsight and NDIF: Democratizing Access to Foundation Model Internals. arXiv 2024. arXiv:2407.14561
146. Bhaskar, A., Wettig, A., Friedman, D., Chen, D. Finding Transformer Circuits with Edge Pruning. NeurIPS 2024. arXiv:2406.16778
147. McGrath, T., Rahtz, M., Kramár, J., Mikulik, V., Legg, S. The Hydra Effect: Emergent Self-repair in Language Model Computations. arXiv 2023. arXiv:2307.15771
148. Rushing, C., Nanda, N. Explorations of Self-Repair in Language Models. ICML 2024. arXiv:2402.15390
149. Wei, J., Bosma, M., Zhao, V., et al. Finetuned Language Models Are Zero-Shot Learners. ICLR 2022. arXiv:2109.01652
150. Ouyang, L., Wu, J., Jiang, X., et al. Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022. arXiv:2203.02155
151. Rafailov, R., Sharma, A., Mitchell, E., et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. arXiv:2305.18290
152. Lambert, N., Morrison, J., Pyatkin, V., et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv 2024. arXiv:2411.15124
153. Sarkar, S., Feng, S., Ney, J., et al. Designing Role Vectors to Improve LLM Inference Behaviour. arXiv 2025. arXiv:2502.12055
154. Tan, D., Chanin, D., Lynch, A., et al. Analysing the Generalisation and Reliability of Steering Vectors. NeurIPS 2024. arXiv:2407.12404
155. Wu, Z., Arora, A., Geiger, A., et al. AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders. ICML 2025. arXiv:2501.17148
156. Kang, D., Liu, Z., Ma, N., Huang, Y., Tan, Z., Jiang, M. Prompt-Activation Duality: Improving Activation Steering via Attention-Level Interventions. arXiv 2026. arXiv:2605.10664
157. Rahn, N., D’Oro, P., Bellemare, M. Controlling Large Language Model Agents with Entropic Activation Steering. arXiv 2024. arXiv:2406.00244
158. Shkolnikov, Y. P. Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices. arXiv 2026. arXiv:2603.04428
159. Braun, J., Eickhoff, C., Krueger, D., Bahrainian, S. A., Krasheninnikov, D. Understanding (Un)Reliability of Steering Vectors in Language Models. arXiv 2025. arXiv:2505.22637
160. Cunningham, H., Ewart, A., Riggs, L., Huben, R., Sharkey, L. Sparse Autoencoders Find Highly Interpretable Features in Language Models. ICLR 2024. arXiv:2309.08600
161. Bricken, T., Templeton, A., Batson, J., et al. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread 2023. transformer-circuits.pub
162. Templeton, A., Conerly, T., Marcus, J., et al. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread 2024. transformer-circuits.pub
163. Lieberum, T., Rajamanoharan, S., Conmy, A., et al. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2. BlackboxNLP 2024. arXiv:2408.05147