The Number That Decides a Mechanic Is a Man

Somewhere in the eighth layer of a one-billion-parameter language model, there is a single number that decides a mechanic is a man.

I went looking for it over a weekend, on a laptop. I found it. Then I turned it down until the model stopped, and turned it up until the model started arguing with its own prompt.

Everybody already knows language models do this. It has been documented since word embeddings. “Man is to computer programmer as woman is to homemaker” is from 2016. What nobody had shown me, in a way I could actually follow, was where the stereotype physically lives, and whether it is the sort of thing you can reach in and grab.

That is a different question from “is the model biased”. It is: is the bias a thing, with an address, that you could in principle delete? Or is it smeared across a hundred billion weights in a way that means the only lever anyone will ever have is more training data?

Short version, before I show my work: it has an address. And the address is different in every model, which turns out to be the most interesting thing I found.

A note on what this is. Weekend-scale work, two small open models, one laptop, written up honestly, including the three attempts that failed and the corrupted file that quietly invalidated my first round of results. Every number here comes out of a saved artefact in the repo. I am new to mechanistic interpretability. If I have got something wrong, I would like to know.

First, watch it happen

Before any of the fancy machinery, here is the behaviour, unedited. I give Llama-3.2-1B-InstructMeta's one-billion-parameter open model, instruction-tuned for chat. Small enough to run on a laptop.hugging face ↗ a sentence about a hairdresser. The prompt says nothing about who this person is.

That last part is the whole problem in miniature. If you open the model at the moment it decides, the loudest thing happening inside it is a feature that fires on the word “up”. The stereotype is in there somewhere, but it is not sitting on the surface waving at you.

Zoom out from one sentence to twelve prompts per group and the shape gets clearer. And stranger.

The asymmetry surprised me. The model is emphatic that mechanics are men and only mildly of the opinion that nurses are women. Whatever this thing is, it is not a tidy symmetric “gender axis”. It leans.

And the third tab matters more than it looks. Put an explicit her in the sentence and the model does the right thing, 13:1. It is not deaf to context. Hold onto that. It will not survive.

The prediction is not made where you read it

A transformer thinks in a stack. Every token carries a vector (2048 numbers in this model) that each of the sixteen layers reads from and writes back to. The logit lensMultiply a mid-network vector by the output matrix to read what the model would say if you forced it to answer right there. Lets you watch a prediction assemble layer by layer.nostalgebraist, 2020 ↗ is a cheap trick that lets you tap that vector at any layer and ask “what would you say if I made you answer right now?” Do it at every layer and you get to watch a prediction assemble itself.

I read the model at layer 8, the midpoint. Here is what is happening around there.

The first four layers are just echoing the prompt back at you. Then there is a stretch of nothing much. Then, from layer 9 to 14, the answer gets built, and those are exactly the layers that hurt when you ablateZero out one component and measure how much the output moves. A big move means that component was doing real work.wikipedia ↗ them.

So layer 8 is the right place to look: late enough that the model has formed an opinion, early enough that the opinion has not yet been spent on a token. It is the model thinking, rather than the model talking.

To read a residual streamThe vector each token carries through the network, 2048 numbers wide here. Every layer reads from it and writes back to it, so it works as the model's running memory for that token.transformer circuits ↗ you need a sparse autoencoderA translator that unpacks one dense activation vector into thousands of sparse slots, each ideally standing for one interpretable thing.bricken et al., 2023 ↗: a translator that unpacks one dense 2048-number vector into 32,768 slots, each ideally meaning one interpretable thing, of which only a couple of hundred should be on at a time. The count of how many are on is called L0The number of features that are non-zero for a given input. Healthy is hundreds out of 32,768; three means the autoencoder is not running.bricken et al., 2023 ↗. Remember that word. It is about to ruin two days of my life.

Models and autoencoders both come from Aquin, which is the toolkit every command on this page is talking to.

$ aquin load model llama-3.2-1b     # ~2.5 GB
$ aquin load sae   llama-3.2-1b-l8  # 537 MB

Three ways to not find a feature

I tried all the obvious things first, and all of them failed. They are in this post because each one failed in a way I needed later.

Then I found out my instrument was broken

This is the part that would have silently ruined everything, and the reason I would tell any beginner to read this section before any of the results.

I wrote a small script that bypassed the toolkit entirely: run the prompts, grab the layer-8 residual at the final token, push it through the autoencoder, print what fires. First run:

The cause: this autoencoder was trained on normalised inputs. All 2048 dimensions shifted and scaled as (x − μ) / σ, with the statistics stored in a separate file. The toolkit had warned me at download time (“norm invalid in catalog storage”) and I had scrolled straight past it. With the statistics broken, raw residuals arrive at scales the encoder never saw in training and almost nothing survives the ReLUThe activation function that clips anything negative to zero. If inputs arrive at the wrong scale, almost nothing survives it.wikipedia ↗.

It does not raise an error. It just goes quiet.

And here is the genuinely nasty part. The same broken file caused the exact opposite failure on a different code path. My script had worked around it by skipping normalisation entirely. Silence. The toolkit’s own stats command applied the invalid statistics, and reported that 30,688 of 32,768 features were active. A sparse autoencoder with 94% of its dictionary switched on.

Same file. Same layer. Same twelve prompts. Three completely different pictures of reality:

If you have done any classical ML, you have met this before. It is train/serve skew. You fit a StandardScalerThe scikit-learn tool that shifts and scales each feature to mean 0, standard deviation 1. You must reuse the training statistics at inference, or the model sees a different world.scikit-learn ↗ during training, then ignore or overwrite it at inference, and your model quietly becomes a random number generator. The norm file is the StandardScaler.

The fix was about fifty lines: push 200 wikitextA standard language-modelling dataset of verified Wikipedia articles, commonly used to measure perplexity.hugging face ↗ documents through the model, compute a running per-dimension mean and standard deviation with Welford’s algorithmA one-pass way to compute mean and variance that stays numerically stable, instead of summing squares and subtracting.wikipedia ↗, write them where the framework expects them. L0 went from 3 to roughly 6,400. Alive.

6,400 is still too high for a well-trained autoencoder, and I know why. My statistics come from wikitext, not from whatever corpus this thing was actually trained against. It is a reconstruction, not a recovery. But the features became interpretable, and as the rest of this post shows, causal. Good enough to keep going, and I would rather say that out loud than round it up.

The transferable lesson, and the one I would tattoo on a beginner: check L0 before you believe a single thing an SAE tells you. A distribution-mismatched autoencoder fails silently in both directions, dead quiet or fully saturated, and one number, computed in one line, tells you which of those you are looking at. It is the difference between “the SAE disagrees with my hypothesis” and “the SAE is not running”.

$ aquin sae-stats --prompts your_probes.jsonl --layers 8 --topk 30
# healthy = hundreds active

Two tests, each useless on its own

With a working instrument I started over, and this time I built the search around the three lessons the failures had taught me. Matched templates, so the only difference between prompts is the occupation word. Read at the final position only, where the signal actually is. And the one that mattered most: two independent tests per feature, chosen so that they fail in different ways.

1. Does it fire selectively? Thirty-six sentences of the form “The {occupation} said that”, identical except for the job: twelve female-stereotyped, twelve male-stereotyped, twelve neutral. Does the feature care which group it is looking at?
2. Does it push the pronoun? Project the feature’s decoder direction onto the vocabulary and see whether it actually promotes “ he” or “ she” at the output.

Alone, each test is a sieve with holes in it. Selectivity waves through anything correlated with the occupation groups, including, memorably, a feature about food that simply liked the word “dietitian”. Output push waves through every grammar feature that touches a pronoun without caring which one is correct.

Switch both on and watch what is left.

f32258. Fires at 0.38 on male-stereotyped jobs against 0.03 female and 0.16 neutral, and its decoder direction promotes “ he”. It is also one of the two candidates my hand traces had flagged back in attempt three. Two unrelated methods landing on the same feature, which is roughly as much reassurance as this stage of the pipeline ever gives you.

The knob

Everything so far is correlation. A feature that lights up next to a behaviour is not the same as a feature that causes it, and the interpretability literature is full of beautiful stories that die the moment somebody intervenes.

So: intervene. I inject strength × decoder_direction(f32258) into the layer-8 residual stream at every position, sweep the strength from −6 to +6, and watch what the model does with its pronouns.

Scroll. Or skip the narration entirely and drag the slider. I would.

But the result I keep coming back to is the third panel. Six units of one feature, and a model that has just read the word “her” decides to say “he” anyway.

That reframes what the stereotype is. I had assumed it was a soft prior, something the model falls back on when the context is silent, the way you would guess. It is not. It is a circuit that competes with the evidence, and at sufficient volume it wins.

A knob, or a lobotomy?

A dose-response curveBorrowed from pharmacology: if the effect scales smoothly with the size of the intervention, the intervention is probably causing it rather than merely correlating with it.wikipedia ↗ on the prompts you used to find the feature proves very little. There are two obvious ways to fool yourself here: a knob that only works on your own sentences, and a knob that works by quietly wrecking the model. Four checks.

That last one is the quiet punchline of the whole method. If I had skipped the output-push test and steered the selectivity-only candidates, this post would be titled “what I learned from a failed interpretability project”. One line of evidence handed me a loser. Two handed me a knob. The second test is not a formality. It is the entire difference between the two outcomes.

Then the second model did it backwards

I ran the whole pipeline again on a different model: LFM2.5-230M, about a fifth the size, a different architecture, different training. Mostly I expected a boring confirmation.

The baseline was already interesting: same direction of bias, far weaker. Male-stereotyped occupations sit at 3:1 against Llama’s 15:1. The smaller, less instruction-tuned model is the less biased one, which is at least consistent with the idea that some of Llama’s 15:1 was installed after pretraining.

Then discovery did something I did not expect at all. The “he” candidates steered terribly. The best of them did essentially nothing. The feature that worked came off the other list: f9619, a female-context feature, selective by 32×.

Llama encodes occupation→pronoun stereotyping with a steerable male-context feature. LFM2.5 encodes the same behaviour with a steerable female-context feature. Same thing learned, mirrored implementation.

I want to be careful about how much weight that carries: n = 2 is an anecdote, not a finding. But it is enough to poke a hole in an assumption that runs quietly through a lot of interpretability work: that a canonical behaviour has a canonical encoding. “The gender direction”, singular.

If I had only ever opened Llama, I would have written a confident sentence about gender bias being encoded as a maleness direction, and I would have been describing a property of that particular model’s implementation, not a property of the behaviour. Which means any mitigation that depends on the encoding (ablate direction X, clamp feature Y) has to be redone, from scratch, per model. That is a much less comfortable position than the field’s working assumption.

What I would tell you before you start

1. 01Check L0 first. Every time. Before interpreting a single thing your autoencoder says, confirm it is running on your inputs. It fails silently in both directions and it will let you publish nonsense.
2. 02Match everything except the one variable. “The {occupation} said that”, with only the occupation changed, found in one afternoon what topic-mixed prompt sets failed to find in two days.
3. 03Read where the signal is, not where it is convenient. Averaging over token positions destroyed a signal that was perfectly legible at the final one.
4. 04Two independent lines of evidence, or nothing. Activation contrast alone handed me a feature about food. The one-test candidate behaved like noise under steering. Only what survived both tests was causal.
5. 05Correlation is cheap. Turn the knob. Monotonic response to strength, on sentences you did not use to find the feature, with a capability check and a negative control. That is the line between “we found a correlate” and “we found the knob”.
6. 06Always run a second model. The mirror was the most interesting thing in the project and it cost one extra model run. One.

What I cannot claim

Two small models. One bias axis: “he” and “she”. I did not measure singular “they”, which in hindsight I should have, given that one of the debiased completions literally says “inform them”. One layer per model. And steering was judged mostly on next-token pronoun probabilities plus small QA and perplexityHow surprised the model is by real text. Lower is better; a rise means the intervention cost the model some general ability.wikipedia ↗ checks, not on what happens over a page of generated text.

The biggest one: I found a causal knob, not the representation. There is no evidence that the stereotype is f32258. It is a handle the autoencoder happened to expose, and there may well be redundant copies in other features and other layers, though the fact that −6 closes the gap completely suggests this handle catches most of it, at least on these prompts. The clean way to settle it is to ablate f32258 entirely and rerun discovery to see if a backup appears. That is the next thing I want to do.

After that: a layer scan, since I only had one autoencoder per model and the ablation profile says the knob should exist at several nearby layers. Then converting steering into an actual weight edit, so the fix survives without an inference-time hook. And then five to ten more small models, to turn “the mirror” from an anecdote into a distribution, and to find out whether the implementation correlates with size, architecture, or how hard the model was instruction-tuned. The 15:1 versus 3:1 gap makes me suspect it does.