Fine-tuning Qwen — distilling a frontier panel into a local model

Our research pipeline pays three frontier models — a Claude, a GPT, and a Grok — to independently analyze a company from the same data briefing. That panel is good, and it costs real money per report. A local 27 B model on a consumer card costs $0 per read — but out of the box it agrees with the paid panel's verdict direction only about 60% of the time (up from 30% before its reasoning mode was enabled — that single switch was the largest improvement anything has produced so far).

The obvious lever — rewriting the prompts to suit the smaller model — is deliberately off the table. The local model sits the same exam as the paid panel, byte for byte; change the question and you can no longer compare the answers. What remains is changing the model: showing it thousands of the panel's own graded answers and training it to answer in that register. That technique has a name, and it is the industry's open secret.

Knowledge distillation trains a small model on a larger model's outputs rather than on raw internet text: same prompt in, teacher's answer as the target. It is how the major labs produce the small models they sell — the giant is the teacher, the mini is the student. Our variant has one twist worth stealing: the teacher is not one model but a three-model panel, so the student learns a register three independent frontier models converge on, not one model's quirks.

The glamour is in the training run; the quality lives in the dataset. Ours is 1,991 briefing→answer pairs from 679 archived reports across 549 tickers (~3,000 tokens per example), and the number started higher — what was removed is the instructive part:

One caveat we log rather than hide: the reconstructed briefings are near-identical but not byte-identical to what the teachers read (live inputs drift in the minutes between a seat's read and the archive freeze). Only 15 of ~2,000 reconstruct to the exact recorded checksum. The drift is small and the fundamentals identical — but "small and identical" is a claim, so every pair carries a match flag instead of a shrug.

Retraining 27 billion parameters is a datacenter job. LoRA sidesteps it: freeze the model, bolt small trainable adapter layers onto it (~0.2% of the parameter count), and train only those — backpropagation against the teacher answers decides which weights move and by how much; nobody hand-tunes billions of anything. QLoRA adds the second trick: hold the frozen base in compressed 4-bit form during training, which is what makes the whole job fit on a 24 GB-class consumer card at all. The adapter's size is not the effect's size — a 0.2% steering surface can move behavior dramatically, the same way the zero-weight reasoning toggle doubled panel agreement.

Two practicalities our smoke test surfaced, offered here so you can skip learning them the hard way. First, a thinking model trained on answer-only data must have its examples rendered in the "empty reasoning block" form the model family documents for non-reasoning samples — otherwise you either train away its thinking or teach it a prompt you never send. Second, measure your token lengths before trusting a context setting: our p95 example length exceeded the planned training context and would have silently truncated ~4% of examples. The census cost one line of code; the silent truncation would have cost a confused week.

Before any training run, we built the measuring stick: a comparison harness that runs the local model through the entire production analysis on a ticker and diffs it against the paid panel's run — verdict directions, per-dimension scores, and the prose side by side. Judged on a fixed cohort of tickers chosen for diversity (a conviction buy, a borderline case, a clear avoid), never on cherry-picked singles. Tickers used for tuning are excluded from training; the adapter's final grade comes only from tickers it has never seen.

The early findings that survived scrutiny: the local model has never once reversed the panel's verdict direction across the cohort — it either matches or softens toward neutral. Its prose is near-parity and occasionally sharper (it caught a stale growth figure the paid seat repeated). Its weakness is commitment: where the panel calls a conviction extreme, the local model hedges toward the middle. Which defines the adapter's mission precisely — the teacher answers are full of frontier models committing with numbers.

The best question anyone asked of this experiment came from outside it: "the base model has natural variance — how do you measure improvement over that?" So we measured the variance instead of assuming it: the same ticker, the same configuration, byte-identical settings, run twice. At temperature 0 — greedy decoding, the most deterministic setting that exists — the expectation was near-zero drift.

The result: 11.9% of all numeric values changed between the two runs (93 of 779 compared). One analytical dimension's score swung 2× (−37 to −74). A composite score moved 20 points. The narrative text came back only 21–47% similar. Temperature 0 makes a large model mostly deterministic, not deterministic — GPU arithmetic reorders, one token flips somewhere, and the divergence cascades from there.

That one $0 control run rewrote the experiment's rules — and demoted some of our own early findings from "pattern" to "anecdote" in the same afternoon:

If you run local-model experiments and have never measured your repeat-run variance, this section is the one to steal. It cost one spare run and it is the difference between tuning a model and tuning your own confirmation bias.

As of the stamp above, the adapter has not been trained. The machine is ready, the dataset is exported, the harness and noise floor exist; the training run comes after the serving configuration stops moving, so its before/after is measured against a stable baseline. This section will carry exactly three things when they exist, and nothing before: the panel-agreement rate before and after, measured on held-out tickers only; whether the "commitment" gap moved; and the noise-floor-adjusted verdict on whether a distilled 27 B closes enough of the gap to matter — because if it does not, that is a finding too, and it points at bigger models rather than more training.

A page that promised numbers before measuring them would violate the one rule every page in this series runs on: every figure is a dated, verified snapshot. The blank space here is that rule, applied to ourselves.

LoRA — Hu et al., 2021 (arXiv:2106.09685). QLoRA — Dettmers et al., 2023 (arXiv:2305.14314). Knowledge distillation — Hinton et al., 2015 (arXiv:1503.02531). Qwen3.8-27B weights — Hugging Face model card. Training tooling — Unsloth (QLoRA on consumer GPUs), Ollama (local serving). The rest of this series — the model, the hardware, the interface.