SYSTEM B · QLoRA FINE-TUNE · CLOSED BOOK

Gemma 2 2B · Yu-Gi-Oh QA

Our QLoRA adapter over Gemma 2 2B, trained on 2,683 grounded Yu-Gi-Oh QA pairs — still closed-book. Fine-tuning teaches the answer shape: it answers like a rulings expert, but still cannot recall facts it was never given in its weights.

HuggingFace adapter →

Section 1 · Ask this system live, and see where it lands against the other two.

Ask it live · System B

connecting…

Ask it and read its answer from the fine-tuned weights, in the right ruling register — but still closed-book. No facts are looked up, so the shape is right long before the facts are.

POST /ask · k=5 · this page shows answer B

Answer B

The three-system ladder · biased 60

Where this system lands against the other two on the 60 held-out biased questions, scored blind by a reference-grounded judge out of 10. Green is this system; tan are the other two.

1.32×System B's mean vs the base (5.25 / 3.98). Fine-tuning lifts the score, but modestly — the facts are still missing.

Measured mean judge /10 on the 60 held-out (biased set), scored blind by a reference-grounded judge (gemini-3.1-flash-lite). This system in green; the other two in tan.

System	Biased 60	Unbiased 60
A · base	3.98	1.93
B · fine-tune	5.25	2.62
C · +retrieval	8.05	8.25

Mean /10 on the biased and the equal-split unbiased 60. Paired deltas — fine-tuning A→B: +1.27 (p=0.007 biased; +0.68 marginal, p=0.041 unbiased). Retrieval B→C: +2.80 (p<0.001 biased; +5.63 unbiased).

The lesson in both: fine-tuning teaches the shape of a good answer; retrieval supplies the facts. System A guesses fluently but is right only 1.83/5 of the time. Fine-tuning (System B) fixes the register — groundedness climbs 0.18 → 0.87 — yet correctness barely moves (1.83 → 2.35), because the facts were never in the 20.8M trained weights. Only when the retriever puts real passages in the prompt (System C) does correctness jump to 3.85/5 and the mean reach 8.05/10.

Section 2 · What's inside it, and what built it.

Architecture · Gemma 2 2B

All three systems are the same Gemma 2 2B — 26 layers, grouped-query attention (8 heads sharing 4 KV), a sliding 4,096-token window, and a GeGLU feed-forward. Run 4-bit at inference.

Class	Gemma2ForCausalLM
Layers	26
Hidden size	2,304
Attention	8 heads / 4 KV · dim 256 · GQA
Feed-forward	GeGLU · inner 9,216
Attention window	sliding 4,096 · logit soft-cap
Norm	RMSNorm
Context	8,192 tokens
Vocabulary	256,128
Embeddings	tied input/output
Inference	4-bit NF4

The Gemma 2 2B configuration shared by all three systems.

Training & corpus · what built it

How this system was built: a teacher wrote QA pairs from a curated corpus, a blind judge filtered them, and a QLoRA adapter (0.8% of the weights) was trained on what survived.

Init from	gemma-2-2b-it · 4-bit NF4
Method	QLoRA · rank-16 / α-32
Targets	all linear modules
Trainable params	20.8M LoRA (~0.8%)
Schedule	LR 2e-4 cosine · seq 512 · bf16
Data	2,683 QA pairs · early-stopped ~1 epoch
Val perplexity	3.87 (ref 4.26)
Hardware	Modal L4
Cost	$3.26 · Modal $0.26 + Gemini $3.00

QLoRA rank-16 / α-32 adapter over Gemma 2 2B — 20.8M trainable params, ~0.8% of the model, one early-stopped epoch on Modal L4.

The lesson in both: all three systems are the same 2.6B model — what changes is only what you feed it. The architecture is identical Gemma 2 2B; the QLoRA fine-tune rewrites just 0.8% of the weights and never sees a card fact at test time. The corpus and pipeline built the fine-tune's 2,683 QA pairs, but the decisive facts arrive at answer time through retrieval, not through training.