StoryForge — Text-Style LoRA

The question

Style LoRAs are usually trained on raw text. I wanted to test something specific: if you annotate each scene with structured control tags — situation, tone, setting, dynamics, and so on — and train on instruction → scene, does the model actually become steerable? Can you ask for Tone: tender, Setting: docks and get it, in a way a plain-text LoRA can’t? To answer that honestly you need a controlled comparison, not a vibe check — so the project is built around an A/B where two models differ in exactly one variable.

The pipeline

Most of the work is upstream of training — building and curating the data, then setting up a clean comparison:

raw scenes (SQLite)
   │
   ▼
 profile + filter        keyword filter, length gating, per-scene scoring
   │
   ▼
 dedup + diversify       MinHash/LSH near-dup removal, MMR selection
   │
   ▼
 plain corpus + manifest scene = one example; manifest records scene_id + split
   │
   ▼
 tag the SAME scenes     discover → curate closed vocab → closed classification
   │                     (LLM tagger, guided-JSON, ~210 labels across 7 facets)
   ▼
 tagged corpus           {prompt: instruction-from-tags, completion: scene},
   │                     aligned 1:1 to the plain corpus, same split + seed
   ▼
 train two QLoRAs        identical hyperparameters + seed; only the corpus differs
   │
   ▼
 compare + export        pick best checkpoint, merge, GGUF for local (ollama) use

A few decisions I think are the point

• Tagging as classification, not free extraction. An open discovery pass harvests candidate descriptors, I curate the vocabulary by hand, then a closed pass assigns only in-vocabulary labels with guided-JSON decoding. That keeps labels consistent and learnable instead of a long tail of one-off phrasings.
• A clean A/B. Both corpuses are built from the same selected scenes, in the same train/val split, with the same seed — aligned by scene_id — so the two trained models differ in one thing: the presence of tags.
• Scene = one training example. No chunk-splitting, so the tag↔example mapping stays 1:1 and there are no splice artifacts; scenes are length-gated to fit a 2048 window with their tag prompt.
• Overfitting-aware training. Attention-only LoRA targets (q/k/v/o, not MLP), alpha == r, dropout, one epoch, validation watched. An earlier multi-epoch run overfit and broke scene logic; this config didn’t — val loss fell to a plateau without rising.
• Cutting what didn’t work. An earlier coherence-scoring stage got removed after a control test (injected-splice detection) caught 0/10 — it added no signal, so I dropped it rather than keep it for appearance.

Honest limitations

I’d rather be straight about where this stands. The A/B isn’t perfectly sterile — the plain corpus trains as raw text and the tagged one as a masked instruction, so the two differ in format as well as tags; it’s a comparison of two approaches more than a pure ablation. Evaluation is currently qualitative (reading generations); the obvious next step is a quantitative metric — a classifier checking whether generations obey the requested tags, or a blind win-rate. And the scale is modest, ~1900 scenes: enough for style, not “big data”.

Where it landed

Both runs trained healthily — train and validation loss fell together and plateaued without overfitting — on a 12 GB RTX 3060 (8B QLoRA, 4-bit, 2048 window), with an end-to-end path from raw data to a merged GGUF model runnable locally through ollama. Whether the tagged model is meaningfully more steerable is the open question the qualitative results point at and the next, quantitative pass is meant to settle. Code and full write-up are on GitHub.

I can’t share dataset or model because of its source.