适合该工作的正确工具

Good collaborative storytelling involves an enormous number of complex decisions, all interdependent and changing on the fly as the situation shifts. At Hidden Door, our narrative systems break that down into a modular, interrelated set of tasks that draw on structured story and world data to make these decisions and render them for the player—a storytelling harness.

A matter of perspective

Sometimes the tasks break down very simply. Here’s a bite-sized one: convert some text from third person perspective to second-person perspective, for a given character. Easy, right? Wrong! Often it is trivially straightforward, requiring simple pronoun replacement and verb tense changes; it’s easy even if you have no additional context about the scene or its characters.

Jamie sets off from the platform, her shoes crunching on gravel as the station lamps flicker behind her. The village road unspools through hedgerows heavy with blackberry, the smell of woodsmoke drifting from somewhere ahead. A painted sign swings on rusted chains: Melete Café — ½ mile.

Changes to:

You set off from the platform, your shoes crunching on gravel as the station lamps flicker behind you. The village road unspools through hedgerows heavy with blackberry, the smell of woodsmoke drifting from somewhere ahead. A painted sign swings on rusted chains: Melete Café — ½ mile.

Unfortunately, it’s not always so simple. When one or more NPCs share a pronoun with the target character, groups of characters and they/them pronouns coexist, or an action is genuinely ambiguous, it gets trickier:

Behind Sage, two porters haul luggage past, their conversation fragmenting as they catch sight of them: "—said they found him in the library stacks, didn't they? No mark on him—" They fall silent, averting their eyes and scuttling further into the building.
As their footsteps fade Jamie glances at Frederika, raising a jagged eyebrow, but Frederika only takes a measured sip of her coffee and shrugs, as if to say she should expect such things to happen regularly here.
“Jamie, you should be prepared for rumors that it’s your fault,” Sage points out. “Happened to me last year. Frederika got lucky on timing, but they found other things to blame her for.”
They are all unwelcome here, just in different ways. The scarred newcomer’s mind whirls.

VS:

Behind Sage, two porters haul luggage past, their conversation fragmenting as they catch sight of your group: "—said they found him in the library stacks, didn't they? No mark on him—" They fall silent, averting their eyes and scuttling further into the building.
As their footsteps fade you glance at Frederika, raising a jagged eyebrow, but Frederika only takes a measured sip of her coffee and shrugs, as if to say you should expect such things to happen regularly here.
“Jamie, you should be prepared for rumors that it’s your fault,” Sage points out. “Happened to me last year. Frederika got lucky on timing, but they found other things to blame her for.”
You are all unwelcome here, just in different ways. Your mind whirls.

Here we have: three different groups of characters referred to as they/them collectively; two gender-neutral names, where Sage’s pronoun could either be unrendered or a singular they; two characters who happen to share a pronoun, each referred to by pronoun within the same sentence; AND names and pronouns in dialogue.

In some cases, there isn’t even one right answer! The first “them,” referring to the group of Sage, Frederika, and Jamie, could be converted to “you” (plural), “the three of you,” or “your group” and similar variants. There’s also a noun phrase referring to the perspective character, which often trips up language models.

This example was lightly edited to pack most of the trickiest cases into one turn, but it’s not uncommon to see them occur with this much density in a complex multi-character scene.

What tool to use?

Frontier models like Claude Fable or GPT Astra can handle many complex steps and directives simultaneously, but output quickly starts to feel aimless and same-y, and latency also becomes a big problem when you’re playing an interactive game. We’re quite certain that LLMs aren’t good at storytelling, people are. So even though a variety of tasks use LLMs within our story systems, filling in the gaps or bending the genre around our deliberately handwritten narrative structures, reaching for the latest-and-greatest model is very rarely the best way to solve a problem. In this case, frontier models would be a bit like using a really, really big sledgehammer to grind pepper: slow, expensive, and absurdly overbuilt for the task.

Why do that when we’ve got a beautiful workbench full of gleaming tools, and the expertise to select the best one for the job? The first step is breaking down larger pieces of the system into the right-size tasks, so we can use those tools with precision and accuracy.

We always reach for the simplest, least-effort intervention that gives us the results we want.

In this case, simple heuristics and string replacement, maybe with some basic NLP techniques like stemming, can handle the simple cases beautifully but will immediately crash and burn on more complex sentences.

Classical NLP could do the job: parse the sentence into tagged structures with something like spaCy; named entity recognition to pull out the different characters by name, pronoun, and epithet; traverse the dependency tree to determine which verbs need conjugated differently, and which type of pronoun it is (possessive, personal, etc) so we can convert it to the appropriate you/your/yours; detect and split out dialogue so we can avoid converting names and pronouns within it.

Building out that pipeline is straightforward, but it’s not an insubstantial engineering project. It would come with a long tail of edge cases, likely require some custom tuning of the models with hand-tagged examples, and by the time you run the sentence-parsing and named entity recognition pipeline on the text, it’s going to be much slower than a smaller LLM. Often breaking a task down into a classical NLP framework makes it faster and more reliable, but not this time.

Because the perspective conversion task is fundamentally a flexible language manipulation problem requiring just a little bit of judgement, it actually is a great candidate for using a language model — it just doesn’t suit a huge one. So we turned to our more standard menu of LLMs that offer very low latency and high throughput. What we reach for changes over time, but the options for most tasks tend to range from the smaller Qwen and gpt-oss models up to gpt-oss-120b, the most capable flagship model that reliably puts out the kind of speed we need for smaller tasks.

Squeezing out more performance

Unfortunately, on sentences with high complexity and ambiguity, even gpt-oss-120b consistently made mistakes. At a full second per conversion on average, even though it runs in parallel with other tasks, we weren’t willing to trade off any more latency for better performance—it’s a small, specific mechanical task that should never slow down a turn.

The first step is always structured evals: gather real-world examples into a dataset, score it systematically, and get real human eyeballs on the results to give feedback on pass/fail, which you can then group into patterns for improvement.

Improving the context fed into the prompt with retrieval-augmented generation (RAG) is a typical next step, but in this case there wasn't much to do. Other than the sentence to convert, the input just needs basic context on the characters, their pronouns, and the current situation. For prompt-tuning we tried the standard set of tools: different prompting strategies, few-shot examples, layering in heuristics for the obvious cases or to offload from the generative part of the task. We saw some improvement, but still had enough conversion mistakes to be noticeable. What to do?

In between prompt tuning and throwing up your hands and training your own model is the broad spectrum of fine-tuning: updating some or all of the learned parameters of a model to change its outputs, improving its performance at a particular task. That used to be a pretty expensive proposition. Fine-tuning the whole model meant a significant investment in engineering infrastructure, compute time, training data, experimentation, and maintenance.

A couple years ago, especially for small tasks like this, the wise bet was almost always to just focus on prompt adjustments. Models were changing and improving quickly enough that your fine-tuning efforts were likely to get you some improvement in behavior at great expense and the cost of lock-in to a now-custom model that'd be superseded shortly by an unmodified new model that could one-shot your task.

But language models have come a long way in the last few years! That tradeoff doesn’t necessarily hold true anymore.

There’s now a much more diverse ecosystem of open-weight models to choose from. Advances in attention mechanisms and context handling, sparse architectures like mixture-of-experts, and improved post-training and reasoning techniques mean that relative to a few years ago, much smaller models can punch way above their weight in capability. A parameter is not a parameter anymore.

A tool for fine adjustments

LoRA (Low-Rank Adaptation) is an excellent next step up the intervention ladder because you don’t need to update the weights of the entire model (though you do still need access to them). Instead, you attach a small set of additional trained parameters that adjust the output behavior in a new direction. It’s like sliding a physical filter onto a camera to change what comes out of the shot, rather than rebuilding the whole camera from scratch; it’s smaller, cheaper, reversible, and you can have different filters for different purposes, swapping them out or even stacking them as the task requires.

That means that for the same small or medium-sized open-weights base model, you can train several different LoRA adapters of negligible size, each trained for a specific task (or style, or IP). This makes it an excellent tool for achieving exceptional task-specific performance with little additional cost in model size or overfitting. In the extreme, imagine a tiny language model that fits on a phone, running locally, with different tiny adapters to help it specialize at each step of the story system flow—swapping out the adapter layer when comparing plot options vs. adhering to canon vs. revising the style for different genres.

Our sentence-conversion task is an excellent use case for small-scale fine-tuning with LoRA. It’s easy to judge, with heuristics to catch the straightforward failures so humans can evaluate the ambiguous cases. We have plenty of historical examples from gameplay on Hidden Door. And it’s easy to manually add synthetic data — constructed or transformed examples that cover the most difficult edge cases we care about improving performance on.

Modern tooling makes local fine-tuning achievable even on fairly ordinary hardware, like Apple’s MLX stack. At such small scales, the experiment loop fits on a few overnights on a dev laptop.

What we did

If we’re going to put in the effort to train and fine-tune, we want to use the smallest model possible for the job. We started our initial scoping with the Qwen and llama families, both of them familiar to us, open-weight, and with a few different options for generation and number of parameters. For models of similar size, the llama models performed significantly worse on the task at baseline, so we focused on Qwen, starting with Qwen3-1.7b and Qwen3-4B.

We used all-linear LoRA adapters and trained overnight on an M5 Macbook Pro (!!), using a much more minimal prompt than our production system—if we’re committing to the effort of fine-tuning, we can let the adapter itself learn the pattern of all the rules, instructions, and edge case corrections that would normally need to go into the prompt. Instead we just feed in the basics: original sentence, characters present and their pronouns, target perspective character, and the text to convert.

Before tuning, both models performed much worse than the production prompt on gpt-oss-120b. In particular, the 1.7B model regularly produced completely useless results. After training, the LoRA-adapted Qwen3-1.7B model scored approximately 97% passing on a held-out evaluation set of real production text conversion samples, beating the gpt-oss-120b version (94% passing) with about 70x fewer parameters and a far shorter prompt.

The whole experiment took two engineering half-days of effort to set up and score the data, one night of crunching on a dev laptop, and a bit of time to look at the results. And now we have a very small LoRA adapter that lets a 1.7B dense model outperform a 120B mixture-of-experts model; this substantially reduces the infrastructure demands of hosting and serving this model while simultaneously improving the quality of results.

The specific task in this example is one tiny slice of a complex system. But the same approach that helps a small model nail tricky pronoun conversions can also let it specialize in a house style or a specific canon's rules. When the cost of specialization is a couple days of engineering and compute time, custom-tuning for each story world is a genuinely achievable proposition! As always, the hard part is knowing which parts of a story need that kind of tool, and when to reach for something else from our well-maintained tool chest.