为改变而打造

The AI hype cycle would like you to be swept away by a continuous wave of new model releases. Every new version is the latest and greatest! Better than ever, at everything! But our team thinks that's the wrong thing to focus on when building with AI.

Models are a commodity. The more capable they get, the more interchangeable they become. This is counterintuitive, but ends up being true for most general language tasks. If you use language models instead as just another capable tool, you can get down to the important part: making cool things using your real human opinions and taste.

And we would know! We've been building story systems that use machine learning as a tool, including “classic” NLP methods and large language models, since the GPT-2 era. One of our very first language generation systems used offline generation and an embeddings ranking workflow to find the best fit across authored templates. Today that same step is a small LLM task with programmatic output wrappers, and it’s gone through half a dozen different LLMs over time as the available options have changed.

Here’s how we do that across the entire product without losing our minds.

Models and modularity

Most importantly, the modular architecture of our narrative system makes it highly adaptable. For example, the AI DM in our agentic story harness is driven by one very capable model, but the tools and information it uses are drawn from the output of many smaller tasks — themselves consisting of heuristics, classical ML, and other LLMs acting on Hidden Door’s structured narrative world model.

Each of those specialized tasks gets its own parameters, fallback behaviors, and distinct definition of what “good” looks like. That bite-sized focus means testing and evals are fast. We can therefore change one piece of the system without applying the same change to all the others — either reactively, as when a bread-and-butter model has an upcoming deprecation, or more often proactively, when interesting new models are released (or when we decide we’d like to fine-tune something of our own).

Splitting out those tasks also means room to make finer-grained choices about quality and latency from the user’s perspective. This is where product design choices can both buy you more tolerance and point to whether your fallbacks should pay in time, money, or quality. In the story systems running on our narrative world model:

• A normal turn has the tightest time budget, because every second is directly felt by the player. It runs on the fast timeline as described in A time-traveling story harness, and most of its quality comes from planning that already happened in the slow timeline. We prioritize instruction-following, low time-to-first-token, and fast throughput.
• Sidebar chat with NPCs or the Narrator must be conversational in the moment, so it uses smaller, quicker models and gets reconciled with everything else afterward. We can pull this off because the context and task are very focused.
• Starting a story means the player has little existing text to read while they wait, and it can also be a player's very first interaction with the system, so latency disproportionately hurts here. Here, we hedge early against slow providers and do speculative pre-work as much as possible, spending a little more to get guaranteed speed and quality.
• A scene ending has more room, both because the player has much more to read in the final beats and because a moment of transition doesn’t feel as bad to wait for. While the player reads those final beats, we’re wrapping up the scene and preparing options for the next one.
• Transforming narrative into other media, such as letting a combat action kick off a fighting minigame (and then rendering the minigame’s results back in the narrative!), is another moment of transition where you get a little more patience in exchange for the significance and magic of the outcome. These kinds of transformations are one of the coolest capabilities of our narrative world model.

A division of tasks that works for you

Automated evaluations and well-delineated tasks simplify our team’s ability to make small adjustments for incremental model changes. Once in a while there’s enough of a leap that something new becomes possible, which is when we might really shake up the tasks into new combinations or architectures. Explore and exploit! For example, the recent resurgence of interest in fast classifiers and transformers-based architectures that return probabilities rather than text (a la Jev) inspired us to explore shifting some tasks from embeddings-based similarity + LLM choice to a one-step classifier.

It’s easy to get excited by these kinds of newly released models and architectures, especially when there’s tangible jumps in what they can do — it really is exciting sometimes! But at the end of the day, they are just a tool. Newer, “more capable” models are not necessarily better for specific tasks, and certainly not for creative tasks.

Our storytelling system not only needs to make coherent, satisfying decisions about dozens of interdependent narrative tasks, it must also write and edit prose that real humans want to read. Model and provider churn could be a constant source of re-tuning and woe when new models are released or a deeply relied-upon version gets deprecated. But it’s not, because we didn't build a system around model capabilities.

We built a system grounded in clear understanding of the problems we were trying to solve and a team of thoughtful people with opinions about how to best do that, which means we can always reach for the most appropriate tool for the job.

Everyone working in AI wants to make better models – us included! And our system lets our platform benefit not just from our own innovation but from the progress made by the frontier labs and the open source community. It’s a wild and wonderful time to be building products with AI at the core, and a bit of thoughtful architecture means that everyone else’s progress becomes your headwinds, too.