D1Does it need AI at all?
What I decide
The problem comes first, then the reason a model is the right tool for it. A model earns its place when the task is too varied for rules, when a wrong answer can be caught or does little harm, and when it moves one outcome that is measured.
Applied
Yexit: the first slice ships reviewed questions and no model. Generation comes once the save flow is proven.
D2What does good mean, in numbers?
What I decide
A minimum quality bar, agreed before the build with engineering and the people who will use the feature, and raised release by release. An eval set measures it: 30 to 50 real inputs with the expected output, labelled by people who know the domain. Model scores sit under one product outcome, and the outcome decides. If the bar is never reached, the feature does not ship.
Applied
MSc thesis: the evaluators got identity and oracle tests before any score was trusted.
D3Prototype first, with real inputs
What I decide
A working prototype in front of users before the spec is final. It stays a prototype while requirements keep moving, value is unproven or failures are common. It moves to engineering when users ask to use it for real work, failures are the exception and success is defined. The handoff is the prototype, the eval set, the known failure modes and the prompt that worked.
Applied
Yexit: prototypes with their failure states (a send failing, an upload failing, offline, microphone denied) and a handoff that says what is functional and what is simulated.
D4Design for the wrong answer
What I decide
What the user sees when the model is wrong, unsure or blocked. AI output is labelled and editable, users are told what errors to expect, and uncertain cases fall back to something safe. Prompt injection and bias tests are written into the spec, so that they get done.
Applied
Yexit: model drafts are candidates only. Shared questions pass software checks and an editor. Custom topics fall back when too few good questions come out, and the personalised suggestions are labelled Concept.
D5Draw the data boundary
What I decide
What the model may receive, what it must never receive, and a test that proves it. Whether user data may train anything is settled up front, and the data practice has to be one the team would describe in public. Before user data goes into any AI tool, its privacy terms are checked.
Applied
Yexit: the question generator never sees answers. The first acceptance test checks that no answer content reaches a question or inference endpoint.
D6Price it before it ships
What I decide
Cost per request, per user and per successful task, at the expected volume, as part of choosing the model. AI margins are thinner than those of ordinary software, so a spending limit is agreed with whoever owns the budget before launch. A feature with several model steps loses reliability at every step, and every retry costs money.
Applied
Sprint tool: a planning page for AI features that turns running cost and maintenance effort into the sprint capacity left, with optional math for agents.
D7Launch small, watch, roll back
What I decide
A staged release: first in the shadow of the current version, then to a small group, then wider. A weekly sample of real outputs graded with the eval rubric, and an alert when a metric crosses its threshold. Model versions pinned. Prompt changes treated like code: versioned, run against the eval set, with a named owner for the rollback.
Applied
Yexit: custom topics are scoped as a pilot behind a flag. Production monitoring has not started, because the product is pre-launch.