AI Workflow Design
This doc is the technical-product bridge between “we use AI” and “here’s exactly how the AI behaves, when it’s trusted, and what it costs.” It’s the document I’d expect to defend line-by-line with an ML engineer in a design review.
Shared pipeline for creative generation and performance recommendation:
Input → Model Call(s) → Confidence Scoring → Lane Assignment → Human Review (if required) → Action → Feedback Loop
1. Purpose
Two AI-driven capabilities exist: creative generation and performance recommendation. Both share the pipeline above. This doc defines each stage, the model limitations we design around, and the cost-control mechanisms baked into the pipeline rather than bolted on after.
2. Model selection & routing
Principle: match model cost/capability to the risk and reversibility of the output — not “use the best model everywhere.”
| Task | Model tier | Rationale |
|---|---|---|
| First-pass creative generation (3–8 candidates) | Lower-cost, faster | High volume, low individual stakes — filtered before a human sees them |
| Refinement of top-N candidates | Higher-capability | Low volume; higher quality bar since these reach the reviewer |
| Drift detection & recommendation reasoning | Higher-capability, strict output schema | Needs defensible rationale over noisy time-series — what a marketer will actually judge |
| Brand-safety wordlist screening | Deterministic rules (not a model) | Compliance-critical checks must not depend on probabilistic behavior |
Known model limitations we design around
- Inconsistent scoring: never surface a raw model confidence % — smooth against a calibration set and bucket High/Medium/Low.
- Hallucinated specificity: reasoning prompts may only reference fields present in input data; outputs validated against source metrics before display.
- Prompt injection via product feed: feed content is untrusted input — delimited data section, never concatenated into instructions; brand-safety still runs.
3. Prompt design principles
- Structured in, structured out — JSON-schema-constrained responses so code can validate rather than regex-parse.
- Few-shot from the account’s own top performers — not generic industry examples.
- Explicit negative constraints — no unverifiable superlatives, no pricing claims absent from the feed, tone bounded by config.
- Separate reasoning prompts from generation prompts — different templates and (per §2) different model tiers.
4. Confidence scoring methodology
Confidence is not the model’s raw self-reported score. It’s a composite:
- Model self-score — one weak signal among several; never sufficient alone.
- Historical acceptance rate — how often similar past suggestions were approved. New accounts default conservative (enforces “can’t reach Auto-apply”).
- Data volume — <7 days or <100 conversions → confidence-capped regardless of model claim.
- Signal strength — how far from target, and how stable over the lookback (one-hour spike ≠ three-day trend).
This composite feeds lane assignment (§5) and is recalibrated periodically against outcomes (§7).
5. Review lane logic
| Lane | Criteria | Example |
|---|---|---|
| Auto-apply | High confidence AND blast radius below cap AND trust tier ≥ established | 5% budget nudge on a campaign with 60 days of stable accepted history |
| Suggest | Medium confidence, OR blast radius above cap, OR newer account | Most v1 traffic — Suggest-only by design |
| Escalate | Low confidence, OR insufficient data, OR compliance-sensitive action | Recommendation with <100 conversions of underlying data |
Lane thresholds are configurable per account (US-11) but bounded by hard platform-level ceilings (e.g. no account can enable Auto-apply above a fixed % of daily budget). That ceiling is not exposed as configurable.
6. Cost controls
- Monthly generation cap enforced at the request layer before any model call (FR-13).
- Tiered routing — if 8 first-pass candidates and top-2 refined, refinement cost is ~25% of “best model everywhere.”
- Caching — cache key = feed-content hash + prompt template version (prompt updates invalidate correctly).
- Circuit breaker — on provider latency/error spikes, degrade to “generation delayed” rather than retry into an incident.
7. Feedback loop & evaluation
- Every approve/edit/reject feeds the historical-acceptance term in confidence scoring.
- Calibration reviewed on a cadence against held-out outcomes; adjust thresholds before blaming “need a bigger model.”
- Prompt template changes are versioned and logged against every output (NFR-4) so regressions are traceable.
8. Deliberately out of scope
- Model fine-tuning / custom training — v1 uses off-the-shelf APIs.
- Multi-language support.
- Sub-minute recommendation latency — hourly refresh is intentional.
More in this series