Phase 2 · Portfolio Extension · Agentic AI Product Design
DORA DevOps Copilot
Agentic AI extension for the DORA Metrics Explorer — multi-step reasoning, tool orchestration, voice interaction, and stakeholder-ready DevOps insights
Agentic AI
Tool Orchestration
LLM-RAG
Voice UX
DevOps Analytics
Flask · Gunicorn · Render
Launch production app
Standalone Flask app in its own GitHub repo, deployed on Render. This page is the portfolio case study.
Production app lives in a separate GitHub repo (dora-devops-copilot), deployed on Render.
This portfolio page is the case study; the app runs at its own URL.
Three repos, one story. Phase 1 RAG is in AI-Powered-DORA-Metrics-Explorer. Phase 2 copilot is a standalone app repo (dora-devops-copilot). This portfolio page is the case study and launch pad — not where the app code lives.
1. Phase 1 vs Phase 2 — What Changes
Phase 1 (current prototype)
- Single-turn NL query → RAG → LLM answer
- User must know what to ask
- No multi-step analysis
- Text UI only
- RAG is the entire brain
Phase 2 (agentic extension)
- Goal-oriented session → plan → tools → synthesis
- Copilot proposes next analyses
- Chains metric lookup + benchmark + recommendation
- Optional voice in / voice out
- RAG becomes one tool among several
2. Product Vision
DORA DevOps Copilot helps engineering leaders and platform teams understand delivery performance through conversational, multi-step analysis — not just answering “what is our CFR?” but “how is our deployment performance this quarter, where are we vs industry, and what should we fix first?”
Example goal: “Review our Q2 DORA performance and tell me the top bottleneck.”
The copilot fetches metrics → compares benchmarks → retrieves best-practice guidance → ranks improvement actions → returns a concise briefing (spoken or written).
3. How It Would Work — Runtime Flow
User (text or voice)
│
▼
┌─────────────────┐ Phase 2 addition: optional STT / TTS
│ Flask /agent │◄──── Web Speech API or Whisper (input)
└────────┬────────┘
│
▼
┌─────────────────────────────────────────┐
│ DORA DevOps Copilot (Orchestrator) │
│ · Parse intent & goal │
│ · Build step plan (ReAct-style loop) │
│ · Session memory (metrics discussed) │
│ · Guardrails (no fabricated metrics) │
└────────┬────────────────────────────────┘
│ calls tools
┌────┴────┬────────────┬──────────────┬─────────────┐
▼ ▼ ▼ ▼ ▼
get_dora compare_ rag_dora detect_ format_
metrics benchmark best_practice anomaly briefing
(CSV/API) (tiers) (vector DB) (trends) (voice/text)
│
▼
LLM synthesis (Ollama / LLaMA 3) — grounded on tool outputs only
│
▼
Response + optional follow-ups (“Want a breakdown by team?”)
4. Agent Capabilities (7 tools)
4.1 get_dora_metrics
Reads deployment frequency, lead time for changes, change failure rate (CFR), and mean time to restore (MTTR) from CSV/API for a given period or team.
Why agentic: Copilot decides which metrics to pull based on the user goal — not the user spelling out every metric.
4.2 compare_benchmark
Maps current values to DORA performance tiers (Elite / High / Medium / Low) using industry reference bands.
Output: “CFR 18% — below High tier target (≤10%).”
4.3 rag_best_practices (Phase 1 RAG, repackaged)
Retrieves DevOps/DORA guidance from embedded docs (Sentence Transformers + vector store). Phase 1 RAG logic moves here as a tool, not the whole app.
4.4 analyze_trend
Compares current period vs prior period; flags improving / degrading metrics.
4.5 identify_bottleneck
Ranks metrics by gap-to-target and trend severity; returns top 1–3 focus areas with rationale.
4.6 generate_action_plan
Produces 3–5 prioritised recommendations grounded in tool outputs (not hallucinated metrics).
4.7 format_briefing
Formats response for channel: voice (≤45 sec, short sentences) or dashboard (structured markdown + numbers).
5. Intent Library
| Intent | Example utterance | Tool chain |
|---|
| INT-01 Metric status | “What’s our change failure rate this month?” | get_dora_metrics → format_briefing |
| INT-02 Performance review | “How did we perform in Q2?” | get_dora_metrics (all four) → compare_benchmark → format_briefing |
| INT-03 Bottleneck diagnosis | “What should we fix first?” | get_dora_metrics → analyze_trend → identify_bottleneck → rag_best_practices → generate_action_plan |
| INT-04 Benchmark compare | “Are we elite on deployment frequency?” | get_dora_metrics → compare_benchmark |
| INT-05 Best practice | “How do teams reduce lead time?” | rag_best_practices → format_briefing |
| INT-06 Follow-up | “Break that down by service.” | Session memory → get_dora_metrics (filtered) → format_briefing |
6. Sample Conversation
User: Review our last quarter DORA metrics and tell me the biggest problem.
Copilot (planning): Fetch Q2 metrics → compare to benchmarks → identify worst gap → retrieve improvement guidance.
Copilot (response): In Q2, your change failure rate was 18% — above the High-performer threshold of 10% and worsening vs Q1 (+4 pts). Deployment frequency and lead time improved. The primary bottleneck is release quality in the payments service. Top actions: (1) expand pre-prod validation, (2) tighten rollback automation, (3) add CFR monitoring per team. Want a detailed breakdown or an action plan for engineering leads?
7. Voice Layer (optional Phase 2B)
| Input |
Web Speech API (browser) or Whisper API/local — same /agent endpoint receives transcribed text |
| Output |
Browser speechSynthesis or TTS API; copilot uses format_briefing(mode=voice) for shorter responses |
| UX |
Push-to-talk on Flask UI; transcript shown alongside spoken answer for accessibility |
| Portfolio value |
Demonstrates voice-as-primary interaction for DevOps standups / manager briefings |
8. Proposed Code Changes (GitHub repo — for your review)
These changes extend the existing Flask app without replacing Phase 1 behaviour. You can keep /ask (RAG) and add /agent (orchestrator).
8.1 Suggested folder structure
AI-Powered-DORA-Metrics-Explorer/
├── app.py # add /agent route; keep existing /ask
├── agent/
│ ├── orchestrator.py # plan → tool loop → synthesize
│ ├── intents.py # intent routing rules
│ ├── session.py # Flask session / conversation state
│ └── tools/
│ ├── metrics.py # get_dora_metrics, analyze_trend
│ ├── benchmarks.py # compare_benchmark, identify_bottleneck
│ ├── rag.py # move Phase 1 RAG here
│ └── briefing.py # generate_action_plan, format_briefing
├── rag/ # existing embeddings + vector store
├── data/ # DORA CSV inputs
├── templates/
│ └── index.html # add voice button + agent chat UI
└── static/
└── voice.js # Web Speech STT/TTS hooks
8.2 New API contract
POST /agent
Request: { "message": "Review Q2 DORA performance", "session_id": "...", "channel": "text|voice" }
Response: {
"answer": "...",
"metrics_used": ["cfr", "deployment_frequency"],
"tools_called": ["get_dora_metrics", "compare_benchmark", "identify_bottleneck"],
"follow_ups": ["Want a breakdown by team?"],
"voice_summary": "..." // optional short form for TTS
}
8.3 Orchestrator loop (conceptual)
def run_agent(user_message, session):
intent = classify_intent(user_message, session)
plan = build_tool_plan(intent) # e.g. [get_dora_metrics, compare_benchmark, ...]
tool_results = []
for step in plan:
result = execute_tool(step, session) # returns structured JSON, not prose
tool_results.append(result)
if result.requires_clarification:
return ask_clarifying_question(result)
answer = synthesize_with_llm(user_message, tool_results) # Ollama/LLaMA 3
return format_briefing(answer, channel=session.channel)
8.4 What stays from Phase 1
- Flask app shell, deployment (Render/Railway)
- Ollama + LLaMA 3 for generation
- Sentence Transformers + RAG corpus (now invoked via
rag_best_practices tool)
- Pandas/CSV metric loading
- Existing UI screenshot / dashboard — extended with agent chat panel
8.5 What Phase 2 adds
- Agent orchestrator with tool calling and session memory
- Structured metric tools (grounding — reduces hallucination)
- Multi-step plans for complex questions
- Follow-up suggestions
- Voice input/output hooks
- Logging: tools called, metrics referenced (audit/debug)
9. Guardrails
- Metrics must come from tools — LLM cannot invent CFR/MTTR numbers; synthesis only uses
tool_results JSON.
- Citation for best practices — RAG tool returns source snippets; copilot references them in recommendations.
- Confidence fallback — if intent unclear, ask clarifying question instead of guessing.
- Voice brevity — voice responses capped at ~3 sentences unless user asks for detail.
- Human review — action plans labelled as “suggested”; not auto-applied to CI/CD systems.
10. Success Metrics
| Metric | Phase 1 baseline | Phase 2 target |
|---|
| Questions requiring multiple manual prompts | High (user chains asks) | ≥70% resolved in one agent session |
| Metric hallucination rate | Risk in pure RAG | 0% — tools-only numbers |
| Time to bottleneck insight | Manual analysis | < 30 seconds per review goal |
| Follow-up engagement | N/A | ≥40% sessions use suggested follow-up |
11. Implementation Roadmap
| Phase | Scope | Effort (est.) |
|---|
| 2A | Tool modules + /agent route + orchestrator (text only) | 1–2 weeks |
| 2B | Session memory + follow-ups + bottleneck ranking | 3–5 days |
| 2C | Voice layer (Web Speech) + voice briefing formatter | 2–3 days |
| 2D | Polish UI, logging, demo video for portfolio | 2–3 days |
12. Relationship to Phase 1
| Phase 1 |
LLM-RAG prototype — natural-language Q&A over DORA metrics and best practices. Hands-on Flask/Ollama build. |
| Phase 2 (this project) |
Agentic copilot — standalone repo dora-devops-copilot, deployable on Render. |
13. Repo structure
| Repo | Purpose |
|---|
portfolio | Main website, navigation, this case study page |
dora-devops-copilot | Runnable Flask app — deploy this to Render |
AI-Powered-DORA-Metrics-Explorer | Phase 1 LLM-RAG prototype |
14. Publish the app (separate repo)
# 1. Create repo on github.com → name: dora-devops-copilot
# 2. Push from your Mac:
cd /Users/knottrail/Documents/GitHub/dora-devops-copilot
git init
git add .
git commit -m "Initial commit: DORA DevOps Copilot"
git branch -M main
git remote add origin https://github.com/YOUR_USERNAME/dora-devops-copilot.git
git push -u origin main
# 3. Render.com → New Blueprint → select dora-devops-copilot repo
# (or deploy via Docker if GitHub OAuth fails)
# 4. Update portfolio/assets/apps-config.js with your Render URL
15. Tech stack
| Component | Technology |
|---|
| App backend | Flask 3 + Gunicorn + pandas |
| Hosting | Render (or Docker) |
| Portfolio | GitHub Pages (links to app — no app code here) |
| Voice | Browser Web Speech API |
Skills Demonstrated
Agentic AI Design
Tool Orchestration
LLM Grounding
RAG as Tool
Voice UX
DevOps Product Thinking
Flask Architecture
Technical Specification
App repo: ·
Live app: