Master Blueprint: The 6 AI Engineering Capstone Projects

Building production-grade AI systems requires moving beyond isolated tutorials and static script execution. The most effective way to prove engineering capability is to build a single, evolving application over a 6-month period. Each monthly milestone adds a substantive engineering layer—transitioning a simple command-line interface (CLI) into a fully automated, evaluated, retrieved, agentic, containerized, and fine-tuned AI platform.

The core application built across this curriculum is the SRE Runbook Copilot & Incident Operations Platform. This project addresses real-world enterprise infrastructure challenges: parsing technical operational documentation, retrieving precise procedures under incident stress, executing safe read-only diagnostics via agent tools, and delivering audited, low-latency responses.

---

Portfolio Architecture Evolution Summary

---

1. Project 1 (Month 1): SRE Runbook Copilot CLI

1.1 Overview & Architecture

The Month 1 capstone project establishes the software engineering foundation. You build an asynchronous command-line tool that accepts operational incident questions alongside raw Markdown runbooks, executing parallelized LLM completions while guaranteeing structured data validation and token cost auditing.

1.2 Tech Stack & Dependencies

• Runtime: Python 3.12+ managed via uv
• API Client: openai (AsyncOpenAI)
• Validation: pydantic v2 (BaseModel, Field)
• Concurrency: Standard library asyncio

1.3 Directory Structure

1.4 Core Implementation Specifications

1. Schema Definition (src/schemas.py): Define RunbookResponse enforcing three fields:
   • reply (string): The explicit troubleshooting steps or direct answer.
   • cited_section (string): The exact header or section from the Markdown text.
   • covered_by_doc (boolean): True if answered directly by the runbook; False if unmentioned.
2. Asynchronous Engine (src/copilot.py): Implement async def analyze_runbook(...) using AsyncOpenAI. Set response_format={"type": "json_object"} and temperature=0.0.
3. Cost Auditor (src/cost_calculator.py): Extract response.usage.prompt_tokens and completion_tokens. Compute exact financial cost using pricing formulas ($0.15 / 1M input tokens, $0.60 / 1M output tokens for standard mini models).
4. Parallel CLI Entry Point (main.py): Use asyncio.gather() to evaluate multiple incident queries simultaneously without blocking the I/O loop.

1.5 Verification & Success Criteria

• Executing uv run main.py parses Markdown runbooks without throwing unhandled parsing errors.
• Sending an unanswerable query correctly returns "covered_by_doc": false without LLM hallucination.
• Terminal output displays a formatted summary of prompt tokens, completion tokens, total cost in USD, and JSON payload.

---

2. Project 2 (Month 2): Automated LLM Evaluation & Regression Test Suite

2.1 Overview & Architecture

Month 2 introduces continuous evaluation—the defining discipline of professional AI engineering. You build an automated test framework that runs a suite of 30 to 50 real-world incident test cases against your prompt pipeline, scoring answers using deterministic assertions and an LLM-as-a-Judge grading rubric integrated into GitHub Actions CI/CD.

2.2 Tech Stack & Dependencies

• Testing Framework: pytest, pytest-asyncio
• Eval Orchestration: promptfoo CLI or native DeepEval / Ragas
• Judge LLM: gpt-4o (or Claude 3.5 Sonnet) as the evaluator model
• CI/CD Platform: GitHub Actions (.github/workflows/evals.yml)

2.3 Evaluation Dataset Schema (evals/questions.yaml)

2.4 Implementation Specifications

1. Deterministic Test Suite (tests/test_deterministic.py): Assert that output strictly matches Pydantic schemas, response latency is under defined limits, and required technical keywords (e.g., commands, flags) are preserved.
2. LLM-as-a-Judge Evaluator (src/evaluators/judge.py): Construct a grading prompt that passes the Question, Context Document, Ground Truth, and Actual Output to an evaluator model. The judge assigns scores (1–5) across three metrics:
   • Faithfulness: Does the answer contain claims not supported by the document?
   • Completeness: Does the answer include all critical steps from the ground truth?
   • Citation Accuracy: Is the cited section heading correct?
3. CI Pipeline Integration (.github/workflows/evals.yml): Configure a workflow that executes on pull requests. The job fails if the aggregate pass rate falls below 90% or if any critical safety assertion fails.

2.5 Verification & Success Criteria

• Pull requests automatically trigger test execution in GitHub Actions.
• Intentionally degrading the system prompt causes the evaluation suite to fail and block PR merging.
• A formatted evaluation report is generated, displaying per-metric accuracy and latency distributions.

---

3. Project 3 (Month 3): Enterprise Hybrid RAG Engine

3.1 Overview & Architecture

Month 3 transforms the single-file copilot into a enterprise Retrieval-Augmented Generation (RAG) platform capable of indexing hundreds of runbooks, post-mortems, and architectural logs. You implement document chunking, hybrid search (combining dense vector embeddings with sparse keyword matching), and cross-encoder reranking.

3.2 Tech Stack & Dependencies

• Vector Database: Qdrant (via Docker container) or PostgreSQL with pgvector
• Embedding Model: OpenAI text-embedding-3-small or Hugging Face bge-small-en-v1.5
• Reranking Engine: FlashRank or Cohere Rerank API
• Document Parsing: langchain-text-splitters or custom Markdown AST parser

3.3 Database Infrastructure (docker-compose.yml)

3.4 Implementation Specifications

1. Document Ingestion Pipeline (src/rag/ingest.py):
   • Parse Markdown files using structural header boundaries (#, ##, ###).
   • Append document metadata (file path, section heading, last modified date) to each chunk payload.
   • Generate dense vector embeddings and upsert payload records to Qdrant.
2. Hybrid Search Retrieval Engine (src/rag/retriever.py):
   • Execute simultaneous dense vector search (semantic similarity) and sparse search (BM25 keyword match).
   • Merge candidate pools using Reciprocal Rank Fusion (RRF).
3. Cross-Encoder Reranking (src/rag/reranker.py):
   • Pass the top 20 retrieved candidates through a reranker model to compute definitive query-document relevance scores.
   • Select the top 3 highest-scoring chunks to populate the LLM context window.
4. Citation Engine (src/rag/generator.py): Enforce explicit inline source citations in every generated answer, referencing source filename and section header.

3.5 Verification & Success Criteria

• Qdrant web dashboard (http://localhost:6333/dashboard) displays indexed collections and vector payloads.
• Retrieval evaluation metrics achieve > 85% Hit Rate @ K=3 and Mean Reciprocal Rank (MRR) > 0.75.
• Answers accurately synthesize information across multiple separate runbook documents.

---

4. Project 4 (Month 4): Agentic MCP Server & Infrastructure Tooling

4.1 Overview & Architecture

Month 4 shifts the copilot from passive question answering to active incident diagnosis. You build a standardized Model Context Protocol (MCP) server that exposes read-only infrastructure tools (e.g., checking Kubernetes pod logs, querying Prometheus metrics), orchestrated by an autonomous agent loop with strict prompt injection guardrails.

4.2 Tech Stack & Dependencies

• Agent Protocol: Model Context Protocol (MCP) SDK (mcp, FastMCP)
• Agent Framework: langgraph or pydantic-ai
• Guardrails: NeMo Guardrails or custom Pydantic input validators
• Mock Infrastructure: kubernetes Python client or mock SRE environment

4.3 Implementation Specifications

1. MCP Server (src/tools/mcp_server.py): Implement an MCP server exposing three read-only operational tools:
   • get_service_status(service_name: str): Returns current pod status, restart counts, and uptime.
   • fetch_error_logs(service_name: str, lines: int): Retrieves the last N log entries filtering for ERROR or FATAL.
   • check_metric_threshold(metric_name: str): Queries CPU/Memory utilization percentages.
2. Stateful Agent Graph (src/agent/graph.py):
   • Construct a state graph using langgraph featuring nodes: agent_reasoning, execute_mcp_tool, and verify_output.
   • Enforce maximum execution iteration limits (e.g., max 5 loop iterations) to prevent infinite loops.
3. Security & Guardrail Layer (src/agent/security.py):
   • Strip executable commands or shell injection characters from tool arguments.
   • Implement RBAC: restrict MCP server credentials to read-only service accounts (get, list, watch).
   • Require explicit human-in-the-loop (HITL) approval for any hypothetical state-modifying action.

4.4 Verification & Success Criteria

• The agent successfully diagnoses an simulated outage by calling tools in logical sequence (Status $\rightarrow$ Logs $\rightarrow$ Metric).
• Tool-calling evaluations confirm the agent selects the correct tool > 95% of the time.
• Attempting prompt injection attacks via malicious document context fails to hijack agent tool execution.

---

5. Project 5 (Month 5): Production FastAPI Microservice, Tracing & MLOps Pipeline

5.1 Overview & Architecture

Month 5 wraps the RAG and agent engine into an enterprise-grade REST microservice. You containerize the application with Docker, implement distributed tracing using OpenTelemetry and Langfuse, set up semantic prompt caching, and establish fallback model routing for high availability.

5.2 Tech Stack & Dependencies

• Web Framework: fastapi, uvicorn, gunicorn
• Containerization: Docker, Docker Compose
• Observability: OpenTelemetry SDK, langfuse / Arize Phoenix
• Resilience Gateway: LiteLLM Proxy / fallback routing

5.3 Microservice Architecture Specs

1. REST Endpoints (src/api/routes.py):
   • POST /v1/copilot/query: Main streaming and JSON response endpoint.
   • GET /healthz: Liveness and readiness probes for Kubernetes.
   • GET /v1/copilot/metrics: Internal telemetry (latency, token costs, cache hit ratio).
2. Distributed Tracing (src/telemetry/tracer.py):
   • Instrument every API request with OpenTelemetry spans tracking LLM generation time, vector retrieval latency, and tool execution duration.
   • Export execution traces directly to Langfuse or Arize Phoenix dashboards.
3. Semantic Prompt Caching (src/cache/semantic.py): Implement an embedding-based cache layer. If an incoming question has $\ge 0.95$ similarity to a recently answered query, serve the cached answer instantly.
4. Containerization (Dockerfile): Build a multi-stage, non-root Docker container image optimized for minimal security surface area and fast startup times.

5.4 Verification & Success Criteria

• Docker container builds cleanly and runs locally via docker-compose up.
• Langfuse dashboard displays end-to-end execution traces with token counts, latency breakdowns, and step costs for every request.
• Stress testing using locust or wrk verifies p95 response latency stays below 2.5 seconds under concurrent load.

---

6. Project 6 (Month 6): Fine-Tuned Open Model Serving & Benchmark Platform

6.1 Overview & Architecture

The final capstone project demonstrates model autonomy and cost optimization. You deploy open-weights foundation models (e.g., LLaMA-3, Qwen-2.5) locally using vLLM/Ollama, fine-tune a lightweight model for specialized incident classification using Parameter-Efficient Fine-Tuning (PEFT / LoRA), and publish a complete portfolio repository comparing hosted APIs versus self-hosted open models.

6.2 Tech Stack & Dependencies

• Serving Engines: vLLM (production GPU serving) or Ollama / llama.cpp (local CPU/GPU)
• Fine-Tuning Framework: Hugging Face transformers, peft, trl, PyTorch
• Benchmarking & Evaluation: lm-evaluation-harness or custom benchmark script

6.3 Implementation Specifications

1. Local Model Serving Pipeline (src/models/serving.py):
   • Deploy an open-weights model using vLLM with PagedAttention enabled for high-throughput batch inference.
   • Expose an OpenAI-compatible API endpoint on port 8000.
2. Task-Specific Fine-Tuning (src/models/train_lora.py):
   • Collect and format a dataset of 500 incident log summaries labeled with severity levels (CRITICAL, WARNING, INFO).
   • Train a Low-Rank Adaptation (LoRA) adapter module on top of a 3B/8B parameter base model using Hugging Face PEFT.
   • Save and merge the fine-tuned adapter weights.
3. Comprehensive Benchmark Suite (src/benchmarks/evaluate.py):
   • Run identical evaluation datasets across Hosted APIs (GPT-4o-mini, Claude 3.5 Haiku) and Self-Hosted Open Models (vLLM LLaMA-3-LoRA).
   • Generate comparative analysis metrics across three dimensions: Accuracy Score (%), p95 Latency (ms), and Cost per 1,000 Inferences ($).
4. Portfolio Showcase Repository (README.md): Compile the complete 6-month system documentation into an executive-ready GitHub portfolio repository containing architectural diagrams, live metrics, docker deployment commands, and recorded video demos.

6.4 Verification & Success Criteria

• vLLM server successfully initializes and serves requests via OpenAI-compatible API endpoints.
• Fine-tuned LoRA model achieves equal or superior accuracy on incident severity classification compared to base foundation models while reducing inference cost by over 70%.
• Portfolio GitHub repository is fully documented with badges, benchmark tables, setup guides, and system design decision rationales.

---

7. Master Summary: Milestone Deliverables & Portfolio Blueprint