AI Strategy and Data-Driven Decisions

1. AI Opportunity Assessment Matrix

2. Build vs. Buy vs. Partner Decision Tree

flowchart TD
    A[Business decision and baseline] --> B{Does AI outperform a<br/>non-AI option in a bounded test?}
    B -->|No or unknown| N[Redesign, stage learning, or stop]
    B -->|Yes| C{Are use risk, data rights,<br/>security, and legal scope acceptable?}
    C -->|No| N
    C -->|Yes| D{Is proprietary behavior or<br/>workflow control strategically material?}
    D -->|No| E{Can a vendor meet lifecycle cost,<br/>integration, portability, and assurance needs?}
    E -->|Yes| Buy[BUY with exit and monitoring controls]
    E -->|No| Partner[PARTNER or staged procurement]
    D -->|Yes| F{Can we operate, evaluate,<br/>secure, and maintain it?}
    F -->|Yes| Build[BUILD with governed release]
    F -->|No| Partner
    Buy --> R[Reassess on evidence, incidents, cost, and lock-in]
    Partner --> R
    Build --> R
    R --> A

3. AI Maturity Model

flowchart LR
    O[Business outcomes and risk boundary] --> A[Assess strategy, data, technology, workflow, talent, governance, monitoring, and realized value]
    A --> C[Identify the binding capability constraints]
    C --> I[Choose the smallest governed improvement]
    I --> E[Deploy or test with success and stop criteria]
    E --> R[Measure adoption, value, incidents, and residual risk]
    R --> A

4. Use Case Prioritization Framework

Illustrative score = (Impact + Confidence + Ease) / 3

Impact (1-10): Business value if successful
Confidence (1-10): Probability of technical success
Ease (1-10): Speed to production, resource requirements

Illustrative AI Score = (Impact + Confidence + Ease + Data Quality) / 4

Data Quality (1-10):
- 10: Clean, abundant, labeled data available
- 5: Data exists but needs cleanup/labeling
- 1: Data doesn't exist or very sparse

5. ROI Calculation for AI Projects

ROI = (Gain - Cost) / Cost × 100%

Gain = (Revenue Increase + Cost Reduction + Risk Mitigation Value)
Cost = (Development + Deployment + Maintenance)

6. Ethical AI Framework

7. Data Readiness Assessment

8. MLOps Pipeline and Change-Control Framework

flowchart LR
    D[Governed data, labels, provenance, and access] --> B[Versioned build, features, training, and configuration]
    B --> E[Evaluate technical, business, safety, fairness, security, privacy, accessibility, latency, and cost criteria]
    E --> A{Authorized for staged release?}
    A -->|No| X[Revise, document, or stop]
    A -->|Yes| P[Stage deployment with fallback and rollback]
    P --> M[Monitor inputs, outputs, use, outcomes, incidents, drift, latency, and cost]
    M --> C{Material signal or planned change?}
    C -->|No| M
    C -->|Yes| R[Diagnose and open controlled change]
    R --> B
    R --> Y[Rollback, remediate, restrict, or retire]

9. Agentic AI Operating and Control Model

flowchart LR
    O[Principal defines goal, identity, authority, data, tools, limits, approvals, and stop rules] --> P[Agent proposes bounded plan]
    P --> G{Plan and next action within current authority?}
    G -->|No| H[Request human decision or stop]
    G -->|Yes| T{Consequential action requires approval?}
    T -->|Yes| A[Named approver reviews evidence and scope]
    A -->|Denied| H
    A -->|Approved| X[Execute through allowlisted tool]
    T -->|No| X
    X --> L[Record inputs, outputs, tool calls, approvals, state, and outcome]
    L --> M{Signal, incident, limit, or goal reached?}
    M -->|Continue| P
    M -->|Goal reached| V[Validate outcome and close authority]
    M -->|Risk or limit| R[Interrupt, revoke, contain, fallback, rollback, or remediate]
    R --> I[Incident review and control update]
    I --> O

10. AI Governance Structure

11. Change Management for AI Adoption

How To Get Started

Most organizations struggle not with understanding AI's potential, but with execution: which use case to pursue first, how to build governance without bureaucracy, and how to deliver value before the pilot budget runs out. This guide has three deliberately different layers, with a clear hierarchy:

• Quick Version: front-end triage and pilot-readiness handoff; it does not authorize production deployment.
• Detailed Version: strategy, business-case, governance, and pilot-design work; it is a design path, not a second production runbook.
• Operating Manual: the canonical detailed execution and decision-gate template; use it when implementing a pilot, adapting its schedule to local evidence. If the templates conflict, the approved local project record and applicable control-owner decision govern.

Constructed execution-template boundary: The schedules, counts, scores, budgets, team sizes, sample ranges, product examples, thresholds, and outputs in this guide are illustrative planning placeholders. Replace them with evidence from the specific workflow, population, risk profile, staffing model, legal and policy review, procurement terms, and measurement design. A template cannot establish that an AI use case is feasible, safe, compliant, or economically attractive.

Quick Version (3-4 Weeks; illustrative path): Rapid AI Opportunity Assessment & Pilot Selection

Goal: Identify and evidence a small set of opportunities on a bounded local schedule; a pilot launch date follows the decision, data, control, and staffing evidence rather than a universal 30-day promise.

Who Should Use This: Organizations new to AI, need a fast evidence triage, or have limited AI resources; use the Detailed Version and canonical Operating Manual before any consequential deployment.

Week 1: AI Opportunity Assessment

Objective: Generate 20+ potential AI use cases and score them on feasibility/value.

Activities:

Day 1-2: Use Case Discovery Workshops

• Facilitate 3 workshops with different departments (Sales, Operations, Customer Service)
• Use prompt: "What manual, repetitive, or data-heavy work consumes 5+ hours/week?"
• Capture 20-30 raw ideas (target: 7-10 per workshop)

Day 3-4: Preliminary Scoring

• Score each use case on 2x2 matrix (Value: Low/High, Feasibility: Low/High)
• Value criteria: Revenue impact, cost reduction, customer experience improvement
• Feasibility criteria: Data exists, <6 months to value, no major technical blockers
• Identify 5-8 "Quick Win" candidates (High Value + High Feasibility)

Day 5: Stakeholder Review

• Present 2x2 matrix to leadership
• Get buy-in on top 5 use cases for deeper assessment

Deliverable: AI Opportunity Assessment Matrix with 20+ use cases mapped, 5 Quick Win candidates identified.

Week 2: ICP Use Case Selection

Objective: Prioritize top 3 use cases using ICE scoring + data readiness check.

Activities:

Day 1-2: Deep Dive on Top 5

• For each Quick Win candidate, assess:
  • Impact (1-10): Quantify business value (e.g., "$500K cost savings/year")
  • Confidence (1-10): Technical feasibility + team capability
  • Ease (1-10): Time to production, resource requirements
  • Data Quality (1-10): Data exists, clean, sufficient volume
• Calculate the illustrative AI Score as (Impact + Confidence + Ease + Data Quality) / 4; preserve the raw evidence and test sensitivity rather than treating the arithmetic as a decision rule.

Day 3: Data Readiness Spot Checks

• For top 3 scored use cases, validate data actually exists:
  • Request representative sample datasets
  • Check for missing values, labeling, data quality issues
  • Flag material missingness, inconsistent labels, or undocumented transformations as red flags

Day 4: Build/Buy/Partner Decision

• For each top 3, decide approach:
  • Buy: a currently available off-the-shelf solution after capability, assurance, portability, data-use, and commercial diligence
  • Partner: a platform plus implementation or domain support after authority, security, IP, and exit review
  • Build: In-house if proprietary data + core differentiator
• Estimate cost for each approach

Day 5: Final Prioritization

• Rank top 3 by AI Score
• Select #1 for immediate pilot (highest score + stakeholder support)

Deliverable: Use Case Prioritization Matrix with top 3 ranked, #1 selected for pilot, Build/Buy/Partner recommendation documented.

Week 3: Pilot Planning

Objective: Scope MVP, assign team, set success metrics for first pilot.

Activities:

Day 1-2: Scope MVP

• Define minimum viable product:
  • Problem: What specific pain point does this solve?
  • Success metric: How do we measure if it worked? (e.g., materially reduce ticket resolution time)
  • Scope: What's in/out? (Start narrow: one customer segment, one product line)
  • Timeline: 6-8 weeks to MVP (model training + initial deployment)

Day 3: Assign Team

• Pilot Owner: Business stakeholder who owns the problem (accountable for adoption)
• Data Scientist: 1-2 people (model development)
• ML Engineer: 1 person (deployment, infrastructure)
• Domain Expert: SME who understands the data (e.g., sales manager for churn prediction)
• Part-time: Product manager (requirements), legal (data privacy review)

Day 4: Data Access & Infrastructure

• Provision access to required datasets (get IT/security approvals)
• Set up development environment (cloud sandbox, Jupyter notebooks, etc.)
• Document data schema, glossary, known issues

Day 5: Kick-off Meeting

• Present 6-8 week timeline with milestones:
  • Weeks 1-2: Data prep, EDA, baseline model
  • Weeks 3-4: Model training, hyperparameter tuning
  • Weeks 5-6: Deployment, A/B test, monitoring
  • Weeks 7-8: Results analysis, iteration
• Set weekly check-in cadence (every Friday)

Deliverable: Pilot Scope Document (1-pager: problem, MVP, success metrics, team, 6-8 week timeline).

Week 4: Pilot-Readiness Handoff

Objective: Begin only the non-production evidence work approved by the local gate and hand off to the canonical Operating Manual if a pilot remains justified.

Activities:

Day 1-3: Data Preparation

• Extract data from source systems (CRM, database, logs)
• Clean data: Handle missing values, remove duplicates, fix outliers
• Label data (if supervised learning): Use SME review and documented labeling guidelines
• Split data into training, validation, and test sets using a method appropriate for the use case

Day 4-5: Baseline Model

• Train simple baseline (e.g., logistic regression, decision tree)
• Evaluate performance: Accuracy, precision, recall, F1
• Document baseline performance and the production target in business and model terms
• Set a target for iteration

Deliverable: Clean dataset (versioned), baseline model performance metrics, week 5-8 plan to improve model.

Output After Week 4:

1. AI Opportunity Assessment: candidate set and 2×2 matrix visualization
2. Use Case Prioritization: documented comparison, constraints, and decision owner
3. Pilot Scope: 1-page scope doc with MVP definition
4. Pilot handoff: evidence, scope, controls, and open questions for the canonical Operating Manual
5. Decision status: go, redesign, stage, or stop; no production commitment is implied

Detailed Version (12-16 Weeks; illustrative path): Strategy, Governance, and Pilot Design

Goal: Design a comprehensive AI strategy, governance, and first-pilot case. Execute and gate any pilot through the canonical Operating Manual and the approved local project record.

Who Should Use This: Organizations serious about AI transformation, have executive buy-in, and resources for 3-4 month engagement.

Phase 1: AI Opportunity Assessment (Weeks 1-2)

Objective: Identify 20-30 use cases across organization, score on feasibility/value, and create AI strategy roadmap.

Week 1: Use Case Discovery

Activities:

• Use Case Workshops (5 sessions): Facilitate with Sales, Marketing, Operations, Finance, Customer Service
• Competitive AI Analysis: Research what competitors are doing with AI (annual reports, press releases, demos)
• Industry Best Practices: Identify top 3 AI use cases in your industry (e.g., retail: personalization, pricing, inventory)
  • Technology Scan: Review AI capability classes relevant to the use case (current foundation models for generative AI; conventional machine learning for prediction)

Deliverables:

• 20-30 use case ideas documented (template: Problem, Current State, AI Solution, Expected Value)
• Competitive AI landscape summary (1-page)

Week 2: Preliminary Scoring & Roadmap

Activities:

• 2×2 Matrix Mapping: Plot all use cases on Value/Feasibility matrix
• Identify Quick Wins: 5-8 High Value + High Feasibility candidates
• Identify Strategic Bets: 3-5 High Value + Low Feasibility long-term investments
• Stakeholder Interviews: Validate assumptions with 5-7 exec stakeholders
• 3-Year Roadmap: Draft high-level AI roadmap (Year 1: Quick Wins, Year 2: Scale, Year 3: Strategic Bets)

Deliverables:

• AI Opportunity Assessment Matrix (visualization with all use cases plotted)
• 3-Year AI Roadmap (1-page timeline)

Phase 2: Use Case Prioritization & Business Case (Weeks 3-4)

Objective: Deep dive on top 3-5 use cases, build ROI models, select first 2 pilots.

Week 3: Deep Dive Analysis

Activities:

• Top 5 Use Case Deep Dives (1 day each):
  • Map current process (flowchart, identify pain points)
  • Define AI solution (specific model type, input/output)
  • Data assessment (volume, quality, labeling requirements)
  • Technical feasibility (complexity, integration requirements)
  • Risk assessment (bias, privacy, regulatory)

Example Deep Dive - Customer Churn Prediction:

• Current State: material churn in a measurable customer segment
• AI Solution: Predict churn in advance using customer behavior data
• Data: customer records, usage features, support history, and NPS or equivalent signal
• Success Metric: reduce churn enough to create measurable retained revenue
• Risks: depends on data sensitivity, explainability needs, and downstream customer treatment

Deliverables:

• 5 Use Case Deep Dive Reports (3-5 pages each)

Week 4: ROI Modeling & Prioritization

Activities:

• Build ROI Models (3-year NPV):
  • Benefits: Revenue lift + Cost reduction + Risk mitigation (quantified annually)
  • Costs: Development, deployment, monitoring, change management, and maintenance
  • ROI Calculation: NPV using the organization's normal hurdle rate

Example ROI - Churn Prediction:

• Year 1 Benefit: retained revenue from the first customer segment

• Year 2-3 Benefit: expansion benefit if the model scales to additional segments

• Total Cost: development, deployment, monitoring, and maintenance

• 3-Year NPV: calculate from the actual business case; do not reuse generic target values

• Prioritization worksheet: Compare the selected use cases using the documented local factors and sensitivity analysis; do not treat a rank as an automatic launch order.

• Build/Buy/Partner Decision: For each use case, decide optimal approach

• Risk Assessment: Flag high-risk use cases requiring ethics review

Deliverables:

• ROI Models (Excel/Google Sheets) for top 5 use cases
• Final Prioritization (ranked 1-5, top 2 selected for pilots)
• Build/Buy/Partner recommendations with cost estimates

Phase 3: AI Governance & Ethics (Weeks 5-6)

Objective: Establish AI governance structure, ethical guidelines, and data access controls before building production models.

Week 5: Governance Structure

Activities:

• Define AI Governance Roles:

  • AI Steering Committee: CEO, CTO, CDO, business unit heads (cadence set by decision rights and portfolio risk)
  • AI Center of Excellence: Chief AI Officer, ML leads, product managers (operating cadence set locally)
  • AI Ethics Board or equivalent review authority: Legal, compliance, ethicist, AI leads (review triggers and cadence set by use and risk)
  • Project Teams: Data scientists, ML engineers, product, domain experts (coordination cadence set by delivery needs)

• Charter Each Body:

  • Document responsibilities, decision rights, escalation paths
  • Set meeting cadence and agenda templates

• AI Policy Documentation:

  • Data Usage Policy: What data can be used for AI? (PII handling, customer consent, retention)
  • Model Approval Process: Low/Medium/High risk categorization, who approves each level
  • Incident Response: What happens if AI causes harm? (investigation, remediation, communication)

Deliverables:

• AI Governance Charter (5-7 pages: roles, responsibilities, processes)
• AI Policy Pack (Data, Model Approval, Incident Response)

Week 6: Ethics & Bias Assessment

Activities:

• Ethical AI Framework Implementation:

  • Train AI teams on 6 principles (Fairness, Transparency, Privacy, Accountability, Safety, Beneficence)
  • Create ethics checklist for all high-risk AI projects

• Bias Testing Plan:

  • For selected pilot use cases, identify protected classes to test (race, gender, age, etc.)
  • Define fairness metrics (e.g., equal opportunity, demographic parity)
  • Plan for bias mitigation (data balancing, fairness constraints, post-processing)

• Privacy & Security:

  • Data minimization: Only use data necessary for AI task
  • Anonymization: Remove PII where possible (differential privacy techniques)
  • Access controls: Role-based access to training data, model outputs
  • Compliance check: identify applicable privacy, sector, accessibility, employment, and AI obligations with qualified legal and control owners; examples may include GDPR, CCPA, or HIPAA where their scope applies

Deliverables:

• Ethical AI Checklist (1-page, used for every AI project)
• Bias Testing Protocol (for pilot use cases)
• Privacy & Security Controls Documentation

Phase 4: First Pilot Planning (Weeks 7-8)

Objective: Detailed planning for first pilot - scope, team, data pipeline, timeline, success metrics.

Week 7: Pilot Scoping & Team

Activities:

• MVP Definition:

  • Problem Statement: Specific pain point (e.g., "Support tickets taking 48 hours to resolve, target 24 hours")
  • AI Solution: Model type, version, inputs, outputs, and controls (e.g., "approved generative model + governed knowledge base → draft responses for human review")
  • Scope: Boundaries (e.g., "Start with 10 most common ticket types, English only, US customers")
  • Success Metrics:
    • Model Metrics: quality, safety, latency, and reliability thresholds appropriate to the use case
    • Business Metrics: ticket resolution time, customer satisfaction, cost, or revenue change
    • Adoption Metrics: eligible workflow volume handled safely by the system over a defined period

• Team Assembly:

  • Pilot Owner: named business owner for the workflow (accountable for business results)
  • Data Science Lead: 1 senior data scientist (model development)
  • ML Engineers: 2 engineers (infrastructure, deployment, monitoring)
  • Product Manager: Owns user experience, requirements
  • Domain Experts: 2 support managers (data labeling, validation, change management)
  • Part-time: Legal (privacy review), Security (access controls), IT (integration)

• Stakeholder Alignment:

  • Present pilot plan to AI Steering Committee
  • Get budget approval based on the approved scope, loaded labor, infrastructure, vendor, governance, and change costs
  • Align with IT on infrastructure (cloud budget, security approvals)

Deliverables:

• Pilot Charter (3-5 pages: problem, MVP, team, success metrics, budget)
• Stakeholder sign-off (Steering Committee approval)

Week 8: Data Pipeline & Infrastructure

Activities:

• Data Pipeline Design:

  • Data Sources: Identify systems (CRM, support ticketing, product database)
  • Data Extraction: Batch vs. streaming, frequency (daily ETL, real-time APIs)
  • Data Storage: Data lake (raw), data warehouse (cleaned), feature store (ML-ready)
  • Data Quality: Validation rules (schema checks, null checks, outlier detection)

• Infrastructure Setup:

  • Cloud Environment: Provision AWS/GCP/Azure resources (compute, storage)
  • MLOps Tooling: Set up experiment tracking (MLflow, W&B), model registry, orchestration (Airflow)
  • Development Environment: Jupyter notebooks, VS Code, version control (GitHub)
  • Security: VPN access, IAM roles, data encryption at rest/in transit

• Data Access Approvals:

  • Legal review for data usage (customer consent, GDPR compliance)
  • Security review for access controls (who can see what data)
  • IT provisioning (database credentials, API keys)

Deliverables:

• Data Pipeline Architecture Diagram (data flow, systems, storage)
• Infrastructure Setup Guide (how to access, security protocols)
• Data Dictionary (all fields, definitions, known issues)

Phase 5: Pilot Execution (Weeks 9-12)

Objective: Build, train, validate, and iterate on first AI model.

Week 9: Data Preparation & EDA

Activities:

• Data Extraction: Pull a period and sample sufficient for the task, population, seasonality, subgroup and tail coverage, uncertainty, and validation design; do not reuse a universal duration or sample count

• Exploratory Data Analysis (EDA):

  • Univariate analysis: Distributions, missing values, outliers
  • Bivariate analysis: Feature correlations with target variable
  • Multivariate analysis: Feature interactions, multicollinearity
  • Data quality summary: missingness, errors, duplicates

• Data Cleaning:

  • Handle missing values: impute, investigate, or exclude based on documented rules
  • Remove duplicates: De-duplicate based on unique keys
  • Fix outliers: Cap at 99th percentile or investigate (could be legitimate)
  • Normalize/standardize: Scale features for model training

• Feature Engineering:

  • Create new features: Aggregations (sum, avg over time), ratios, lags (for time-series)
  • Encode categoricals: One-hot encoding, target encoding
  • Feature selection: Remove low-variance, highly correlated features

Deliverables:

• Clean Dataset (versioned in feature store)
• EDA notebook with visualizations
• Feature Documentation (what each feature represents)

Week 10: Model Training & Validation

Activities:

• Baseline Model:

  • Train simple model (logistic regression, decision tree)
  • Evaluate on validation set: Accuracy, precision, recall, F1, AUC
  • Document baseline performance in the agreed model and business metrics

• Advanced Models:

  • Compare a small set of plausible model classes selected for the task, baseline, data, controls, and operating constraints
  • Hyperparameter tuning: Grid search, random search, Bayesian optimization
  • Use a validation design appropriate to the data-generating process and leakage risks; fold count is a local methodological choice

• Model Evaluation:

  • Test set performance: document final metrics on held-out test set
  • Business metric translation: connect model performance to workflow or financial outcomes
  • Error analysis: Where does model fail? (certain ticket types, edge cases)

• Model Explainability:

  • SHAP/LIME: Which features drive predictions?
  • Feature importance: Top 10 most predictive features
  • Document for transparency (especially if high-risk use case)

Deliverables:

• Trained candidate models, as justified by the validation plan, versioned in the model registry
• Model evaluation memo with metrics, comparison table, and recommendation
• Model explainability memo with SHAP plots or feature importance

Week 11: Bias Testing & Refinement

Activities:

• Bias Testing:

  • Segment test set by protected classes (if applicable: gender, age, race, geography)
  • Measure performance disparity across approved comparison groups
  • Fairness metrics: Demographic parity, equal opportunity, equalized odds

• Bias Mitigation (if issues found):

  • Data balancing: Oversample underrepresented groups
  • Algorithmic fairness: Add fairness constraints to loss function
  • Post-processing: Adjust decision thresholds per group
  • Re-evaluate: Did mitigation work? Trade-offs in overall accuracy?

• Model Refinement:

  • Incorporate feedback from domain experts (does model make sense?)
  • A/B test variations: Different feature sets, model types
  • Select final production model: Best balance of performance, explainability, fairness

Deliverables:

• Bias testing memo with performance by relevant group
• Final Production Model (selected, documented, approved by ethics board if high-risk)

Week 12: Deployment Preparation

Activities:

• Model Deployment:

  • Containerize model: Docker image with model, dependencies, API endpoint
  • Deploy to staging: Test in non-production environment
  • Integration testing: Ensure model integrates with application (API calls, latency <SLA)

• Monitoring Setup:

  • Performance Monitoring: Track accuracy, latency, error rate in production
  • Data Drift Monitoring: Detect material input changes and trigger diagnosis against the intended population, data pipeline, outcomes, and monitoring design. Retraining is one possible response only after evidence review and approved change control.
  • Model Drift Monitoring: Detect if accuracy degrades over time
  • Alerting: Slack/PagerDuty alerts if metrics fall below thresholds

• Rollout Plan:

  • Shadow Mode (Week 13): Model makes predictions but doesn't affect users (compare to human decisions)
  • A/B Test (Week 14): limited treatment group against current process
  • Gradual Rollout (Week 15): if the test succeeds, ramp through approved exposure stages
  • Rollback Plan: If issues arise, instant rollback to previous process

Deliverables:

• Deployed Model (in staging environment)
• Monitoring Dashboard (real-time metrics)
• Rollout Plan (3-week go-live timeline)

Phase 6: Deployment & Scaling (Weeks 13-16)

Objective: Production deployment, monitoring, iteration, and planning for second use case.

Week 13: Shadow Mode Deployment

Activities:

• Deploy to Production (Shadow Mode):

  • Model runs in background, makes predictions, but doesn't affect user experience
  • Compare model predictions to actual human decisions
  • Measure model-human agreement rate and review disagreement quality

• Data Collection:

  • Log all predictions, inputs, outputs for analysis
  • Collect feedback from domain experts: "Was model right/wrong? Why?"

• Issue Identification:

  • Identify failure modes: Edge cases where model underperforms
  • Prioritize fixes: Critical (blocks go-live) vs. Nice-to-have (future iteration)

Deliverables:

• Shadow-mode performance memo with agreement rate and failure modes
• Go/No-Go Decision (ready for A/B test or need iteration?)

Week 14: A/B Test Deployment

Activities:

• A/B Test Setup:

  • Limited traffic to AI treatment group
  • Remaining traffic to current-process control group
  • Random assignment, stratified by key variables (customer segment, ticket type)

• Metrics Tracking:

  • Model Metrics: Accuracy, latency, error rate (real-time dashboard)
  • Business Metrics: Ticket resolution time, customer satisfaction, support cost
  • Experiment Design: Pre-specify the estimand, assignment unit, primary and guardrail outcomes, minimum decision-relevant effect, sample-size/power assumptions, analysis plan, stopping/monitoring rule, attrition and interference checks, subgroup questions, and practical decision threshold. Duration follows the design and operational cycle; a p-value or fixed number of weeks is not a rollout rule. See Chapter 22.

• User Feedback:

  • Interview support agents: "Is AI helpful? Where does it fail?"
  • Collect customer feedback: NPS, qualitative comments

Deliverables:

• A/B Test Results (after 2 weeks: treatment vs. control performance)
• Recommendation (expand rollout, iterate, or pivot)

Week 15: Full Deployment

Activities:

• Ramp to approved production scope:

  • If A/B test succeeds against the pre-specified decision rule, ramp through approved exposure stages
  • Monitor closely for issues during ramp

• Training & Change Management:

  • Train support team on how to use AI tool (demos, documentation, office hours)
  • Celebrate wins: share concrete success metrics with the team
  • Address concerns: "AI helps you focus on complex cases, not replacing you"

• Handoff to Operations:

  • Production model owned by ML engineering team (monitoring, retraining)
  • Business owner (VP Customer Service) owns adoption, success metrics
  • Weekly sync: Review metrics, prioritize improvements

Deliverables:

• Production deployment to approved scope
• Change management memo with adoption rate and user feedback
• Operations Runbook (how to monitor, troubleshoot, retrain)

Week 16: Retrospective & Scale Planning

Activities:

• Pilot Retrospective:

  • What worked? (Data quality, team collaboration, executive support)
  • What didn't? (Underestimated data labeling effort, integration complexity)
  • Lessons learned: Document for next pilot

• ROI Validation:

  • Compare actual vs. projected ROI
  • Compare projected and actual value, then explain the variance
  • Update ROI model for future use cases

• Scale Planning:

  • Select second use case from prioritized list (Week 3-4 output)
  • Apply learnings: Use same MLOps infrastructure, governance process
  • Set timeline: Faster execution (8-10 weeks vs. 12) due to foundation in place

• AI Strategy Update:

  • Update 3-year roadmap based on pilot results
  • Present to AI Steering Committee: "Pilot success, here's the next 3 use cases for Year 1"
  • Get budget approval for scaling

Deliverables:

• Pilot retrospective memo with successes, failures, and lessons learned
• ROI Validation (actual vs. projected)
• Next 3 Use Cases Roadmap (Q2-Q4 timeline)

Output After 16 Weeks:

1. Production AI Model: First use case deployed, delivering measurable business value
2. AI Governance: Established structure, ethics framework, policies operationalized
3. MLOps Infrastructure: Data pipelines, model registry, monitoring in place (reusable for next use cases)
4. Team Capability: Trained team, documented processes, ready to scale
5. Momentum: Success story to evangelize, roadmap for next 2-3 use cases, executive support

Common Pitfalls

1. Pursuing Low-Value Use Cases (Chasing AI Hype, Not Business Value)

Problem:

• Teams pick use cases because they're "cool" or "cutting-edge" (e.g., generative art, chatbots that don't solve real problems)
• No clear ROI or business metric improvement
• Success = "We built an AI!" (not "We saved $X or grew revenue by Y%")

Example:

• Company builds an AI-powered meeting summarizer because a current model can generate summaries
• No one uses it (calendar integrations don't work, summaries mediocre)
• 6 months, $300K spent, zero business impact

How to Avoid:

• Start with business problems, not AI solutions: "What costs us the most money/time?" → Then ask "Can AI help?"
• Require quantified business case: "This will save $X or grow revenue by Y%"
• Reject use cases without clear success metrics

2. Skipping Data Readiness (Building Models on Bad Data)

Problem:

• Teams assume data is "good enough" without validation
• Discover too late: Missing values, wrong labels, not enough volume, biased samples
• "Garbage In, Garbage Out" - model performance stalls because data quality is poor

Example:

• Churn prediction model using CRM data
• Discover: many churned customers don't have churn reason logged, dates inconsistent, key features missing
• Spend 3 months cleaning data before can even train model (should've been Week 1 task)

How to Avoid:

• Week 1 of every pilot: Data quality assessment (100-1000 sample records, spot check manually)
• Use the Data Readiness Checklist (Section 7) and define owner-set acceptance criteria for the intended use, population, risk, and validation design; no universal score is required
• Budget a substantial share of pilot time for data preparation

3. No AI Governance (Bias Issues, Regulatory Problems, Ethical Failures)

Problem:

• Teams deploy AI without ethics review, bias testing, or compliance checks
• Model discriminates against protected classes (gender, race, age)
• Regulatory violations (GDPR, CCPA) or PR disasters when bias discovered

Constructed example:

• Hiring AI screens resumes, trained on historical data
• Discovers: Model penalizes women (because historically fewer women in tech roles)
• Possible consequences include discriminatory impact, workforce harm, legal exposure, loss of trust, or withdrawal of the system; the actual disposition depends on facts, controls, and applicable obligations

How to Avoid:

• Establish AI Ethics Board before deploying first model (Week 5-6 in Detailed Version)
• Mandate bias testing for all high-risk use cases (hiring, lending, criminal justice)
• Use Ethical AI Checklist (Section 6): 6 principles must be addressed

4. Piloting Without Clear Success Metrics (Can't Tell If It Worked)

Problem:

• Teams launch pilot with vague goals: "Improve customer experience"
• No baseline measurement, no quantified target
• At end of pilot: "Did it work?" → "Uh... maybe? Customers seem happier?"
• Can't justify scaling or budget for next use case

Example:

• Gen AI chatbot deployed to customer service
• No measurement of: Ticket deflection rate, resolution time, customer satisfaction before/after
• Anecdotal feedback: "Some customers like it, some don't"
• Can't prove ROI, leadership loses confidence in AI

How to Avoid:

• Define 3 success metrics before pilot starts (Week 3 or Week 7):
  • Model Metric: Accuracy, latency (technical success)
  • Business Metric: Revenue, cost, time savings (quantified $$ impact)
  • Adoption Metric: % users actively using AI (change management success)
• Measure baseline before pilot: "Current ticket resolution time = 48 hours" (target: 24 hours)
• Track metrics weekly during pilot and brief stakeholders

5. Overpromising ROI (Model Never Achieves Expected Lift in Production)

Problem:

• Business case assumes best-case model performance from external examples
• Reality: performance is lower because of messier data, edge cases, and integration issues
• Projected $1M savings becomes $300K, leadership disappointed, future AI projects harder to fund

Example:

• Predictive maintenance model: "Will materially reduce downtime and save money"
• Reality: model misses edge cases and saves much less than projected
• Still positive ROI, but overpromise damages credibility

How to Avoid:

• Conservative ROI Assumptions:
  • Use 50th percentile (median) performance, not 90th percentile (research paper best-case)
  • Assume gradual adoption rather than instant usage
  • Apply an explicit uncertainty adjustment derived from a defensible range and sensitivity analysis; do not use a universal multiplier
• Iterate ROI Model:
  • Update after pilot with actual performance (Week 16)
  • Show "Projected vs. Actual" transparently (builds trust even if missed target)
• Underpromise, Overdeliver:
  • Business case: "Save $500K" (conservative)
  • Actual: Save $700K → Exceeds expectations, easier to get next project funded

Measurement Framework

Weekly Metrics (During Pilot):

Discovery Phase (Weeks 1-4):

• Use Case Discovery Count: Target 20+ use cases identified
• Stakeholder Engagement: # workshops conducted (target: 5+), # departments involved (target: 5+)
• Data Assessment Progress: # use cases with data spot-checked (target: top 5)

Development Phase (Weeks 9-12):

• Data Quality: missing values and labeled-data coverage against approved thresholds
• Model Performance: Accuracy, F1, AUC (track improvement weekly)
• Development Velocity: Story points completed, blockers resolved

Deployment Phase (Weeks 13-16):

• Shadow Mode Agreement: model-human agreement and disagreement review quality
• A/B Test Results: Estimated business effect and uncertainty against the pre-specified decision threshold, plus guardrails, attrition, interference, subgroup behavior, operational incidents, and practical significance; do not use p < 0.05 as an automatic ship rule
• Adoption Rate: share of eligible users actively using AI against approved target

Pilot Success Metrics (End of Pilot):

Model Metrics (Technical Success):

• Accuracy/F1/AUC: Did model hit the approved performance target?
• Latency: Response time within SLA? (e.g., "<5 seconds")
• Uptime: Model availability against production requirement

Business Impact (Value Delivered):

• Cost Savings: Quantified reduction in labor, errors, downtime
• Revenue Lift: Increased conversions, upsells, retention
• Time-to-Value: Weeks from kickoff to production deployment (target: <16 weeks for first pilot)
• ROI: value relative to cost against the approved hurdle

Adoption Metrics (Change Management Success):

• User Adoption: share of target users actively using AI against approved adoption targets
• User Satisfaction: NPS or satisfaction score from end users against approved target
• Support Tickets: # support requests about AI tool (lower = more intuitive)

Production Metrics (Ongoing Post-Deployment):

Model Health:

• Prediction Accuracy: Track accuracy on production data weekly (detect model drift)
• Data Drift: KL divergence, PSI (Population Stability Index) to detect input distribution changes
• Latency & Errors: p95 latency and error rate against approved alert thresholds

Business Impact Realization:

• Actual vs. Projected ROI: Track realized savings/revenue monthly
• Cumulative Value: Total $$ saved or earned since deployment
• Payback Period: Months to break even on pilot cost

MLOps Health:

• Retraining Frequency: How often model retrained (monthly? quarterly?)
• Deployment Speed: Time from model retrained to deployed (target: <1 week)
• Incident Response: MTTD (Mean Time to Detect) and MTTR (Mean Time to Resolve) for model issues

Dashboards:

Executive Dashboard (Monthly):

• AI projects in flight (#), in production (#), pipeline (#)
• Cumulative ROI from all AI projects ($)
• Top 3 risks/blockers

Operational Dashboard (Weekly):

• Per-project status: On track / At risk / Blocked
• Model performance metrics (accuracy, latency)
• Business impact metrics (cost savings, revenue lift)

Technical Dashboard (Real-time):

• Model accuracy, prediction volume, error rate
• Data drift detection, model drift detection
• Infrastructure health (CPU, memory, API latency)

Red Flags: When AI Strategy Is Failing

The numerical cutoffs in this checklist are illustrative local operating thresholds, not universal AI-practice benchmarks. Set and document thresholds for the specific use case, risk level, baseline performance, and governance obligations.

Data Quality Red Flags:

• High Missingness: Data too sparse to train accurate model -> fix data collection first
• Labels Inconsistent: agreement is below the approved threshold -> improve labeling guidelines and retrain annotators
• Data Drift Signal: Input distributions or relationships appear to be changing → diagnose data pipelines, population, concept, outcome, and monitoring validity; then use approved change control to continue monitoring, correct data or workflow, recalibrate, retrain, revert, restrict, or retire
• Evidence Coverage Concern: Sample size, subgroup or tail coverage, label quality, or uncertainty is inadequate for the decision → the method owner chooses additional data, a simpler method, transfer or few-shot techniques, narrower use, a pilot, or no deployment based on context-specific validation

Model Performance Red Flags:

• Accuracy Plateau: model stalls despite trying multiple algorithms -> likely data quality issue, need better features
• Overfitting: training performance is far better than test performance -> model memorizing, not generalizing
• Bias Detected: material performance disparity across groups -> requires mitigation before production deployment
• Explainability Issues: Can't explain why model makes predictions → Red flag for high-risk use cases (regulatory, ethical concerns)

Adoption Resistance Red Flags:

• Low Usage: adoption below target after launch -> users don't trust, don't understand, or tool not integrated into workflow
• Negative Feedback: NPS <50 (illustrative local planning threshold), with recurring error complaints → investigate model performance and user experience
• Shadow IT: Users building their own tools (Excel macros) instead of using AI → AI tool doesn't solve real problem
• Lack of Executive Support: Leadership not promoting AI, no budget for scaling → Need to re-engage with success stories, ROI proof

Execution Red Flags:

• Pilot Stuck >6 Months: If the first pilot takes >6 months to production (illustrative local planning threshold) → review scope, data readiness, ownership, and delivery constraints
• No ROI Measurement: Can't quantify business impact → Can't justify next project, AI program will die
• Governance Bottleneck: Ethics reviews taking >1 month (illustrative local planning threshold) → review process capacity and pre-approve eligible low-risk use cases
• Talent Churn: Data scientists leaving, can't hire → Compensation not competitive, projects not impactful, or tech debt too high

Strategic Red Flags:

• Competitor AI Advantage: Competitors deploying AI faster, delivering better customer experiences → Need to accelerate or risk losing market share
• No AI Roadmap: One-off projects without strategic plan → AI will remain experimental, won't scale
• Siloed AI Teams: Each department building own models, no shared infrastructure → Inefficient, duplicative work, hard to scale
• Regulatory Concerns: Legal team blocking AI due to compliance worries → Need to engage legal early, build compliant-by-design systems

Recovery Actions:

If Data Quality Issues:

• Pause model development, invest 4-8 weeks in data cleanup
• Hire data engineers to build pipelines (prevent future issues)
• Use data quality tools (Great Expectations, dbt tests)

If Model Performance Issues:

• Get domain expert feedback: "What's missing? What features would help?"
• Try different model types (ensemble, neural nets, transformer models)
• Consider buying pre-trained models (transfer learning) instead of building from scratch

If Adoption Resistance:

• User research: Interview 10 non-adopters, understand barriers
• Improve UX: Make AI tool easier to use, integrate into existing workflows
• Change management: study non-adoption, redesign the workflow, train affected users, and use accountable escalation rather than coercive adoption targets

If Execution Stalled:

• Ruthlessly cut scope: MVP → Minimum Minimum Viable Product
• Assign dedicated owner: Someone accountable for go-live, not part-time
• Executive escalation: If blockers (IT, legal, budget), escalate to Steering Committee

Contrarian Reality Check: What They Don't Tell You About AI

Most AI strategy guides assume your project will succeed. This section starts from a more useful premise: broad AI adoption does not automatically translate into measurable business value. Current public reports show widespread adoption alongside uneven value capture, so understanding why projects stall is more valuable than copying success stories. ^{[4] [3]}

The Uncomfortable Truths About AI in 2026

Diagnostic #1: AI Activity Without Decision-Grade Value Evidence

The issue: An organization can adopt AI tools, announce a strategy, and run pilots without demonstrating attributable business value. Public surveys document broad adoption and uneven value capture, but they do not establish the motives of any executive, team, or adviser. Diagnose the operating evidence instead: ownership, baselines, controls, adoption, and measured outcomes. ^{[4] [3]}

How to Detect the Pattern:

• Symptom 1: Pilots that run for 12+ months without production deployment
  • Stronger evidence: pilot has a time-boxed path to a controlled production test, redesign, or explicit stop decision
  • Warning signal: no decision owner, decision date, or documented reason for continuing
• Symptom 2: AI budget spent on conferences, PR, and buzzword consulting instead of talent, data, infrastructure, and governance
  • Stronger evidence: budget maps to business-owned use cases, production infrastructure, evaluation, change work, and risk controls
  • Warning signal: spending cannot be reconciled to a business case or control plan
• Symptom 3: No measurable business outcome after the pilot window
  • Stronger evidence: can estimate retained revenue, avoided cost, quality improvement, risk reduction, or cycle-time change against a baseline and counterfactual
  • Warning signal: capability claims have no measurement design or decision rule
• Symptom 4: Innovation lab isolated from core business
  • Stronger evidence: operating teams share accountability for workflow adoption, quality, risk, and economics
  • Warning signal: the pilot has no receiving owner, integration plan, or operating budget
• Symptom 5: Chief AI Officer hired but given no budget or authority
  • Stronger evidence: accountable leaders have a defined mandate, budget, escalation path, and decision rights
  • Warning signal: formal titles exist without documented authority or ownership

These indicators identify a governance and value-realization problem. They do not prove bad faith. Causes may include weak data, unclear authority, learning-stage uncertainty, security or legal constraints, workflow friction, or a business case that no longer holds.

How to Avoid It:

• Set hard deadlines: pilot has a production path, redesign decision, or stop decision
• Require ROI measurement: Define success metrics upfront, measure monthly
• Embed AI in business units: No separate labs, AI reports to P&L owners
• Decide explicitly: if evidence misses agreed business, safety, readiness, or adoption criteria, stop, redesign, or extend only with a documented learning objective and owner

Cross-Reference: See Chapter 17 "Digital Transformation Theater" for broader organizational transformation patterns. AI theater is a specific instance of the general phenomenon where organizations confuse activity (launching initiatives) with progress (delivering value).

Truth #2: The Real Gap Is Adoption Versus Value

The Data:

• Stanford AI Index: AI adoption is widespread, but the report should be used for trend context rather than as proof that any specific project will succeed. ^[4]
• McKinsey State of AI: many organizations report AI use, but measurable EBIT impact is more limited and context-dependent. ^[3]
• Governance implication: treat value realization as an operating discipline: use-case selection, data readiness, risk controls, product ownership, deployment, and change management. ^{[1] [2]}

Root Causes (Not What Vendors Tell You):

Cause #1: Poor Data Quality

• The Lie: "We have lots of data, we're ready for AI"
• The Truth: Having data ≠ having good data
  • Material missingness can make the model learn collection artifacts instead of customer or operational behavior
  • Inconsistent labels? Model learns noise, not patterns
  • Biased historical data? Model automates discrimination
• Example: Bank builds credit risk model, discovers that income data is frequently missing because sales teams were not required to collect it. The project becomes a data-process repair effort before model training can start.
• Fix: Assess data quality BEFORE building models (Framework #7: Data Readiness)

Cause #2: Unclear Business Case

• The Lie: "AI will transform our business"
• The Truth: Vague goals = no ROI
  • "Improve customer experience" is not a success metric
  • "Reduce churn enough to retain a defined amount of revenue" is measurable
• Example: Retailer builds "AI demand forecasting" without a pre-agreed success metric. After deployment, leaders cannot tell whether it beats the previous planning process, so adoption stalls.
• Fix: Quantify ROI before pilot (Framework #5: ROI Calculation)

Cause #3: Lack of Full-Stack Delivery Talent

• The Lie: "We'll hire a data scientist and do AI"
• The Truth: One data scientist can't do it alone
  • Need: data science, ML engineering, data engineering, product ownership, domain expertise, security, and compliance input
• Example: Startup hires one data scientist. The model works in a notebook but cannot be deployed, monitored, or integrated into the workflow because the rest of the delivery system is missing.
• Fix: Staff full team or buy vendor solution (Framework #2: Build vs. Buy)

The Projects That Succeed Do This Differently:

1. Start with business problem, not AI capability ("We lose $5M to churn" not "Let's use deep learning")
2. Validate data quality early, not after model development
3. Set hard success metrics upfront and define the stop condition
4. Staff the full delivery system or partner with a vendor
5. Keep pilots time-boxed; no perpetual experiments

Cross-Reference: For culture change needed to support AI adoption, see Chapter 17 "Culture Change Is About Incentives, Not Values." AI projects fail when incentives reward old behaviors (manual work, gut decisions) instead of new behaviors (data-driven, automated).

Truth #3: When AI Is Overkill (Simple Solutions Beat Complex Models)

The Dirty Secret: Many business problems don't need AI. A well-designed rule-based system or simple regression often beats a complex model on cost, transparency, and operational reliability.

When NOT to Use AI:

Case 1: Problem Can Be Solved with Rules

• Example: Email spam filter
  • AI approach: Train deep learning model on millions of emails
  • Simple approach: Block known spam domains, keyword filters, user reports
  • Result: simple rules may be good enough without ML infrastructure
• When rules work: the problem has clear decision criteria, edge cases are rare, and rules do not change constantly

Case 2: Insufficient Data Volume

• The operating test: supervised learning needs enough labeled examples to cover the variation that matters for the decision
• Example: Manufacturing defect detection
  • Scenario: company has a small, narrow defect-image set and wants AI to detect issues
  • Reality: the model may overfit if the examples do not cover real-world variation
  • Better approach: Manual inspection or partner with vendor with larger dataset
• When to avoid: labels are too sparse, inconsistent, or unrepresentative for the operational task

Case 3: Problem Requires Near-Perfect Accuracy

• AI Reality: model errors are inevitable; the question is whether the workflow can tolerate and catch them
• Example: Medical diagnosis for rare diseases
  • AI model: a strong aggregate metric may still hide clinically important errors
  • Reality: rare-disease mistakes can create patient harm and liability
  • Better approach: AI as decision support (flags cases for human review), not autonomous decision
• When to avoid: high-stakes decisions without human oversight, appeal, monitoring, and risk controls

Case 4: Explainability Required by Law

• Example: U.S. credit decisions may involve adverse-action notice and reason requirements under ECOA/Regulation B depending on the actor, product, decision, and facts; the legal owner must determine applicability and evidence duties.
  • Complex AI: Deep learning model can't explain "why" application rejected
  • Legal risk: failure to meet applicable notice, discrimination, recordkeeping, or model-risk obligations can create legal and operational exposure
  • Possible response: choose a simpler or more explainable method where it fits, or validate an explanation and adverse-action process with qualified legal, compliance, and model-risk owners; SHAP alone does not establish compliance
• When to pause or choose a simpler method: where applicable obligations or workflow harm require evidence, auditability, human review, or explanation that the proposed system cannot provide

Case 5: Problem Changes Faster Than You Can Retrain

• Example: News recommendation during breaking events
  • AI model: Trained on historical data (yesterday's news patterns)
  • Reality: Breaking news changes patterns instantly, model outdated in hours
  • Better approach: Real-time rules-based system, human curation, or hybrid
• When to avoid: Highly dynamic environments where patterns shift daily

The Cost of Over-Engineering:

• Simple rule-based system: usually cheaper, easier to explain, and easier to maintain
• AI model: usually requires data, evaluation, deployment, monitoring, security, and change management
• ROI question: Is AI meaningfully better than rules after full lifecycle cost? If not, don't build it.

Pragmatic Decision Tree:

Does problem have clear rules? → YES → Use rules, not AI
Do you have enough representative labeled examples? → NO → Don't use supervised learning
Can you tolerate model error? → NO → Don't use autonomous AI (use human-in-loop)
Is explainability legally required? → YES → Use simple models (logistic regression, decision trees)
Do patterns change faster than monthly? → YES → Consider real-time rules over static models

Use the branch that matches the decision rather than counting Yes/No answers. Clear deterministic rules point toward rules; insufficient labels point toward another method or more evidence; intolerable error points toward manual or human-in-the-loop control; applicable explanation or audit duties may favor a simpler or specially validated method; and rapidly changing patterns may favor rules, human curation, or a monitored adaptive design. Consider AI only when the intended task has an evidence-supported pattern advantage over alternatives and the error, control, lifecycle, and remedy boundary is acceptable.

Truth #4: LLM Strategy in 2026 (Model Choice Is a Moving Target)

The LLM Hype vs. Reality:

• Hype: "ChatGPT will replace all knowledge workers"
• Reality: LLMs are powerful tools for specific use cases, but they need evaluation, retrieval, security, monitoring, and fallback design for production use. ^{[6] [9]}

When LLMs Actually Work (High ROI Use Cases):

Use Case 1: Content Generation (Copywriting, Summarization)

• Best for: marketing copy, email drafts, meeting summaries, documentation
• Cost: depends on model, token volume, latency requirements, retrieval, review, and monitoring
• Example: customer support team uses an LLM to draft responses
  • Input: customer complaint + company policy
  • Output: professional response draft for human review
  • Benefit: faster drafting and more consistent tone if reviewers stay accountable
• When to use: repetitive writing tasks where human review is practical and quality can be evaluated

Use Case 2: Information Extraction (Structured Data from Unstructured Text)

• Best for: Parsing contracts, extracting invoice data, analyzing customer feedback
• Example: Legal team extracts key terms from 1,000 contracts
  • Manual: slow review by trained staff
  • LLM-assisted: batch extraction plus human validation of exceptions
  • Savings: depends on volume, error tolerance, review burden, and rework rate
• When to use: Large volume of unstructured documents, need structured output

Use Case 3: Customer Support Chatbots (Tier 1 Deflection)

• Best for: Answering FAQs, routing requests, simple troubleshooting
• Example: E-commerce chatbot using Claude with RAG (retrieval-augmented generation)
  • Handles well-documented repetitive questions and routes uncertain cases to people
  • Cost includes model usage, retrieval, monitoring, escalation design, and support
  • Savings depend on safe deflection rate, rework, customer satisfaction, and escalation quality
• When to use: High ticket volume, repetitive questions, clear documentation to reference

When LLMs Are Expensive Overkill:

Overkill Scenario 1: Simple Classification (Use Traditional ML)

• Problem: Email spam detection
• LLM approach: paid model call per email
  • cost scales with message volume and model choice
• Traditional ML: Logistic regression or Naive Bayes
  • training and inference are usually much cheaper at scale
• Lesson: don't use a high-cost language model when a simpler model meets the requirement

Overkill Scenario 2: Deterministic Tasks (Use Rules)

• Problem: Data validation (check if email format valid, phone number correct)
• LLM approach: model checks validity on every record
  • cost and latency scale with validation volume
• Rules approach: Regex + validation libraries
  • cost is negligible after implementation
• Lesson: LLMs can't beat regex for deterministic tasks

LLM Decision Matrix (2026):

Table 16.5: Constructed model-selection comparison aid (Use case | Model approach | Cost | When to use). Product names, prices, and performance are intentionally omitted because they change; validate current options and controls before procurement.

Fine-Tuning vs. RAG vs. Prompt Engineering:

Prompt Engineering (Cheapest, Start Here):

• Cost: often lower than model adaptation, but not zero; include staff time, evaluation data, review, model usage, security testing, monitoring, and maintenance
• Use when: default model performance is close to acceptable and the gap is instruction clarity
• Example: Customer support bot
  • Bad prompt: "Answer this question: {question}"
  • Good prompt: "You are a helpful customer support agent. Use the following documentation: {docs}. Answer the question professionally: {question}. If you don't know, say 'I'll escalate to a human agent.'"
  • Improvement: better instructions can improve consistency without changing models

RAG (Retrieval-Augmented Generation) (Medium Cost):

• Cost: vector store, retrieval pipeline, evaluation, monitoring, and operations
• Use when: Need model to reference specific company knowledge base
• Example: Internal HR chatbot
  • Challenge: A general model does not reliably know the organization's current policies
  • Solution: Use an authorized, versioned retrieval store; retrieve relevant approved content per query; provide it to the evaluated model; cite the retrieved source; and retain fallback and access controls
  • Goal: higher accuracy on company-specific questions with traceable retrieved context
• Better than fine-tuning for: Dynamic knowledge (policies change monthly), lower cost

Fine-Tuning (Highest Cost, Rare Use):

• Cost: data labeling, training, evaluation, hosting, and maintenance
• Use when: generic models fail, you have enough high-quality labeled examples, and the business case justifies cost
• Example: Legal contract analysis (highly specialized language)
  • Generic model: weak performance on specialized clause extraction
  • Fine-tuned or domain-adapted model: better fit if training data and evaluation support it
  • Use case: very high volume where quality and unit economics justify specialization
• Only if: Volume justifies cost, performance critical, proprietary data exists

The 2026 LLM Strategy:

1. Start with prompt engineering and evaluation
2. Add RAG if the system needs company knowledge or traceable context
3. Fine-tune only if volume, quality, privacy, or unit economics justify it
4. For simple tasks (classification, validation), use traditional ML, not LLMs

Cost Management (Don't Go Broke on API Bills):

• Cache repetitive queries: cache stable responses where freshness and policy allow it
• Use cheaper models for simple tasks: route by task difficulty, risk, and latency requirement
• Batch processing: Process 1,000 documents overnight (cheaper than real-time)
• Set monthly budget caps: alert before spend exceeds the approved operating envelope

Truth #5: Build, Buy, or Partner Is a Contingent Sourcing Decision

The Myth: "We have unique needs, we must build custom AI"

The Reality: Buying or partnering may reduce time and delivery risk for a mature capability; building may be justified when proprietary data, workflow control, product differentiation, security, or economics materially change the decision. Compare the options on the same lifecycle, risk, and strategic assumptions. ^{[1] [2]}

When to BUILD Custom AI:

Build Criteria:

1. Core competitive differentiator: model behavior or the AI-enabled workflow is central to the value proposition
2. Proprietary data moat: You have data competitors can't access (user behavior, sensor data, transaction history)
3. Can't buy equivalent: No vendor offers comparable solution (truly novel use case)
4. Cost justifies: full lifecycle cost is lower than buying or produces clearly superior strategic value
5. Have talent: can hire, retain, or partner for the engineering, data, product, security, and governance work

Constructed examples of when to build:

• A media service has unique interaction data and makes ranking quality central to retention.
• A manufacturer operates a safety-critical system whose sensor, validation, and incident-response requirements cannot be delegated cleanly.
• A marketplace has proprietary transaction data and needs tightly controlled pricing or routing logic.

When to BUY Vendor Solutions:

Buy Criteria:

1. Commodity capability: Every company needs it (CRM insights, email security, HR analytics)
2. Mature vendor market: established vendors with credible implementations
3. Non-differentiating: Competitors using same tools won't hurt you (accounting AI, IT support chatbots)
4. Fast time-to-value: need production deployment faster than a custom team can deliver
5. Lower risk: vendor has solved similar problems repeatedly and supports compliance, security, and operations

Constructed examples of when to buy:

• A CRM add-on meets documented sales-workflow, security, integration, and economics requirements.
• A support product handles a bounded class of low-risk requests with acceptable quality, escalation, and unit cost.
• A human-resources analytics product meets applicable employment-law, privacy, validation, and human-review requirements; vendor purchase does not transfer accountability.

The Build vs. Buy Cost Reality:

Build Custom AI:

• Team: product, data science, ML engineering, data engineering, security, and domain expertise
• Infrastructure: compute, data storage, MLOps, evaluation, monitoring, and incident response
• Year 1 Cost: build, integration, validation, and deployment
• Ongoing maintenance: staffing, retraining, model updates, vendor/API changes, security, and support
• Multi-year total cost: calculate with full lifecycle assumptions, not only prototype cost

Buy Vendor Solution:

• Subscription: usage, seat, workflow, or platform fees
• Implementation: configuration, integration, data migration, and change management
• Year 1 Cost: subscription plus implementation and governance setup
• Ongoing cost: subscription, support, monitoring, vendor management, and internal ownership
• Multi-year total cost: compare against build on the same time horizon

Cost comparison: neither option is inherently cheaper. Compare implementation, integration, evaluation, monitoring, data movement, switching, failure, exit, and internal-ownership costs over the same horizon.

When Building Makes Financial Sense:

• Scenario: the use case creates strategic value a vendor cannot capture for you
• Example: proprietary pricing, ranking, recommendation, or routing logic tied to unique data can justify custom investment
• Threshold: if a vendor can meet the need at materially lower risk and acceptable differentiation loss, buy instead

The "Build vs. Buy" Mistakes Companies Make:

Mistake #1: "We need custom AI" without testing alternatives

• Constructed example: a bank begins custom fraud detection before documenting which requirements vendors cannot meet
• Risk: the team commits to an architecture before comparing quality, control, lifecycle cost, and exit options
• Lesson: make the sourcing case falsifiable; do not infer motives from the decision

Mistake #2: "This vendor is too expensive" (False Economy)

• Example: retailer rejects an expensive-looking vendor solution and builds a custom tool
• Why: the subscription looks expensive compared with the prototype estimate
• Reality: ongoing maintenance, integration, monitoring, and staffing erase the expected savings
• Lesson: factor in maintenance cost and operating risk before declaring build cheaper

Mistake #3: "We'll buy initially, then build later" without an exit design

• Example: Company buys chatbot vendor solution, plans to "build custom later when we scale"
• Reality: switching costs, data portability, contract terms, and operating priorities may make a later rebuild unattractive
• Lesson: define portability, ownership, trigger conditions, and exit economics before signing; continuing to buy may still be the correct outcome

The Pragmatic Approach (Hybrid Strategy):

1. Buy for commodity use cases (CRM, support, HR, IT, security, analytics)
2. Build for core differentiators (unique competitive moat, proprietary data)
3. Partner for complex domains (supply chain optimization, fraud detection – vendor + your data)

Decision Framework:

Is AI your core product? → YES → Build
Do you have proprietary data competitors can't access? → YES → Build
Does vendor solution meet the need at materially lower lifecycle risk? → YES → Buy
Is this a commodity capability (every company needs it)? → YES → Buy
Can you hire/retain ML team? → NO → Buy or Partner
Need production faster than a custom team can safely deliver? → YES → Buy

The result may differ by use case. Record the assumptions, owners, uncertainty, and conditions that would change the sourcing decision.

Truth #6: AI ROI Measurement Reality (Vanity Metrics vs. Real Value)

The Problem: Model deployment, tool usage, or accuracy does not by itself establish attributable business impact. In McKinsey's July 2024 respondent survey, reported gen-AI use was widespread while most respondents reported no tangible enterprise-level EBIT impact; the result is self-reported and survey-specific, not a causal estimate or universal rate. ^[3]

Vanity Metrics (What Companies Emphasize):

• "Deployed many AI models to production" (So what? Do they deliver value?)
• "AI accuracy is high" (Accuracy on what? Does it translate to business results?)
• "Reduced processing time materially" (Did revenue increase? Did costs drop? Or just faster at doing low-value work?)
• "Employees are using AI tools" (Usage is not value unless productivity, quality, or cycle time improves)

Real ROI Metrics (What Actually Matters):

Revenue Impact:

• Measurable: "AI recommendations increased average order value enough to create attributable revenue"
• Measurable: "AI lead scoring improved conversion enough to create attributable closed deals"
• NOT measurable: "AI improved customer engagement" (What does that mean? Show me the $$$)

Cost Savings:

• Measurable: "AI chatbot safely deflected a defined class of Tier 1 tickets and reduced support cost"
• Measurable: "AI inventory forecasting reduced stockouts enough to lower lost sales"
• NOT measurable: "AI improved operational efficiency" (By how much? In what process? Prove it.)

Risk Mitigation:

• Measurable: "AI fraud detection caught attributable fraudulent transactions"
• Measurable: "AI predictive maintenance reduced downtime enough to avoid measurable production loss"
• NOT measurable: "AI improved risk management" (Too vague to be useful)

How to Calculate TRUE AI ROI:

Step 1: Establish Baseline (Before AI)

• Measure current state over a representative baseline period
• Example: support costs, ticket volume, first-contact resolution, handle time, escalation rate

Step 2: Deploy AI, Measure Impact (After AI)

• Measure the same metrics after deployment
• Example: same support metrics, plus safety, escalation, and customer-satisfaction effects

Step 3: Calculate Value Created

• Cost savings: cost avoided after adjusting for volume and mix
• Efficiency gain: time saved on eligible work, net of review and escalation
• Quality improvement: fewer escalations, better resolution, or fewer errors
• Total annual value: sum only the benefits with credible attribution

Step 4: Factor in Costs

• Year 1 investment: team time, infrastructure, integration, validation, change management
• Ongoing costs: maintenance, monitoring, retraining, support, security, and governance

Step 5: Calculate ROI

• Year 1: value minus build and deployment investment
• Year 2: value minus operating cost
• Year 3: value minus operating cost
• Multi-year total: cumulative net value across the evaluation horizon
• Multi-year ROI: cumulative net value divided by cumulative investment

Reality Check: many AI projects have delayed payback because production integration, adoption, and governance take time. If the organization cannot tolerate that payback profile, do not invest.

The AI ROI Mistakes Companies Make:

Mistake #1: Counting Pilot Success as Production ROI

• Example: "Our pilot saved time; if we scale to the whole company, savings will multiply."
• Reality: Pilots often succeed because of handpicked data and expert oversight. Production rarely achieves pilot results.
• Lesson: Measure ROI in production at scale, not pilot performance.

Mistake #2: Ignoring Opportunity Cost

• Example: spent heavily building AI, then saved less than a simpler commercial or process-change alternative would have produced
• Reality: AI must beat the next-best use of the same budget and management attention
• Lesson: AI ROI must beat alternative investments (hiring, marketing, product features)

Mistake #3: Cherry-Picking Metrics

• Example: "AI improved model accuracy" (sounds useful)
• Reality: error volume, exception handling, review cost, and customer harm may erase the benefit
• Lesson: Measure business impact, not just model metrics

Pragmatic AI ROI Framework:

1. Measure baseline before AI
2. Deploy AI, then measure production impact, not pilot-only performance
3. Calculate value: Revenue increase + Cost savings + Risk avoided
4. Calculate costs: Development + Ongoing maintenance
5. Demand a return that beats the organization's hurdle rate and alternatives
6. Compare to alternatives: Would other investments yield better ROI?

The Honest ROI Conversation:

• Weak AI projects: visible activity without attributable business value
• Good AI projects: measurable value after deployment and adoption
• Exceptional AI projects: strategic differentiators with durable data, workflow, or product advantages

If your AI strategy team claims spectacular ROI without a baseline, counterfactual, adoption evidence, and operating-cost model, treat it as unproven.

See Also: Chapter 9, Problem Structuring, to define feasible alternatives, decision and chance nodes, gates, and evidence needs; then use Chapter 22, Data Analysis and Insights for expected value or utility, break-even probability, Bayesian updating, value of information, sensitivity, and explicit decision rules. These calculations do not authorize deployment or override the governance in this chapter.

Why This Matters: Mental Models & AI Wisdom

Understanding why AI strategy frameworks work is as important as knowing how to use them. This section explores the mental models that make AI successful, real-world failure cases that reveal what happens when strategy is missing, competing schools of thought, and how AI maturity changes your approach.

Mental Models: Why AI Strategy Works

1. Why AI Opportunity Assessment Works: Systematic Evaluation vs. Hype-Driven Selection

The AI Opportunity Assessment Matrix (Value × Feasibility) is a screening aid for challenging hype-driven selection. It does not prevent failure or replace security, legal, ethical, operational, and financial review.

The Problem: AI hype creates "solution looking for problem" syndrome. Teams see a new frontier, open, or task-specific model and immediately think "We need this!" without asking what decision or workflow it improves. The result can be visible activity without attributable business value.

The Mental Model: Value creation happens at the intersection of capability and need. AI capability alone is worthless. Business need alone doesn't justify AI if you can solve it cheaper/faster without AI. The 2×2 matrix forces explicit trade-offs.

• High Value + High Feasibility = Quick Wins (do immediately)
• High Value + Low Feasibility = Strategic Bets (long-term investment)
• Low Value + High Feasibility = Experiments (capability building)
• Low Value + Low Feasibility = Avoid (waste of resources)

Why it can help: It makes value and feasibility assumptions visible enough to challenge. Estimates such as savings, growth, data readiness, and delivery time should carry an owner, range, evidence source, and sensitivity test.

Constructed example: a company compares an AI strategy assistant with bounded contract-clause extraction. The first has uncertain trust, evaluation, and workflow fit; the second has a clearer corpus and review workflow but still requires confidentiality, privilege, quality, and human-review controls. The matrix supports a staged test; it does not justify invented savings or delivery dates.

Key Insight: AI strategy succeeds when you start with business problems and evaluate AI as one potential solution, not when you start with AI and search for problems.

2. Why Governance Matters Early: Preventing Bias, Safety, and Regulatory Issues

Governance introduced only after deployment may discover design, data, authority, or control problems late. Earlier risk work can reduce exposure and improve detectability, but it cannot guarantee that bias, harm, regulatory breach, or operational failure will not occur. ^{[1] [6] [2]}

The Problem: AI failures may be ordinary software defects, model-performance failures, security incidents, unsafe automation, discrimination, privacy violations, or workflow failures. Consequences depend on the use, affected people, jurisdiction, controls, and response.

The mental model: AI governance is an accountable risk-management system, not an insurance policy or legal safe harbor. Risk review, testing, documentation, oversight, incident response, and ownership can reduce risk and improve decisions; residual risk remains and requires human acceptance by the appropriate authority. ^{[1] [2]}

Think of it like building codes: You don't build a skyscraper and then check if it's safe. You design safety in from day one. Same with AI.

What it is intended to improve:

• Pre-deployment evaluation: tests defined performance, subgroup, safety, and failure-mode questions before release
• Ethical review: identifies affected parties, rights, foreseeable harms, benefit distribution, and remedy options
• Compliance review: identifies applicable obligations and evidence; counsel determines legal applicability
• Accountability structure: assigns decision, monitoring, escalation, and residual-risk owners

Hypothetical example: healthcare AI startup building a diagnostic support tool. No governance initially. Deployed model to hospitals. Later discovered: model trained on data from one demographic group and underperformed on others. Hospitals pulled it and the company faced legal, clinical, and commercial risk.

Governance questions that could have detected or mitigated the risk include:

• whether subgroup performance and uncertainty were evaluated on representative deployment data
• whether the clinical role, human oversight, escalation, and evidence requirements were approved by qualified owners
• whether data provenance, fitness, missingness, and distribution shift were documented

The cost and effectiveness of controls are context-specific. Compare prevention, monitoring, response, and residual-risk options rather than assuming governance is inexpensive or fully protective.

Key insight: build governance into the lifecycle early enough to influence design, sourcing, deployment, and stop decisions—and continue it after release.

3. Why Pilots Matter: Validating ROI Before Full Investment

The biggest AI mistake isn't building the wrong model; it's scaling the wrong model. Pilots validate assumptions before major investment.

The Problem: business cases are projections, not reality. You may assume AI will materially reduce churn, but you do not know whether customers will trust recommendations, sales will use the tool, or production performance will be sufficient.

The Mental Model: pilots are bounded hypothesis tests. Spend a small amount to validate a larger investment. If the pilot fails, you avoid scaling the wrong thing. If it succeeds, you have evidence to justify scaling.

Think of pilots like clinical trials: Phase I (does it work at all?), Phase II (does it work on real users?), Phase III (does it work at scale?). You don't skip to Phase III.

Why it works:

• Reality check: projected model quality can fall in production because of messy data, edge cases, and workflow mismatch
• User feedback: Discover adoption barriers ("Too complex," "Don't trust it," "Workflow doesn't fit")
• ROI validation: Measure actual savings/revenue lift, not projected
• Risk mitigation: Fail small, not big

Hypothetical example: retailer projected that AI demand forecasting would materially reduce inventory costs. It built the full system before piloting. After deployment, forecasts worked for high-volume SKUs but failed on long-tail inventory, so projected savings never arrived.

Better approach: run a bounded pilot on a representative SKU set. Validate accuracy, workflow fit, and actual cost reduction. If it works, scale. If not, pivot or stop before major investment.

Pilot ROI calculation:

• Pilot cost: bounded discovery and implementation spend
• If it kills a bad project: it saves the larger rollout budget
• If it validates a good project: it unlocks a better-supported scale decision
• Risk-adjusted return: positive when the pilot is designed to test the true scale assumptions

Key Insight: Pilots aren't delays—they're insurance policies. Better to spend 3 months validating than 18 months building the wrong thing.

4. Why Data Readiness Matters: Garbage In, Garbage Out Principle

One common reason AI projects fail is bad data. You can't reliably fix a data problem with a better model. ^[7]

The Problem: teams assume "data exists, we're good to go" without checking quality. They may discover too late that missingness, inconsistent labels, insufficient volume, or biased samples make the model unusable.

The Mental Model: AI is only as good as its training data. A mediocre algorithm on clean data beats a sophisticated algorithm on dirty data every time.

Analogy: You can't bake a great cake with rotten ingredients, no matter how skilled the chef. Data quality is your ingredients.

Why it works:

• Volume: need enough representative labeled samples for the decision and model type
• Quality: missingness, errors, duplicates, and outliers need documented handling
• Relevance: Features must correlate with target (garbage features = garbage predictions)
• Labeling: labels must be consistent enough for the model to learn the intended construct
• Recency: Recent data reflects current patterns (2-year-old data may be obsolete)

The 5 Data Readiness Dimensions (from Section 7) are not bureaucracy; they are the survival checklist. If several critical dimensions are weak, stop building models and fix data first.

Hypothetical example: bank building a credit risk model. It assumes CRM data is clean, starts model training, and performance stalls. Investigation finds:

• Many loan applications are missing income data because salespeople were not required to collect it
• "Employment status" has many inconsistent values
• Default labels are inconsistent across products and time periods

The team spends the next phase cleaning data before retraining. Lesson: assess data quality before starting model work.

Data Readiness ROI:

• Time spent on assessment: short compared with a failed model build
• Time saved avoiding bad models: substantial when gaps are found early
• Performance improvement: depends on whether the data fixes address the true error source

Key Insight: Assess data readiness before writing a single line of code. If data isn't ready, your project will fail no matter how good your team is.

Failure Case Studies: What Happens When AI Strategy Fails

Case Study 1: Amazon's Biased Hiring AI (2014-2018) ^[11]

What Happened: Reuters reported that Amazon worked on an AI recruiting tool that was ultimately scrapped after it showed bias against women. The case is useful because it shows how historical hiring data can encode historical bias. ^[11]

The source supports a recruiting-bias cautionary case, not a complete account of Amazon's testing, review, data-governance, or monitoring practices. Treat the controls below as general operating lessons for high-impact recruiting systems, not as claims about Amazon's internal process.

General Control Lessons:

1. Test for material group disparities before decision use

   • Evaluate ranking and selection outcomes across relevant groups.
   • Define documented remediation and escalation paths for material disparities.

2. Require an independent governance review

   • Treat recruiting as a high-impact use case requiring defined accountability.
   • Review intended use, affected populations, controls, and approval criteria before deployment.

3. Audit training data for historical bias and representativeness

   • Document known gaps, limitations, and mitigation choices before model release.
   • Do not assume historical hiring outcomes are a suitable target without review.

4. Monitor outcomes after deployment

   • Track agreed performance, fairness, and process indicators.
   • Pause or revise the system when approved guardrails are breached.

Key Lesson: Historical data can encode historical bias. Recruiting systems should be tested, governed, data-audited, and monitored before their outputs affect applicants.

Case Study 2: Google's Diabetic Retinopathy Screening Deployment Lesson ^[12]

What Happened: Google Research and collaborators evaluated deep-learning diabetic retinopathy screening in Thailand's national screening program. The peer-reviewed case is best read as a deployment and workflow lesson: real-time AI can perform well, but field use still depends on image quality, clinical workflow, staffing, and escalation design. ^[12]

The source supports the importance of deployment and workflow context; it does not establish a case-specific account of omitted pilots, user research, data-quality assessment, or monitoring. Use the following as general deployment controls.

General Deployment Controls:

1. Pilot in the target environment

   • Test with the intended workflows, equipment, staffing model, and escalation paths before scaling.
   • Compare deployment inputs and operating conditions with development assumptions.

2. Assess workflow readiness with users

   • Identify handoffs, time constraints, training needs, and exceptions that affect practical use.
   • Revise the design when the system does not fit the live workflow.

3. Assess field-data quality and coverage

   • Test whether expected production inputs, edge cases, and failure modes are represented in evaluation.
   • Define an escalation path for inputs that do not meet quality requirements.

4. Monitor performance and operational outcomes

   • Track agreed quality, safety, workflow, and escalation indicators after deployment.
   • Investigate material differences between evaluation and field performance.

Key Lesson: Offline evaluation does not by itself establish field readiness. Plan for deployment context, workflow fit, data quality, and monitoring before scaling.

Case Study 3: IBM Healthcare Data and Analytics Asset Sale: Commercialization Checklist ^[13]

What Happened: IBM announced in January 2022 that Francisco Partners would acquire its healthcare data and analytics assets, including Health Insights, MarketScan, Clinical Development, Social Program Management, Micromedex, and imaging software assets. ^[13]

The announcement documents the transaction, not the causes of it or the performance of IBM's healthcare offerings. The following is a hypothetical commercialization checklist for any organization evaluating a broad industry-AI proposition; it is not a causal account of IBM's strategy.

Hypothetical Commercialization Checklist:

1. Define a narrow opportunity

   • Identify the specific user, workflow, data, decision, and intended outcome.
   • Test value and feasibility before funding an expansion thesis.

2. Validate a pilot before scaling

   • Measure adoption, outcome, and operational performance in the target workflow.
   • Use pre-agreed continuation, redesign, and exit criteria.

3. Choose a capability-sourcing approach deliberately

   • Compare building, buying, and partnering against the organization's differentiated capabilities and operating constraints.
   • Limit commitments until the chosen model has adequate use-case evidence.

4. Set measurable commercialization criteria

   • Tie investment decisions to defined user adoption, outcome, risk, and economic measures.
   • Review the evidence at agreed decision points rather than relying on broad category narratives.

Key Lesson: A broad industry label is not a commercialization plan. Define a specific use case, measure value and feasibility, and decide whether to scale from evidence.

Competing Schools of Thought in AI Strategy

1. Build vs. Buy vs. Partner: Capability Development vs. Speed

The Debate: How should you acquire AI capabilities?

Build School (Own Everything)

• Philosophy: AI is core differentiator → Must build in-house to own IP and customize
• Illustrative pattern: firms whose core product or workflow depends on proprietary data or behavior may choose internal control; this is a hypothesis to test, not evidence that any named company endorses a school.
• Strengths:
  • Full control over algorithm, data, roadmap
  • Competitive advantage (competitors can't replicate)
  • Data moat (proprietary data + proprietary models = defensibility)
• Weaknesses:
  • Expensive (hire ML team, build infrastructure, 12-18 months to production)
  • Risky (might fail to build better than vendors)
  • Slower (build takes longer than buy)

Buy School (Use Vendors)

• Philosophy: AI is commodity → Buy best-in-class tools, focus on business problems
• Illustrative pattern: organizations may use vendor capabilities when the use is non-differentiating and the vendor meets lifecycle, assurance, portability, and exit requirements; current product capability requires fresh diligence.
• Strengths:
  • Fast (deploy in weeks, not years)
  • Lower cost (subscription vs. building team)
  • Lower risk (proven solutions)
• Weaknesses:
  • No differentiation (competitors use same tools)
  • Vendor lock-in (hard to switch once integrated)
  • Less customization (generic solutions may not fit your specific problem)

Partner School (Co-Develop)

• Philosophy: Combine vendor expertise with your data → Best of both worlds
• Illustrative pattern: organizations may partner when they need specialist capability plus control of domain data and workflow evidence; partner claims require diligence.
• Strengths:
  • Faster than build (leverage partner's expertise)
  • More customized than buy (tailored to your problem)
  • Shared risk (partner invested in success)
• Weaknesses:
  • Expensive (consulting fees)
  • IP ownership ambiguity (who owns the model?)
  • Dependency (partner leaves, knowledge leaves)

When to Use Each:

• Build: Core differentiator + proprietary data + ability to operate the lifecycle, with internal control justified by evidence
• Buy: Commodity capability + mature vendor market + non-differentiating (e.g., email AI, CRM AI)
• Partner: Need expertise + your unique data + complex problem (e.g., supply chain optimization, fraud detection)

The Pragmatic Approach: Start with Buy/Partner (fast, low risk), build selectively where you have proprietary data + core differentiator. Don't build everything.

2. Centralized vs. Distributed AI Governance: Control vs. Agility

The Debate: Who should control AI development?

Centralized School

• Philosophy: AI too risky to leave to individual teams → Central AI CoE (Center of Excellence) controls all AI
• Model:
  • AI Steering Committee approves all AI projects
  • Central AI team (data scientists, ML engineers) builds all models
  • Standardized MLOps platform, tools, processes
• Strengths:
  • Consistent standards (same bias testing, security, compliance across all models)
  • Economies of scale (shared infrastructure, talent)
  • Risk management (central oversight sets policy, escalation, and independent challenge; it reduces but does not eliminate unauthorized or unsafe use)
• Weaknesses:
  • Slower (bottleneck = central team must build everything)
  • Less innovation (business units can't experiment freely)
  • Disconnect from business (central team may not understand domain problems)

Distributed School

• Philosophy: Innovation requires autonomy → Let business units build their own AI
• Model:
  • Each business unit has AI team or budget to hire vendors
  • Minimal central oversight (guidelines, not approval)
  • Teams choose their own tools, build their own models
• Strengths:
  • Faster (no central bottleneck)
  • More innovation (teams experiment freely)
  • Domain expertise (business unit understands their problem)
• Weaknesses:
  • Inconsistent standards (some teams don't test for bias, security)
  • Duplication (multiple teams solving same problem)
  • Higher risk (rogue AI deployed without oversight)

The Middle Ground: Federated Governance

• Model:
  • Central CoE: Sets standards (bias testing, MLOps, security), provides shared infrastructure, reviews high-risk projects
  • Business Units: Build models for their domain, using central infrastructure, subject to central standards
• Strengths:
  • Balances speed (distributed) with safety (central standards)
  • Economies of scale (shared infra) + domain expertise (business units)
  • Innovation with guardrails
• Illustrative federated pattern: a central function sets standards and assurance while product teams retain delivery authority; use organization-specific evidence rather than treating any named company as a universal model.

When to Use Each:

• Centralized: Early in AI maturity (Level 1-2), high-risk industries (healthcare, finance), or small company (1 AI team)
• Distributed: Mature AI organization (Level 3-4), low-risk use cases, or large company (many teams)
• Federated: a possible design when business units need delivery authority and a central function retains standards, assurance, and escalation

3. Experimentation-First vs. Strategy-First: Learning vs. Planning

The Debate: Should you plan AI strategy upfront or learn through experimentation?

Experimentation-First School (Lean Startup for AI)

• Philosophy: AI too uncertain to plan → Run experiments, learn fast, pivot
• Model:
  • Launch 10 small AI pilots ($50K each)
  • See which ones work
  • Scale winners, kill losers
  • Strategy emerges from experiments
• Strengths:
  • Faster learning (real data beats projections)
  • Lower risk (small failures vs. big bets)
  • Adaptability (pivot based on results)
• Weaknesses:
  • Scattered effort (no focus)
  • Missed big opportunities (no one experiments on hard, high-value problems)
  • No infrastructure investment (experiments don't justify MLOps platform)

Strategy-First School (Traditional Planning)

• Philosophy: AI too expensive to waste → Plan strategy, prioritize, execute
• Model:
  • 3-month strategy phase (opportunity assessment, use case prioritization, roadmap)
  • Select top 3 use cases (high value + feasible)
  • Build pilots for top 3 (not 10 random experiments)
  • Scale winners
• Strengths:
  • Focused effort (resources on highest-value use cases)
  • Justifies infrastructure (roadmap justifies MLOps investment)
  • Alignment (exec buy-in on strategy)
• Weaknesses:
  • Slower (3 months planning before first experiment)
  • Analysis paralysis (over-planning, under-doing)
  • Missed opportunities (strategy may miss emerging use cases)

The Middle Ground: Strateg experimentation

• Model:
  • Month 1-2: Quick strategy (opportunity assessment, prioritize top 5 use cases)
  • Month 3-8: Pilot top 3 use cases in parallel (learn fast)
  • Month 9+: Scale winners, kill losers, update strategy based on learnings
• Strengths:
  • Fast start (2 months, not 6)
  • Focused experiments (top 5, not random 10)
  • Learning informs strategy (iterate based on results)

When to Use Each:

• Experimentation-First: Early-stage startups, high uncertainty, culture of rapid testing
• Strategy-First: Enterprises, high-risk domains (need exec buy-in), clear use cases
• Strategic Experimentation: useful when uncertainty is material and experiments have explicit learning, risk, budget, and stop rules

Stage Dependency: How AI Strategy Changes with Maturity

Your AI approach should evolve as your organization matures. What works at Level 1 (Ad Hoc) fails at Level 4 (Strategic).

Early Stage (AI Maturity Level 1-2): Proof-of-Concept Focus

Characteristics:

• Few AI projects
• Small/no AI team
• No MLOps infrastructure
• Exec skepticism ("Does AI work for us?")

AI Strategy:

• Goal: Prove AI works (deliver one successful use case)
• Approach:
  • Pick one Quick Win (High Value + High Feasibility)
  • Lightweight governance matched to risk, with escalation for high-impact use cases
  • Buy or Partner (don't build; too slow)
  • Measure ROI obsessively (prove value to get next project funded)
• Governance: lightweight but explicit: manager approval, basic security, data-use review, and risk escalation
• Infrastructure: Use vendor platforms (AWS SageMaker, Google Vertex AI; don't build MLOps)
• Success = at least one AI project in production delivering measurable value

Example: retailer wants to prove AI value. Picks a customer-support chatbot as a quick win, buys a vendor solution, deploys on a narrow ticket category, and measures safe deflection, customer satisfaction, and support-cost change.

Mistake to Avoid: building custom AI infrastructure before proving AI works.

Growth Stage (AI Maturity Level 2-3): Portfolio Management

Characteristics:

• Multiple AI projects in flight
• Growing AI team
• Emerging governance (starting ethics reviews)
• Multiple pilots, some in production

AI Strategy:

• Goal: scale from first success to a managed portfolio
• Approach:
  • Use Case Prioritization (rank quick wins and strategic bets explicitly)
  • Emerging governance (ethics board for high-risk use cases, bias testing protocols)
  • Build + Buy (build where proprietary data, buy for commodities)
  • MLOps investment (need infrastructure to support repeatable deployment and monitoring)
• Governance:
  • AI Steering Committee (quarterly reviews)
  • Ethics Board (reviews high-risk projects)
  • Policies (data usage, model approval, incident response)
• Infrastructure: Build MLOps capabilities for model registry, monitoring, reproducible evaluation, approved change workflows, staged release, rollback, and evidence retention. Workflow automation must not turn a drift alert into automatic retraining or deployment.
• Success = multiple AI projects in production with measurable cumulative value

Example: bank has a growing AI portfolio. It builds a centralized MLOps platform, establishes review for credit-risk and fraud models, and tracks production value by use case.

Mistake to Avoid: No governance → Bias/compliance issues → Model shut down, regulatory fines.

Scale Stage (AI Maturity Level 4-5): Enterprise AI

Characteristics:

• Broad AI portfolio
• Large or deeply networked AI organization
• Mature governance (formal ethics frameworks)
• AI embedded in most business processes

AI Strategy:

• Goal: AI as competitive moat
• Approach:
  • Portfolio management (balance Quick Wins, Strategic Bets, Experiments)
  • Full governance (ethics review for all high-risk, ongoing bias monitoring)
  • Build for core differentiators (proprietary models on proprietary data)
  • Data flywheel (more users → more data → better models → more users)
• Governance:
  • AI Steering Committee (monthly reviews, budget oversight)
  • AI Ethics Board (reviews all high-risk, audits production models quarterly)
  • Comprehensive policies (fairness, transparency, accountability, safety)
  • Incident response (process for when AI causes harm)
• Infrastructure: MLOps capabilities proportionate to the portfolio, such as governed feature pipelines, controlled experiments, versioned evaluation, explanation tooling where useful, and approved retraining and release workflows. Preserve human ownership of material model changes.
• Success = AI drives material revenue, cost, quality, risk, or competitive-advantage outcomes

Example: large AI-native platform companies embed AI across products, governance, infrastructure, and data feedback loops.

Mistake to Avoid: Treating AI like science project (no accountability, no business impact) → Expensive team delivering no ROI.

Operating Manual: The Canonical Constructed 16-Week AI Use-Case Pilot

This operating manual provides a constructed week-by-week template for moving an AI use case from opportunity assessment through a possible production decision. It is designed for teams who want detailed, actionable steps with decision gates and red flags, but it is not a universal delivery promise or a substitute for the shorter or longer path justified by the use case.

Constructed operating-manual boundary: Every week, hour, count, score, budget, sample size, percentage, product name, threshold, role allocation, example result, and rollout step below is a teaching placeholder unless it is explicitly tied to a source or the local project record. Replace it with the organization's evidence, method owner, legal and control-owner review, staffing, procurement terms, and approved decision rule. The template can produce a stop, redesign, stage, or no-AI decision; it must not be used to manufacture a go decision.

Overview: 16-Week Timeline

The operating manual is structured in 6 phases:

• Weeks 1-2: Opportunity Assessment (10 days)
• Weeks 3-4: Business Case Development (10 days)
• Weeks 5-6: Data Assessment & Governance (10 days)
• Weeks 7-10: Pilot Development (20 days)
• Weeks 11-14: Pilot Deployment (20 days)
• Weeks 15-16: Pilot Review & Scaling Plan (10 days)

Total time investment: depends on scope, data condition, risk profile, and staffing model. Financial investment: estimate from team time, infrastructure, tools, vendor costs, governance, and change management.

Phase 1: Opportunity Assessment (Weeks 1-2)

Goal: Identify and select the highest-value, most feasible AI use case for your pilot.

Week 1: Use Case Brainstorming & Scoring

Day 1-2: Cross-Functional Workshop (16 hours)

• Activities:
  • Assemble AI pilot team (sponsor, product owner, data scientist, engineer, business stakeholder)
  • Run a time-boxed brainstorming workshop with a representative set of business stakeholders
  • Generate a sufficiently broad set of potential AI use cases using Framework #1 (Opportunity Assessment)
  • Categories: Automate (reduce costs), Augment (increase productivity), Innovate (new capabilities)
• Workshop structure:
  • Hour 1: Educate on relevant capability classes and validate current options as of the review date
  • Hour 2: Brainstorm by business function (sales, marketing, operations, finance, product)
  • Hour 3: Categorize use cases (automate vs augment vs innovate)
  • Hour 4: Initial scoring on value + feasibility
• Output: use case ideas documented in a reviewable decision record
• Red flag: idea coverage or participation is below the locally set minimum → broaden participation or improve problem-framing before ranking

Day 3-4: Use Case Scoring & Prioritization (16 hours)

• Activities:
  • Score each use case on Framework #4 (Use Case Prioritization)
  • Dimensions: locally defined evidence of value, feasibility, data readiness, and time or capacity constraints
  • Use Framework #4's documented factors and sensitivity analysis; do not introduce a second composite formula without recording its rationale
  • Rank candidates only after constraint, risk, and non-AI alternative review
• Scoring criteria:
  • Value: locally defined evidence of revenue, cost, quality, risk, customer, or workforce impact
  • Feasibility: technical, operational, data, control, and staffing constraints defined for the use case
  • Data: readiness evidence for authority, relevance, quality, labels, coverage, and infrastructure
  • Speed: a range based on scope, dependencies, review, validation, and rollout requirements
• Output: ranked candidates with scores, evidence, constraints, and unresolved questions
• Red flag: no candidate clears the locally approved feasibility and control rule → revisit the problem, scope, or non-AI alternatives

Day 5: Stakeholder Alignment (8 hours)

• Activities:
  • Present top 10 use cases to executive sponsor
  • Validate business value assumptions with business unit leaders
  • Confirm data availability with data/IT teams
  • Select top 3 finalists for detailed assessment in Week 2
• Selection criteria:
  • Top local near-term candidates: high value and feasible under the approved evidence and control boundary
  • Top local strategic candidate: high value with explicitly owned dependencies and staged learning
• Output: 3 finalist use cases approved for deep dive
• Red flag: no accountable sponsor or decision owner → pause selection until authority and resources are clear

Week 2: Detailed Use Case Assessment

Day 1-2: Business Value Deep Dive (16 hours)

• Activities:
  • For each of 3 finalists, quantify business value in detail:
    • Revenue increase: how much value would a credible conversion lift create?
    • Cost reduction: estimate the affected work, loaded cost, service impact, transition cost, and uncertainty; do not assume that automation equals headcount reduction
    • Risk reduction: how much value would lower fraud loss or downtime create?
  • Interview a representative set of business stakeholders and affected workers appropriate to the use case
  • Document current state vs. AI-enabled future state
• Output: Detailed business value assessment for each finalist
• Red flag: Can't quantify value beyond "it would be cool" → Not ready for investment

Day 3-4: Technical Feasibility Assessment (16 hours)

• Activities:
  • For each finalist, assess technical requirements:
    • Data requirements: What data needed? Is it available? Quality?
    • Model approach: Supervised learning, LLM, computer vision, etc.?
    • Infrastructure: Can run on existing cloud or need new setup?
    • Third-party solutions: Buy vs. build assessment
  • Consult with the technical, data, security, privacy, legal, and domain owners needed for the use case
  • Research vendor solutions (if buy option considered)
• Technical feasibility checklist:

  USE CASE: [Name]
  
  DATA:
  □ Data exists (internal or can acquire)
  □ Data quality acceptance criteria are defined for the decision, population, features, labels, missingness, measurement error, and harm of mistakes
  □ Evidence volume and coverage are justified for the model class, task complexity, subgroup and tail performance, uncertainty, and validation design
  □ Label quality, authority, provenance, and timing are feasible for the pilot; owner-set values and deadlines are documented rather than treated as universal cutoffs
  
  TECHNOLOGY:
  □ Proven approach exists (not research project)
  □ Open-source models/tools available OR vendor solution available
  □ Required skills, independent review, and accountable owners are available on the locally approved schedule
  □ Infrastructure, security, privacy, monitoring, rollback, and incident capabilities can meet the locally approved pilot plan
  
  DECISION: □ Feasible (proceed)  □ Risky (needs mitigation)  □ Infeasible (reject)
  

• Output: Technical feasibility scorecard for each finalist
• Red flag: All 3 finalists marked "infeasible" → Go back to use case list, select simpler options

Day 5: Final Use Case Selection (8 hours)

• Activities:
  • Compare the finalists using Framework #4's documented factors, sensitivity analysis, constraints, and non-AI alternatives
  • Select one use case for the constructed pilot template, or stop if no option meets the local gate
  • Document selection rationale (why this one vs. others?)
  • Get executive sponsor sign-off
• Selection criteria:
  • Strongest documented case after sensitivity and constraint review
  • Clear business sponsor who will champion
  • Data access, authority, and evidence plan fit the approved schedule
  • Can demonstrate a decision-relevant result within the approved schedule, which may be shorter or longer than this template
• Output: Selected use case with executive approval

Decision Gate #1: End of Week 2

• Go criteria:
  • One use case selected with a documented rationale that survives the approved score and sensitivity review
  • Quantified business value beats the approved materiality threshold
  • Technical feasibility confirmed (proven approach, data available)
  • Executive sponsor committed (budget + resources approved)
• No-Go criteria:
  • No use case clears the locally approved decision rule → go back to problem framing or choose no AI
  • Value not quantifiable → Not ready for AI investment
  • No accountable sponsor, resources, or decision authority → pause until the gap is resolved

Contingency: If No-Go:

1. Expand brainstorming: bring in more business units and improve coverage of the decision space
2. Lower ambition: test a narrower, lower-risk option or a non-AI alternative
3. Get executive buy-in: Present AI strategy to C-suite before proceeding

Phase 2: Business Case Development (Weeks 3-4)

Goal: Build comprehensive business case with ROI projection, resource plan, and risk assessment.

Week 3: ROI Modeling & Resource Planning

Day 1-2: Baseline Metrics (16 hours)

• Activities:
  • Document current state performance:
    • Current process: How does it work today? (manual, rules-based, etc.)
    • Current performance: What's the baseline? (accuracy, speed, cost)
    • Current cost: FTEs, tools, error costs
  • Example: document support staffing cost, ticket volume, first-contact resolution, and handle time
  • Gather a historical period sufficient to establish the baseline, seasonality, mix, and uncertainty for the decision
• Baseline documentation template:

  CURRENT STATE:
  - Process: [Describe current workflow]
  - Volume: [Transactions/tickets/tasks per month]
  - Performance: [Key metric - accuracy, speed, quality]
  - Cost: [FTEs × salary + tools + error costs]
  - Pain points: [What's broken or inefficient?]
  
  BASELINE METRICS (approved baseline window):
  - [Metric 1]: [Value] (e.g., First-contact resolution: 80%)
  - [Metric 2]: [Value] (e.g., Average handle time: 12 minutes)
  - [Cost]: [Value] (e.g., $500K annually)
  

• Output: Current state baseline documented with 6 months of data
• Red flag: Can't access historical data → Data governance issue, may block pilot

Day 3-4: ROI Calculation (16 hours)

• Activities:

  • Project AI-enabled future state:
    • Target performance improvement in operational terms
    • Value created: Revenue gain or cost savings
    • Implementation cost: Team time, infrastructure, tools, vendors
    • Ongoing costs: Maintenance, monitoring, retraining
  • Calculate 3-year ROI using Framework #5:
    • Year 1 net value: partial value during the production window - implementation costs
    • Year 2-3 net value: full annual value - ongoing costs
    • ROI = (Total value - Total costs) / Total costs

• ROI model template:

  The figures in the following block are a constructed arithmetic illustration only. Replace every amount, timing, ramp, cost, and hurdle with the approved business case and report a range where uncertainty is material.

  INVESTMENT (Year 1):
  - Team time: 1,200 hours × $150/hr = $180K
  - Infrastructure: Cloud GPU, MLOps tools = $20K
  - Vendor costs: API calls, licenses = $10K
  - Total Year 1 investment: $210K
  
  VALUE CREATED:
  - Year 1 (6 months): $250K (50% of annual value)
  - Year 2: $500K (full annual value)
  - Year 3: $500K
  - 3-year total value: $1.25M
  
  ONGOING COSTS (Years 2-3):
  - Maintenance: $30K/year
  - Monitoring/retraining: $20K/year
  - Total ongoing: $50K/year
  
  3-YEAR ROI:
  - Total value: $1.25M
  - Total costs: $210K + $100K = $310K
  - Net value: $940K
  - ROI: ($940K / $310K) = approximately 303% net ROI; gross value/cost is approximately 4.03x
  - Payback illustration: with a six-month build period and $500K/year of linear production value, $210K is recovered about 5.0 months after production starts, or about 11 months from project start; state the convention used
  

• Output: 3-year ROI model with sensitivity analysis on key assumptions

• Red flag: ROI misses the approved hurdle -> use case not economically compelling

Day 5: Resource Planning (8 hours)

• Activities:
  • Identify team roles needed:
    • Product owner (0.5 FTE): Define requirements, prioritize features
    • Data scientist (1 FTE): Model development, experimentation
    • ML engineer (0.5 FTE): Infrastructure, deployment
    • Data engineer (0.5 FTE): Data pipeline, quality
    • Business analyst (0.25 FTE): Metrics, reporting
  • Assess build vs. buy:
    • Build: Custom model on proprietary data (higher cost, more control)
    • Buy: Vendor solution or API (lower cost, less customization)
  • Document resource needs and availability
• Resource plan template:

  TEAM COMPOSITION (16-week pilot):
  - Product owner: [Name], 0.5 FTE (20 hrs/week)
  - Data scientist: [Name/Hire], 1 FTE (40 hrs/week)
  - ML engineer: [Name], 0.5 FTE (20 hrs/week)
  - Data engineer: [Name], 0.5 FTE (20 hrs/week)
  - Business analyst: [Name], 0.25 FTE (10 hrs/week)
  
  BUILD vs. BUY DECISION:
  □ Build (custom model on proprietary data)
  □ Buy (vendor solution: [Vendor name])
  □ Hybrid (buy base model, fine-tune on our data)
  
  Rationale: [Why this approach?]
  

• Output: Resource plan with named team members and build/buy decision
• Red flag: Can't staff 1 FTE data scientist → Pilot will fail without technical expertise

Week 4: Risk Assessment & Business Case Finalization

Day 1-2: Risk Identification & Mitigation (16 hours)

• Activities:
  • Use Framework #6 (Ethical AI) to assess ethical risks:
    • Bias: Could model discriminate? (protected classes, historical bias)
    • Privacy: Does it use sensitive data? (PII, health data)
    • Transparency: Can we explain decisions? (regulatory requirement?)
    • Safety: Could it cause harm? (physical, financial, reputational)
  • Document risk level (Low/Medium/High) and mitigation plan
  • Involve legal/compliance team for high-risk use cases
• Risk assessment template:

  RISK ASSESSMENT:
  
  BIAS RISK: [Low/Medium/High]
  - Description: [Could model treat groups differently?]
  - Mitigation: [Bias testing, diverse training data, fairness constraints]
  
  PRIVACY RISK: [Low/Medium/High]
  - Description: [What sensitive data is used?]
  - Mitigation: [Data anonymization, consent, encryption, access controls]
  
  TRANSPARENCY RISK: [Low/Medium/High]
  - Description: [Do we need to explain decisions?]
  - Mitigation: [Explainability tools, human review, audit trail]
  
  SAFETY RISK: [Low/Medium/High]
  - Description: [Could AI cause harm?]
  - Mitigation: [Human-in-loop, confidence thresholds, fallback to manual]
  
  OVERALL RISK: [Low/Medium/High]
  ACTION: □ Proceed  □ Requires ethics review  □ Too risky
  

• Output: Risk assessment with mitigation plan
• Red flag: High risk across multiple dimensions → May need ethics board review before proceeding

Day 3-4: Business Case Document (16 hours)

• Activities:
  • Compile comprehensive business case (10-15 pages):
    • Executive summary (1 page): Problem, solution, ROI, ask
    • Use case description (2 pages): Current vs. future state
    • ROI model (2 pages): Investment, value, payback
    • Resource plan (1 page): Team, build/buy, timeline
    • Risk assessment (2 pages): Risks + mitigation
    • Success metrics (1 page): How we'll measure success
    • 16-week plan (2 pages): Week-by-week milestones
  • Review with stakeholders (product, data, legal, finance)
  • Incorporate feedback
• Output: Business case document ready for approval
• Red flag: Stakeholders raise major objections → May need to revise use case or approach

Day 5: Approval & Kickoff (8 hours)

• Activities:
  • Present business case to executive sponsor and steering committee
  • Get budget approval based on the approved scope, loaded labor, infrastructure, vendor, governance, and change costs
  • Get resource commitments (team members allocated)
  • Schedule kickoff meeting for Week 5
• Approval checklist:

  □ Executive sponsor approved budget
  □ Team members committed (product owner, data scientist, engineer)
  □ Data access approved (if needed)
  □ Infrastructure budget allocated (cloud, tools)
  □ Success metrics agreed upon
  □ 16-week timeline accepted
  

• Output: Approved business case, budget allocated, team assembled

Decision Gate #2: End of Week 4

• Go criteria:
  • Business case approved with ROI above the hurdle rate
  • Budget allocated for the pilot scope
  • Team staffed with product ownership and technical delivery capacity
  • Data access confirmed (can get data within 2 weeks)
  • Risk mitigation plan in place
• No-Go criteria:
  • ROI misses hurdle -> not economically viable
  • Can't staff team → Delay until resources available
  • High risk with no mitigation → Requires ethics review or use case change

Contingency: If No-Go:

1. Reduce scope: Smaller pilot with lower cost and faster timeline (8-10 weeks)
2. Adjust timeline: Push start date until resources available
3. Select different use case: Go back to Week 2 finalists

Phase 3: Data Assessment & Governance (Weeks 5-6)

Goal: Validate data quality and establish governance framework for responsible AI deployment.

Week 5: Data Discovery & Quality Assessment

Day 1-2: Data Source Identification (16 hours)

• Activities:
  • Map all data sources needed for use case:
    • Internal databases: CRM, ERP, data warehouse, logs
    • External sources: Third-party APIs, public datasets, vendor data
    • Unstructured data: Documents, images, audio (if applicable)
  • For each source, document:
    • Location: Where is data stored?
    • Volume: How much data? (rows, GB)
    • Access: Who owns it? How to get access?
    • Refresh: Real-time, daily, weekly batch?
• Data inventory template:

  DATA SOURCES:
  
  Source 1: [Name, e.g., "CRM customer records"]
  - Location: [Salesforce, PostgreSQL, S3, etc.]
  - Volume: [500K records, 2GB]
  - Fields: [Customer_ID, Purchase_History, Support_Tickets]
  - Owner: [Sales team, IT contact: owner@example.com] <!-- illustrative -->
  - Access: [API, database query, CSV export]
  - Refresh: [Real-time, daily batch]
  
  [Repeat for each source]
  

• Output: Complete data inventory (5-10 sources typical)
• Red flag: Data owner says "no access" → Escalate to executive sponsor immediately

Day 3-5: Data Quality Assessment (24 hours)

• Activities:
  • Use Framework #7 (Data Readiness) to assess quality:
    • Completeness: how many records meet the approved completeness threshold?
    • Accuracy: What evidence supports correctness? (use a sample and review design justified by the decision)
    • Consistency: Are formats consistent? (dates, names, categories)
    • Timeliness: How fresh is data? (real-time, daily, weekly, stale?)
  • Pull a representative sample or other evidence set justified by the task, population, tail coverage, uncertainty, and validation design
  • Run data quality scripts:
    • Missing values: df.isnull().sum()
    • Duplicates: df.duplicated().sum()
    • Outliers: Statistical analysis (z-score, IQR)
  • Document data quality issues
• Data quality scorecard:

  CONSTRUCTED DATA QUALITY ASSESSMENT ILLUSTRATION:
  
  Dataset: [Name]
  Sample size: [methodologically justified evidence set]
  
  COMPLETENESS:
  - Field 1 (Customer_ID): 100% complete ✓
  - Field 2 (Email): 85% complete ⚠ (15% missing)
  - Field 3 (Purchase_History): 60% complete ⚠ (40% missing)
  
  ACCURACY (sampled 100 records):
  - Email format valid: 95% ✓
  - Phone number valid: 80% ⚠
  - Address complete: 70% ⚠
  
  CONSISTENCY:
  - Date format: 3 different formats found ⚠ (need to standardize)
  - Category values: 85% use standard taxonomy ✓
  
  TIMELINESS:
  - Last updated: Daily ✓
  - Lag: <24 hours ✓
  
  OVERALL QUALITY SCORE: 75/100 (MEDIUM)
  ISSUES TO ADDRESS:
  1. Standardize date formats
  2. Fill missing emails (or exclude from training)
  3. Validate phone numbers (or drop field)
  

• Output: Data quality memo with specific issues identified
• Red flag: quality score below threshold -> significant data engineering work needed before modeling

Week 6: Data Governance & Compliance

Day 1-2: Data Privacy & Compliance (16 hours)

• Activities:
  • Identify sensitive data:
    • PII: Names, emails, SSN, addresses
    • Protected: Health data (HIPAA), financial data (PCI-DSS), children's data (COPPA)
    • Regulated: Industry-specific (GDPR for EU, CCPA for California)
  • Consult with legal/compliance team
  • Determine handling requirements:
    • Anonymization: Remove identifiers
    • Pseudonymization: Replace identifiers with tokens
    • Encryption: At rest + in transit
    • Access controls: Who can see data?
    • Retention: How long to keep? When to delete?
  • Document compliance requirements
• Privacy & compliance checklist:

  DATA PRIVACY ASSESSMENT:
  
  SENSITIVE DATA:
  □ Contains PII (names, emails, SSN, etc.)
  □ Contains protected health information (HIPAA)
  □ Contains payment data (PCI-DSS)
  □ Contains children's data (COPPA)
  □ Subject to GDPR (EU residents)
  □ Subject to CCPA (California residents)
  
  COMPLIANCE REQUIREMENTS:
  □ Legal review completed
  □ Data anonymization plan in place
  □ Encryption configured (at rest + in transit)
  □ Access controls defined (role-based access)
  □ Data retention policy documented
  □ User consent obtained (if required)
  
  APPROVAL:
  □ Legal team approved
  □ Compliance team approved
  

• Output: Privacy & compliance plan approved by legal
• Red flag: Legal blocks data use → May need to anonymize or select different data source

Day 3-4: Data Pipeline Development (16 hours)

• Activities:
  • Design data pipeline:
    • Extract: Pull data from sources (APIs, databases, files)
    • Transform: Clean, standardize, feature engineering
    • Load: Store in ML-ready format (feature store, data warehouse)
  • Build initial pipeline (ETL scripts or tools like Airflow, dbt)
  • Validate: Does pipeline produce clean, consistent data?
  • Schedule: Set up automated refresh (daily, weekly)
• Pipeline architecture:

  DATA PIPELINE:
  
  EXTRACT:
  - Source 1 (CRM): API pull, daily at 2am
  - Source 2 (Support): Database query, daily at 3am
  - Source 3 (Logs): S3 batch, daily at 4am
  
  TRANSFORM:
  - Remove duplicates
  - Fill missing values (median for numeric, mode for categorical)
  - Standardize formats (dates, phone numbers)
  - Feature engineering (RFM score, sentiment analysis)
  
  LOAD:
  - Destination: Feature store (Feast) or Data warehouse (Snowflake)
  - Format: Parquet files, partitioned by date
  - Refresh: Daily at 6am
  
  VALIDATION:
  - Row count check against an approved local range and known source changes
  - Data quality check against approved completeness and fitness criteria
  - Alert on failure: Email to team + Slack
  

• Output: Data pipeline producing clean ML-ready data
• Red flag: Pipeline fails validation → Data engineering issues, may delay pilot by 1-2 weeks

Day 5: Governance Framework (8 hours)

• Activities:
  • Establish AI governance for pilot:
    • Roles: Who approves model changes? Who monitors production?
    • Policies: Model approval process, bias testing requirements, incident response
    • Tools: Model registry (MLflow), monitoring (Evidently, Fiddler)
  • Document governance plan (Framework #10)
  • Set up weekly review meetings (product owner, data scientist, business stakeholder)
• Governance framework:

  AI GOVERNANCE (Pilot):
  
  ROLES:
  - Product owner: Defines requirements, approves model for production
  - Data scientist: Develops model, runs bias tests
  - Business stakeholder: Validates business impact
  - Legal/compliance: Approves high-risk models
  
  APPROVAL PROCESS:
  1. Model development complete → Data scientist
  2. Appropriate performance and fairness evaluation passed against pre-specified metrics, groups, uncertainty bounds, and context-specific thresholds approved by the methodological, legal, and business owners → Evaluation owner
  3. Business validation (performance meets targets) → Product owner
  4. Compliance review (if high-risk) → Legal
  5. Production deployment approval → Product owner + Stakeholder
  
  MONITORING:
  - Performance: Daily (accuracy, latency, errors)
  - Performance and fairness: at a cadence matched to use, volume, detectability, and harm; review applicable group definitions and lawful data use with Legal/Privacy
  - Business impact: Weekly (KPIs vs. baseline)
  
  INCIDENT RESPONSE:
  - Performance crosses an approved guardrail → contain exposure, preserve the version and evidence, investigate, and use the documented rollback or escalation path
  - A fairness or harm signal crosses an approved guardrail → pause or restrict the affected use when warranted, investigate causes, assess affected people and remedy, and revalidate any proposed change
  - Compliance violation → Immediate shutdown, notify legal
  

• Output: Governance framework documented and approved

Decision Gate #3: End of Week 6

• Go criteria:
  • Data quality evidence meets the approved use-case threshold and unresolved gaps are owned
  • Data pipeline producing clean ML-ready data
  • Privacy/compliance approval obtained
  • Governance framework in place
• No-Go criteria:
  • Blocking data gaps remain → extend data work, reduce scope, use another method, or stop according to the approved decision rule
  • Legal blocks data use → Need to anonymize or change approach
  • Can't build data pipeline → Technical blocker, may need vendor solution

Contingency: If No-Go:

1. Extend data phase: Add 2 weeks for data engineering (cleaning, pipelines)
2. Reduce data scope: Use subset of data with higher quality
3. Buy data: If internal data insufficient, acquire third-party dataset

Phase 4: Pilot Development (Weeks 7-10)

Goal: Develop, validate, and prepare AI model for deployment.

Weeks 7-8: Model Development Sprint 1

Week 7: Baseline Model (40 hours)

• Activities:
  • Define success metrics:
    • Performance: Accuracy, precision, recall, F1 (for classification)
    • Business: Revenue impact, cost savings, efficiency gain
    • Latency: owner-approved response-time and tail-latency objectives derived from the user workflow, load, safety, cost, dependencies, and fallback behavior
  • Build baseline model:
    • Simple approach first (logistic regression, decision tree, rule-based)
    • Establishes performance floor to beat
  • Split data into train, validation, and test sets using the approved validation design
  • Train baseline model on training set
  • Evaluate on validation set
• Baseline model scorecard:

  BASELINE MODEL:
  
  Approach: [Logistic regression, decision tree, heuristic]
  Training data: [50K examples, balanced classes]
  
  PERFORMANCE (Validation Set):
  - Accuracy: 75%
  - Precision: 72%
  - Recall: 70%
  - F1 score: 71%
  - Latency: 50ms
  
  BUSINESS METRIC:
  - Current state: 80% first-contact resolution
  - Baseline model: 82% first-contact resolution
  - Improvement: +2% (modest)
  
  CONCLUSION: Baseline works but needs improvement to hit target (+10%)
  

• Output: Baseline model with target performance comparison
• Red flag: Baseline performs worse than current state → Data or feature engineering issue

Week 8: Advanced Model Development (40 hours)

• Activities:
  • Experiment with advanced approaches:
    • ML models: Random forest, XGBoost, neural networks
    • Language models: compare current approved models and non-model alternatives for the text-heavy use case
    • Ensembles: Combine multiple models for better performance
  • Feature engineering:
    • Create new features from raw data (e.g., RFM score, time since last event)
    • Test feature importance (which features matter most?)
  • Hyperparameter tuning:
    • Grid search or Bayesian optimization
    • Use cross-validation or another leakage-resistant validation design appropriate to the data-generating process
  • Iterate the number of model versions justified by the evidence and review capacity
  • Select best model based on validation performance
• Model experiment tracking:

  EXPERIMENT LOG:
  
  Experiment 1: Random Forest
  - Features: 20 (all available)
  - Validation accuracy: 82%
  - Latency: 80ms
  - Result: +7% improvement over baseline ✓
  
  Experiment 2: XGBoost
  - Features: 15 (removed low-importance features)
  - Validation accuracy: 85%
  - Latency: 60ms
  - Result: +10% improvement, faster ✓✓
  
  Experiment 3: Neural Network
  - Features: 20
  - Validation accuracy: 86%
  - Latency: 200ms ⚠ (too slow)
  - Result: Best accuracy but latency fails requirement
  
  SELECTED MODEL: XGBoost (Experiment 2)
  - Best balance of accuracy (85%) + latency (60ms)
  

• Output: Best-performing model selected, ready for testing
• Red flag: can't beat baseline by the approved minimum improvement -> may need more data, better features, or different approach

Weeks 9-10: Model Validation & Testing

Week 9: Bias Testing & Explainability (40 hours)

• Activities:
  • Bias testing (Framework #6: Ethical AI):
    • Test model performance across protected classes (race, gender, age)
    • Measure group-level outcome and error differences using counsel-approved fairness criteria
    • Example: if one group is materially disadvantaged, pause and investigate
  • If bias detected:
    • Re-balance training data (oversample underrepresented groups)
    • Use fairness constraints (ensure equal opportunity across groups)
    • Retrain and re-test
  • Explainability:
    • Use SHAP or LIME to explain individual predictions
    • Validate: Do explanations make business sense?
    • Example: "Approved because high income + low debt + long credit history" (makes sense ✓)
• Bias testing memo:

  BIAS TESTING RESULTS:
  
      Groups tested: counsel-approved protected or risk-relevant groups
  
      GROUP COMPARISON:
      - Group A approval and error rates: within approved tolerance
      - Group B approval and error rates: within approved tolerance
      - Group C approval or error rate: outside tolerance - review required
  
      MITIGATION:
      - Re-balanced training data (oversampled underrepresented groups)
      - Retrained model with fairness constraint
      - Re-test: group-level results returned within approved tolerance
      ```
  

• Output: Bias testing passed, model ready for deployment
• Red flag: can't mitigate material group-level harm → legal, ethics, and business review before deployment

Week 10: Business Validation & User Testing (40 hours)

• Activities:
  • Test model on a holdout test set never seen during training:
    • Confirm performance matches validation results
    • Example: validation and test performance are close enough to support deployment
  • Business validation:
    • Run model predictions on last month's real data
    • Compare AI predictions to actual human decisions
    • Measure: How often does AI agree with humans? Where does it differ?
  • User acceptance testing:
    • Show model outputs to representative end users
    • Ask: "Do these predictions make sense?" "Would you trust this?"
    • Collect feedback on UX, explanations, confidence levels
• Validation results:

  TEST SET PERFORMANCE:
      - Accuracy: meets approved target
      - Precision: meets approved target
      - Recall: meets approved target
      - F1: meets approved target
      - Latency: meets workflow requirement
  
      BUSINESS VALIDATION (Last month's data):
      - AI agreement with human decisions: tracked
      - AI disagreement with human decisions: reviewed
      - Analysis of disagreements:
        - AI more accurate in some cases
        - Human decision more appropriate in some cases
        - Some cases unclear and need more evidence
      - Conclusion: AI value depends on disagreement quality, not agreement alone
  
      USER ACCEPTANCE:
      - Most users: "Predictions make sense"
      - Most users: "I would trust this with the right controls"
      - Feedback: "Add confidence score to predictions" (implemented)
  

• Output: Model validated on business metrics and user acceptance
• Red flag: Users don't trust model → UX issue or model not ready for production

Decision Gate #4: End of Week 10

• Go criteria:
  • Model performance beats baseline by a meaningful, pre-defined amount
  • Bias testing passed under the approved fairness protocol
  • Test set performance consistent with validation
  • Users trust model outputs enough for the intended workflow
  • Latency meets requirements
• No-Go criteria:
  • Can't beat baseline → Need more data, different approach, or abandon use case
  • Bias can't be mitigated → Legal/ethical risk too high
  • Users don't trust → Need to improve explainability or UX

Contingency: If No-Go:

1. Extend development: 2 more weeks of feature engineering and model iteration
2. Reduce scope: Deploy to subset of use cases where model performs well
3. Human-in-loop: Deploy as recommendation system (human makes final decision) instead of fully automated

Phase 5: Pilot Deployment (Weeks 11-14)

Goal: Deploy model to production, monitor performance, and validate business impact.

Week 11-12: Production Deployment

Week 11: Infrastructure Setup (40 hours)

• Activities:
  • Set up production infrastructure (Framework #8: MLOps):
    • Model serving: Deploy model API (FastAPI, TensorFlow Serving, SageMaker)
    • Monitoring: Set up dashboards (Evidently, Grafana)
    • Logging: Track predictions, latency, errors
    • Alerting: team alerts on performance degradation
  • Configure model registry (MLflow, SageMaker Model Registry):
    • Version control: Track model versions, who deployed, when
    • Rollback plan: Can revert to previous version if issues arise
  • Security:
    • Authentication: API keys, OAuth
    • Authorization: Role-based access control
    • Encryption: TLS for API, encrypted storage for model artifacts
  • Load testing:
    • Simulate production traffic (100-1000 requests/second)
    • Measure: Can infrastructure handle expected load?
    • Service objective: set and approve a context-specific tail-latency target from user needs, workflow timing, safety, cost, load, upstream/downstream dependencies, and fallback behavior; record the percentile, measurement window, and owner
• Infrastructure checklist:

  PRODUCTION SETUP:
  
  DEPLOYMENT:
  □ Model API deployed (endpoint: https://api.company.com/model/predict)
  □ Load balancer configured (auto-scaling enabled)
  □ Health checks configured (ping every 30 seconds)
  
  MONITORING:
  □ Performance dashboard (Grafana): Accuracy, latency, throughput
  □ Business dashboard: KPIs (first-contact resolution, cost per ticket)
  □ Alerts configured:
        - Quality drops beyond threshold -> alert team immediately
        - Latency exceeds workflow threshold -> investigate scaling
        - Error rate exceeds threshold -> check logs
  
  LOGGING:
  □ Prediction logs: Store all predictions + features (for debugging)
  □ Audit trail: Track who made changes, when
  
  SECURITY:
  □ API authentication (OAuth)
  □ TLS encryption (in transit)
  □ Encrypted storage (at rest)
  
  ROLLBACK:
  □ Previous model version tagged (can revert in <5 minutes)
  

• Output: Production infrastructure ready, tested, and monitored
• Red flag: Load testing fails (latency >200ms) → Need to optimize model or scale infrastructure

Week 12: Gradual Rollout (40 hours)

• Activities:
  • Start with a small slice of traffic (shadow mode):
    • AI makes predictions but doesn't affect user experience
    • Compare AI predictions to human decisions
    • Monitor for errors, bias, performance issues
  • Increase exposure only if quality, safety, latency, and user-feedback gates are met
  • Monitor daily:
    • Performance metrics: Accuracy, latency, errors
    • Business metrics: First-contact resolution, cost per ticket
    • User feedback: Are users noticing AI? Any complaints?
• Rollout log:

  GRADUAL ROLLOUT:
  
      Day 1-2 (shadow/small traffic slice):
      - Performance: quality and latency within approved range
      - Business: target metric moving in the expected direction
      - Issues: None
  
      Next exposure step:
      - Performance: quality and latency still acceptable
      - Business: target metric remains stable or improving
      - Issues: edge cases investigated and fixed
  
      Larger exposure step:
      - Performance: quality and latency still acceptable
      - Business: target metric remains stable or improving
      - Issues: None
  
      DECISION: Proceed to wider rollout only if gates remain green
      ```
  

• Output: model handling a larger traffic slice with no major issues
• Red flag: performance degrades as exposure increases → infrastructure or model issue; rollback to prior safe exposure

Week 13-14: Full Rollout & Monitoring

Week 13: Full Rollout (40 hours)

• Activities:
  • Increase to the approved full-production scope
  • Communicate to end users:
    • Internal announcement: "We've deployed AI to improve [use case]. You may notice [changes]."
    • Training materials: How to use AI outputs, how to override if needed
    • Feedback channel: where users should send issues
  • Monitor intensively:
    • Check dashboards frequently during the first production window
    • Respond to alerts within the agreed service level
    • Weekly review meeting with team + stakeholders
  • Collect user feedback:
    • Ask representative users: "How is AI performing?" "Any issues?"
    • Track support tickets related to AI
• Full-rollout checklist:

  FULL ROLLOUT:
  
  PREPARATION:
  □ User communication sent (email to all users)
  □ Training materials published (docs, video)
  □ Feedback channel created (Slack #ai-feedback, support email)
  
      MONITORING (First production window):
      - Day 1: quality, latency, and volume within approved range
      - Day 2: quality, latency, and volume remain stable
      - Day 3: quality, latency, and volume remain stable
  
      USER FEEDBACK (Week 13):
      - Positive: users report recommendations are helpful
      - Neutral: users report no meaningful workflow change
      - Negative: users report incorrect or confusing outputs; investigate and document
  
      DECISION: continue rollout only if production and user-feedback gates remain acceptable
      ```
  

• Output: model deployed to approved production scope
• Red flag: user complaints spike → UX issue or model error; consider rollback

Week 14: Performance Validation (40 hours)

• Activities:
  • Collect 2 weeks of production data (Week 13-14)
  • Compare to baseline (pre-AI):
    • Business metric: compare target metric before and after AI
    • Cost metric: compare cost before and after AI, adjusted for volume and mix
    • User satisfaction: compare user-satisfaction signal before and after AI
  • Validate ROI:
    • Projected value: from approved business case
    • Actual value: annualized only when the sample is stable enough to justify it
  • Document lessons learned:
    • What went well? (data quality, team collaboration)
    • What didn't? (initial latency issues, edge cases)
    • What would we do differently? (earlier load testing)
• Performance validation memo:

  PILOT PERFORMANCE (Weeks 13-14):
  
      BASELINE (Pre-AI):
      - Target business metric
      - Workflow cycle-time metric
      - Cost metric
  
      WITH AI (Weeks 13-14):
      - Target business metric changed in expected direction
      - Workflow cycle-time metric changed in expected direction
      - Cost metric changed in expected direction
  
      ROI VALIDATION:
      - Projected annual value: from approved business case
      - Actual annualized value: calculated only if evidence is stable
      - Investment: Year 1 cost
      - Payback period: calculated from actual value and cost
  
      CONCLUSION: Pilot successful, ready to scale
  

• Output: Performance validated, ROI confirmed, ready for Week 15-16 planning

Decision Gate #5: End of Week 14

• Go criteria:
  • Business metric improved by a meaningful, pre-defined amount
  • Model stable in production (accuracy consistent, latency acceptable)
  • User feedback positive enough for production workflow
  • ROI is close enough to projection to justify the next investment
• No-Go criteria:
  • Business metric declined or did not improve enough → pilot did not deliver value
  • Model unstable (frequent errors, performance degradation)
  • User feedback negative beyond the approved tolerance

Contingency: If No-Go:

1. Extend pilot: 2 more weeks to allow performance to stabilize
2. Adjust model: Fix identified issues and re-deploy
3. Pivot scope: Focus on subset of use cases where model performs well

Phase 6: Pilot Review & Scaling Plan (Weeks 15-16)

Goal: Review pilot results, document learnings, and plan for scaling AI across organization.

Week 15: Pilot Retrospective

Day 1-2: Data Collection & Analysis (16 hours)

• Activities:
  • Compile full pilot results:
    • Quantitative: Performance metrics, business KPIs, ROI
    • Qualitative: User feedback, team experiences, lessons learned
  • Interview stakeholders (10-15 people):
    • Executive sponsor: "Did we achieve objectives?"
    • Product owner: "What would you do differently?"
    • Data scientist: "Technical challenges and solutions?"
    • End users: "How has AI changed your work?"
  • Analyze what worked and what didn't
• Retrospective template:

  PILOT RETROSPECTIVE:
  
  WHAT WENT WELL:
  - Data quality higher than expected (75/100 vs. 60 feared)
  - Team collaboration excellent (weekly syncs kept everyone aligned)
  - Bias testing caught issues early (avoided deployment problems)
  - User adoption smooth (80% positive feedback)
  
  WHAT DIDN'T GO WELL:
  - Initial latency issues (200ms → fixed with optimization)
  - Edge cases not caught in testing (5% of users reported odd behavior)
  - Rollout communication delayed (should have sent earlier)
  
  WHAT WE'D DO DIFFERENTLY:
  - Start load testing earlier (Week 9 instead of Week 11)
  - Involve end users in Week 8 (not just Week 10)
  - Budget 2-week buffer (16 weeks was tight)
  

• Output: Retrospective document with lessons learned
• Red flag: Team says "pilot was failure" → Need honest assessment of whether to scale

Day 3-5: Business Case Update (24 hours)

• Activities:
  • Update business case with actuals:
    • Actual investment: $180K (vs. $210K budgeted) ✓
    • Actual value: $480K/year (vs. $500K projected) ✓
    • Actual timeline: 16 weeks (vs. 16 planned) ✓
  • Recalculate 3-year ROI with actuals:
    • Year 1: $180K investment - $240K value (6 months) = $60K net
    • Year 2-3: $50K/year ongoing costs - $480K/year value = $430K net/year
    • 3-year total: $60K + $430K + $430K = $920K net value
    • ROI: actual value divided by total investment
  • Present updated business case to executive sponsor
• Output: Updated business case with actual results
• Red flag: actual ROI misses the approved hurdle -> pilot successful but less valuable than hoped

Week 16: Scaling Plan

Day 1-2: Next Use Cases (16 hours)

• Activities:
  • Review original top 10 use case list from Week 1
  • Select next 2-3 use cases to pilot in parallel:
    • Use learnings from Pilot 1 to accelerate (8-12 weeks instead of 16)
    • Prioritize Quick Wins (high value, high feasibility)
  • For each, draft mini business case:
    • Value: $X annually
    • Investment: $Y (likely less than Pilot 1 due to reusable infrastructure)
    • Timeline: 8-12 weeks
• Next use cases:

  USE CASE #2: [Name, e.g., "Sales lead scoring"]
  - Value: $300K/year (10% sales productivity increase)
  - Investment: $100K (reuse infrastructure, 8-week pilot)
  - Timeline: Q2 2025
  - Team: Same team + 1 sales analyst
  
  USE CASE #3: [Name, e.g., "Inventory demand forecasting"]
  - Value: $500K/year (reduce stockouts by 15%)
  - Investment: $120K (new data sources needed, 10-week pilot)
  - Timeline: Q3 2025
  - Team: New team (data scientist + supply chain analyst)
  

• Output: 2-3 next use cases prioritized with mini business cases
• Red flag: Executive sponsor says "no budget for more pilots" → Scale current pilot instead of new ones

Day 3-4: Scaling Roadmap (16 hours)

• Activities:
  • Build 12-month AI roadmap:
    • Q2: Scale Pilot 1 to more users/use cases + Launch Pilot 2
    • Q3: Scale Pilot 2 + Launch Pilot 3
    • Q4: Portfolio review, plan for 5-10 pilots in Year 2
  • Resource plan:
    • Team growth: Hire 2 data scientists, 1 ML engineer (if budget approved)
    • Infrastructure: Invest in MLOps platform (if >5 models in production)
    • Governance: Establish AI Ethics Board (if deploying high-risk models)
  • Budget request:
    • Year 2 AI investment: $500K-$1M
    • Expected return: $2-3M annually (assuming 3-5 successful pilots)
• 12-month roadmap:

  AI SCALING ROADMAP (12 months):
  
  Q2 2025 (Months 1-3):
  - Scale Pilot 1 to 100% of support tickets (done ✓)
  - Launch Pilot 2: Sales lead scoring (8 weeks)
  - Hire: 1 data scientist
  
  Q3 2025 (Months 4-6):
  - Scale Pilot 2 to all sales reps
  - Launch Pilot 3: Inventory forecasting (10 weeks)
  - Establish AI Steering Committee (quarterly reviews)
  
  Q4 2025 (Months 7-9):
  - Scale Pilot 3 to all warehouses
  - Launch Pilots 4-5: [TBD based on portfolio review]
  - Invest in MLOps platform (if >5 models)
  
  Q1 2026 (Months 10-12):
  - Portfolio review: 5+ models in production, $2M+ annual value
  - Plan Year 2: 10-15 pilots, build AI Center of Excellence
  - Budget request: $1-2M for Year 2
  
  METRICS:
  - Year 1 target: 3-5 models in production, $1.5M value, ROI >200%
  

• Output: 12-month scaling roadmap with resource and budget plan
• Red flag: Can't get budget for Year 2 → Limited to maintaining current pilots, no growth

Day 5: Final Presentation & Approval (8 hours)

• Activities:
  • Present pilot results + scaling plan to executive leadership:
    • Pilot 1 results: annual value, ROI, and payback period from actual data
    • Lessons learned: Data quality critical, user feedback valuable, timeline realistic
    • Scaling plan: 2-3 more pilots in next 12 months, $2-3M total value
    • Budget request: $500K-$1M for Year 2
  • Get approval for:
    • Next use cases (Pilots 2-3)
    • Resource plan (hire 1-2 people)
    • Budget allocation ($500K-$1M)
  • Celebrate success with team (pilot delivered value!)
• Final presentation outline:

  PILOT REVIEW & SCALING PLAN (30-minute presentation)
  
  SLIDE 1: Executive Summary
  - Pilot: [Use case name]
  - Result: $480K annual value, ROI 329%, 4.5-month payback
  - Recommendation: Scale to 2-3 more use cases in next 12 months
  
  SLIDES 2-5: Pilot Results
  - Business impact: +5% FCR, -10% cost, +5 NPS
  - Technical performance: 84% accuracy, 78ms latency, stable
  - User feedback: 80% positive
  - ROI validation: 96% of projected value
  
  SLIDES 6-8: Lessons Learned
  - What worked: Data quality, team collaboration, bias testing
  - What didn't: Latency issues, edge cases, communication
  - What's next: Apply learnings to accelerate Pilots 2-3
  
  SLIDES 9-12: Scaling Plan
  - Next use cases: Sales lead scoring ($300K), Inventory forecasting ($500K)
  - 12-month roadmap: 3-5 models, $2M+ value
  - Resource request: 2 hires, $500K-$1M budget
  - Expected return: $2-3M annually, ROI >200%
  
  SLIDE 13: Ask
  - Approve Pilots 2-3 (Q2-Q3 2025)
  - Approve budget ($500K-$1M)
  - Approve hiring (2 data scientists, 1 ML engineer)
  

• Output: Executive approval to scale AI program

Decision Gate #6: End of Week 16

• Scale criteria:
  • Pilot 1 delivered enough of projected value to justify scaling
  • Executive approval for 2-3 more pilots
  • Budget allocated ($500K-$1M for Year 2)
  • Team growth approved (hiring plan)
  • Action: Launch Pilots 2-3, build AI program
• Maintain criteria:
  • Pilot 1 successful but limited budget
  • Approval to scale Pilot 1 but not launch new pilots
  • Action: Focus on maximizing value from Pilot 1, revisit in 6 months
• Pause criteria:
  • Pilot 1 did not deliver enough projected value to justify scaling
  • No executive support for further investment
  • Action: Maintain Pilot 1 if still positive ROI, pause new initiatives

Contingency: If Pause:

1. Document learnings: What went wrong? Can we fix it?
2. Maintain Pilot 1: Continue monitoring, optimize for value
3. Revisit in 6 months: Market conditions, technology, leadership priorities may change

Red Flags by Week (Warning Signals)

Week 1-2 (Opportunity Assessment):

• <20 use cases brainstormed → Need broader stakeholder participation
• No executive sponsor identified → Project will fail without top-down support
• All use cases score <6/10 → Need to educate team on AI capabilities or look at different problems

Week 3-4 (Business Case):

• Can't quantify business value → Not ready for AI investment
• ROI misses hurdle -> use case not economically viable
• Can't staff 1 FTE data scientist → Pilot will fail without technical expertise

Week 5-6 (Data Assessment):

• Data readiness evidence misses the approved threshold → diagnose the blocking gap, revise scope or method, and record whether to continue, stage, or stop
• Legal blocks data use → May need to anonymize or abandon use case
• Data pipeline fails validation → Technical blocker, may delay 1-2 weeks

Week 7-10 (Pilot Development):

• Baseline model worse than current state -> data or feature engineering issue
• Can't beat baseline by the approved minimum improvement -> need more data, better features, or different approach
• Bias can't be mitigated under the approved fairness protocol -> legal/ethical risk too high
• Users don't trust model outputs -> UX or explainability issue

Week 11-14 (Pilot Deployment):

• Load testing fails against workflow requirement -> need to optimize or scale infrastructure
• Performance degrades as traffic expands -> infrastructure or model issue, rollback
• User complaints spike beyond tolerance -> UX issue or model error
• Business metric does not improve enough -> pilot not delivering expected value

Week 15-16 (Review & Planning):

• Actual ROI misses the approved hurdle -> pilot may be useful but less valuable than projected
• Team says "pilot was failure" -> need honest assessment
• No budget for Year 2 -> can only maintain current pilot, no growth

Resource Requirements (Detailed)

Human Resources (16-week pilot):

Core Team:

• Product Owner: part-time owner across the pilot

  • Weeks 1-4: Define requirements, stakeholder management
  • Weeks 5-10: Feature prioritization, user testing
  • Weeks 11-16: Deployment oversight, success tracking
  • Cost: estimate from loaded labor rate and actual allocation

• Data Scientist: primary modeling and evaluation lead

  • Weeks 1-6: Data assessment, pipeline development
  • Weeks 7-10: Model development, testing
  • Weeks 11-14: Deployment support, monitoring
  • Weeks 15-16: Documentation, knowledge transfer
  • Cost: estimate from loaded labor rate and actual allocation

• ML Engineer: deployment and monitoring lead

  • Weeks 5-10: Data pipeline, infrastructure setup
  • Weeks 11-14: Production deployment, monitoring
  • Cost: estimate from loaded labor rate and actual allocation

• Data Engineer: data pipeline and quality lead

  • Weeks 5-6: Data pipeline development
  • Weeks 7-10: Pipeline maintenance, data quality
  • Cost: estimate from loaded labor rate and actual allocation

• Business Analyst: metrics and business validation support

  • All weeks: Metrics tracking, reporting, business validation
  • Cost: estimate from loaded labor rate and actual allocation

Extended Team (part-time):

• Executive Sponsor: strategic oversight and approvals
• Legal/Compliance: privacy, regulatory, and risk review
• Business Stakeholders: interviews, validation, feedback, and adoption support

Total Team Cost: sum loaded labor cost for each role based on real allocation.

Infrastructure & Tools:

• Cloud compute: training and inference cost based on model type and volume
• MLOps tools: registry, monitoring, evaluation, observability, and alerting
• Data storage: source, feature, evaluation, and logging storage
• API costs: if using model APIs, estimate by volume, model mix, latency, and caching
• Total infrastructure: compute from expected usage and retention requirements

Vendor Costs (if applicable):

• Third-party data: include only if needed and licensed for AI use
• Consulting: include only if outsourcing parts of delivery, governance, or validation
• Total vendor costs: estimate from quotes, not generic ranges

Total Budget Range:

• Low end: internal team, limited tooling, narrow scope
• Mid range: standard tooling, moderate data engineering, production monitoring
• High end: vendor support, premium tooling, complex data, regulated workflow

Realistic budget: calculate from actual labor allocation, infrastructure, vendor quotes, governance work, and change-management needs.

Decision Gates (Detailed)

Gate #1 (Week 2): Proceed with Use Case?

• Criteria: strong composite score, quantified value, credible feasibility, and executive sponsor
• Options:
  • YES → Proceed to business case
  • NO (low score) → Go back to brainstorming, select different use case
  • NO (no sponsor) → Get executive buy-in before proceeding

Gate #2 (Week 4): Proceed to Pilot?

• Criteria: ROI beats the hurdle rate, team staffed, data accessible, and risks mitigated
• Options:
  • YES → Proceed to data assessment
  • NO (economics) → Adjust scope or pricing assumptions
  • NO (team) → Delay start until resources available
  • NO (risk) → Get ethics review or change approach

Gate #3 (Week 6): Proceed to Development?

• Criteria: data quality acceptable, pipeline working, privacy approved, and governance in place
• Options:
  • YES → Proceed to model development
  • NO (data quality) → Extend data phase 2 weeks for engineering
  • NO (privacy) → Anonymize data or change approach
  • NO (pipeline) → Simplify pipeline or use vendor solution

Gate #4 (Week 10): Proceed to Deployment?

• Criteria: model beats baseline meaningfully, bias review passed, users trust outputs, and latency acceptable
• Options:
  • YES → Proceed to production deployment
  • NO (performance) → Extend development 2 weeks
  • NO (bias) → Retrain with fairness constraints or human-in-loop
  • NO (trust) → Improve explainability and UX

Gate #5 (Week 14): Pilot Successful?

• Criteria: business metric improved enough, model stable, user feedback positive, and ROI close enough to projection
• Options:
  • YES → Proceed to scaling planning
  • NO (value) → Extend pilot briefly to allow stabilization
  • NO (stability) → Fix issues and re-deploy
  • NO (feedback) → Improve UX or reduce scope

Gate #6 (Week 16): Scale or Maintain?

• Criteria: Based on actual ROI and executive support
• Options:
  • SCALE (ROI beats hurdle, exec support) → Launch more pilots
  • MAINTAIN (positive ROI, limited budget) → Focus on Pilot 1
  • PAUSE (ROI misses hurdle or no support) → Maintain if positive, pause new initiatives

Contingency Triggers

Trigger 1: If data quality is below the approved threshold by Week 5

• Action: Extend data phase or reduce scope to subset with higher quality data
• Rationale: Poor data = poor model; can't skip data engineering

Trigger 2: If technical staffing is insufficient by Week 7

• Action: Delay pilot start OR hire contractor/consultant
• Rationale: Can't develop AI model without technical expertise

Trigger 3: If bias can't be mitigated by Week 10

• Action: Legal review → May need human-in-loop OR abandon use case
• Rationale: Deploying biased model = legal/ethical risk

Trigger 4: If business metric improvement misses the approved threshold by Week 14

• Action: Extend pilot briefly or reduce scope to where model performs well
• Rationale: Pilot not delivering expected value, need to diagnose why

Trigger 5: If executive sponsor leaves during pilot

• Action: Find new sponsor immediately OR pause pilot
• Rationale: Without exec support, pilot will fail to scale even if successful

Timeline Variance (Adapt to Your Situation)

Rapid Mode (10-12 weeks):

• When to use: Simple use case (well-defined problem, clean data, proven approach)
• Changes:
  • Weeks 1-2 → 1 week (rapid use case selection, pre-validated by exec)
  • Weeks 3-4 → 1 week (simplified business case)
  • Weeks 5-6 → 1 week (data already clean and accessible)
  • Weeks 7-10 → 3 weeks (use pre-built models or vendor APIs)
  • Weeks 11-14 → 3 weeks (faster rollout)
  • Weeks 15-16 → 1 week (streamlined review)
• Risk: less validation means higher risk of deploying the wrong solution
• Best for: proven use cases using a current model or vendor API, with data already clean

Standard Mode (16 weeks):

• When to use: Moderate complexity, some data engineering needed, custom modeling
• Changes: Follow plan as written above
• Best for: Most enterprise AI pilots (custom models on internal data)

Thorough Mode (20-24 weeks):

• When to use: Complex use case (high-risk, regulated industry, novel approach)
• Changes:
  • Weeks 1-2 → expanded discovery and stakeholder interviews
  • Weeks 5-6 → expanded data engineering and source integration
  • Weeks 7-10 → expanded modeling and validation
  • Weeks 11-14 → extended beta with representative users
  • Add 2-week buffer for unforeseen issues
• Best for: Healthcare (HIPAA), finance (regulatory), autonomous systems (safety-critical)

Measurement Dashboard (Track Weekly)

Weekly Metrics Tracker:

Table 16.6: Constructed pilot measurement dashboard (Week | Phase | Activity | Deliverable | Status | Red flags). Timing, thresholds, status values, and red flags are local planning inputs; define them against the approved use case and risk boundary.

Milestone Metrics (End of Week 16):

Model Performance:

• Model performance improves meaningfully over baseline
• Latency meets workflow requirement
• Bias testing passed under approved fairness protocol
• Uptime meets production stability requirement

Business Impact:

• Key metric improved by the approved threshold
• Cost savings or revenue gain is material to the business case
• User satisfaction or workflow acceptance improved
• ROI beats the approved hurdle rate

Operational Readiness:

• Model monitoring dashboard operational
• Alerting configured (performance, errors, bias)
• Rollback plan tested within the required recovery window
• Documentation complete (model card, user guide, runbook)

Governance & Compliance:

• Privacy/compliance approval obtained
• Bias testing documented and passed
• Model registry updated (version, owner, approval)
• Incident response plan documented

Readiness Assessment:

PASS (Ready to Scale):
- most milestone metrics hit
- actual ROI beats hurdle or is on track
- executive support for more pilots
- Team ready to scale (hiring approved)
-> ACTION: Launch next pilots, build AI program

MAINTAIN (Successful but Limited Growth):
- several milestone metrics hit
- actual ROI positive but expansion case limited
- Limited budget for new pilots
-> ACTION: Scale Pilot 1, revisit expansion at the next planning cycle

PIVOT (Needs Improvement):
- too few milestone metrics hit
- actual ROI misses hurdle
- Business impact unclear
-> ACTION: Investigate root cause, improve model, or extend pilot only if the next test is clear

Success Criteria & Benchmarks

What "Good" Looks Like at Week 16:

• Model: quality, latency, and reliability meet the approved production requirements
• Business: the key metric improves enough to matter commercially
• ROI: projected and observed value beat the approved hurdle rate
• Users: user feedback is positive enough for continued production use
• Readiness: Documentation complete, monitoring operational, governance in place

What "Struggling" Looks Like at Week 16:

• Model: quality, latency, or reliability miss production requirements
• Business: improvement is too small or value is unclear
• ROI: ROI misses the approved hurdle rate or payback is too slow
• Users: feedback shows low trust, confusion, or harmful workflow friction
• Readiness: Incomplete docs, monitoring gaps, governance ad-hoc

What to Do if Struggling:

1. Diagnose root cause: Is it model performance? Data quality? UX? Deployment issues?
2. Extend pilot 4 weeks: Give time to fix identified issues
3. Reduce scope: Focus on subset where model performs well
4. Consider pivot: Different use case or approach
5. Continue only when the expected learning value, risk, cost, and decision deadline justify another iteration

What to Do if Succeeding:

1. Document playbook: What worked? How to replicate for Pilots 2-3?
2. Celebrate with team: Acknowledge hard work and success
3. Secure budget for scaling: Present results to exec team, get Year 2 funding
4. Plan next pilots: Apply learnings to accelerate (aim for 10-12 weeks vs. 16)
5. Invest in infrastructure: If scaling to 5+ models, build MLOps platform

Chapter Summary

AI Strategy frameworks covered:

1. Opportunity Assessment - Identify high-value, feasible use cases
2. Build/Buy/Partner - Decide how to acquire AI capabilities
3. Maturity Model - Assess and advance organizational AI readiness
4. Use Case Prioritization - Focus on quick wins and strategic bets
5. ROI Calculation - Justify AI investments financially
6. Ethical AI - Build responsible, fair AI systems
7. Data Readiness - Ensure foundation for AI success
8. MLOps and Change Control - Operationalize AI with versioning, monitoring, and release gates
9. Agentic AI Operating and Control Model - Bound delegated execution, authority, evidence, and recovery
10. Governance - Structure oversight and decision-making
11. Change Management - Drive adoption and overcome resistance

2026 AI Landscape:

• Generative AI is now part of mainstream enterprise experimentation and product design. ^{[4] [6]}
• Multimodal models enable text, image, audio, and structured-data workflows. ^[4]
• Smaller and specialized models can reduce cost, latency, or privacy exposure when they fit the task.
• AI agents require stronger evaluation, monitoring, and secure-development controls before production use. ^[9]
• Regulation and governance increasingly require risk-based controls, documentation, and accountable ownership. ^{[14] [2]}

Success Factors:

• Start bounded, scale on evidence: expand only after business, technical, adoption, safety, and control gates are met
• Data quality > Algorithm sophistication: GIGO still applies
• Change management = technical implementation: Both critical
• Ethical considerations upfront: Easier to build in than bolt on
• Continuous learning: AI evolving rapidly, stay current

Next Chapter: Leading Digital Transformation - Operating-model change, transformation leadership, architecture, governance, and execution

Authored connections:

• Chapter 4, Financial Analysis and Valuation: baselines, unit economics, cash-flow effects, and investment appraisal.
• Chapter 7, Organizational Behavior and Leadership: power, participation, incentives, psychological safety, and change leadership.
• Chapter 17, Leading Digital Transformation: operating-model and transformation dependencies.
• Chapter 19, Cybersecurity and Risk Management: security governance, threat modeling, and incident response.
• Chapter 20, The Ethics of AI and Data: stakeholder, rights, justice, fairness, and remedy analysis.
• Chapter 21, Product Management: discovery, product economics, release gates, and post-launch learning.
• Chapter 9, Problem Structuring: feasible alternatives, non-compensable gates, decision/chance-node structure, and evidence needs.
• Chapter 22, Data Analysis and Insights: causal assumptions, expected value or utility, break-even probability, Bayesian updating, information value, sensitivity analysis, experiments, and decision rules.