Data Analysis and Insights

Overview

The Pyramid Principle is a top-down communication structure: state the answer first, then support it with grouped arguments and evidence. Minto's work is widely used in consulting because it helps teams convert messy analysis into a logical executive argument. ^[1]

The pyramid is not a decoration for slides. It is a forcing mechanism. If you cannot state the answer at the top, the analysis is not ready for a decision.

How to Apply

When To Use It

Use the Pyramid Principle when:

• you are presenting a recommendation to executives;
• the analysis has multiple workstreams;
• stakeholders are asking for "the so what";
• the team is confusing evidence with argument;
• the conclusion will be challenged.

The Structure

Top: Governing thought.

"We should pause expansion into Region B until unit economics improve."

Middle: The small set of reasons.

1. Current contribution margin is below the approved threshold.
2. The gap is driven by logistics and service cost, not customer acquisition.
3. Two operating changes may close most of the gap, but they need additional validation.

Bottom: Evidence for each reason.

Each supporting point needs evidence, but the evidence should not lead the story. Data supports the argument; it does not replace it.

Practical Template

Manager Checklist

• Can the recommendation be read without the appendix?
• Are the supporting arguments mutually distinct?
• Does each argument support the answer directly?
• Does the evidence resolve a decision uncertainty?
• Have you named what would change the answer?

Common Failure Modes

• Bottom-up tour: The presenter walks through the analysis in the order it was performed. The audience waits too long for the answer.
• Evidence pile: The slides contain many facts but no governing thought.
• False certainty: The top-line answer hides a material uncertainty that should be part of the recommendation.
• Non-MECE support: The same argument appears in multiple branches, making the case feel padded.

Worked Mini-Example

Weak version: "We analyzed sales performance by channel, customer type, region, and sales-rep tenure."

Constructed Pyramid version: "We should shift the next planning cycle's incremental sales capacity to enterprise renewals because the current growth constraint is expansion within existing accounts, not new-logo demand."

Support:

1. Renewal expansion has the clearest near-term revenue path.
2. New-logo conversion is constrained by sales-cycle length.
3. Support capacity is the main risk to expansion quality.

The second version makes the decision visible. The analysis can now be judged by whether it supports that recommendation.

So What for Managers

• Put the decision, owner, timing, and threshold at the top of the recommendation.
• Require each supporting argument to resolve a distinct decision uncertainty and link to inspectable evidence.
• Show the strongest alternative interpretation and what evidence would change the recommendation.

Limits and Critiques

• A communication structure does not validate the data, causal design, model, or business case beneath it.
• Grouping arguments as mutually exclusive and collectively exhaustive is a judgment that can be wrong or contested.
• Answer-first writing can conceal affected groups, dissent, uncertainty, or implementation constraints if the review process rewards brevity over evidence.

Connections

Use Chapter 9 for problem structuring; Chapter 20 for ethics, affected-party voice, and remedy; and Frameworks 2–5 here for causal, statistical, regression, and visual evidence that supports the recommendation.

Overview

Minto names the setup Situation, Complication, Question (SCQ). The communication then supplies the answer. In this chapter, SCQA is an explicit shorthand for the combined SCQ setup plus answer, not a claim that Minto gave the framework that four-part name. The sequence gives an audience shared context, the tension, the decision question, and the recommendation in a compact structure. ^[1]

How to Apply

The Four Parts

How To Use SCQA Before Analysis

Write SCQA before you build the model. This does three useful things:

1. It exposes whether the team agrees on the problem.
2. It prevents exploratory work from becoming endless.
3. It defines what evidence would be useful.

SCQA Diagnostic Questions

Situation

• What is already agreed?
• What has changed?
• What scope is included and excluded?

Complication

• What is now difficult, risky, or inconsistent?
• What constraint makes the obvious answer insufficient?
• What stakeholder tension must be resolved?

Question

• What decision must be made?
• Who owns it?
• When does it need to be made?

Answer

• What should we do?
• What evidence supports the action?
• What risk remains?

Strong SCQA vs. Weak SCQA

Weak SCQA: "We looked at churn because leadership asked for churn analysis."

Constructed strong SCQA: "Retention has become the constraint on growth. New bookings are healthy, but expansion is being offset by preventable churn in one segment. Should we prioritize new sales capacity or retention intervention in the next planning cycle? We should fund the retention intervention first because it addresses the highest-leakage part of the revenue engine."

Common Failure Modes

• No complication: The story has context but no tension.
• Question is too broad: "How do we grow?" is not a decision question.
• Answer is analytical, not operational: "Churn is correlated with low usage" is a finding, not an action.
• Answer arrives too late: Executives should not have to wait for the final slide to learn the recommendation.

So What for Managers

• Write the decision question before analysis so the team can distinguish useful evidence from exploratory detail.
• Separate shared facts, changed conditions, the decision, and the recommendation; do not smuggle assumptions into the Situation.
• Give the audience a route to challenge the complication, the options, the evidence, and the answer.

Limits and Critiques

• SCQ or SCQA improves communication but does not prove that the question is the right one or that the answer follows from the evidence.
• A compelling narrative can create anchoring, omit minority views, or make a contested causal story feel settled.
• The structure should be adapted for safety, legal, scientific, or affected-party review where a short executive answer cannot carry the necessary uncertainty.

Connections

Use Framework 1 for the answer-first pyramid; Chapter 9 for problem definition; and Frameworks 3–5 for the design, statistical, and model evidence that should populate the story.

Overview

Correlation means two things move together. Causation means changing one thing would change another thing. Pearl's structural causal-inference work is one influential source for distinguishing association from intervention logic through explicit assumptions, causal models, confounding, and counterfactual reasoning. ^[2]

For managers, the practical question is not philosophical. It is this: can we act on the relationship, or should we only use it for prediction and monitoring?

How to Apply

Required Mermaid Diagram: Decision Tree

Figure 22.1. Routing from association to prediction or causal decision. The author-created tree distinguishes predictive use from intervention claims and routes causal questions through confounding, timing, design, testing, and explicit uncertainty. A path through the tree does not itself identify an effect. Source basis: causal-model and intervention reasoning. ^[2]

Text equivalent: Start with an observed relationship. If the decision is prediction only, validate the predictive model. If the decision intervenes on X to change Y, examine confounding, timing, selection, overlap, spillovers, measurement, and design. Prefer a feasible ethical randomized experiment; otherwise justify a quasi-experimental or observational design. End with act, test, hold, or redesign and a named methods owner.

flowchart TD
    A[Observed relationship between X and Y] --> B{Is the decision only prediction?}
    B -->|Yes| C[Validate predictive use, leakage, calibration, and distribution shift]
    B -->|No| D[Define intervention, outcome, estimand, units, and causal graph]
    D --> E[Assess confounding, selection, overlap, timing, and measurement]
    E --> F[Assess spillovers, interference, attrition, and affected stakeholders]
    F --> G{Is randomization feasible and ethical?}
    G -->|Yes| H[Design experiment with guardrails and valid measurement]
    G -->|No| I{Is a credible quasi-experimental or observational design available?}
    I -->|Yes| J[Justify design and test identifying assumptions]
    I -->|No| K[Hold causal claim; redesign, collect evidence, or use only directionally]
    H --> L[Validate assumptions, sensitivity, heterogeneity, and external limits]
    J --> L
    K --> M[Methods-owner decision: act, test, hold, or redesign]
    L --> M
    C --> M

How To Use The Tree

1. Name the relationship. Example: accounts with more product usage renew more often.
2. Decide the use case. If the goal is prediction, a non-causal model may be useful. If the goal is intervention, causal evidence matters.
3. List confounders. Larger customers may both use more product and renew more often. That does not prove product usage caused renewal.
4. Check timing. If renewal commitment happens before usage rises, the causal story may be backwards.
5. Look for design strength. Randomization, natural experiments, phased rollouts, discontinuities, and credible controls are stronger than simple before-after comparisons.
6. Pick the action level. Act now, run a pilot, monitor, or decline to act.

Methods and Ownership Boundary

A causal diagram, decision tree, or regression does not create identification. Prefer randomized assignment when ethical and feasible. Otherwise justify the observational or quasi-experimental design—such as a credible natural experiment, discontinuity, difference-in-differences, instrumental variable, matching/weighting, synthetic control, or phased rollout—without choosing a method merely because it is available. Document the estimand, comparison group, assumptions, timing, overlap, interference/spillovers, attrition, missingness, measurement, heterogeneous effects, diagnostics, sensitivity, and external-validity limits. A named human methodological owner must approve the causal claim and its limits.

Decision Categories

Practical Tests For Causality

Confounding test: What else could explain both variables?

Reverse-causality test: Could Y be causing X?

Selection test: Did the people exposed to X differ from the people not exposed to X before the analysis began?

Mechanism test: Can an operator explain how X would change Y?

Intervention test: If we deliberately changed X, would we expect Y to move?

Example

The data shows that customers attending onboarding sessions have higher renewal rates. That relationship may be useful for predicting renewal risk, but it does not prove onboarding caused renewal. Customers who attend may already be more motivated, better staffed, or larger. The decision is not "onboarding works"; the decision is "should we invest in an onboarding intervention, and what design would prove incremental value?" ^[2]

Managerial Rule

When the decision changes behavior, budget, pricing, staffing, or customer treatment, do not treat correlation as causation. Treat it as a hypothesis to test.

So What for Managers

• Label every material claim as descriptive, predictive, diagnostic, or causal before selecting a method or action.
• Require a named methods owner to approve the estimand, design, assumptions, diagnostics, sensitivity, and external-validity limits.
• Use association for bounded prediction or triage only when validation, fairness, leakage, drift, and decision-loss checks support that use.

Limits and Critiques

• A decision tree organizes questions; it cannot identify an effect or substitute for design, data quality, overlap, or domain knowledge.
• Randomization may be infeasible or unethical, while quasi-experimental designs depend on assumptions that can fail invisibly.
• Causal effects may vary across people, time, treatment versions, spillovers, and institutions; an average estimate can hide material harm or benefit.

Connections

Use Chapter 9 for problem structure; Chapter 16 for AI evaluation and deployment governance; Chapter 20 for fairness, privacy, safety, and remedy; and Frameworks 4–5 here for statistical and regression interpretation.

Overview

Statistical significance is often misused in business settings. The ASA statement on p-values says a p-value is not the probability that the hypothesis is true, is not the probability that the result occurred by chance alone, and should not be used as a bright-line substitute for scientific or practical reasoning. ^[3]

For managers, the goal is not to become a statistician. The goal is to ask the right questions before turning a test result into a decision.

How to Apply

The Three Questions To Ask

1. Is the effect real enough? Look at uncertainty and design quality.
2. Is the effect big enough? Compare effect size with the business threshold.
3. Is the action worth it? Include cost, risk, reversibility, and strategic fit.

What A p-Value Means

A p-value is a statement about the compatibility of the observed data with a specified statistical model and null hypothesis. It is not a direct measure of business importance. ^[3]

Manager translation: "If the null model were the right model, how surprising would this result be?"

What A Confidence Interval Means

A confidence interval gives a range of values compatible with the data and model assumptions. For decision makers, the most useful feature is often the range width, not the label.

Ask: Does the entire plausible range support the same decision, or does part of the range imply a different action?

Statistical vs. Practical Significance

Common Misuses

Misuse 1: "p less than 0.05 proves it works."

No. A threshold does not prove truth, causality, or business value. The ASA statement specifically warns against using p-values as a mechanical decision rule. ^[3]

Misuse 2: "Not significant means no effect."

No. It may mean the analysis was underpowered, noisy, badly designed, or measuring the wrong outcome.

Misuse 3: "The smaller p-value is the better project."

No. A smaller p-value can come from a larger sample, a cleaner design, or a trivial but precisely measured effect. The business decision still needs effect size and economics.

Misuse 4: "The result is significant, so we should scale."

No. Scaling requires practical significance, implementation feasibility, and downside analysis.

Worked Example

An experiment estimates that a new checkout flow increases completed orders by 0.4 points, with a plausible range from 0.1 to 0.7 points. The p-value is below the pre-specified threshold.

Decision interpretation: If the engineering cost is low and the flow has no customer-experience downside, the decision may be to launch. If the change creates operational risk, the same result may justify another test instead.

Manager Checklist

• Was the hypothesis specified before seeing the result?
• What is the effect size in business units?
• What is the uncertainty range?
• Does the uncertainty range cross the decision threshold?
• Are there multiple comparisons or repeated looks at the data?
• Is the finding causal, predictive, or descriptive?
• What would we do differently if the result were half as large?

So What for Managers

• Translate effect size and uncertainty into the units of the decision, including cost, capacity, risk, reversibility, and affected groups.
• Pre-specify the primary question, practical threshold, analysis plan, stopping rule, and multiplicity family before reading the result.
• Treat a result that crosses the decision threshold as a prompt for judgment, not as automatic authorization to scale.

Limits and Critiques

• P-values and confidence intervals are conditional on the design, model, sampling, measurement, and analysis choices that produced them.
• Statistical significance can be unstable under low power, repeated looks, selective reporting, multiple comparisons, or post-hoc subgroup analysis.
• A statistically precise estimate can still be operationally irrelevant, harmful, inequitable, or too uncertain for a high-stakes decision.

Connections

Use Framework 3 for causal design; Framework 5 for regression/model boundaries; Chapter 4 for economic thresholds; and Frameworks 9–13 here for sensitivity, decision analysis, experimentation, and optimization.

Overview

Regression analysis estimates relationships between an outcome and one or more predictors. Gelman and Hill present regression as an applied modeling tool that must be interpreted with research design, model assumptions, and substantive meaning in view. ^[4]

First declare the task. A model optimized for prediction is evaluated on out-of-sample performance, calibration, leakage, decision loss, subgroup behavior, robustness, and distribution shift. A model used for coefficient inference needs a defensible sampling/design and uncertainty model. A regression used for causal estimation additionally needs an identification strategy and the methods-owner gate in Section 3. Good prediction does not imply causal validity; an interpretable coefficient does not imply good prediction.

For managers, regression output should answer four questions:

1. What outcome are we explaining?
2. What predictor is being interpreted?
3. What unit change does the coefficient represent?
4. What uncertainty and design limitations remain?

How to Apply

The Regression Output Translation Table

How To Read A Coefficient

Read every coefficient with a unit.

Weak reading: "Usage has a coefficient of 1.8."

Decision-grade reading: "For accounts that differ by one additional active user, the model estimates 1.8 more outcome units, holding the included controls constant."

That sentence still does not prove causality. It states what the model estimates.

The Five Manager Questions

1. Unit: What exact change does the coefficient represent?
2. Baseline: What is the starting level?
3. Uncertainty: How wide is the plausible range?
4. Controls: What has the model adjusted for, and what is missing?
5. Decision threshold: Would we act if the true effect were at the low end of the range?

Common Regression Traps

Trap 1: Coefficient without units

If the unit is unclear, the result is not ready for decision use.

Trap 2: Controls treated as magic

Controls reduce some alternative explanations. They do not automatically remove bias or prove causation.

Trap 3: R-squared worship

A model can have high fit and be useless for intervention. A model can have lower fit and still answer a narrow decision question.

Trap 4: Extrapolation

Do not use the model outside the range where the data supports it.

Trap 5: Average effect hides segments

The average effect may be positive while the effect is negative for a key segment. Ask for segment checks when the decision will be applied unevenly.

Regression Decision Memo Template

Decision: Should we fund the retention intervention?

Model: Regression of renewal outcome on product usage, account size, tenure, segment, support tickets, and onboarding status.

Finding: Higher product usage is associated with higher renewal likelihood after included controls.

Causal stance: Directional, not causal. Customers may self-select into higher usage.

Business implication: Run a targeted usage intervention pilot before scaling.

What would change the answer: If the pilot shows no incremental lift in the target segment, do not scale.

Minimum Model Documentation

Before acting on a regression, require:

• variable definitions;
• sample inclusion and exclusion rules;
• time window;
• missing-data handling;
• model formula;
• declared task: prediction, description, inference, or causal estimation;
• functional form, nonlinearities, interactions, and dependence structure;
• coefficient table;
• residual or diagnostic review;
• leakage checks, holdout or cross-validation plan, calibration and decision-loss evaluation when predictive;
• multiplicity, model-selection, and researcher/analyst degrees-of-freedom record;
• subgroup and heterogeneous-effect checks appropriate to the decision;
• deployment drift and monitoring plan where the model will operate repeatedly;
• sensitivity checks;
• causal interpretation limits.

So What for Managers

• Ask what task the model serves—prediction, description, inference, or causal estimation—before interpreting a coefficient.
• Require units, uncertainty, validation, diagnostics, leakage checks, subgroup behavior, decision loss, and drift monitoring when the model will operate repeatedly.
• Treat controls as adjustments under assumptions, not as proof that omitted variables, selection, measurement error, or reverse causality have disappeared.

Limits and Critiques

• A regression coefficient is conditional on the model, data, design, controls, functional form, and population; it is not automatically a causal effect.
• Good fit or predictive accuracy can coexist with poor calibration, unfair error distribution, leakage, unstable drift, or unusable decisions.
• Average coefficients can conceal heterogeneous effects, nonlinearities, interactions, and harms concentrated in a subgroup.

Connections

Use Framework 3 for causal stance; Framework 4 for uncertainty and multiple testing; Chapter 16 for predictive and AI governance; and Chapter 22 Frameworks 9–13 for sensitivity, decision, experiment, and optimization boundaries.

Overview

Data visualization helps people compare, diagnose, and decide. Tufte's work emphasizes graphical integrity, data density, small multiples, and avoiding visual distortion, while Few's work translates table and graph design into practical business communication. ^{[5] [6]}

The purpose of a visualization is not to look sophisticated. The purpose is to make the comparison obvious without hiding uncertainty or context.

How to Apply

Chart Selection Guide

Visualization Rules For Managers

1. Lead with the question. The title should say what the viewer should learn.
2. Make the comparison visible. Sort bars, align scales, and reduce non-data ink.
3. Show the denominator. A rate without a base can mislead.
4. Show uncertainty when it affects the decision.
5. Use consistent scales for comparison.
6. Avoid decoration that competes with the data.
7. Label directly when possible.
8. Separate diagnostic charts from executive charts.

Executive Chart Template

Title: "Enterprise churn improvement is concentrated in accounts with completed onboarding."

Subtitle: "Pilot cohort outperformed matched comparison accounts, but support-load risk remains."

Chart: Line or bar comparison with direct labels.

Decision note: "Scale to enterprise accounts only; do not expand to SMB until support process is redesigned."

Common Visualization Failure Modes

Failure 1: The chart answers the wrong question.

If the decision is whether to invest, a ranking of activity volume may be less useful than a chart of margin impact.

Failure 2: The chart hides the base.

A large movement in a small population can look more important than a small movement in the core business.

Failure 3: The chart uses inconsistent scales.

Small multiples are useful only when the scales support honest comparison.

Failure 4: The dashboard becomes a storage unit.

Dashboards should separate operating control metrics from investigative diagnostics.

The "One Chart, One Job" Rule

Every executive chart should have one job:

• prove a recommendation;
• diagnose a root cause;
• compare alternatives;
• show a risk range;
• track a committed KPI.

If a chart tries to do all five, split it.

So What for Managers

• Start with the decision and choose the chart, denominator, baseline, uncertainty display, and accessible alternative that answer it.
• Inspect scales, missingness, subgroup coverage, aggregation, annotation, and source definitions before accepting a visual conclusion.
• Treat a chart, table, or dashboard as a communication and control artifact with an owner, refresh rule, and escalation path.

Limits and Critiques

• Visual integrity cannot repair biased sampling, weak causal design, poor measurement, or an invalid denominator.
• Simplification can improve comprehension while hiding uncertainty, distributional effects, small groups, or operational exceptions.
• Tufte/Few-inspired design guidance is not a universal aesthetic or accessibility standard; test the rendered artifact with affected users and assistive technology where relevant.

Connections

Use Frameworks 1–5 for the argument and analytical evidence; Chapter 20 for fairness, privacy, and affected-party review; and Chapter 22 Frameworks 7–10 for metric, benchmark, sensitivity, and simulation visuals.

Overview

A KPI tree is a practical author synthesis that decomposes a top-level outcome into hypothesized drivers, sub-drivers, controllable operating metrics, and guardrails. Kaplan and Norton's balanced-scorecard work supports linked, balanced measures, but the exact tree and every arrow below are hypotheses to validate rather than canonical causal structure. ^[7]

A good KPI tree does not merely display metrics. It shows how the business works.

How to Apply

Required Mermaid Diagram: KPI Tree

Figure 22.2. Constructed KPI hypothesis tree for profitable growth. The diagram links a strategic outcome to financial and operating measures. Each arrow is a proposed relationship to define, measure, and test; ownership does not make it causal. Source basis: balanced-measurement logic, adapted as author synthesis. ^[7]

Text equivalent: Profitable growth is decomposed into revenue growth, margin expansion, and capital efficiency. Those branches are further decomposed into acquisition, expansion, retention, margin, cost, discount, working-capital, and productivity measures. Each metric needs a definition, owner, guardrail, and evidence showing whether changing it can improve the higher-level outcome without unacceptable harm elsewhere.

graph TD
    A[North Star Outcome: Profitable Growth] --> B[Revenue Growth]
    A --> C[Margin Expansion]
    A --> D[Capital Efficiency]
    B --> E[New Customers]
    B --> F[Expansion Revenue]
    B --> G[Retention]
    C --> H[Gross Margin]
    C --> I[Service Cost]
    C --> J[Discount Discipline]
    D --> K[Working Capital]
    D --> L[Sales Productivity]
    E --> M[Qualified Pipeline]
    E --> N[Win Rate]
    F --> O[Seat Expansion]
    F --> P[Cross-Sell]
    G --> Q[Logo Renewal]
    G --> R[Usage Health]

How To Build A KPI Tree

1. Start with the outcome. Pick one strategic result, such as profitable growth, retention, or cash conversion.
2. Decompose into hypothesized drivers. Use business logic, accounting identities, process knowledge, and evidence—not dashboard availability—and label identity, association, prediction, and causal hypotheses separately.
3. Separate outcomes from levers. Revenue is an outcome; price, volume, renewal, and mix are drivers.
4. Assign ownership. Every controllable metric needs an owner.
5. Define the grain. Decide whether the metric is measured by customer, product, region, week, or month.
6. Identify tradeoffs. A metric that improves one branch may damage another.
7. Set review cadence. Some metrics are daily operating controls; others are monthly strategy measures.
8. Validate and revise the tree. Test definitions, accounting reconciliation, lag structure, causal assumptions, trade-offs, and whether interventions on a lever actually change the intended outcome.

KPI Tree Template

KPI Quality Tests

• Decision-linked: Does this metric influence a real choice?
• Owned: Is one person accountable for movement?
• Actionable: Can the owner change it?
• Timely: Is it available at the cadence of action?
• Comparable: Is the definition stable across periods and teams?
• Balanced: Does it include guardrails against local optimization?

Common KPI Tree Failure Modes

Failure 1: Metric pile instead of driver logic

If the tree is just every available metric, it will not guide action.

Failure 2: No owner

A KPI without an accountable owner is an observation, not a management tool.

Failure 3: Financial-only view

Financial outcomes matter, but they often lag operational drivers. Balanced-scorecard logic asks managers to connect financial, customer, process, and learning dimensions. ^[7]

Failure 4: No guardrails

If the tree only rewards speed, cost, or conversion, teams may sacrifice quality, trust, or long-term value.

So What for Managers

• Treat every driver arrow as a hypothesis with a definition, owner, lag, evidence plan, and guardrail rather than a proven causal chain.
• Reconcile KPI definitions to financial, customer, process, and quality records before using them to allocate resources or evaluate people.
• Review trade-offs and affected groups so local optimization does not improve a metric by shifting cost, risk, or harm elsewhere.

Limits and Critiques

• A KPI tree can impose a tidy causal story on a complex system; accounting identities, associations, predictions, and causal claims must be distinguished.
• A single North Star or balanced scorecard does not capture every mission, obligation, distributional effect, or non-compensable limit.
• Metrics create incentives and can be gamed; measurement changes may reflect instrumentation, selection, or behavior adaptation rather than real performance.

Connections

Use Chapter 8 for execution and KPI governance; Chapter 20 for ethics and guardrails; Chapter 21 for product metrics; and Frameworks 3–5 here for causal, statistical, and visual validation.

Overview

Benchmarking compares performance, practices, or capabilities against a relevant reference group. APQC's Benchmarking Basics is a current institutional resource for launching a benchmarking initiative; it brings together materials for planning, collection, analysis, adaptation, partner selection, and data normalization. ^[8]

Benchmarking is useful when it creates a learning agenda. It is dangerous when it becomes status theater. These practical cautions are an author synthesis for decision use, not claims of an average performance benefit. ^[8]

How to Apply

Types Of Benchmarking

This classification is an author synthesis to help choose a reference group; it does not label any external peer as universally best-performing. ^[8]

The Benchmarking Process

APQC groups its methodology materials around plan, collect, analyze, and adapt. The seven-step manager workflow below translates those phases into a decision-oriented sequence; it does not demonstrate that a transferred practice will succeed. ^[8]

1. Define the question. What decision will the comparison influence?
2. Choose the peer set. The reference group should match the decision.
3. Normalize definitions. Align metric definitions, time windows, mix, and scope.
4. Find the gap. Compare performance and practice, not just numbers.
5. Diagnose causes. Identify what the better performer does differently.
6. Translate to action. Decide which practice can be adopted, adapted, or rejected.
7. Track implementation. The value comes from operational change.

Benchmarking Questions Managers Should Ask

• Are we comparing like with like?
• Is the peer group relevant to our strategy?
• Are definitions consistent?
• Does the comparison control for mix, geography, scale, or maturity?
• Is the gap caused by practice, context, or measurement?
• What action would we take if the comparison is valid?

Common Benchmarking Traps

Trap 1: Bad peer group

A fast-growing startup, mature enterprise, and regulated utility may all use the same metric name while operating under different constraints.

Trap 2: Definition mismatch

"Active customer" can mean login, purchase, subscription, transaction, or account status.

Trap 3: Copying without context

The best practice may depend on scale, brand, channel, capital structure, or regulation.

Trap 4: External comparison used to avoid internal accountability

If every external comparison becomes an excuse, the work has become defensive rather than diagnostic.

Practical Output

End every benchmarking effort with a decision table:

Rows above are illustrative, not external performance claims.

So What for Managers

• Define the comparison population, metric, denominator, time window, mix, data authority, and purpose before looking for a gap.
• Use benchmarks to generate learning questions and testable adaptations, not to copy another organization or assign blame.
• Record which differences are evidence-backed, which are hypotheses, and who owns the decision to adopt, adapt, reject, or learn more.

Limits and Critiques

• Benchmark comparability can fail through definition mismatch, selection, scale, geography, regulation, maturity, accounting, or unobserved context.
• A better observed performer may benefit from a different strategy or constraints; association does not establish that its practice caused the gap.
• External data may be stale, permission-limited, confidential, or too aggregated to support a fair operational decision.

Connections

Use Chapter 3 for competitive context; Chapter 4 for financial definitions; Chapter 6 for operating processes; Chapter 8 for execution; and Frameworks 7 and 9 here for KPI and sensitivity follow-up.

Overview

Sensitivity analysis asks how much the decision changes when key assumptions move. Saltelli and coauthors frame sensitivity analysis as a way to understand how uncertainty in model inputs affects model outputs. ^[9]

Managers should use sensitivity analysis before arguing about precision. If the decision is robust across plausible assumptions, you can move faster. If the decision flips, you need better information or a more reversible plan.

How to Apply

Sensitivity Analysis Grid

All entries in this grid are placeholders for a worked business case, not external facts.

How To Build It

1. Start with the decision model. Identify the output that matters: NPV, margin, payback, service level, retention, or risk.
2. List the assumptions. Include volume, price, cost, timing, adoption, churn, conversion, and operational constraints.
3. Choose plausible ranges. Do not use fantasy upside or performative downside.
4. Move one input at a time first. This reveals which assumptions matter.
5. Then test combinations. Real downside cases often combine several adverse assumptions.
6. Mark decision sensitivity. High sensitivity means the assumption can change the answer.
7. Turn sensitivity into action. Gather more evidence, redesign the project, stage the commitment, or hedge the risk.

The Three Sensitivity Questions

1. Which assumption matters most?
2. How wrong can we be before the decision changes?
3. What is the cheapest way to reduce the most important uncertainty?

Tornado Logic Without The Chart

A tornado chart ranks assumptions by how much each one changes the output. You do not need the chart to use the logic. Rank assumptions by decision impact, then spend analytical effort only on the top few.

Common Failure Modes

Failure 1: Sensitivity theater

The model includes ranges, but the recommendation ignores them.

Failure 2: Narrow ranges

The downside case is barely different from the base case, so the analysis creates false comfort.

Failure 3: Equal attention to every input

Most inputs do not change the decision. Focus effort on the assumptions that do.

Failure 4: No operational response

If a high-sensitivity assumption has no owner or mitigation, the analysis is incomplete.

So What for Managers

• Identify the assumptions that can change the decision, not every input that can vary.
• Use ranges, combinations, and coherent scenarios that reflect evidence, operational constraints, and affected-party consequences.
• Convert sensitivity into an information, staging, mitigation, or stop decision with an owner and trigger.

Limits and Critiques

• One-at-a-time sensitivity can miss interactions, dependence, nonlinearities, structural uncertainty, and tail behavior.
• Plausible ranges are judgments; narrow or asymmetric ranges can create false robustness or manufactured volatility.
• A model can be sensitive to an input without that input being controllable, measurable, or worth improving.

Connections

Use Chapter 4 for valuation and cash-flow thresholds; Framework 8 for benchmark definitions; Framework 10 for simulation; and Frameworks 11–13 for decision, experiment, and optimization choices.

Overview

Monte Carlo simulation models uncertainty by assigning ranges or distributions to uncertain inputs, running many simulated outcomes, and examining the resulting range of possible outputs. Hubbard's measurement work uses Monte Carlo thinking to make uncertain business cases more decision-ready, while sensitivity-analysis sources clarify how input uncertainty propagates through model outputs. ^{[10] [9]}

Use Monte Carlo when a single base case hides too much risk.

How to Apply

Figure 22.3: Monte Carlo Setup Flow

Figure 22.3. Monte Carlo decision-model workflow. The author-created flow moves from a deterministic decision model through uncertain inputs, distributions and dependencies, simulation, validation, outcome ranges, sensitivity, and action. Simulation outputs remain conditional on the structure and assumptions. ^{[9] [10]}

Text equivalent: Define and validate the base decision model; identify uncertain inputs; choose evidence-based ranges or distributions and their dependencies; run enough simulations for stable summaries; compare results with known data or limiting cases; inspect the full outcome distribution and tail; identify decision drivers; then act, stage, hedge, redesign, or collect higher-value information.

flowchart LR
    A[Decision Model] --> B[Identify Uncertain Inputs]
    B --> C[Assign Plausible Ranges]
    C --> D[Choose distributions and encode input dependence]
    D --> E[Run simulations]
    E --> V[Validate base case, limiting cases, stability, and known evidence]
    V --> F[Review full outcome range and tail scenarios]
    F --> G[Find sensitivity drivers and failure regions]
    G --> R[Decision, guardrail, owner, and stakeholder review]
    R --> H[Act, stage, hedge, redesign, or learn more]
    H --> A

When To Use Monte Carlo

Use it when:

• the decision is material;
• several uncertain inputs interact;
• downside risk matters;
• a single base case is misleading;
• leaders need to understand range, not just point estimate;
• reversibility is limited.

Setup Steps

1. Define the decision output. Example: expected cash flow, payback period, capacity shortfall, or service-level breach.
2. Build the deterministic model first. If the base model is unclear, simulation will only add confusion.
3. Identify uncertain inputs. Focus on the few variables that plausibly move the decision.
4. Set ranges or distributions. Use historical data, expert estimates, contract terms, or pilot evidence.
5. Preserve relationships. Encode evidence-based correlation, conditional dependence, common drivers, and tail dependence; independence is a model assumption, not a default fact.
6. Validate the model. Reconcile the deterministic case, test known/limiting cases, inspect input/output behavior, compare with held-out or historical evidence where possible, and obtain independent review.
7. Run enough iterations for stability. The point is a stable outcome range, not false precision.
8. Analyze the output distribution. Look at downside, upside, median, tail, decision thresholds, and subgroup or stakeholder consequences where relevant.
9. Translate to action. Stage investment, change design, hedge risk, gather more evidence, or proceed.

A named human model owner and decision owner must approve the structure, inputs, dependence, validation, thresholds, and limits. Software output does not own the assumptions.

Distribution Selection For Managers

Examples are generic setup patterns, not universal modeling rules.

Monte Carlo Output Questions

• What is the central case?
• How bad is the downside tail?
• Which inputs drive the downside?
• How often does the result cross the decision threshold?
• Which mitigation reduces downside most cheaply?
• Does the decision still work if the most important input disappoints?

Common Failure Modes

Failure 1: Fancy model, weak assumptions

Monte Carlo does not rescue bad inputs.

Failure 2: Hidden correlations

Revenue, churn, price, and service cost may move together. Treating them as independent can make the range look safer than it is.

Failure 3: Output without decision threshold

If the model does not show where the decision changes, it is just a distribution.

Failure 4: Overprecision

Simulation output can look exact. Communicate rounded ranges and decision categories, not spurious decimal points.

So What for Managers

• Approve the deterministic model, input evidence, dependence assumptions, validation plan, iterations, thresholds, and model/decision owners before interpreting output.
• Report the full decision-relevant range, tail risk, sensitivity drivers, stability, and stakeholder consequences rather than a single simulated average.
• Use simulation to choose a staged action, mitigation, hedge, redesign, or information purchase; do not let software output choose the action.

Limits and Critiques

• A simulation is conditional on model structure, distributions, dependence, data quality, validation, and the decision threshold.
• More iterations reduce simulation noise but do not correct misspecification, omitted consequences, false independence, or biased input ranges.
• Expected or percentile outputs can hide catastrophic low-probability harms, subgroup effects, liquidity constraints, or non-compensable obligations.

Connections

Use Framework 9 for global sensitivity; Framework 11 for expected value, utility, and information value; Chapter 20 for rights, privacy, and ethics; and Chapter 16 for AI model and deployment governance.

Overview

Managerial decision analysis separates choices from uncertain events, makes consequences and probabilities inspectable, and tests whether more information can improve an authorized decision. It is a decision aid, not a forecast or a substitute for legal, safety, rights, ethical, finance, methods, or operational review.

How to Apply

Purpose and Boundary

Decision analysis separates what an owner can choose from what remains uncertain. A decision node contains feasible actions; a chance node contains mutually exclusive uncertain outcomes with probabilities that sum to one; a terminal branch contains the consequence of that action-outcome path. HM Treasury's 2026 appraisal guidance uses probability-weighted expected values and decision trees for complex, sequential, or difficult-to-reverse choices. ^[11]

Apply non-compensable gates before ranking options. If an option fails an applicable legal requirement, safety limit, rights obligation, ethical standard, or other authorized minimum, remove or redesign it rather than allowing favorable money or a weighted score to offset the failure. Official UK analytical guidance warns that compensatory models permit good performance on one criterion to offset poor performance on another and describes absolute minima for eliminating unsuitable options before scoring. ^[12]

Expected monetary value is a risk-neutral comparison, not a promise, forecast, or complete decision rule. Use expected utility when an authorized decision maker's risk preference or the severity and distribution of consequences could change the ranking. Utility must be elicited and sensitivity-tested; it must not be invented to rationalize a preferred answer. ^[12]

Core Calculations

1. Expected monetary value (EMV): for each feasible option, multiply every monetary consequence by its probability and sum the products.
2. Break-even probability: for an upside U and downside D, both measured relative to the alternative, solve p x U + (1 - p) x D = 0. When U > 0 and D < 0, p* = -D / (U - D).
3. Bayesian update: begin with the relevant base rate or prior probability, then update it with evidence: P(H|E) = P(E|H) x P(H) / P(E). Penn State's open probability lesson derives this conditional-probability relationship. ^[13]
4. Value of information (VOI): compare the expected value of the best action after observing information with the expected value of the best action now. Subtract testing, delay, implementation, privacy, safety, and other information-acquisition costs. Information has decision value only when it can change an action or improve its timing, targeting, safeguards, or consequence. USGS illustrates this logic with Bayesian decision trees comparing decisions with and without new scientific information. ^[14]
5. Reversibility: model later decision points explicitly. A staged choice may preserve an option to expand, revise, or stop as evidence arrives, but only if rollback, obligations, affected parties, and path-dependent costs make that flexibility real. ^[11]

Worked Example: Test Before a Bounded AI-Support Launch

The following company, probabilities, test characteristics, and monetary consequences are constructed teaching assumptions, not benchmarks. Amounts are in thousands of dollars and exclude any consequence that cannot responsibly be monetized.

An organization is considering a bounded AI assistant for low-risk customer-support drafting. Before analysis, Privacy, Security, Legal, Accessibility, Product, and Support owners must approve the data authority, use boundary, evaluation, human-review design, incident path, and rollback. If a required gate fails, the organization redesigns or stops regardless of EMV. See Chapter 16 for AI/non-AI sourcing, evaluation, deployment, and change control.

For the feasible financial comparison, the team estimates a 45 percent base-rate probability of a high-value outcome. A bounded launch would produce +$600 if high value materializes and -$400 otherwise. An evaluation costing $40 has an assumed 80 percent pass rate when the opportunity is genuinely high value and a 20 percent false-positive rate when it is low value.

Figure 22.4. Evidence-gated decision tree for a constructed AI-support decision. Decision and chance nodes are labeled in text as well as by shape. The tree compares immediate launch, testing, and deferral only after non-compensable gates pass. It is an author-created worked example based on expected-value, decision-tree, Bayesian-update, utility, information-value, and reversibility principles. ^{[11] [12] [14] [13]}

Text equivalent: First determine whether legal, safety, rights, security, privacy, accessibility, and authority gates are satisfied. If not, redesign or stop. If they are satisfied, compare immediate launch, an evaluation followed by a later launch/defer decision, and deferral. Immediate launch faces a 45 percent high-value outcome and 55 percent low-value outcome. The evaluation passes 47 percent of the time; after a pass, the updated high-value probability is 76.6 percent and launch has a positive conditional EMV. After a fail, the updated probability is 17.0 percent and deferral dominates launch in the constructed monetary model.

flowchart TD
    G{DECISION GATE:<br/>all non-compensable requirements satisfied?}
    G -->|No| R[Redesign or stop;<br/>do not offset the failed gate]
    G -->|Yes| D{DECISION:<br/>launch, test, or defer?}
    D -->|Launch now| C((CHANCE:<br/>value outcome))
    C -->|High value: 45%| H[Consequence: +$600k]
    C -->|Low value: 55%| L[Consequence: -$400k]
    D -->|Buy evaluation for $40k| T((CHANCE:<br/>test result))
    T -->|Pass: 47%| P{DECISION:<br/>launch or defer?}
    P -->|Launch; posterior 76.6%| PE[Conditional EMV: +$366k]
    P -->|Defer| P0[Consequence: $0]
    T -->|Fail: 53%| F[Defer;<br/>launch EMV would be -$230k]
    D -->|Defer| Z[Consequence: $0]

Calculation Table

The risk-neutral monetary ranking is therefore test ($132 after test cost), launch now ($50), defer ($0). This is not a deployment authorization. If a $400 downside threatens solvency, harms a protected group, breaches a duty, or exceeds risk authority, expected utility or a non-compensable gate can change or eliminate the choice. Vary the base rate, test accuracy, consequences, delay, and test cost; obtain independent review when small changes reverse the result.

Manager Checklist

• Are all options lawful, safe enough, authorized, and consistent with non-compensable rights and policy minima?
• Which nodes are decisions and which are uncertain events?
• Are branches mutually exclusive, collectively sufficient for the decision, and probability-normalized?
• What evidence supports each base rate, likelihood, consequence, and dependency?
• What is the break-even probability, and how far is the current estimate from it?
• Would risk preference, tail harm, distributional impact, or liquidity change an EMV ranking?
• Can the proposed information actually change the decision, and does its net value remain positive after all costs?
• Is the proposed staged action genuinely reversible, with an owner, monitor, rollback, remedy, and review date?

So What for Managers

• Apply non-compensable legal, safety, rights, ethical, policy, authority, and feasibility gates before ranking monetary or weighted-score outcomes.
• Separate decision nodes, chance nodes, probabilities, consequences, evidence, risk preference, information costs, and reversibility in the record.
• Use sensitivity and an authorized owner to decide whether to launch, test, defer, stage, redesign, or stop.

Limits and Critiques

• Expected monetary value assumes a risk-neutral comparison and can obscure distribution, liquidity, tail harm, rights, and unequal consequences.
• Probabilities and utilities are judgment-laden; false precision or invented utility can make a preferred answer look mathematically inevitable.
• Value of information is only useful when the information can change an action and its privacy, safety, delay, implementation, and opportunity costs are included.

Connections

Use Chapter 4 for financial thresholds; Chapter 9 for decision structure; Chapter 16 for AI deployment and change control; Chapter 20 for non-compensable ethical boundaries; and Frameworks 9–10 for sensitivity and simulation.

Overview

Experimentation estimates a decision-relevant contrast under a precommitted estimand, design, uncertainty plan, guardrails, stopping rule, and decision rule. A statistically detectable result is not automatically an economically valuable, safe, fair, private, accessible, or deployable result.

How to Apply

Start With The Estimand, Not The Test

An experiment should estimate a decision-relevant effect, not merely produce a p-value. Before assignment begins, write the estimand: the precise effect the analysis is intended to estimate. The ICH E9(R1) framework defines this discipline through the treatment conditions, population, outcome variable, handling of events after assignment that complicate interpretation, and population-level summary. The structure transfers usefully to business experiments even though business tests are not clinical trials. ^[15]

For a retention experiment, a defensible estimand might be: “the 30-day difference in retained-customer rate among eligible new customers assigned to the new onboarding flow versus the current flow, regardless of whether they complete onboarding.” That statement resolves five questions before results arrive:

Do not quietly replace this estimand with “effect among completers” after attrition, “effect on exposed users” after non-use, or “effect in the winning segment” after looking at the data. Those are different questions and may require different designs or assumptions.

MDE, Power, and Sample Size

The minimum detectable effect (MDE) is the smallest effect the design is intended to detect with its specified power and error rate. It is not the smallest effect that matters. The team should first set a minimum practically important effect from economics, customer value, risk, or capacity; then size the experiment so a result near that threshold is informative. ICH E9 requires sample-size justification tied to the primary objective, type I error, power, and assumptions. ^[16]

For an illustrative equal-allocation, two-arm comparison of means with a two-sided 5 percent error rate, 80 percent power, and outcome standard deviation sigma, a common normal approximation is:

n per arm = ceiling(2 x (1.96 + 0.84)^2 x sigma^2 / MDE^2)

The approximation below uses sigma = 1, so MDE is in standardized units. Smaller effects require sharply larger samples. Binary or ratio outcomes, clustering, covariate adjustment, unequal allocation, repeated looks, multiplicity, noncompliance, and interference require design-specific calculations.

Accessible data table for Figure 22.5:

The Precommitted Experiment Contract

Before launch, the decision owner and qualified methods owner should approve one inspectable contract:

1. Decision and rule: what happens for benefit, inconclusive evidence, or guardrail harm; who can stop or override; and the deadline.
2. Estimand and assignment: population, unit of randomization, treatment contrast, outcome, horizon, post-assignment events, and analysis population.
3. Effect and precision: practical threshold, MDE, error rate, power, variance or baseline-rate assumptions, sample size, and any attrition, clustering, or noncompliance allowance.
4. Metric hierarchy: one primary outcome or a pre-defined primary family; diagnostic metrics; data-quality checks; and guardrails with explicit non-inferiority or harm limits. Microsoft distinguishes overall-evaluation, diagnostic, data-quality, and guardrail metrics in its experimentation practice. ^[17]
5. Timing and stopping: fixed horizon or a valid group-sequential/always-valid design; planned looks; efficacy, futility, safety, and operational stop rules. Repeatedly checking ordinary fixed-horizon p-values and stopping when one crosses a threshold invalidates their usual interpretation. ^[18]
6. Multiplicity: the family of outcomes, variants, looks, and confirmatory subgroups to be protected; the error-control method; and which analyses are exploratory. More tests create more opportunities for chance findings. ^{[16] [18]}
7. Attrition and missingness: expected loss, reasons, participant or unit flow, differential loss by arm, telemetry failure, analysis handling, and sensitivity analysis. Transparent flow and loss reporting is necessary because post-assignment exclusions can change the comparison. ^[19]
8. Interference and spillovers: whether one unit's treatment can change another unit's outcome through teams, households, networks, inventory, pricing, or shared capacity. If so, redefine the estimand and consider cluster, geographic, network, or two-stage assignment; direct and spillover effects are distinct causal quantities. ^[20]
9. Novelty and duration: whether initial curiosity, learning, fatigue, seasonality, or survivorship could make the early effect differ from the steady-state effect; pre-plan time-segment checks or a longer holdout. A changing effect across date segments may indicate novelty, but the pattern is a diagnostic rather than proof. ^[17]
10. Subgroups: a small number of theory-backed, pre-specified hypotheses with interaction tests. Do not infer that subgroups differ merely because one is statistically significant and another is not; label post-hoc segment mining exploratory. ^{[16] [19]}

Decision Rules That Resist Result Shopping

Use a rule that combines effect, uncertainty, and guardrails. For example: “Adopt only if the 95 percent interval for the primary effect excludes zero, the point estimate exceeds the practical threshold, no pre-specified guardrail crosses its harm limit, data-quality checks pass, and the confirmatory analysis follows the stopping and multiplicity plan. Otherwise continue only under the precommitted inconclusive rule, redesign, or stop.”

That rule is intentionally stricter than “p < 0.05.” A statistically detectable effect can be economically trivial; an economically attractive point estimate can remain too uncertain; and a primary-metric gain can still be unacceptable when reliability, safety, fairness, privacy, cost, or customer-experience guardrails deteriorate.

So What for Managers

• Approve the estimand, practical threshold, assignment, primary outcome, guardrails, power/precision, stopping, multiplicity, attrition, interference, novelty, subgroup, and decision plan before assignment.
• Interpret the result against effect size, uncertainty, data quality, exposure, implementation cost, guardrails, and affected groups—not p-value alone.
• Treat inconclusive, heterogeneous, unsafe, or invalid results as reasons to continue under the precommitted rule, redesign, or stop rather than shop for a winning segment.

Limits and Critiques

• Randomization does not cure poor measurement, noncompliance, attrition, interference, novelty, selective reporting, or an ill-defined estimand.
• Fixed-horizon, sequential, clustered, networked, adaptive, and subgroup analyses require design-specific methods; a simple sample-size formula is not universal.
• A positive incremental estimate can be inappropriate to scale when capacity, accessibility, privacy, fairness, safety, security, or legal conditions fail.

Connections

Use Frameworks 3–5 for causal and statistical interpretation; Chapter 13 for experimentation; Chapter 16 for AI evaluation; Chapter 20 for ethics and guardrails; and Chapter 21 for product decisions and metrics.

Overview

Optimization translates a controllable choice into decision variables, an objective, constraints, units, scenarios, and a feasible solution. A solver can optimize the mathematical model supplied; it cannot establish that the objective, data, constraints, omissions, or consequences represent the real decision.

How to Apply

From Prediction To A Feasible Choice

Prediction estimates what may happen. Prescriptive analytics recommends a feasible action under an explicit model. A linear optimization model has:

Google OR-Tools' official examples use the same variable-constraint-objective-solve structure. A solver's “optimal” status means optimal for the mathematical model provided; it does not prove that the objective, data, constraints, or omitted consequences represent the real decision. ^[21]

Accessible vertex table for Figure 22.6:

Continuous LP, Integer Models, and Scenarios

Use a continuous linear program when fractional decisions are meaningful: tons, hours, budget shares, or flow. Use an integer or mixed-integer model when choices are indivisible or logical: facilities, people, trucks, production batches, open-or-close decisions, or yes/no assignments. Integer restrictions can create an integrality gap between the relaxed continuous solution and the best implementable whole-number choice, and they generally require different solution methods. ^[22]

In the constructed model, the continuous optimum is (39.67, 20.67) with value 2,206.67; the best whole-unit solution is (40, 20) with value 2,200. If a supply scenario adds x <= 35, the continuous optimum becomes (35, 23) with value 2,090. Scenario constraints should represent coherent states or policies—such as demand caps, emissions ceilings, staffing floors, service requirements, or supplier failure—not arbitrary stress multipliers.

Shadow-Price Intuition and Limits

A shadow price is the local improvement in the optimal objective from relaxing a binding constraint by one unit, while the relevant solution structure remains unchanged. In this example, local dual values are approximately 16.67 per additional labor unit and 6.67 per additional machine unit. That makes the first extra labor unit more valuable in the model, but it does not mean the organization should pay any price indefinitely. Shadow prices can change when a constraint stops binding, a different corner becomes optimal, integrality matters, or the model changes. ^[23]

Optimality Is Conditional

Before acting on a solver output, ask:

• Are the variables actually controllable, and are implementation lags represented?
• Does the objective include the decision owner's real economics and non-compensable safety, legal, rights, quality, and service limits?
• Are key nonlinearities, uncertainty, queues, network effects, and human behavior material enough to invalidate a linear model?
• Are constraints measured in compatible units and validated by process owners?
• Is the model feasible under base, downside, and operational-disruption scenarios?
• Does the recommendation remain useful with forecast error, parameter ranges, and integer or policy restrictions?
• Has a human owner compared the result with a baseline, inspected surprising allocations, and documented omitted effects?

Optimization does not manufacture certainty. It makes the assumed decision logic inspectable.

Reproducible Two-Part Exercise

All inputs and expected outputs are stored in docs/evidence-packets/ch22-experiment-optimization-visual-data.json; node scripts/verify-ch22-experiment-optimization.js recomputes the results.

Part A — experiment sizing. Using the stated normal approximation, two-sided 5 percent error, 80 percent power, standardized MDE 0.25, and 10 percent anticipated attrition:

1. Raw size is ceiling(15.68 / 0.25^2) = 251 per arm.
2. Recruit ceiling(251 / 0.90) = 279 per arm, or 558 total.
3. Write the estimand, practical threshold, primary outcome, guardrails, fixed or sequential stopping rule, hypothesis family, attrition/interference plan, novelty check, subgroup interactions, and adopt/redesign/stop rule before assignment.

Part B — optimization. For maximize 40x + 30y, subject to 2x + y <= 100, x + 2y <= 81, and x,y >= 0:

1. Solve the two binding equalities to obtain the continuous intersection (119/3, 62/3), or (39.67, 20.67), with value 6,620/3, or 2,206.67.
2. Enumerate feasible whole-number points to obtain integer optimum (40, 20) with value 2,200.
3. Add scenario constraint x <= 35; the new optimum is (35, 23) with value 2,090.
4. Explain why the continuous answer, integer answer, and scenario answer differ, and name one omitted real-world constraint that could change the recommendation.

So What for Managers

• Confirm that variables are controllable, units and constraints are validated, non-compensable limits are encoded, and a human owner can implement the solution.
• Compare continuous, integer, scenario, baseline, and sensitivity results; inspect surprising allocations and omitted effects before acting.
• Treat “optimal” as conditional on the model and use staging, monitoring, and rollback when forecast error or operational disruption matters.

Limits and Critiques

• Optimization can produce a precise answer to the wrong objective or an infeasible answer when constraints, lags, nonlinearities, behavior, or data are misspecified.
• Shadow prices are local and conditional; they can change when a constraint stops binding, integrality matters, or the model structure changes.
• A mathematical optimum can externalize quality, labor, environmental, accessibility, privacy, safety, or distributional costs unless the decision record includes them.

Connections

Use Chapter 6 for capacity and operating constraints; Chapter 4 for financial objectives; Chapter 8 for execution; Chapter 20 for rights and non-compensable boundaries; and Frameworks 9–12 for uncertainty, decision, and experiment evidence.