Right prediction. Wrong reason.

Shape defines the answer. Background is a shortcut. An unused badge can still receive attribution. Inspect all three in a synthetic world we can enumerate exactly.

A synthetic experiment · Hand-specified model · No training or external AI service

Start with the experiment ↓

Start by swapping only the background. The true shape stays the same.

ONE RECORD · TWO WORLDS

What changes the prediction?

Both views use the same model. Only the background changes between them.

Selected record

—P(circle)

Calculating…

Background swapped

—P(circle)

Calculating…

The pictures encode three binary features; the model reads those codes, not pixels. Classify as circle when P(circle) ≥ 50%. The actual class is always the shape.

Inside this model

Adjust the weights. The badge coefficient is fixed at zero.

Shape weight 1.0

Background weight 2.4

Shape × background 0.8

z = 1.0s + 2.4c + 0.8scP(circle) = 1 / (1 + e^−z)

s: circle +1 / square −1
c: blue +1 / sand −1
p: badge +1 / −1; absent from the formula

SAME PREDICTION · DIFFERENT QUESTIONS

What deserves the credit?

Explanation population

Badge agreement with shape 90%

Identical in both cohorts. Not a model input weight.

Marginal replacementConditional observation

Pushes away from circlePushes toward circle

Exact Shapley contributions in percentage points of P(circle). With the common baseline, both explanations reconstruct the same prediction. Shared scale: −100 to +100 pp, centered on zero. A negative bar is not an error rate.

Two meanings of “missing”

Marginal replacement: hold the revealed features fixed; average the others over the chosen population. This can break their relationship with the revealed features.

Conditional observation: keep only population records that match the revealed features. A badge can convey information about shape even when the model ignores the badge.

Population average —+ feature contributions= prediction —

Inspect all eight coalitions and the exact accounting

The three features have eight possible subsets. Each contribution averages its marginal addition over all six feature orders. The table shows the model's expected P(circle) after revealing each subset.

All coalition values, as percentages
Revealed features	Marginal	Conditional

Export results for full floating-point values. Displayed contributions are rounded to two decimals; a displayed sum may differ by 0.01 percentage points.

ONE MODEL · TWO POPULATIONS

Does the explanation survive a shift?

Change which clues travel together. The model stays fixed.

Reference background agreement 90%

Blue with circles; sand with squares.

Shifted background agreement 10%

Only this association changes across cohorts.

Reference cohort

—

Shifted cohort

—

● Correct prediction · × Wrong prediction. Each symbol is one synthetic record. Outlined symbols match the selected three-feature combination. Each cohort contains exactly 800 records with 400 circles and 400 squares; repeated records are not independent real-world observations.

Inspect the complete population and select a record

All eight distinct combinations are shown. Select a row to update the linked prediction, counterfactual and explanations. Background and badge are independent conditional on shape in this construction.

Synthetic cohort counts and model predictions
Record	Ref.	Shifted	P(circle)

Selection changes the inspected record, not the population counts.

WHAT THIS ESTABLISHES

An attribution is an answer to a specified question.

In the shortcut scenario, accuracy falls when the background stops tracking shape. A locally accurate explanation does not guarantee a robust model. In the unused-feature scenario, the badge can receive conditional credit because it reveals information about shape. Flipping the badge alone still changes the prediction by exactly zero.

Neither set of bars identifies a real-world causal effect. “Marginal” and “conditional” define two different expectation games. The appropriate question depends on what you want to investigate. The distinction is discussed by Janzing and colleagues and Aas and colleagues; additive feature attribution is developed by Lundberg and Lee.

Mathematical specification and reproducibility

For a selected record x and subset S of revealed features, marginal replacement uses v_m(S) = E[f(x_S, X_−S)]. Conditional observation uses v_c(S) = E[f(X) | X_S = x_S]. The marginal version retains dependencies among the missing features.

φ_{j} = \sum_{S \subseteq N ∖ {j}} w (S) Δ_{j} (S)

w (S) = \frac{| S |! (2 - | S |)!}{3!}

Δ_{j} (S) = v (S \cup {j}) - v (S)

There are three features. For subset sizes 0, 1 and 2, the weights are 1/3, 1/6 and 1/3. All eight coalition values are enumerated; there is no Monte Carlo sampling or fitted explanation.

Each shape has 400 records. If a background/badge combination agrees with shape with percentages a and b, its integer count is (a/5)(b/5). For a disagreement use 100−a or 100−b. Five-point controls make every count integral, and positive counts ensure every conditional expectation exists.

The model is specified directly, not trained. Its output is a bounded score called P(circle), not evidence of empirical calibration. The population is completely synthetic. The shown background intervention is a feature replacement in this toy system, not a treatment effect in the world.

For a real model review

State the explanation question and reference population. Probe counterfactual changes that preserve the intended label. Evaluate representative shifts. Inspect model outputs and underlying counts alongside the explanation. A plausible chart is a starting point for investigation, not a governance sign-off.

Keep an experiment

Every control is encoded in the URL. The export includes both explanations, all coalitions, population counts and counterfactuals.