# Reporting This document records the methodology choices embedded in `BusinessReport` and `DiagnosticReport` — the convenience layer that produces plain-English stakeholder narratives from any diff-diff result. Methodology for estimators lives in `REGISTRY.md`. This file is the single source for reporting-layer decisions; `REGISTRY.md` cross-links here rather than duplicating content. ## Module - `diff_diff/business_report.py` — `BusinessReport`, `BusinessContext`. - `diff_diff/diagnostic_report.py` — `DiagnosticReport`, `DiagnosticReportResults`. Both modules dispatch by `type(results).__name__` lookup to avoid circular imports across the 16 result classes. They do no estimator fitting and do not re-derive any variance from raw data; every effect, SE, p-value, CI, and sensitivity bound is either read from the fitted result or produced by an existing diff-diff utility (`compute_honest_did`, `HonestDiD.sensitivity`, `bacon_decompose`, `check_parallel_trends`, `compute_pretrends_power`). When the caller passes the raw panel + column kwargs, `DiagnosticReport` may call those utilities on the supplied data (2x2 PT via `check_parallel_trends`, Goodman-Bacon decomposition via `bacon_decompose`, and the EfficientDiD Hausman PT-All vs PT-Post pretest via `EfficientDiD.hausman_pretest`). The `design_effect` section of `DiagnosticReport.to_dict()` is a read-only surface: it echoes `survey_metadata.design_effect` and `effective_n` from the fitted result along with a `band_label` enum classifying the deviation from 1. The enum values are: - `"improves_precision"` for `deff < 0.95` (effective N is LARGER than nominal N — a precision-improving design); - `"trivial"` for `0.95 <= deff < 1.05` (effectively no effect on inference); - `"slightly_reduces"` for `1.05 <= deff < 2`; - `"materially_reduces"` for `2 <= deff < 5`; - `"large_warning"` for `deff >= 5`; - `None` when `deff` is missing or non-finite. The section does not call `compute_deff_diagnostics` (that helper needs per-fit internals the result objects do not expose). The report layer **does** compose a few cross-period summary statistics from per-period inputs already produced by the estimator — specifically the joint-Wald / Bonferroni pre-trends p-value from pre-period event-study coefficients (see `_pt_event_study`), the MDV-to-ATT ratio for power-tier selection, and the heterogeneity dispersion block (CV / range / sign- consistency over post-treatment group / event-study / group-time effects, pre-period and reference-marker rows excluded). These are reporting-layer aggregations of inputs already in the result object, not new inference. ## Target parameter The BusinessReport and DiagnosticReport schemas both carry a top-level `target_parameter` block that names what scalar the headline number actually represents. The 16 result classes have meaningfully different estimands — a stakeholder reading `overall_att = -0.0214` on a Callaway-Sant'Anna fit cannot tell whether that is the simple-weighted average across `ATT(g,t)` cells, an event-study-weighted aggregate, or a group-weighted aggregate. Baker et al. (2025) Step 2 is "Define the target parameter"; BR/DR does that work for the user. Schema shape: ```json "target_parameter": { "name": "overall ATT (cohort-size-weighted average of ATT(g,t))", "definition": "A cohort-size-weighted average of group-time ATTs ...", "aggregation": "simple", "headline_attribute": "overall_att", "reference": "Callaway & Sant'Anna (2021); REGISTRY.md Sec. CallawaySantAnna" } ``` Field semantics: - `name` — short stakeholder-facing name. Rendered verbatim in BR's summary paragraph and DR's overall-interpretation paragraph. Always non-empty. - `definition` — plain-English description of what the scalar is and how it is aggregated. Rendered in BR's and DR's full-report markdown (under "## Target Parameter") but omitted from the summary paragraph so stakeholder prose stays within the 6-10- sentence target. - `aggregation` — machine-readable tag dispatching agents can branch on. Complete enumeration per estimator: - `"did_or_twfe"` (DiDResults / TwoWayFixedEffects both route here — neutral tag; ambiguous at the result-class level until estimator provenance is persisted) - `"event_study"` (MultiPeriodDiDResults) - `"simple"` (CallawaySantAnna / Imputation / TwoStage / Wooldridge) - `"iw"` (SunAbraham) - `"stacked"` (StackedDiD) - `"pt_all_combined"` / `"pt_post_single_baseline"` (EfficientDiD branched on `pt_assumption`) - `"dose_overall"` (ContinuousDiD) - `"ddd"` / `"staggered_ddd"` (TripleDifference / StaggeredTripleDiff) - `"spillover"` (SpilloverDiD — Butts 2021 total effect on the treated `tau_total`, identified off far-away controls; headline `att`) - dCDH dynamic branches follow the exact `overall_att` contract: `"M"` / `"M_x"` / `"M_fd"` / `"M_x_fd"` for `L_max=None`; `"DID_1"` / `"DID_1_x"` / `"DID_1_fd"` / `"DID_1_x_fd"` for `L_max=1`; `"delta"` / `"delta_x"` for `L_max>=2` without trend suppression; and `"no_scalar_headline"` when `trends_linear=True` AND `L_max>=2` (the scalar is intentionally NaN). - `"synthetic"` (SyntheticDiD) / `"factor_model"` (TROP) / `"twfe"` (BaconDecomposition read-out) / `"unknown"` (default fallback). - `headline_attribute` — the raw result attribute the scalar comes from (`"overall_att"` / `"att"` / `"avg_att"` / `"twfe_estimate"`), OR `None` when `aggregation == "no_scalar_headline"` (the dCDH `trends_linear=True, L_max>=2` branch where `overall_att` is intentionally NaN by design). Agents dispatching on this field must handle `None` by inspecting `headline.reason` (BR) / `headline_metric.reason` (DR), which distinguishes two subcases: - **Populated-surface subcase** (per-horizon `linear_trends_effects` dict is non-empty): `reason` directs callers to `results.linear_trends_effects[l]` for per-horizon cumulated level effects. - **Empty-surface subcase** (`linear_trends_effects is None` because no horizons survived estimation): `reason` names the empty state explicitly and directs callers toward re-fit remediation (larger `L_max` or `trends_linear=False`) rather than a nonexistent dict. The dCDH native estimand label is also branched — on this subcase `_estimand_label()` returns `DID^{fd}_l (no cumulated level effects survived estimation)` (or `DID^{X,fd}_l (...)` when covariates are active). Different result classes use different attribute names; agents that want to re-read the raw value can dispatch on `headline_attribute`. - `reference` — one-line citation pointer to the canonical paper and the REGISTRY.md section. Per-estimator dispatch lives in `diff_diff/_reporting_helpers.py::describe_target_parameter`. Each branch is sourced from the corresponding estimator's section in REGISTRY.md; new result classes must add an explicit branch (the exhaustiveness test `TestTargetParameterCoversEveryResultClass` locks this in). A few branches read fit-time config from the result object: - `EfficientDiDResults.pt_assumption`: `"all"` (over-identified combined) vs `"post"` (just-identified single-baseline) branches `aggregation` between `"pt_all_combined"` and `"pt_post_single_baseline"`. - `StackedDiDResults.clean_control`: `"never_treated"` / `"strict"` / `"not_yet_treated"` varies the `definition` clause describing which units qualify as controls. - `ChaisemartinDHaultfoeuilleResults.L_max` + `covariate_residuals` + `linear_trends_effects`: branches the dCDH estimand tag per the exact `overall_att` contract in `chaisemartin_dhaultfoeuille.py:2602-2634` and `chaisemartin_dhaultfoeuille.py:2828-2834`: - `L_max=None` → `DID_M` (Phase 1 per-period aggregate; `aggregation="M"`). - `L_max=1` → `DID_1` (single-horizon per-group estimand, Equation 3 of the dynamic companion paper; `aggregation="DID_1"`). - `L_max>=2` → cost-benefit `delta` (Lemma 4 cross-horizon aggregate; `aggregation="delta"`). - `trends_linear=True` AND `L_max>=2` → `overall_att` is intentionally NaN (no scalar aggregate; per-horizon level effects live on `results.linear_trends_effects[l]`). `aggregation="no_scalar_headline"` and `headline_attribute` is `None`. Covariates (`has_controls`) and/or linear trends (`has_trends`, when `L_max < 2`) add `_x` / `_fd` / `_x_fd` suffixes to the `aggregation` tag and the corresponding `^X` / `^{fd}` / `^{X,fd}` superscripts to the `name` (e.g. `DID^X_1`, `delta^X`, `DID^{fd}_M`), matching the result class's own `_estimand_label()` helper at `chaisemartin_dhaultfoeuille_results.py:454-490`. A few branches emit a fixed tag regardless of fit-time config — notably `CallawaySantAnna`, `ImputationDiD`, `TwoStageDiD`, and `WooldridgeDiD`. For these estimators the `overall_att` (or `att` / `avg_att`) scalar is ALWAYS the simple weighted aggregation; the fit-time `aggregate` kwarg populates additional horizon / group tables on the result object but does not change the headline scalar. Disambiguating those tables in prose is tracked under BR/DR gap #9 (per-cohort narrative rendering). `ContinuousDiDResults` emits a single `"dose_overall"` tag with a disjunctive definition (`ATT^loc` under PT; `ATT^glob` under SPT) because the PT-vs-SPT regime is a user-level assumption, not a library setting. ## Design deviations - **Note:** No hard pass/fail gates. `DiagnosticReport` does not produce a traffic-light verdict. Severity is conveyed through natural-language phrasing ("robust", "fragile", "material share"). This is an explicit deviation from the strategy document's Gap 4 ("traffic-light assessment (green/yellow/red)"); the choice is motivated by the well-known risk of naive thresholds producing false confidence. A `ConservativeThresholds` opt-in layer remains available as a future addition if practitioner demand materialises. - **Note:** Placebo battery is opt-in (`run_placebo=False` by default). `run_all_placebo_tests` on a typical panel (500 permutations times one DiD fit per permutation) adds tens of seconds of latency, which would be surprising as the default on a convenience wrapper. The schema reserves the `"placebo"` key; it is always rendered with `{"status": "skipped", "reason": "..."}` in MVP so agents parsing the schema see a stable shape. - **Note:** `DiagnosticReport` does not call `check_parallel_trends` on event-study or staggered result objects. `check_parallel_trends` in `diff_diff/utils.py` assumes a single binary treatment with universal pre-periods; for staggered and event-study designs, DR reads the pre-period event-study coefficients directly and constructs a joint Wald statistic (or Bonferroni fallback when `vcov` is missing). This mirrors the guidance in `practitioner._parallel_trends_step(staggered=True)`. - **Note:** Survey-design threading for fit-faithful Bacon replay. `DiagnosticReport(survey_design=...)` and `BusinessReport(survey_design=...)` accept the original `SurveyDesign` object and forward it to `bacon_decompose(survey_design=...)` so the Goodman-Bacon decomposition is computed under the same design as the weighted estimate. When `survey_metadata` is set but `survey_design` is not supplied, Bacon skips with an explicit reason rather than replaying an unweighted decomposition for a design that differs from the weighted estimate; users can alternatively pass `precomputed={'bacon': ...}` with a survey-aware result. The simple 2x2 parallel-trends helper (`utils.check_parallel_trends`) has no survey-aware variant. On a survey-backed `DiDResults` the check is skipped **unconditionally**, regardless of whether `survey_design` is supplied, because the helper cannot consume the design even when it is available. Users must pass `precomputed={'parallel_trends': ...}` with a survey-aware pretest result to opt in. Event-study PT on staggered estimators is unaffected — it reads the weighted pre-period coefficients directly off the fitted result and uses the finite-df reference described below, so no second replay is needed. - **Note:** Survey finite-df PT policy. When the fitted result carries a finite `survey_metadata.df_survey`, `_pt_event_study` computes `F = W / k` (numerator df = k pre-period coefficients) against an F(k, df_survey) reference distribution rather than chi-square(k). The design-based SE already reflects the effective sample size, so the chi-square reference would systematically over-reject under the finite-sample correction the SE captures. The schema surfaces the survey branch via the `method` suffix `_survey` (e.g., `joint_wald_survey`, `joint_wald_event_study_survey`) and exposes the denominator df as `df_denom`, so BR / DR prose can flag the finite-sample correction rather than silently presenting a chi-square-style result. Non-finite `df_survey` (NaN / inf / non-positive) falls back to the chi-square path. - **Note:** Estimator-native validation surfaces are surfaced rather than duplicated. `SyntheticDiDResults` routes parallel-trends to `pre_treatment_fit` (the RMSE of the synthetic-control fit on the pre-period), and routes sensitivity to `in_time_placebo()` + `sensitivity_to_zeta_omega()`. `TROPResults` surfaces factor-model diagnostics (`effective_rank`, `loocv_score`, selected `lambda_*`) under `estimator_native_diagnostics`. `SyntheticControlResults` routes parallel-trends to the `scm_fit` analogue (`pre_rmspe`, verdict `design_enforced_pt`) and surfaces `pre_rmspe`, donor-weight concentration, the in-space placebo permutation p-value, the ADH-2015 §4 leave-one-out (`leave_one_out`), in-time placebo (`in_time_placebo`), regression-weight extrapolation (`regression_weights`) and sparse-SC subset-search (`sparse_synthetic_control`) blocks, the Firpo-Possebom (2018) test-inversion `confidence_set`, and the Chernozhukov-Wüthrich-Zhu (2021) `conformal_inference` block (joint / pointwise / average) under `estimator_native_diagnostics` — each is populated only when the caller has already run the corresponding opt-in method (DR never triggers a refit loop implicitly; otherwise a `status="not_run"` stub), and it omits HonestDiD-style `sensitivity` (significance IS the placebo). `EfficientDiDResults` PT runs through `EfficientDiD.hausman_pretest` (the estimator's native PT-All vs PT-Post check). - **Note:** Pre-trends verdict is a three-bin heuristic, not a field convention. DR maps the joint p-value as follows: - `joint_p >= 0.30` → `no_detected_violation`. - `0.05 <= joint_p < 0.30` → `some_evidence_against`. - `joint_p < 0.05` → `clear_violation`. These thresholds are diff-diff heuristics. The 0.30 upper bound draws on equivalence-testing intuition (Rambachan & Roth 2023 discuss the limitations of pre-tests). The `no_detected_violation` label deliberately avoids "parallel trends hold" language — the test did not detect a violation, but pre-trends tests are commonly underpowered. See the power-aware phrasing rule below. - **Note:** Power-aware phrasing for `no_detected_violation`. DR calls `compute_pretrends_power(results, violation_type='linear', alpha=alpha, target_power=0.80)` for the estimator families that ship a `compute_pretrends_power` adapter: `MultiPeriodDiDResults`, `CallawaySantAnnaResults`, and `SunAbrahamResults` (see `_APPLICABILITY["pretrends_power"]` in `diff_diff/diagnostic_report.py`). Other staggered families with event-study output (`ImputationDiDResults`, `TwoStageDiDResults`, `SpilloverDiDResults`, `StackedDiDResults`, `EfficientDiDResults`, `StaggeredTripleDiffResults`, `WooldridgeDiDResults`, `ChaisemartinDHaultfoeuilleResults`) do not yet have a power adapter and therefore render the `no_detected_violation` tier as `underpowered` with the fallback reason recorded in `schema["pre_trends"]["power_reason"]` (plain-English explanation) while `schema["pre_trends"]["power_status"]` carries the machine-readable enum (`"ran"` / `"skipped"` / `"error"` / `"not_applicable"`). BusinessReport then reads `mdv_share_of_att = max_abs_pre_violation / abs(att)` and selects a tier. The numerator is the **level-scale max pre-period violation under the MDV**, computed as `mdv * max(|violation_weights|)` — NOT the raw `mdv` scalar. Post PR-B Step 4, raw `mdv` for `violation_type='linear'` is in Roth's γ units (a slope on relative time), so comparing it directly to a level-scale `|att|` would mix units on irregular pre-period grids and mis-tier the result. The level-scale quantity is exposed via the new `PreTrendsPowerResults.max_abs_pre_violation` property and the `DiagnosticReport.pretrends_power` block schema field of the same name. Tier thresholds: - `< 0.25` → `well_powered` — "the test has 80% power to detect a violation of magnitude M, which is only X% of the estimated effect; if a material pre-trend existed, this test would likely have caught it." - `>= 0.25 and < 1.0` → `moderately_powered` — "the test is informative but not definitive; see the sensitivity analysis below for bounded-violation guarantees." - `>= 1.0` → `underpowered` — "the test has limited power — a non-rejection does not prove the assumption. See the HonestDiD sensitivity analysis below for a more reliable signal." - Power analysis not runnable → fall back to `underpowered` phrasing; the fallback reason is recorded in `schema["pre_trends"]["power_reason"]` (plain-English explanation; `power_status` carries the enum). Rationale: always-hedging phrasing under-sells well-designed studies; always-confident phrasing over-sells underpowered ones. The library already ships `compute_pretrends_power()`, so using it is the honest default rather than hedging every non-violation. - **Note:** Pre-period covariance routing for staggered-estimator power. As of the PR-B PreTrendsPower implementation audit (Roth 2022), `compute_pretrends_power()` consumes the full `event_study_vcov` sub-block when it is available — non-bootstrap CS fits (`staggered_results.py` populates the matrix) and non-bootstrap SA fits (`sun_abraham.py` builds it via `W @ vcov_cohort @ W.T`). The `PreTrendsPowerResults.covariance_source` field records the actual extraction path (`"full_pre_period_vcov"` vs `"diag_fallback"`), and the `DiagnosticReport.pretrends_power` block surfaces that label unchanged. There are two paths through the report layer with different downgrade semantics: - **New fits** (post-PR-B, `PreTrendsPowerResults.covariance_source` is populated): `DiagnosticReport` reads the persisted label directly. Non-bootstrap CS / SA fits report `"full_pre_period_vcov"` and are NOT downgraded; bootstrap / replicate-weight paths report `"diag_fallback"` and also pass through unchanged (no "available but unused" concern — the estimator did its best with what was available). - **Legacy serialized results** (pre-PR-B, no `covariance_source` field on the object): the report layer falls back to type-based inference in `_infer_cov_source(source_fit)`. For event-study result types (CS / SA / etc.) with populated `event_study_vcov`, the legacy- ambiguous case still emits the conservative `"diag_fallback_available_full_vcov_unused"` sentinel and the `well_powered → moderately_powered` downgrade still applies — because without the persisted provenance we cannot rule out that the stored power was computed from `diag(ses^2)` under PR-A semantics. For `MultiPeriodDiDResults` without `interaction_indices`, the legacy fallback reports `"diag_fallback"` (a genuine fallback, not the "available but unused" case, so no downgrade applies). Remaining `"diag_fallback"` cases on new fits — bootstrap / replicate-weight CS and SA, plus ImputationDiD / Stacked / EfficientDiD / TwoStageDiD — pass through unchanged because nothing better is available on those result types yet. - **Note:** Unit-translation policy. BusinessReport does not arithmetically translate log-points to percents or level effects to log-points. The estimate is rendered in the scale the estimator produced; `outcome_unit="log_points"` emits an informational caveat. The policy avoids guessing the underlying model (no estimator in the library currently exports both log and level coefficients), which would be unsafe in the presence of non-linear link functions (Poisson QMLE, logit). - **Note:** Single-knob `alpha` with preserved-native-CI fallback. BusinessReport exposes only `alpha` (defaults to `results.alpha`); there is no separate `significance_threshold` parameter. When the requested `alpha` matches the fit's native level, it drives both the CI level (`(1 - alpha) * 100`% interval) and the phrasing tier threshold ("statistically significant at the (1 - alpha) * 100% level"). When the requested `alpha` differs from the fit's native level (e.g., the user asks for `alpha=0.10` on a result fit with `alpha=0.05`), BusinessReport does NOT recompute the CI at the requested level, because the stored CI is the only quantile the underlying estimator supplied (bootstrap distributions and finite-df analytical variances are not always retained on the result). Instead, the schema preserves the fit's native CI (with its original level) and uses the requested `alpha` only for the significance-phrasing threshold, and emits an `alpha_override_preserved` caveat describing the mismatch. This is the conservative choice: it avoids silently recomputing CIs under assumptions the estimator may not support. - **Note:** Schema stability policy for the AI-legible `to_dict()` surface. New top-level keys count as additive (no version bump); new values in any `status` enum count as breaking (agents doing exhaustive pattern match will break on unknown enums); renames and removals count as breaking. The `BUSINESS_REPORT_SCHEMA_VERSION` and `DIAGNOSTIC_REPORT_SCHEMA_VERSION` constants bump independently. The v3.2 CHANGELOG marks both schemas experimental so users do not anchor tooling on them prematurely; a formal deprecation policy will land within two subsequent PRs. - **Note:** Schema version 2.0 (both BR and DR). The BR/DR gap #6 target-parameter PR adds the `headline.status` / `headline_metric.status` value `"no_scalar_by_design"` (used for the dCDH `trends_linear=True, L_max>=2` configuration where `overall_att` is intentionally NaN). Per the stability policy above, new enum values are breaking changes, so `BUSINESS_REPORT_SCHEMA_VERSION` and `DIAGNOSTIC_REPORT_SCHEMA_VERSION` bumped from `"1.0"` to `"2.0"`. The schemas remain marked experimental, so the formal deprecation policy does not yet apply. ## Reference implementation(s) The phrasing rules follow the guidance in: - Baker, A. C., Callaway, B., Cunningham, S., Goodman-Bacon, A., & Sant'Anna, P. H. C. (2025). *Difference-in-Differences Designs: A Practitioner's Guide.* (The 8-step workflow enforced through `diff_diff/practitioner.py`.) - Rambachan, A., & Roth, J. (2023). *A More Credible Approach to Parallel Trends.* Review of Economic Studies. (HonestDiD sensitivity; the pre-test power caveat directly shaped the three-tier power phrasing.) - Roth, J. (2022). *Pretest with Caution: Event-study Estimates after Testing for Parallel Trends.* American Economic Review: Insights. (Motivates the power-aware phrasing tiers.)