Interactive notebook

This tutorial is a Jupyter notebook. You can view it on GitHub or download it to run locally.

Pre-Trends Power Analysis (Roth 2022)

A passing pre-trends test doesn’t mean parallel trends holds—it may just mean the test has low power to detect violations. Pre-Trends Power Analysis (Roth 2022) answers a critical question:

“What violations could my pre-trends test have detected?”

This notebook covers:

1. Motivation: Why pre-trends tests can be misleading

2. Basic usage with PreTrendsPower

3. Computing the Minimum Detectable Violation (MDV)

4. Power curves across violation magnitudes

5. Different violation types (linear, constant, last period, custom)

6. Integration with Honest DiD

7. Visualization and reporting

import numpy as np
import pandas as pd

from diff_diff import (
    MultiPeriodDiD,
    PreTrendsPower,
    compute_pretrends_power,
    compute_mdv,
    plot_pretrends_power,
)

# For plots
try:
    import matplotlib.pyplot as plt
    plt.style.use('seaborn-v0_8-whitegrid')
    HAS_MATPLOTLIB = True
except ImportError:
    HAS_MATPLOTLIB = False
    print("matplotlib not installed - visualization examples will be skipped")

1. Motivation: The Problem with Pre-Trends Tests

Standard practice in DiD analysis is to test for parallel trends by checking if pre-treatment coefficients are jointly zero. However, this approach has a fundamental problem:

A non-significant pre-trends test could mean:

1. Parallel trends actually holds ✓

2. There’s a violation, but the test lacks power to detect it ✗

Why does this matter?

If your pre-trends test has low power, it provides little reassurance about the validity of your DiD. Even a “passing” test (p > 0.05) might miss economically meaningful violations.

Roth (2022) introduces two key concepts:

1. Power of the pre-trends test: The probability of rejecting the null (parallel trends) when there is a violation of a given magnitude

2. Minimum Detectable Violation (MDV): The smallest violation your pre-trends test can detect with a target level of power (e.g., 80%)

2. Generate Example Data

We’ll create panel data suitable for event study analysis with multiple pre-treatment periods.

# Generate event study data using the library function
from diff_diff import generate_event_study_data

# Generate panel data for event study analysis:
# - 5 pre-treatment periods (0-4)
# - 5 post-treatment periods (5-9)
# - Half of units are treated starting at period 5
df = generate_event_study_data(
    n_units=300,
    n_pre=5,
    n_post=5,
    treatment_fraction=0.5,
    treatment_effect=5.0,
    unit_fe_sd=2.0,
    noise_sd=2.0,
    seed=42
)

print(f"Generated {len(df)} observations")
print(f"Units: {df['unit'].nunique()} ({df[df['treated']==1]['unit'].nunique()} treated)")
print(f"Periods: {df['period'].nunique()} (5 pre, 5 post)")
print(f"True ATT: 5.0")

3. Fit Event Study

First, we estimate a standard event study to get the pre-period coefficients and their variance-covariance matrix.

# Fit event study with ALL periods (pre and post) relative to reference period
# For pre-trends power analysis, we need coefficients for pre-periods too
mp_did = MultiPeriodDiD()

# Use period 4 as the reference period (last pre-period, excluded from estimation)
# Specify post_periods as the actual post-treatment periods; MultiPeriodDiD
# automatically estimates pre-period coefficients for the event study.
event_results = mp_did.fit(
    df,
    outcome='outcome',
    treatment='treated',
    time='period',
    post_periods=[5, 6, 7, 8, 9]
)

print(event_results.summary())

# Visualize the event study
if HAS_MATPLOTLIB:
    from diff_diff import plot_event_study
    fig, ax = plt.subplots(figsize=(10, 6))
    plot_event_study(
        event_results,
        ax=ax,
        title='Event Study: Pre-Trends Look Good',
        show=False
    )
    plt.tight_layout()
    plt.show()

The pre-period coefficients (periods 0-3, with period 4 as reference) appear close to zero. But how confident should we be that parallel trends holds? Let’s assess the power of this pre-trends test.

4. Basic Pre-Trends Power Analysis

The PreTrendsPower class computes the power of the pre-trends test to detect violations of different magnitudes.

# Create a PreTrendsPower object
pt = PreTrendsPower(
    alpha=0.05,      # Significance level for pre-trends test
    power=0.80,      # Target power for MDV calculation
    violation_type='linear'  # Type of violation to consider
)

# Define the actual pre-treatment periods (those before treatment starts at period 5)
# These are the periods we want to analyze for pre-trends power
pre_treatment_periods = [0, 1, 2, 3]

# Fit to the event study results, specifying which periods are pre-treatment
pt_results = pt.fit(event_results, pre_periods=pre_treatment_periods)

print(pt_results.summary())

Interpreting the Results

Key metrics:

1. MDV (Minimum Detectable Violation): The smallest violation magnitude your pre-trends test can detect with 80% power

   • Smaller MDV = more informative test

   • If MDV is large, even big violations could go undetected

2. Power at specific violations: How likely is the test to reject when there’s a violation?

   • Low power = uninformative “passing” test

   • High power = reassuring “passing” test

3. Test informativeness: Is the MDV small enough to be useful?

# Access key results
print(f"Minimum Detectable Violation (MDV): {pt_results.mdv:.4f}")
print(f"Target power: {pt_results.target_power:.0%}")
print(f"Test informativeness: {'Informative' if pt_results.is_informative else 'Uninformative'}")
print("")
print("Interpretation:")
print(f"  With 80% power, your pre-trends test can detect violations")
print(f"  of magnitude {pt_results.mdv:.3f} or larger.")
print(f"")
print(f"  Violations smaller than {pt_results.mdv:.3f} would likely go undetected.")

5. Power at Specific Violation Magnitudes

You can compute the power to detect a specific violation magnitude:

# Compute power for specific violation magnitudes
violations_to_check = [0.5, 1.0, 2.0, 3.0, 5.0]

print(f"{'Violation':>12} {'Power':>10} {'Detectable?':>15}")
print("-" * 40)

for v in violations_to_check:
    power = pt_results.power_at(v)
    detectable = "Yes" if power >= 0.80 else "No"
    print(f"{v:>12.1f} {power:>10.1%} {detectable:>15}")

6. Power Curves

A power curve shows how the power to detect violations changes with violation magnitude. This is the most useful visualization for understanding your test’s informativeness.

# Generate power curve
curve = pt.power_curve(
    event_results,
    n_points=50,
    pre_periods=pre_treatment_periods
)

# Preview the data
print("Power curve data (first 10 points):")
print(curve.to_dataframe().head(10))

# Plot the power curve
if HAS_MATPLOTLIB:
    fig, ax = plt.subplots(figsize=(10, 6))
    plot_pretrends_power(
        curve,
        ax=ax,
        mdv=pt_results.mdv,  # Show MDV line on plot
        target_power=0.80,
        title='Pre-Trends Test Power Curve',
        show=False
    )
    plt.tight_layout()
    plt.show()

Reading the Power Curve

• X-axis: Violation magnitude (larger = worse violation of parallel trends)

• Y-axis: Power (probability of rejecting when violation exists)

• Horizontal line at 0.80: Conventional target power

• Vertical line at MDV: Minimum detectable violation

Key insight: The curve shows the range of violations your test could miss. If your ATT estimate could be biased by a violation smaller than the MDV, your results may be unreliable.

7. Different Violation Types

Pre-trends violations can take different forms. The violation_type parameter specifies the pattern:

1. Linear (default): Violation grows linearly over time

   • E.g., treated group diverges steadily from control

2. Constant: Same violation in all pre-periods

   • E.g., level shift between groups

3. Last period: Violation only in the period just before treatment

   • E.g., anticipation effects

4. Custom: User-specified violation pattern

# Compare violation types
violation_types = ['linear', 'constant', 'last_period']

print(f"{'Violation Type':>15} {'MDV':>10} {'Power at 2.0':>15}")
print("-" * 45)

for vtype in violation_types:
    pt_v = PreTrendsPower(violation_type=vtype)
    results_v = pt_v.fit(event_results, pre_periods=pre_treatment_periods)
    power_at_2 = results_v.power_at(2.0)
    print(f"{vtype:>15} {results_v.mdv:>10.3f} {power_at_2:>15.1%}")

# Custom violation weights
# Example: Violation concentrated in periods 2 and 3 (approaching treatment)
# We have pre-periods 0, 1, 2, 3 estimated (reference period 4 is excluded)
n_pre = 4  # Periods 0, 1, 2, 3
custom_weights = np.zeros(n_pre)
custom_weights[-2:] = 1.0  # Weight on last two pre-periods (periods 2 and 3)

pt_custom = PreTrendsPower(
    violation_type='custom',
    violation_weights=custom_weights
)
results_custom = pt_custom.fit(event_results, pre_periods=pre_treatment_periods)

print(f"Custom violation (last 2 pre-periods): MDV = {results_custom.mdv:.3f}")

Visualizing Different Violation Types

if HAS_MATPLOTLIB:
    fig, axes = plt.subplots(1, 3, figsize=(15, 5))

    for ax, vtype in zip(axes, ['linear', 'constant', 'last_period']):
        pt_v = PreTrendsPower(violation_type=vtype)
        results_v = pt_v.fit(event_results, pre_periods=pre_treatment_periods)
        curve_v = pt_v.power_curve(event_results, n_points=50, pre_periods=pre_treatment_periods)

        plot_pretrends_power(
            curve_v,
            ax=ax,
            mdv=results_v.mdv,  # Show MDV line on plot
            target_power=0.80,
            title=f'Violation Type: {vtype.replace("_", " ").title()}',
            show=False
        )

    plt.tight_layout()
    plt.show()

8. Integration with Honest DiD

Pre-trends power analysis connects naturally with Honest DiD (Rambachan & Roth 2023). The workflow:

1. Compute MDV from pre-trends power analysis

2. Use MDV to calibrate the violation bound (M) in Honest DiD

3. Compute robust confidence intervals under this calibrated bound

This answers: “If violations could be as large as what my pre-trends test could have missed, would my conclusions still hold?”

from diff_diff import HonestDiD

# First, compute MDV
pt = PreTrendsPower(violation_type='linear')
pt_results = pt.fit(event_results, pre_periods=pre_treatment_periods)

print(f"MDV from pre-trends power analysis: {pt_results.mdv:.3f}")
print("")

# Use MDV to calibrate Honest DiD
# The MDV tells us what violations we couldn't have detected
# So we should check robustness to violations up to the MDV
honest = HonestDiD(method='smoothness', M=pt_results.mdv)
honest_results = honest.fit(event_results)

print("Honest DiD results (M = MDV):")
print(honest_results.summary())

# Use the built-in sensitivity integration
sensitivity_results = pt.sensitivity_to_honest_did(
    event_results,
    pre_periods=pre_treatment_periods
)

print("Joint sensitivity analysis:")
print(f"  MDV: {sensitivity_results['mdv']:.3f}")
print(f"  Max pre-period SE: {sensitivity_results['max_pre_se']:.3f}")
print(f"  MDV / max(SE): {sensitivity_results['mdv_in_ses']:.2f}")
print("")
print("Interpretation:")
print(sensitivity_results['interpretation'])

9. Convenience Functions

For quick calculations, use the convenience functions:

# Quick MDV calculation
mdv = compute_mdv(event_results, target_power=0.80, violation_type='linear', pre_periods=pre_treatment_periods)
print(f"MDV: {mdv:.3f}")

# Quick power calculation at a specific violation
power_result = compute_pretrends_power(event_results, M=2.0, pre_periods=pre_treatment_periods)
print(f"Power at violation=2.0: {power_result.power:.1%}")

10. Working with Real Event Studies

In practice, you’ll apply pre-trends power analysis to your actual event study estimates. Here’s the typical workflow:

# Typical workflow for pre-trends power analysis

# Step 1: Estimate event study with proper pre/post period classification
mp_did = MultiPeriodDiD()

# Specify actual post-treatment periods; pre-period coefficients are
# estimated automatically by MultiPeriodDiD for the event study
pre_treatment_periods = [0, 1, 2, 3]  # Define which are pre-treatment

results = mp_did.fit(
    df,
    outcome='outcome',
    treatment='treated',
    time='period',
    post_periods=[5, 6, 7, 8, 9]
)

# Step 2: Assess power of the pre-trends test
print("Step 2: Pre-Trends Power Analysis")
pt = PreTrendsPower(alpha=0.05, power=0.80, violation_type='linear')
pt_results = pt.fit(results, pre_periods=pre_treatment_periods)
print(f"MDV (80% power): {pt_results.mdv:.3f}")
print("")

# Step 3: Interpret
print("Step 3: Interpretation")
print(f"Your pre-trends test could only detect violations >= {pt_results.mdv:.3f}")
print(f"Violations smaller than this would likely go undetected.")
print("")

# Step 4: Connect to Honest DiD for robust inference
print("Step 4: Robust Inference with Honest DiD")
honest = HonestDiD(method='smoothness', M=pt_results.mdv)
honest_results = honest.fit(results)
print(f"Robust 95% CI (M=MDV): [{honest_results.ci_lb:.3f}, {honest_results.ci_ub:.3f}]")
print(f"Conclusion: {'Effect is robust' if honest_results.is_significant else 'Effect may not be robust'}")

11. Exporting Results

Results can be exported to DataFrames for further analysis or reporting:

# Export single result
print("Single result as DataFrame:")
print(pt_results.to_dataframe())
print("")

# Export power curve
print("Power curve as DataFrame (first 10 rows):")
curve = pt.power_curve(event_results, pre_periods=pre_treatment_periods)
print(curve.to_dataframe().head(10))

# Export to dict for JSON serialization
result_dict = pt_results.to_dict()
print("Result as dictionary:")
for key, value in result_dict.items():
    print(f"  {key}: {value}")

Summary

Key Takeaways:

1. Pre-trends tests can be misleading: A “passing” test (p > 0.05) doesn’t mean parallel trends holds—it may mean the test has low power.

2. MDV quantifies test informativeness: The Minimum Detectable Violation tells you the smallest violation your test could detect with 80% power.

3. Power curves visualize sensitivity: See how detection power changes with violation magnitude.

4. Different violation types matter: Linear, constant, and last-period violations have different detectability.

5. Integration with Honest DiD: Use MDV to calibrate sensitivity analysis bounds.

Best Practices:

• Always report pre-trends power analysis alongside standard pre-trends tests

• Include power curves in supplementary materials

• Use MDV to calibrate Honest DiD sensitivity analysis

• Consider multiple violation types

• Discuss what violation magnitudes would be economically meaningful in your setting

Reference:

Roth, J. (2022). Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends. American Economic Review: Insights, 4(3), 305-322. https://doi.org/10.1257/aeri.20210236

Related Tutorials

• 04_parallel_trends.ipynb - Testing and visualizing parallel trends

• 05_honest_did.ipynb - Sensitivity analysis for parallel trends violations

• 06_power_analysis.ipynb - Power analysis for study design (sample size, MDE)