Power Analysis

Power analysis for DiD study design.

Overview

Power analysis helps researchers design studies with adequate statistical power to detect meaningful treatment effects. This module provides:

Analytical Power Calculations: Fast closed-form power for standard DiD designs
Minimum Detectable Effect (MDE): Smallest effect detectable at target power
Sample Size Calculations: Required sample size for target power
Simulation-Based Power: Monte Carlo power for any DiD estimator

PowerAnalysis

Main class for analytical power calculations.

class diff_diff.PowerAnalysis[source]

Bases: object

Power analysis for difference-in-differences designs.

Provides analytical power calculations for basic 2x2 DiD and panel DiD designs. For complex designs like staggered adoption, use simulate_power() instead.

Parameters:

alpha (float, default=0.05) – Significance level for hypothesis testing.
power (float, default=0.80) – Target statistical power.
alternative (str, default='two-sided') – Alternative hypothesis: ‘two-sided’, ‘greater’, or ‘less’.

Examples

Calculate minimum detectable effect:

>>> from diff_diff import PowerAnalysis
>>> pa = PowerAnalysis(alpha=0.05, power=0.80)
>>> results = pa.mde(n_treated=50, n_control=50, sigma=1.0)
>>> print(f"MDE: {results.mde:.3f}")

Calculate required sample size:

>>> results = pa.sample_size(effect_size=0.5, sigma=1.0)
>>> print(f"Required N: {results.required_n}")

Calculate power for given sample and effect:

>>> results = pa.power(effect_size=0.5, n_treated=50, n_control=50, sigma=1.0)
>>> print(f"Power: {results.power:.1%}")

Notes

The power calculations are based on the variance of the DiD estimator:

For basic 2x2 DiD:

Var(ATT) = sigma^2 * (1/n_treated_post + 1/n_treated_pre

1/n_control_post + 1/n_control_pre)

For panel DiD with T periods:

Var(ATT) = sigma^2 * (1/(N_treated * T) + 1/(N_control * T))

(1 + (T-1)*rho) / (1 + (T-1)*rho)

Where rho is the intra-cluster correlation coefficient.

References

Bloom, H. S. (1995). “Minimum Detectable Effects.” Burlig, F., Preonas, L., & Woerman, M. (2020). “Panel Data and Experimental Design.”

Methods

__init__(alpha=0.05, power=0.8, alternative='two-sided')[source]

Parameters:

alpha (float)
power (float)
alternative (str)

power(effect_size, n_treated, n_control, sigma, n_pre=1, n_post=1, rho=0.0)[source]

Calculate statistical power for given effect size and sample.

Parameters:

effect_size (float) – Expected treatment effect size.
n_treated (int) – Number of treated units.
n_control (int) – Number of control units.
sigma (float) – Residual standard deviation.
n_pre (int, default=1) – Number of pre-treatment periods.
n_post (int, default=1) – Number of post-treatment periods.
rho (float, default=0.0) – Intra-cluster correlation for panel data.

Returns:

Power analysis results.

Return type:

PowerResults

Examples

>>> pa = PowerAnalysis()
>>> results = pa.power(effect_size=2.0, n_treated=50, n_control=50, sigma=5.0)
>>> print(f"Power: {results.power:.1%}")

mde(n_treated, n_control, sigma, n_pre=1, n_post=1, rho=0.0)[source]

Calculate minimum detectable effect given sample size.

The MDE is the smallest effect size that can be detected with the specified power and significance level.

Parameters:

n_treated (int) – Number of treated units.
n_control (int) – Number of control units.
sigma (float) – Residual standard deviation.
n_pre (int, default=1) – Number of pre-treatment periods.
n_post (int, default=1) – Number of post-treatment periods.
rho (float, default=0.0) – Intra-cluster correlation for panel data.

Returns:

Power analysis results including MDE.

Return type:

PowerResults

Examples

>>> pa = PowerAnalysis(power=0.80)
>>> results = pa.mde(n_treated=100, n_control=100, sigma=10.0)
>>> print(f"MDE: {results.mde:.2f}")

sample_size(effect_size, sigma, n_pre=1, n_post=1, rho=0.0, treat_frac=0.5)[source]

Calculate required sample size to detect given effect.

Parameters:

effect_size (float) – Treatment effect to detect.
sigma (float) – Residual standard deviation.
n_pre (int, default=1) – Number of pre-treatment periods.
n_post (int, default=1) – Number of post-treatment periods.
rho (float, default=0.0) – Intra-cluster correlation for panel data.
treat_frac (float, default=0.5) – Fraction of units assigned to treatment.

Returns:

Power analysis results including required sample size.

Return type:

PowerResults

Examples

>>> pa = PowerAnalysis(power=0.80)
>>> results = pa.sample_size(effect_size=5.0, sigma=10.0)
>>> print(f"Required N: {results.required_n}")

power_curve(n_treated, n_control, sigma, effect_sizes=None, n_pre=1, n_post=1, rho=0.0)[source]

Compute power for a range of effect sizes.

Parameters:

n_treated (int) – Number of treated units.
n_control (int) – Number of control units.
sigma (float) – Residual standard deviation.
effect_sizes (list of float, optional) – Effect sizes to evaluate. If None, uses a range from 0 to 3*MDE.
n_pre (int, default=1) – Number of pre-treatment periods.
n_post (int, default=1) – Number of post-treatment periods.
rho (float, default=0.0) – Intra-cluster correlation.

Returns:

DataFrame with columns ‘effect_size’ and ‘power’.

Return type:

pd.DataFrame

Examples

>>> pa = PowerAnalysis()
>>> curve = pa.power_curve(n_treated=50, n_control=50, sigma=5.0)
>>> print(curve)

sample_size_curve(effect_size, sigma, sample_sizes=None, n_pre=1, n_post=1, rho=0.0, treat_frac=0.5)[source]

Compute power for a range of sample sizes.

Parameters:

effect_size (float) – Treatment effect size.
sigma (float) – Residual standard deviation.
sample_sizes (list of int, optional) – Total sample sizes to evaluate. If None, uses sensible range.
n_pre (int, default=1) – Number of pre-treatment periods.
n_post (int, default=1) – Number of post-treatment periods.
rho (float, default=0.0) – Intra-cluster correlation.
treat_frac (float, default=0.5) – Fraction assigned to treatment.

Returns:

DataFrame with columns ‘sample_size’ and ‘power’.

Return type:

pd.DataFrame

Example

from diff_diff import PowerAnalysis

# Create power analysis object
pa = PowerAnalysis(
    effect_size=0.5,
    n_treated=100,
    n_control=100,
    n_pre=4,
    n_post=4,
    sigma=1.0,
    rho=0.5,  # Within-unit correlation
    alpha=0.05
)

# Compute power
power = pa.compute_power()
print(f"Power: {power:.2%}")

# Compute MDE at 80% power
mde = pa.compute_mde(power=0.80)
print(f"MDE: {mde:.3f}")

# Required sample size
n = pa.compute_sample_size(power=0.80)
print(f"Required N per group: {n}")

PowerResults

Results from power analysis.

class diff_diff.PowerResults[source]

Bases: object

Results from analytical power analysis.

power

Statistical power (probability of rejecting H0 when effect exists).

Type:: float

mde

Minimum detectable effect size.

Type:: float

required_n

Required total sample size (treated + control).

Type:: int

effect_size

Effect size used in calculation.

Type:: float

alpha

Significance level.

Type:: float

alternative

Alternative hypothesis (‘two-sided’, ‘greater’, ‘less’).

Type:: str

n_treated

Number of treated units.

Type:: int

n_control

Number of control units.

Type:: int

n_pre

Number of pre-treatment periods.

Type:: int

n_post

Number of post-treatment periods.

Type:: int

sigma

Residual standard deviation.

Type:: float

rho

Intra-cluster correlation (for panel data).

Type:: float

design

Study design type (‘basic_did’, ‘panel’, ‘staggered’).

Type:: str

power: float

mde: float

required_n: int

effect_size: float

alpha: float

alternative: str

n_treated: int

n_control: int

n_pre: int

n_post: int

sigma: float

rho: float = 0.0

design: str = 'basic_did'

__repr__()[source]

Concise string representation.

Return type:: str

summary()[source]

Generate a formatted summary of power analysis results.

Returns:: Formatted summary table.
Return type:: str

print_summary()[source]

Print the summary to stdout.

Return type:: None

to_dict()[source]

Convert results to a dictionary.

Returns:: Dictionary containing all power analysis results.
Return type:: Dict[str, Any]

to_dataframe()[source]

Convert results to a pandas DataFrame.

Returns:: DataFrame with power analysis results.
Return type:: pd.DataFrame

__init__(power, mde, required_n, effect_size, alpha, alternative, n_treated, n_control, n_pre, n_post, sigma, rho=0.0, design='basic_did')

Parameters:

power (float)
mde (float)
required_n (int)
effect_size (float)
alpha (float)
alternative (str)
n_treated (int)
n_control (int)
n_pre (int)
n_post (int)
sigma (float)
rho (float)
design (str)

Return type:

None

SimulationPowerResults

Results from simulation-based power analysis.

class diff_diff.SimulationPowerResults[source]

Bases: object

Results from simulation-based power analysis.

power

Estimated power (proportion of simulations rejecting H0).

Type:: float

power_se

Standard error of power estimate.

Type:: float

power_ci

Confidence interval for power estimate.

Type:: Tuple[float, float]

rejection_rate

Proportion of simulations with p-value < alpha.

Type:: float

mean_estimate

Mean treatment effect estimate across simulations.

Type:: float

std_estimate

Standard deviation of estimates across simulations.

Type:: float

mean_se

Mean standard error across simulations.

Type:: float

coverage

Proportion of CIs containing true effect.

Type:: float

n_simulations

Number of simulations performed.

Type:: int

effect_sizes

Effect sizes tested (if multiple).

Type:: List[float]

powers

Power at each effect size (if multiple).

Type:: List[float]

true_effect

True treatment effect used in simulation.

Type:: float

alpha

Significance level.

Type:: float

estimator_name

Name of the estimator used.

Type:: str

power: float

power_se: float

power_ci: Tuple[float, float]

rejection_rate: float

mean_estimate: float

std_estimate: float

mean_se: float

coverage: float

n_simulations: int

effect_sizes: List[float]

powers: List[float]

true_effect: float

alpha: float

estimator_name: str

bias: float

rmse: float

simulation_results: List[Dict[str, Any]] | None = None

__post_init__()[source]: Compute derived statistics.

__repr__()[source]

Concise string representation.

Return type:: str

summary()[source]

Generate a formatted summary of simulation power results.

Returns:: Formatted summary table.
Return type:: str

print_summary()[source]

Print the summary to stdout.

Return type:: None

to_dict()[source]

Convert results to a dictionary.

Returns:: Dictionary containing simulation power results.
Return type:: Dict[str, Any]

to_dataframe()[source]

Convert results to a pandas DataFrame.

Returns:: DataFrame with simulation power results.
Return type:: pd.DataFrame

power_curve_df()[source]

Get power curve data as a DataFrame.

Returns:: DataFrame with effect_size and power columns.
Return type:: pd.DataFrame

__init__(power, power_se, power_ci, rejection_rate, mean_estimate, std_estimate, mean_se, coverage, n_simulations, effect_sizes, powers, true_effect, alpha, estimator_name, simulation_results=None)

Parameters:

power (float)
power_se (float)
power_ci (Tuple[float, float])
rejection_rate (float)
mean_estimate (float)
std_estimate (float)
mean_se (float)
coverage (float)
n_simulations (int)
effect_sizes (List[float])
powers (List[float])
true_effect (float)
alpha (float)
estimator_name (str)
simulation_results (List[Dict[str, Any]] | None)

Return type:

None

Convenience Functions

compute_power

Quick power computation.

diff_diff.compute_power(effect_size, n_treated, n_control, sigma, alpha=0.05, n_pre=1, n_post=1, rho=0.0)[source]

Convenience function to compute power for given effect and sample.

Parameters:

effect_size (float) – Expected treatment effect.
n_treated (int) – Number of treated units.
n_control (int) – Number of control units.
sigma (float) – Residual standard deviation.
alpha (float, default=0.05) – Significance level.
n_pre (int, default=1) – Number of pre-treatment periods.
n_post (int, default=1) – Number of post-treatment periods.
rho (float, default=0.0) – Intra-cluster correlation.

Returns:

Statistical power.

Return type:

float

Examples

>>> power = compute_power(effect_size=5.0, n_treated=50, n_control=50, sigma=10.0)
>>> print(f"Power: {power:.1%}")

compute_mde

Compute minimum detectable effect.

diff_diff.compute_mde(n_treated, n_control, sigma, power=0.8, alpha=0.05, n_pre=1, n_post=1, rho=0.0)[source]

Convenience function to compute minimum detectable effect.

Parameters:

n_treated (int) – Number of treated units.
n_control (int) – Number of control units.
sigma (float) – Residual standard deviation.
power (float, default=0.80) – Target statistical power.
alpha (float, default=0.05) – Significance level.
n_pre (int, default=1) – Number of pre-treatment periods.
n_post (int, default=1) – Number of post-treatment periods.
rho (float, default=0.0) – Intra-cluster correlation.

Returns:

Minimum detectable effect size.

Return type:

float

Examples

>>> mde = compute_mde(n_treated=50, n_control=50, sigma=10.0)
>>> print(f"MDE: {mde:.2f}")

compute_sample_size

Compute required sample size.

diff_diff.compute_sample_size(effect_size, sigma, power=0.8, alpha=0.05, n_pre=1, n_post=1, rho=0.0, treat_frac=0.5)[source]

Convenience function to compute required sample size.

Parameters:

effect_size (float) – Treatment effect to detect.
sigma (float) – Residual standard deviation.
power (float, default=0.80) – Target statistical power.
alpha (float, default=0.05) – Significance level.
n_pre (int, default=1) – Number of pre-treatment periods.
n_post (int, default=1) – Number of post-treatment periods.
rho (float, default=0.0) – Intra-cluster correlation.
treat_frac (float, default=0.5) – Fraction assigned to treatment.

Returns:

Required total sample size.

Return type:

int

Examples

>>> n = compute_sample_size(effect_size=5.0, sigma=10.0)
>>> print(f"Required N: {n}")

simulate_power

Simulation-based power for any DiD estimator.

diff_diff.simulate_power(estimator, n_units=100, n_periods=4, treatment_effect=5.0, treatment_fraction=0.5, treatment_period=2, sigma=1.0, n_simulations=500, alpha=0.05, effect_sizes=None, seed=None, data_generator=None, data_generator_kwargs=None, estimator_kwargs=None, progress=True)[source]

Estimate power using Monte Carlo simulation.

This function simulates datasets with known treatment effects and estimates power as the fraction of simulations where the null hypothesis is rejected. This is the recommended approach for complex designs like staggered adoption.

Parameters:

estimator (estimator object) – DiD estimator to use (e.g., DifferenceInDifferences, CallawaySantAnna).
n_units (int, default=100) – Number of units per simulation.
n_periods (int, default=4) – Number of time periods.
treatment_effect (float, default=5.0) – True treatment effect to simulate.
treatment_fraction (float, default=0.5) – Fraction of units that are treated.
treatment_period (int, default=2) – First post-treatment period (0-indexed).
sigma (float, default=1.0) – Residual standard deviation (noise level).
n_simulations (int, default=500) – Number of Monte Carlo simulations.
alpha (float, default=0.05) – Significance level for hypothesis tests.
effect_sizes (list of float, optional) – Multiple effect sizes to evaluate for power curve. If None, uses only treatment_effect.
seed (int, optional) – Random seed for reproducibility.
data_generator (callable, optional) – Custom data generation function. Should accept same signature as generate_did_data(). If None, uses generate_did_data().
data_generator_kwargs (dict, optional) – Additional keyword arguments for data generator.
estimator_kwargs (dict, optional) – Additional keyword arguments for estimator.fit().
progress (bool, default=True) – Whether to print progress updates.

Returns:

Simulation-based power analysis results.

Return type:

SimulationPowerResults

Examples

Basic power simulation:

>>> from diff_diff import DifferenceInDifferences, simulate_power
>>> did = DifferenceInDifferences()
>>> results = simulate_power(
...     estimator=did,
...     n_units=100,
...     treatment_effect=5.0,
...     sigma=5.0,
...     n_simulations=500,
...     seed=42
... )
>>> print(f"Power: {results.power:.1%}")

Power curve over multiple effect sizes:

>>> results = simulate_power(
...     estimator=did,
...     effect_sizes=[1.0, 2.0, 3.0, 5.0, 7.0],
...     n_simulations=200,
...     seed=42
... )
>>> print(results.power_curve_df())

With Callaway-Sant’Anna for staggered designs:

>>> from diff_diff import CallawaySantAnna
>>> cs = CallawaySantAnna()
>>> # Custom data generator for staggered adoption
>>> def staggered_data(n_units, n_periods, treatment_effect, **kwargs):
...     # Your staggered data generation logic
...     ...
>>> results = simulate_power(cs, data_generator=staggered_data, ...)

Notes

The simulation approach: 1. Generate data with known treatment effect 2. Fit the estimator and record the p-value 3. Repeat n_simulations times 4. Power = fraction of simulations where p-value < alpha

For staggered designs, you’ll need to provide a custom data_generator that creates appropriate staggered treatment timing.

References

Burlig, F., Preonas, L., & Woerman, M. (2020). “Panel Data and Experimental Design.”

Complete Example

from diff_diff import (
    PowerAnalysis,
    compute_mde,
    simulate_power,
    DifferenceInDifferences,
    plot_power_curve,
)

# Quick MDE calculation
mde = compute_mde(
    n_treated=50,
    n_control=50,
    n_pre=4,
    n_post=4,
    sigma=1.0,
    rho=0.5,
    power=0.80,
    alpha=0.05
)
print(f"MDE: {mde:.3f}")

# Simulation-based power for DiD estimator
sim_results = simulate_power(
    estimator=DifferenceInDifferences(),
    effect_size=0.5,
    n_treated=100,
    n_control=100,
    n_periods=8,
    treatment_start=4,
    sigma=1.0,
    n_simulations=1000
)
print(f"Simulated power: {sim_results.power:.2%}")

# Power curve
pa = PowerAnalysis(n_treated=100, n_control=100, n_pre=4, n_post=4, sigma=1.0)
fig = plot_power_curve(pa, effect_range=(0, 1), n_points=50)
fig.savefig('power_curve.png')

Power Analysis

Overview

PowerAnalysis

Example

PowerResults

SimulationPowerResults

Convenience Functions

compute_power

compute_mde

compute_sample_size

simulate_power

Complete Example

See Also