{ "cells": [ { "cell_type": "markdown", "id": "t20-cell-001", "metadata": {}, "source": [ "# Tutorial 20: HAD for a National Brand Campaign with Regional Spend Intensity\n", "\n", "A practitioner walkthrough for measuring per-dollar lift when every market is treated at a different dose level - treatment varies in intensity, not in status, and no never-treated unit exists. Comparison comes from the dose variation across markets, not from an untreated holdout. The tutorial uses the `HeterogeneousAdoptionDiD` estimator (alias `HAD`), built for this case." ] }, { "cell_type": "markdown", "id": "t20-cell-002", "metadata": {}, "source": [ "## 1. The Measurement Problem\n", "\n", "Your team launched a Q4 brand campaign with a national TV blast that hit every DMA on the same date. On top of that, regional teams allocated incremental ad budget on a per-DMA basis. The lightest-touch markets put in about $5K; the heaviest committed up to $50K. Every DMA got something - the regional teams all participated - but intensity varied widely. Leadership wants the per-dollar lift on weekly site visits attributable to the regional add-on spend." ] }, { "cell_type": "markdown", "id": "t20-cell-003", "metadata": {}, "source": [ "**Why standard DiD doesn't fit here.** Three things make this hard for the usual estimators.\n", "\n", "First, every DMA got the national TV blast simultaneously and every regional team allocated some add-on budget. There is no never-treated unit to serve as the untreated baseline; the comparison structure has to come from the dose variation across markets instead.\n", "\n", "Second, regional add-on spend varies continuously across DMAs - from $5K to $50K. Collapsing to a binary high-spend / low-spend indicator throws away the dose information that's the whole point of the analysis.\n", "\n", "Third, continuous-dose DiD methods that DO exist generally require a never-treated unit in the panel. That requirement fails here.\n", "\n", "diff-diff's `HeterogeneousAdoptionDiD` (alias `HAD`) is built for this case. It identifies the per-dollar marginal effect from the local-linear behavior of outcome changes near the *boundary* of the dose distribution (the lightest-touch DMA's spend), and reports it as a single Weighted Average Slope (WAS) on the dose scale." ] }, { "cell_type": "code", "execution_count": null, "id": "t20-cell-004", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "\n", "from diff_diff import HAD, generate_continuous_did_data\n", "\n", "try:\n", " import matplotlib.pyplot as plt\n", " HAS_MATPLOTLIB = True\n", "except ImportError:\n", " HAS_MATPLOTLIB = False\n", " print('matplotlib not installed - plots will be skipped')" ] }, { "cell_type": "markdown", "id": "t20-cell-005", "metadata": {}, "source": [ "## 2. The Data\n", "\n", "60 DMAs over 8 weeks. Weeks 1-4 are the pre-campaign baseline; the campaign launches at week 5 with the national TV blast hitting every DMA. Regional teams' add-on spend varies from $5K to $50K per DMA - every DMA participates, none sits at $0. The true per-$1K lift on weekly site visits is locked at the seed; the tutorial recovers it." ] }, { "cell_type": "code", "execution_count": null, "id": "t20-cell-006", "metadata": {}, "outputs": [], "source": [ "MAIN_SEED = 87\n", "TRUE_SLOPE = 100.0 # weekly visits per $1K of regional spend (locked)\n", "BASELINE_VISITS = 5000.0\n", "\n", "raw = generate_continuous_did_data(\n", " n_units=60,\n", " n_periods=8,\n", " cohort_periods=[5],\n", " never_treated_frac=0.0,\n", " dose_distribution='uniform',\n", " dose_params={'low': 5.0, 'high': 50.0}, # $K - every DMA spent at least $5K\n", " att_function='linear',\n", " att_intercept=0.0,\n", " att_slope=TRUE_SLOPE,\n", " unit_fe_sd=8.0,\n", " time_trend=0.5,\n", " noise_sd=2.0,\n", " seed=MAIN_SEED,\n", ")\n", "\n", "panel = raw.copy()\n", "# HAD requires D=0 in the pre-launch period for every unit; the\n", "# generator gives constant dose across periods, so we zero out\n", "# pre-launch rows. Post-launch dose values from the generator are\n", "# kept as-is and the outcomes were generated from them, so the\n", "# DGP is internally consistent (no post-hoc relabeling).\n", "panel.loc[panel['period'] < panel['first_treat'], 'dose'] = 0.0\n", "\n", "panel = panel.rename(\n", " columns={\n", " 'unit': 'dma_id',\n", " 'period': 'week',\n", " 'outcome': 'weekly_visits',\n", " 'dose': 'regional_spend_k',\n", " }\n", ")\n", "panel['weekly_visits'] = panel['weekly_visits'] + BASELINE_VISITS\n", "\n", "panel.head()" ] }, { "cell_type": "code", "execution_count": null, "id": "t20-cell-007", "metadata": {}, "outputs": [], "source": [ "post_doses = (\n", " panel.loc[panel['week'] >= 5]\n", " .groupby('dma_id')['regional_spend_k']\n", " .first()\n", ")\n", "\n", "print(f'Panel shape: {panel.shape}')\n", "print(f'DMAs: {panel[\"dma_id\"].nunique()}, weeks: {panel[\"week\"].nunique()}')\n", "print(f'\\nPost-launch regional spend per DMA ($K):')\n", "print(f' min: {post_doses.min():.2f} (lightest-touch DMA)')\n", "print(f' median: {post_doses.median():.2f}')\n", "print(f' max: {post_doses.max():.2f}')\n", "\n", "if HAS_MATPLOTLIB:\n", " fig, ax = plt.subplots(figsize=(8, 4))\n", " ax.hist(post_doses, bins=20, edgecolor='white', color='steelblue')\n", " ax.set_xlabel('Regional add-on spend per DMA ($K)')\n", " ax.set_ylabel('Number of DMAs')\n", " ax.set_title('Regional spend distribution across 60 DMAs (post-launch)')\n", " plt.tight_layout()\n", " plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "t20-cell-008", "metadata": {}, "outputs": [], "source": [ "if HAS_MATPLOTLIB:\n", " tertile = pd.qcut(post_doses, q=3, labels=['low', 'mid', 'high'])\n", " panel_with_tertile = panel.merge(\n", " tertile.rename('spend_tertile'), left_on='dma_id', right_index=True\n", " )\n", " grouped = (\n", " panel_with_tertile.groupby(['week', 'spend_tertile'], observed=True)\n", " ['weekly_visits'].mean().unstack()\n", " )\n", "\n", " fig, ax = plt.subplots(figsize=(8, 4))\n", " colors = {'low': 'lightcoral', 'mid': 'goldenrod', 'high': 'steelblue'}\n", " for tier in ('low', 'mid', 'high'):\n", " ax.plot(\n", " grouped.index, grouped[tier], marker='o',\n", " color=colors[tier], label=f'{tier} regional spend',\n", " )\n", " ax.axvline(4.5, linestyle='--', color='gray', alpha=0.7)\n", " ax.set_xlabel('Week')\n", " ax.set_ylabel('Mean weekly visits per DMA')\n", " ax.set_title('Weekly visits by regional-spend tertile (dashed = launch)')\n", " ax.legend()\n", " plt.tight_layout()\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "t20-cell-009", "metadata": {}, "source": [ "## 3. Fitting HAD: The Headline Per-Dollar Lift\n", "\n", "HAD targets the **Weighted Average Slope** (WAS): roughly, the average per-dollar marginal effect across the dose distribution. Because all DMAs received at least some regional spend (none at exactly zero), HAD anchors the local-linear fit at the *boundary* of the dose distribution - the lightest-touch DMA's spend, called `d_lower`. The estimand is then `WAS_d_lower`: the average per-dollar marginal effect of regional spend *above* `d_lower`.\n", "\n", "HAD's headline mode wants exactly two time points (a pre-launch snapshot and a post-launch snapshot). We collapse the 8-week panel into a 2-row-per-DMA panel by averaging weekly visits within the pre-launch window (weeks 1-4) and the post-launch window (weeks 5-8). The post-period regional spend is each DMA's actual spend; the pre-period is zero by construction." ] }, { "cell_type": "code", "execution_count": null, "id": "t20-cell-010", "metadata": {}, "outputs": [], "source": [ "panel_2pd = panel.copy()\n", "panel_2pd['period'] = (panel_2pd['week'] >= 5).astype(int) + 1 # 1=pre, 2=post\n", "panel_2pd = (\n", " panel_2pd.groupby(['dma_id', 'period'], as_index=False)\n", " .agg(\n", " weekly_visits=('weekly_visits', 'mean'),\n", " regional_spend_k=('regional_spend_k', 'mean'),\n", " )\n", ")\n", "\n", "est = HAD(design='auto')\n", "result = est.fit(\n", " panel_2pd,\n", " outcome_col='weekly_visits',\n", " dose_col='regional_spend_k',\n", " time_col='period',\n", " unit_col='dma_id',\n", ")\n", "print(result.summary())" ] }, { "cell_type": "markdown", "id": "t20-cell-011", "metadata": {}, "source": [ "**Reading the result.** HAD estimates a per-$1K marginal effect of about **100 weekly visits per DMA** above the boundary spend, with a 95% confidence interval running from 98.6 to 101.4. The estimate sits essentially on the true per-$1K lift of 100.\n", "\n", "Under a (locally) linear dose-response, this WAS_d_lower estimate translates to per-DMA dollar lift through the difference between each DMA's spend and the boundary `d_lower` (about $5K). A DMA that allocated $30K of regional add-on spend saw roughly (30 - 5) x 100 = **~2,500 extra weekly visits**; a DMA that allocated $10K saw roughly (10 - 5) x 100 = ~500. Multiply by your revenue per visit to put the lift in dollar terms.\n", "\n", "**A note on the UserWarning above.** The library fired a UserWarning reminding us that this regime (Design 1, with `d_lower > 0`) requires **Assumption 6** from de Chaisemartin et al. (2026) for point identification of `WAS_d_lower`, or **Assumption 5** for sign identification only. Both are about local linearity of the dose-response near `d_lower` - neither is testable from data. In our example, the dose-response is linear by DGP construction, so Assumption 6 holds. In a real analysis, you'd justify this from domain knowledge (e.g., is there reason to believe the marginal effect of the next $1K of regional spend is roughly constant in the $5K-$50K range?)." ] }, { "cell_type": "code", "execution_count": null, "id": "t20-cell-012", "metadata": {}, "outputs": [], "source": [ "print(f'WAS_d_lower estimate (att): {result.att:.4f}')\n", "print(f'Standard error: {result.se:.4f}')\n", "print(f'95% CI: [{result.conf_int[0]:.4f}, {result.conf_int[1]:.4f}]')\n", "print(f'\\nDesign diagnostic:')\n", "print(f' design path: {result.design!r}')\n", "print(f' target parameter: {result.target_parameter!r}')\n", "print(f' d_lower (boundary spend): ${result.d_lower:.2f}K')\n", "print(f' dose mean (D-bar): ${result.dose_mean:.2f}K')\n", "print(f' N units: {result.n_obs}')\n", "print(f' N units above d_lower: {result.n_treated}')" ] }, { "cell_type": "markdown", "id": "t20-cell-013", "metadata": {}, "source": [ "**What the design path tells us.** HAD picked the `continuous_near_d_lower` regime because all DMAs have positive regional spend - no DMA sits at exactly $0 to anchor a different design. The boundary `d_lower` resolves to the lightest-touch DMA's spend (about $5K). From there, HAD identifies the slope at the boundary using a local-linear fit on the differenced outcome, interpreted as the average per-dollar marginal effect of regional spend *above* `d_lower`, integrated across the dose distribution.\n", "\n", "If the lightest-touch DMA had spent exactly $0 (no regional add-on), HAD would have switched to a different identification path (`continuous_at_zero`, Design 1') with target `WAS` instead of `WAS_d_lower`. The auto-detection picks the right one based on the data; you don't have to choose it manually." ] }, { "cell_type": "markdown", "id": "t20-cell-014", "metadata": {}, "source": [ "## 4. Multi-Week Event Study\n", "\n", "The headline WAS_d_lower collapses 4 post-launch weeks into a single number. To see whether the per-dollar lift was immediate, building, or fading across the campaign, fit HAD's event-study mode on the full multi-week panel. It returns a per-week WAS_d_lower for each post-launch week (weeks since launch e=0, 1, 2, 3) plus pre-launch placebo horizons (e=-2, -3, -4) that should sit on zero if pre-trends are parallel.\n", "\n", "(The library fires the same Assumption 5/6 UserWarning on this fit as on the headline fit above; we addressed it there. The filter below keeps the cell output focused on the event-study table.)" ] }, { "cell_type": "code", "execution_count": null, "id": "t20-cell-015", "metadata": {}, "outputs": [], "source": [ "import warnings\n", "\n", "with warnings.catch_warnings():\n", " warnings.filterwarnings(\n", " 'ignore',\n", " message=r\".*continuous_near_d_lower.*Assumption.*\",\n", " category=UserWarning,\n", " )\n", " est_es = HAD(design='auto')\n", " result_es = est_es.fit(\n", " panel,\n", " outcome_col='weekly_visits',\n", " dose_col='regional_spend_k',\n", " time_col='week',\n", " unit_col='dma_id',\n", " first_treat_col='first_treat',\n", " aggregate='event_study',\n", " )\n", "print(result_es.summary())" ] }, { "cell_type": "code", "execution_count": null, "id": "t20-cell-016", "metadata": {}, "outputs": [], "source": [ "es_df = result_es.to_dataframe()\n", "es_df.round(3)" ] }, { "cell_type": "code", "execution_count": null, "id": "t20-cell-017", "metadata": {}, "outputs": [], "source": [ "if HAS_MATPLOTLIB:\n", " event_times = list(result_es.event_times)\n", " atts = list(result_es.att)\n", " ci_lows = list(result_es.conf_int_low)\n", " ci_highs = list(result_es.conf_int_high)\n", "\n", " fig, ax = plt.subplots(figsize=(9, 4.5))\n", " for e, att, lo, hi in zip(event_times, atts, ci_lows, ci_highs):\n", " color = 'steelblue' if e >= 0 else 'gray'\n", " ax.errorbar(\n", " [e], [att],\n", " yerr=[[att - lo], [hi - att]],\n", " fmt='o', color=color, capsize=4,\n", " )\n", " ax.axvline(-0.5, linestyle='--', color='gray', alpha=0.7)\n", " ax.axhline(0, linestyle='--', color='black', alpha=0.5)\n", " ax.set_xlabel('Weeks since campaign launch')\n", " ax.set_ylabel('Per-$1K lift above d_lower (weekly visits)')\n", " ax.set_title('Per-week WAS_d_lower with 95% CIs (gray = pre-launch placebos)')\n", " plt.tight_layout()\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "t20-cell-018", "metadata": {}, "source": [ "**Reading the dynamics.**\n", "\n", "- The pre-launch placebo horizons (weeks -4, -3, -2) all sit at essentially zero - per-$1K effects within ±0.06 with 95% CIs comfortably bracketing zero. Visually consistent with parallel pre-trends. (Note: this is a visual placebo check, not a formal pretest - HAD ships a separate composite pretest workflow we did not run here; see extensions.)\n", "- The per-week post-launch effects (weeks 0, 1, 2, 3) all hover right around 100 visits per $1K with overlapping 95% CIs and lower bounds well above zero. The per-dollar lift is stable across all four weeks of the campaign.\n", "- Practically: the campaign delivered its per-dollar lift on impact and held it across all four post-launch weeks. No ramp-up, no fade." ] }, { "cell_type": "markdown", "id": "t20-cell-019", "metadata": {}, "source": [ "## 5. Communicating the Result to Leadership\n", "\n", "When you bring this back to a non-technical audience, lead with the headline number and ground each follow-on claim in a specific piece of evidence. The template below mirrors the structure used in the brand-awareness and geo-experiment tutorials in this library.\n", "\n", "> **Headline.** Across 60 DMAs, the regional add-on ad spend delivered approximately **100 incremental weekly visits per additional $1,000** of regional spend above the lightest-touch DMA's $5K floor (95% CI: 98.6 to 101.4). On a baseline of about 5,000 weekly visits per DMA, the heaviest-spending DMAs ($50K) saw roughly 4,500 extra weekly visits during the campaign window. *[Source: `result.att` and `result.conf_int` from Section 3.]*\n", ">\n", "> **Sample size and design.** 60 DMAs observed for 8 weeks (480 DMA-weeks). Every DMA received the national TV blast on the same date; regional add-on spend ranged from a $5K floor to $50K (median ~$25K). Method: `HeterogeneousAdoptionDiD` (HAD), built for panels where treatment varies in intensity across units and no never-treated unit is available; comparison comes from the dose variation across markets. *[Source: panel shape and dose distribution from Section 2.]*\n", ">\n", "> **Validity evidence.** Two visual checks. (a) The per-week placebo horizons before launch (weeks -4, -3, -2) sit at essentially zero, with 95% CIs comfortably bracketing zero - DMAs were not already trending differently by spend tier. (This is a visual parallel-trends check, not a formal pretest; HAD also ships a composite pretest workflow we did not run here.) (b) Per-week post-launch effects are stable across all four campaign weeks at ~100 visits per $1K. *[Source: event-study horizons from Section 4.]*\n", ">\n", "> **Business interpretation.** The per-$1K lift applies to spend *above* the $5K floor (HAD's local boundary). Subtracting the floor from each DMA's actual spend and multiplying gives the per-DMA visit lift estimate: a DMA at $30K saw roughly (30 - 5) x 100 = 2,500 extra weekly visits during the campaign; a DMA at $10K saw roughly (10 - 5) x 100 = 500. Translate to your own revenue-per-visit to compare against regional spend.\n", ">\n", "> **Practical significance.** The per-dollar lift was stable across all four post-launch weeks - the campaign delivered its return on impact and held it. Whether the absolute per-DMA dollar lift justifies the regional spend is a business judgment, not a statistical one. Identification of WAS_d_lower additionally requires Assumption 6 (local linearity of the dose-response near the boundary), which is non-testable and should be argued from domain knowledge. *[Source: per-week dynamics from Section 4.]*" ] }, { "cell_type": "markdown", "id": "t20-cell-020", "metadata": {}, "source": [ "Adapt this template by swapping in your own numbers from `result.att`, `result.conf_int`, `result.d_lower`, the per-week event-study table, and your own DMA / spend distribution. The pattern - **headline → sample → validity → business → practical** - is what to keep." ] }, { "cell_type": "markdown", "id": "t20-cell-021", "metadata": {}, "source": [ "## 6. Extensions\n", "\n", "This tutorial covered HAD's headline workflow: the overall WAS_d_lower fit and the multi-week event study. The library also supports several extensions we did not demonstrate here.\n", "\n", "- **Population-weighted (survey-aware) inference**: when some markets or regions carry more weight than others - e.g., DMAs weighted by population - HAD accepts a `SurveyDesign` object on the same `fit()` interface (the deprecated `weights=` and `survey=` kwarg aliases will be removed in the next minor release; use `survey_design=` going forward). [Tutorial 22](22_had_survey_design.ipynb) walks the BRFSS-shape survey-design path end-to-end including the pretest workflow.\n", "- **Composite pretest workflow**: HAD ships a `did_had_pretest_workflow` that combines the QUG support-infimum test (`H0: d_lower = 0`, which adjudicates between the `continuous_at_zero` and `continuous_near_d_lower` design paths) with linearity tests (Stute and Yatchew-HR). On the two-period (`aggregate='overall'`) path this workflow checks QUG and linearity only; the parallel-trends step is closed by the multi-period (`aggregate='event_study'`) joint variants (`stute_joint_pretest`, `joint_pretrends_test`, `joint_homogeneity_test`). The visual placebo check we used in Section 4 is a parallel-trends sanity check, not a substitute for the formal joint pretests; see [Tutorial 21](21_had_pretest_workflow.ipynb) for an end-to-end pretest walkthrough.\n", "- **`continuous_at_zero` design path**: if the lightest-touch DMA had no regional add-on (spend exactly $0), HAD switches to the Design 1' identification path with target `WAS` instead of `WAS_d_lower`. The auto-detection picks it up.\n", "- **Mass-point design path**: if a meaningful chunk of DMAs sit at exactly the same minimum spend (rather than spread continuously near the boundary), HAD switches to a 2SLS estimator with matching identification logic. Auto-detected as well.\n", "\n", "See the [`HeterogeneousAdoptionDiD` API reference](../api/had.html) for the full parameter list." ] }, { "cell_type": "markdown", "id": "t20-cell-022", "metadata": {}, "source": [ "**Related tutorials.**\n", "\n", "- [Tutorial 1: Basic DiD](01_basic_did.ipynb) - the 2x2 building block.\n", "- [Tutorial 2: Staggered DiD](02_staggered_did.ipynb) - Callaway-Sant'Anna for absorbing staggered adoption.\n", "- [Tutorial 14: Continuous DiD](14_continuous_did.ipynb) - the Callaway-Goodman-Bacon-Sant'Anna estimator for continuous-dose settings WHERE you do have a never-treated unit AND want the full dose-response curve, not just the average slope. Use this if your panel has a never-treated unit; use HAD (this tutorial) if every unit is treated at some dose.\n", "- [Tutorial 17: Brand Awareness Survey](17_brand_awareness_survey.ipynb) - for survey data with sampling weights, strata, and PSU.\n", "- [Tutorial 18: Geo-Experiment Analysis](18_geo_experiments.ipynb) - synthetic-DiD for panels with very few treated markets.\n", "- [Tutorial 19: dCDH Marketing Pulse](19_dcdh_marketing_pulse.ipynb) - for panels where treatment turns on AND off over time." ] }, { "cell_type": "markdown", "id": "t20-cell-023", "metadata": {}, "source": [ "**Summary checklist.**\n", "\n", "- HAD is the right tool when treatment varies in intensity across units, the dose distribution is continuous, and no never-treated unit exists; comparison comes from the dose variation across units.\n", "- The WAS_d_lower (Weighted Average Slope at the boundary) reports per-dose-unit lift on the dose scale, anchored at the lightest-touch unit's dose. Multiply by `(actual_dose - d_lower)` to get per-unit treatment effects under (locally) linear dose-response.\n", "- The design auto-detection picks the right identification path from the dose distribution - usually you can leave `design='auto'` alone.\n", "- The Design 1 path requires Assumption 5 or 6 for identification, both about local linearity at the boundary - non-testable, justify from domain knowledge.\n", "- The event-study mode extends the headline WAS_d_lower into per-week dynamics, with pre-launch placebos that double as a parallel-trends visual check (formal pretests are a separate workflow noted in extensions)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }