{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# Measuring Campaign Impact on Brand Awareness with Survey Data\n\nYour company launched a brand awareness campaign in certain markets.\nThe marketing team conducted brand tracking surveys across all markets before and after the\ncampaign, using a stratified sampling design with demographic weighting. Each wave surveyed\n200 respondents — some in campaign markets, some in control markets.\n\nMarketing leadership wants to know:\n\n- Did aided awareness actually increase among respondents in campaign markets?\n- Did consideration move?\n- How confident should we be in these numbers?\n\nThis tutorial shows how to answer these questions using Difference-in-Differences (DiD) with\nproper survey design corrections. DiD compares the change among campaign-exposed respondents\nto the change among control respondents — if awareness went up 8 points in campaign markets\nbut only 2 in control markets, the incremental lift is 6 points.\n\nThe complication: your survey data has a complex sampling design — stratified by region, with\nunequal selection probabilities and geographic clustering. Ignoring this can make you\noverconfident in your results.\n\n**What you'll learn:**\n\n1. Analyzing brand tracking survey data with DiD\n2. Why survey design (weights, strata, clusters) changes your answer\n3. Measuring multiple brand funnel metrics\n4. Checking whether the result is trustworthy\n5. Extending to staggered campaign rollouts\n6. Communicating results to stakeholders",
   "id": "f8cd1807"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Setup"
   ],
   "id": "63c132f7"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "import warnings\n\nimport numpy as np\nimport pandas as pd\nfrom diff_diff import (\n    DifferenceInDifferences,\n    SurveyDesign,\n    check_parallel_trends,\n)\nfrom diff_diff.prep import generate_survey_did_data\nfrom diff_diff.practitioner import practitioner_next_steps\n\n# Suppress numerical artifacts from survey variance computation with\n# extreme weights. These are benign matmul edge cases, not methodology\n# issues — results are unaffected. All other warnings come through.\nwarnings.filterwarnings(\"ignore\", category=RuntimeWarning, module=\"diff_diff.survey\")\n\ntry:\n    import matplotlib.pyplot as plt\n\n    plt.style.use(\"seaborn-v0_8-whitegrid\")\n    HAS_MATPLOTLIB = True\nexcept ImportError:\n    HAS_MATPLOTLIB = False\n    print(\"matplotlib not installed — plots will be skipped.\")",
   "id": "7c6c8ec1"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 2. Data Preparation\n\nWe'll generate synthetic brand tracking data that mirrors a real survey:\n200 respondents across 8 waves, sampled from 5 geographic regions with\ncluster sampling and demographic weighting. The campaign launches at wave 5\nfor respondents in certain markets.",
   "id": "69d0010f"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "# Generate survey data with known treatment effect (~5 percentage points)\nraw = generate_survey_did_data(\n    n_units=200,\n    n_periods=8,\n    cohort_periods=[5],       # Campaign launches at wave 5\n    never_treated_frac=0.6,   # ~60% of respondents are in control markets\n    treatment_effect=5.0,     # True lift: 5 percentage points\n    n_strata=5,               # 5 geographic regions\n    psu_per_stratum=4,        # 4 sampling clusters per region\n    weight_variation=\"high\",  # Substantial demographic weighting\n    informative_sampling=True,\n    return_true_population_att=True,\n    seed=46,\n)\n\n# Create the binary indicators that DiD needs\nraw[\"campaign_respondent\"] = (raw[\"first_treat\"] > 0).astype(int)\nraw[\"post_campaign\"] = (raw[\"period\"] >= 5).astype(int)\n\n# Rename columns to business terms\ndata = raw.rename(columns={\n    \"unit\": \"respondent_id\",\n    \"period\": \"wave\",\n    \"outcome\": \"awareness\",\n    \"stratum\": \"region\",\n    \"psu\": \"cluster\",\n    \"weight\": \"survey_weight\",\n    \"first_treat\": \"campaign_start_wave\",\n    \"treated\": \"campaign_active\",\n})\n\n# Scale awareness to realistic brand metric percentages (~45% baseline)\ndata[\"awareness\"] = data[\"awareness\"] + 45\n\n# Create additional brand funnel metrics\n# Effects attenuate down the funnel: awareness > consideration > purchase intent\nrng = np.random.default_rng(seed=99)\ndata[\"consideration\"] = 25 + (data[\"awareness\"] - 45) * 0.6 + rng.normal(0, 1.0, len(data))\ndata[\"purchase_intent\"] = 12 + (data[\"awareness\"] - 45) * 0.3 + rng.normal(0, 0.8, len(data))\n\nprint(f\"Dataset: {data.shape[0]} observations, {data['respondent_id'].nunique()} respondents, {data['wave'].nunique()} waves\")\nprint(f\"Campaign respondents: {data.groupby('respondent_id')['campaign_respondent'].first().sum()}\")\nprint(f\"Control respondents: {(~data.groupby('respondent_id')['campaign_respondent'].first().astype(bool)).sum()}\")",
   "id": "c6960896"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "# Average brand metrics by group and period\nsummary = data.groupby([\"campaign_respondent\", \"post_campaign\"]).agg(\n    awareness=(\"awareness\", \"mean\"),\n    consideration=(\"consideration\", \"mean\"),\n    purchase_intent=(\"purchase_intent\", \"mean\"),\n).round(1)\nsummary.index = summary.index.set_names([\"Campaign Respondent\", \"Post Campaign\"])\nsummary",
   "id": "53cf1176"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Visual Inspection\n",
    "\n",
    "Before running any analysis, plot awareness over time for campaign vs. control markets.\n",
    "The key question: were the two groups trending similarly *before* the campaign launched?"
   ],
   "id": "b16f8a20"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "trends = data.groupby([\"wave\", \"campaign_respondent\"])[\"awareness\"].mean().unstack()\ntrends.columns = [\"Control\", \"Campaign\"]\n\nif HAS_MATPLOTLIB:\n    fig, ax = plt.subplots(figsize=(10, 5))\n    trends.plot(ax=ax, marker=\"o\", linewidth=2)\n    ax.axvline(x=4.5, color=\"gray\", linestyle=\"--\", alpha=0.7, label=\"Campaign Launch\")\n    ax.set_xlabel(\"Wave\")\n    ax.set_ylabel(\"Aided Awareness (%)\")\n    ax.set_title(\"Brand Awareness Over Time\")\n    ax.legend()\n    plt.tight_layout()\n    plt.show()\nelse:\n    print(trends.to_string())",
   "id": "c03c6b19"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before the campaign launched (waves 1-4), awareness was trending similarly in both groups.\n",
    "After launch (waves 5-8), campaign markets pulled ahead. This is exactly the pattern DiD\n",
    "is designed to measure."
   ],
   "id": "8b4cef1e"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Naive DiD (Ignoring Survey Design)\n",
    "\n",
    "First, run a standard DiD analysis that treats every survey response equally."
   ],
   "id": "7245c127"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "did_naive = DifferenceInDifferences()\nresults_naive = did_naive.fit(\n    data,\n    outcome=\"awareness\",\n    treatment=\"campaign_respondent\",\n    time=\"post_campaign\",\n)\nprint(results_naive)\nprint(f\"\\nThe campaign increased awareness by {results_naive.att:.1f} percentage points\")\nprint(f\"95% CI: ({results_naive.conf_int[0]:.1f}, {results_naive.conf_int[1]:.1f})\")",
   "id": "e5db9120"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks like a strong, precise result. But it treats every survey response as equally\n",
    "informative and ignores the sampling structure. Let's see what happens when we account for\n",
    "the survey design."
   ],
   "id": "81f52c46"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Survey-Aware DiD\n",
    "\n",
    "Brand tracking surveys rarely use simple random sampling. Respondents are sampled in\n",
    "geographic clusters with demographic quotas and weighting. The `SurveyDesign` object\n",
    "tells diff-diff how the survey was conducted."
   ],
   "id": "0bb69515"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "sd = SurveyDesign(\n    weights=\"survey_weight\",  # Accounts for demographic oversampling\n    strata=\"region\",          # Sample was drawn separately within each region\n    psu=\"cluster\",            # Respondents sampled in geographic clusters\n    fpc=\"fpc\",                # Finite population correction\n)\n\ndid_survey = DifferenceInDifferences()\nresults_survey = did_survey.fit(\n    data,\n    outcome=\"awareness\",\n    treatment=\"campaign_respondent\",\n    time=\"post_campaign\",\n    survey_design=sd,\n)\n\nprint(results_survey)\nprint(f\"\\nThe campaign increased awareness by {results_survey.att:.1f} percentage points\")\nprint(f\"95% CI: ({results_survey.conf_int[0]:.1f}, {results_survey.conf_int[1]:.1f})\")",
   "id": "efbf20d6"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What Changed?\n",
    "\n",
    "Let's compare the naive and survey-aware results side by side."
   ],
   "id": "dd92bbc9"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "se_ratio = results_survey.se / results_naive.se\n\ncomparison = pd.DataFrame({\n    \"Naive\": [\n        f\"{results_naive.att:.2f}\",\n        f\"{results_naive.se:.3f}\",\n        f\"({results_naive.conf_int[0]:.1f}, {results_naive.conf_int[1]:.1f})\",\n        f\"{results_naive.p_value:.4f}\",\n    ],\n    \"Survey-Aware\": [\n        f\"{results_survey.att:.2f}\",\n        f\"{results_survey.se:.3f}\",\n        f\"({results_survey.conf_int[0]:.1f}, {results_survey.conf_int[1]:.1f})\",\n        f\"{results_survey.p_value:.4f}\",\n    ],\n}, index=[\"Lift (pp)\", \"Std Error\", \"95% CI\", \"p-value\"])\n\nprint(comparison.to_string())\nprint(f\"\\nSE inflation ratio: {se_ratio:.2f}x\")\nprint(f\"Survey-aware standard errors are {(se_ratio - 1) * 100:.0f}% larger than naive.\")\nprint(f\"\\nThe lift estimate is similar, but the naive analysis makes you think\")\nprint(f\"you know it more precisely than you actually do.\")",
   "id": "387a083f"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The standard errors more than doubled. Respondents within the same geographic cluster\n",
    "tend to answer similarly, so each response carries less independent information than the\n",
    "raw sample size suggests. The naive analysis was overconfident.\n",
    "\n",
    "In this case, both analyses agree the campaign worked — but the survey-aware confidence\n",
    "interval is much wider. In a closer call, ignoring the survey design could lead you to\n",
    "claim a significant result when the evidence is actually inconclusive."
   ],
   "id": "e8a4065d"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Multiple Brand Metrics\n",
    "\n",
    "Brand campaigns don't just move awareness — they should also move consideration and\n",
    "purchase intent. Let's measure the lift across the full brand funnel."
   ],
   "id": "60fc326a"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "outcomes = [\"awareness\", \"consideration\", \"purchase_intent\"]\nfunnel_results = {}\n\nfor outcome in outcomes:\n    did = DifferenceInDifferences()\n    r = did.fit(\n        data,\n        outcome=outcome,\n        treatment=\"campaign_respondent\",\n        time=\"post_campaign\",\n        survey_design=sd,\n    )\n    funnel_results[outcome] = r\n\n# Results table\nfunnel_df = pd.DataFrame({\n    \"Metric\": [\"Awareness\", \"Consideration\", \"Purchase Intent\"],\n    \"Lift (pp)\": [funnel_results[o].att for o in outcomes],\n    \"SE\": [funnel_results[o].se for o in outcomes],\n    \"95% CI Lower\": [funnel_results[o].conf_int[0] for o in outcomes],\n    \"95% CI Upper\": [funnel_results[o].conf_int[1] for o in outcomes],\n    \"p-value\": [funnel_results[o].p_value for o in outcomes],\n}).round(2)\n\nprint(funnel_df.to_string(index=False))",
   "id": "f891e2f1"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if HAS_MATPLOTLIB:\n",
    "    metrics = [\"Awareness\", \"Consideration\", \"Purchase\\nIntent\"]\n",
    "    lifts = [funnel_results[o].att for o in outcomes]\n",
    "    ci_low = [funnel_results[o].conf_int[0] for o in outcomes]\n",
    "    ci_high = [funnel_results[o].conf_int[1] for o in outcomes]\n",
    "    errors = [[l - lo for l, lo in zip(lifts, ci_low)],\n",
    "              [hi - l for l, hi in zip(lifts, ci_high)]]\n",
    "\n",
    "    fig, ax = plt.subplots(figsize=(8, 5))\n",
    "    bars = ax.bar(metrics, lifts, color=[\"#2196F3\", \"#4CAF50\", \"#FF9800\"],\n",
    "                  yerr=errors, capsize=8, edgecolor=\"black\", linewidth=0.5)\n",
    "    ax.axhline(y=0, color=\"black\", linewidth=0.5)\n",
    "    ax.set_ylabel(\"Incremental Lift (percentage points)\")\n",
    "    ax.set_title(\"Campaign Impact Across the Brand Funnel\")\n",
    "\n",
    "    for bar, lift in zip(bars, lifts):\n",
    "        ax.text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 0.3,\n",
    "                f\"+{lift:.1f}pp\", ha=\"center\", va=\"bottom\", fontweight=\"bold\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ],
   "id": "14ff1a06"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The campaign moved awareness the most, consideration less, and purchase intent the\n",
    "least. This is typical funnel attenuation — the message reached people but didn't fully\n",
    "convert to purchase consideration. All three effects are statistically significant."
   ],
   "id": "d3b6008d"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Is This Result Trustworthy?\n",
    "\n",
    "Two diagnostic checks help validate the result."
   ],
   "id": "d68bf901"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "### Parallel Trends Check\n\nDiD assumes campaign and control groups would have continued trending the same way if\nthe campaign hadn't run. `check_parallel_trends()` is a quick informal check that\ncompares pre-campaign slopes — it does not account for survey design, so treat it as\na sanity check rather than a formal test. The formal robustness assessment comes from\nHonestDiD in Section 8.",
   "id": "96bbef84"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "pt = check_parallel_trends(\n    data,\n    outcome=\"awareness\",\n    time=\"wave\",\n    treatment_group=\"campaign_respondent\",\n)\n\nprint(f\"Pre-campaign trend difference: {pt['trend_difference']:.3f}\")\nprint(f\"p-value: {pt['p_value']:.3f}\")\nprint(f\"\\nParallel trends {'consistent with the data' if pt['parallel_trends_plausible'] else 'NOT supported'}\")\nif pt[\"parallel_trends_plausible\"]:\n    print(\"Before the campaign, awareness was trending at a similar rate in both groups.\")",
   "id": "3d718ede"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Placebo Test\n",
    "\n",
    "Run the same DiD analysis on the pre-campaign period only, where no campaign effect\n",
    "should exist. If we find a \"significant\" effect here, something is wrong."
   ],
   "id": "aecefb2f"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "# Use waves 1-4 only; split at wave 3 as a \"placebo\" campaign launch\npre_data = data[data[\"wave\"] <= 4].copy()\npre_data[\"placebo_post\"] = (pre_data[\"wave\"] >= 3).astype(int)\n\n# Use survey_design here too — consistent with the main analysis\ndid_placebo = DifferenceInDifferences()\nr_placebo = did_placebo.fit(\n    pre_data,\n    outcome=\"awareness\",\n    treatment=\"campaign_respondent\",\n    time=\"placebo_post\",\n    survey_design=sd,\n)\n\nprint(f\"Placebo lift: {r_placebo.att:.2f} pp (p = {r_placebo.p_value:.3f})\")\nif r_placebo.p_value > 0.05:\n    print(\"No significant effect in the pre-campaign period — the method isn't picking up spurious patterns.\")\nelse:\n    print(\"WARNING: Significant placebo effect detected — investigate further.\")",
   "id": "ef7db9b1"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "The informal trend check is consistent with parallel trends, and the survey-aware\nplacebo test finds no effect where none should exist. Together these are supportive\nevidence, though neither formally proves the parallel trends assumption — it is always\nan untestable assumption about what *would have* happened.\n\nFor event study designs (Section 8 below), HonestDiD sensitivity analysis provides\na formal assessment of how robust the result is to violations of this assumption.",
   "id": "5a36cef4"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Practitioner Guidance\n",
    "\n",
    "diff-diff includes an automated checklist based on the\n",
    "[Baker et al. (2025)](https://arxiv.org/pdf/2503.13323) practitioner workflow.\n",
    "It suggests diagnostic steps based on your estimator and results.\n",
    "\n",
    "*Note: the code snippets in the output use placeholder column names — substitute\n",
    "your own.*"
   ],
   "id": "ea3733ec"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "practitioner_next_steps(results_survey, verbose=True)"
   ],
   "id": "8e82a98a"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Extension: Staggered Campaign Rollout\n",
    "\n",
    "Many campaigns don't launch in all markets at once — they roll out in waves.\n",
    "Some markets go live in month 2, others in month 4. When this happens, basic\n",
    "DiD can give biased results. The `CallawaySantAnna` estimator handles this\n",
    "correctly.\n",
    "\n",
    "Let's generate data where the campaign rolled out in two waves."
   ],
   "id": "5dd70c53"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "from diff_diff import CallawaySantAnna\nfrom diff_diff.visualization import plot_event_study\n\n# Campaign rolls out in two waves: some markets at wave 3, others at wave 5\nstag_raw = generate_survey_did_data(\n    n_units=200,\n    n_periods=8,\n    cohort_periods=[3, 5],\n    never_treated_frac=0.4,\n    treatment_effect=5.0,\n    dynamic_effects=True,\n    effect_growth=0.1,        # Effect builds 10% per wave (repeated exposure)\n    n_strata=5,\n    psu_per_stratum=4,\n    weight_variation=\"high\",\n    informative_sampling=True,\n    return_true_population_att=True,\n    seed=42,\n)\n\nstag_data = stag_raw.rename(columns={\n    \"unit\": \"respondent_id\", \"period\": \"wave\", \"outcome\": \"awareness\",\n    \"stratum\": \"region\", \"psu\": \"cluster\", \"weight\": \"survey_weight\",\n    \"first_treat\": \"campaign_start_wave\",\n})\nstag_data[\"awareness\"] = stag_data[\"awareness\"] + 45\n\nprint(f\"Campaign cohorts: {sorted(stag_data['campaign_start_wave'].unique())}\")\nprint(f\"  Wave 3 launch: {(stag_data.groupby('respondent_id')['campaign_start_wave'].first() == 3).sum()} respondents\")\nprint(f\"  Wave 5 launch: {(stag_data.groupby('respondent_id')['campaign_start_wave'].first() == 5).sum()} respondents\")\nprint(f\"  Control:       {(stag_data.groupby('respondent_id')['campaign_start_wave'].first() == 0).sum()} respondents\")",
   "id": "7d1c9510"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "stag_sd = SurveyDesign(\n    weights=\"survey_weight\", strata=\"region\", psu=\"cluster\", fpc=\"fpc\",\n)\n\n# base_period=\"universal\" is required for valid HonestDiD sensitivity analysis —\n# it uses a common reference period so pre-treatment coefficients are comparable.\ncs = CallawaySantAnna(base_period=\"universal\")\nstag_results = cs.fit(\n    stag_data,\n    outcome=\"awareness\",\n    unit=\"respondent_id\",\n    time=\"wave\",\n    first_treat=\"campaign_start_wave\",\n    aggregate=\"event_study\",\n    survey_design=stag_sd,\n)\n\nprint(stag_results)\nprint(f\"\\nOverall campaign lift: {stag_results.overall_att:.1f} pp\")\nprint(\"\\nEvent study effects (relative to campaign launch):\")\nprint(stag_results.to_dataframe(level=\"event_study\").round(2).to_string(index=False))",
   "id": "dd607a40"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if HAS_MATPLOTLIB:\n",
    "    fig, ax = plt.subplots(figsize=(10, 6))\n",
    "    plot_event_study(\n",
    "        stag_results,\n",
    "        ax=ax,\n",
    "        title=\"Campaign Effect Over Time (Staggered Rollout)\",\n",
    "        xlabel=\"Waves Relative to Campaign Launch\",\n",
    "        ylabel=\"Awareness Lift (pp)\",\n",
    "    )\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ],
   "id": "372755dd"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "The event study shows the campaign effect building over time — starting around 5pp at\nlaunch and growing to about 7pp with sustained exposure. Pre-campaign periods show no\nsignificant effects, consistent with the parallel trends assumption.",
   "id": "f67896f2"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sensitivity Analysis\n",
    "\n",
    "HonestDiD ([Rambachan & Roth, 2023](https://academic.oup.com/restud/article/90/5/2555/7039335))\n",
    "tells us how much the parallel trends assumption would need to be violated for the\n",
    "result to disappear."
   ],
   "id": "281d0958"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "from diff_diff import compute_honest_did\n\nhonest = compute_honest_did(stag_results, method=\"relative_magnitude\", M=1.0)\n\nprint(honest.summary())\nprint(\"\\nIn plain English: even if the pre-campaign trends were off by as much as\")\nprint(\"the largest observed pre-period fluctuation, the campaign effect remains positive.\")",
   "id": "0d7c096e"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Communicating Results to Leadership\n",
    "\n",
    "Here's how to write up the finding for stakeholders:"
   ],
   "id": "751a8e53"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "r = results_survey  # The main 2x2 result\nn_campaign = data.groupby(\"respondent_id\")[\"campaign_respondent\"].first().sum()\nn_control = (~data.groupby(\"respondent_id\")[\"campaign_respondent\"].first().astype(bool)).sum()\n\nprint(\"=\" * 70)\nprint(\"EXECUTIVE SUMMARY\")\nprint(\"=\" * 70)\nprint(f\"\"\"\nThe brand awareness campaign increased aided awareness by {r.att:.1f}\npercentage points (95% CI: {r.conf_int[0]:.1f} to {r.conf_int[1]:.1f})\namong {n_campaign} campaign-exposed respondents compared to {n_control}\ncontrol respondents.\n\nThis result accounts for the complex survey sampling design and is\nsupported by pre-campaign trend analysis and placebo testing.\n\nImpact across the brand funnel:\n  - Awareness:       +{funnel_results['awareness'].att:.1f} pp\n  - Consideration:   +{funnel_results['consideration'].att:.1f} pp\n  - Purchase Intent: +{funnel_results['purchase_intent'].att:.1f} pp\n\nThe effect attenuates down the funnel, suggesting the campaign\nsuccessfully raised awareness but further investment is needed to\nconvert awareness into purchase consideration.\n\"\"\")",
   "id": "2fa7cc34"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Key points for your write-up:**\n",
    "\n",
    "- Report the **survey-aware** estimate, not the naive one — it reflects the true uncertainty\n",
    "- Include confidence intervals, not just point estimates — leadership should understand the range\n",
    "- Distinguish **statistical significance** (is the effect real?) from **practical significance**\n",
    "  (is it big enough to matter?)\n",
    "- A 5pp lift in awareness from 46% to 51% may or may not justify the campaign spend —\n",
    "  that's a business judgment, not a statistical one"
   ],
   "id": "d4884c2b"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## Summary\n\n**What we covered:**\n\n- **Survey design matters**: Ignoring the complex sampling structure made standard errors\n  more than 2x too small, creating false precision\n- **DiD with survey data**: `SurveyDesign` integrates directly with all diff-diff estimators —\n  just pass `survey_design=sd` to `.fit()`\n- **Brand funnel analysis**: Measuring awareness, consideration, and purchase intent together\n  reveals where the campaign effect attenuates\n- **Diagnostics**: Informal trend checks and survey-aware placebo tests provide supportive\n  evidence; HonestDiD provides formal robustness assessment\n- **Staggered rollouts**: `CallawaySantAnna` handles campaigns that launch in waves, with\n  event study plots showing how the effect builds over time\n- **Sensitivity**: HonestDiD quantifies how robust the result is to assumption violations\n\n**When to use this approach:**\n\n- You have survey data collected before and after a campaign or intervention\n- The campaign ran in some markets/regions but not others\n- Randomized A/B testing wasn't feasible\n- Your survey uses stratified sampling, clustering, or weighting\n\n**Related tutorials:**\n\n- [Tutorial 16: Survey DiD](16_survey_did.ipynb) — deep dive into survey design theory,\n  replicate weights, and design effect diagnostics\n- [Tutorial 02: Staggered DiD](02_staggered_did.ipynb) — more on Callaway-Sant'Anna and\n  staggered adoption designs\n- [Tutorial 05: Honest DiD](05_honest_did.ipynb) — full sensitivity analysis guide",
   "id": "f3b24495"
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}