docs(lwdid): tutorial 31 - rolling-transformation DiD (numbers-locked)

CHANGELOG.md

@@ -8,6 +8,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- **LWDiD tutorial notebook** (`docs/tutorials/31_lwdid.ipynb`): the replacement for
+  the tutorial withdrawn from PR #588, authored via the numbers-locked workflow (every
+  cited number prototyped in scripts first, notebook assembled and executed once).
+  Builds the method from the TWFE failure mode (Goodman-Bacon decomposition on a
+  synthetic staggered panel) through demeaning/detrending and exact small-sample
+  inference, then replicates both papers' applications on the authors' SSC data: Prop
+  99 (LW 2026 Table 3: demean -0.422, detrend -0.227, exact p 0.021; RI under the
+  package's Phipson-Smyth convention) and Walmart entry fitted as the honest
+  staggered design with real first-entry cohorts (WATT(1) = 0.032 vs Appendix Table
+  A4), plus cluster/wild-bootstrap inference and transformation diagnostics.
 - **ImputationDiD `aux_partition` variants now anchored against Stata `did_imputation
   avgeffectsby()` (no library behavior change beyond a docstring wording fix).** The
   coarser `aux_partition="cohort"` / `"horizon"` groupings — previously the

TODO.md

@@ -69,7 +69,6 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m
 
 | Issue | Location | Origin | Effort | Priority |
 |-------|----------|--------|--------|----------|
-| Author a fresh LWDiD tutorial notebook (the contribution's tutorial 27 was WITHDRAWN from PR #588 during CI review: its Walmart empirical narrative was built on a fabricated common onset with jobs translations that did not match the staggered estimate, and per-sentence repair failed across 5 review rounds). Build via the numbers-locked workflow: prototype in scripts, lock the numbers, then assemble and execute once; teach the staggered fit as the causal specification and keep any pooled contrast explicitly diagnostic | `docs/tutorials/` | #588 | Medium |
 | LWDiD staggered fit recomputes cohort-wide work across surfaces: each cohort's rolling transformation is built once per `fit_staggered` cohort loop, but unit-level post summaries, control-eligibility sets, and the composite path's `ydot_by_cohort` are re-derived independently by the cell, aggregate, and `_composite_regression_aggregation` paths — runtime grows ~cohorts × panel size. Cache the per-cohort transformation + post summaries in one pass and reuse; add a many-cohort unbalanced-panel benchmark | `diff_diff/lwdid_staggered.py`, `diff_diff/lwdid.py` | #588 | Mid | Low |
 | Reuse the demeaner's factorized codes in `absorbed_fe_rank`/`absorbed_fe_cr1_k_increment` instead of re-factorizing: at 186k rows the rank helper adds ~1.9 ms per absorbed fit (7.7% of the fastest Rust-served TWFE fit) and the K_reference increment ~3.2 ms per clustered-hc1 absorbed fit (~13%; see `docs/performance-plan.md` "Component-aware absorbed-FE rank"), and the helpers and `demean_by_groups` factorize the same group columns. Threading the codes through the call sites halves the factorize work; the `connected_components` call itself is ~1.1 ms. Deliberately not done in the correctness PRs. | `diff_diff/utils.py` | #variance-inventory | Quick | Low |
 | `EfficientDiD` conditional path: the largest remaining O(n) stage is the sieve/nuisance construction outside the tiled pass (~9s at 10k). (The `_ridge_solve_weights` Python-prep shave landed 2026-07-07 — the `omega_stack[rest]` fancy-index copy and tail scatter are skipped when no row is zero-masked, byte-identical outputs; the `zero_mask` abs scan itself remains, needed for correctness.) | `efficient_did_covariates.py` | CS-scaling | Mid | Low |

benchmarks/stata/README.md

@@ -361,7 +361,7 @@ python -c "from diff_diff import load_prop99, load_walmart; \
   dfs = {'prop99': load_prop99(), 'walmart': load_walmart()}; \
   assert all(df.attrs.get('source') != 'synthetic_fallback' for df in dfs.values()); \
   pins = {'prop99': '16c3ac1da351788817433fc890ec2f502a8bdfcb46cbc8d693653330e71d5a65', \
-          'walmart': '410885572143dceb9daa643a8097768f1bc3493f9437451a9e4d1d5dc1e18d14'}; \
+          'walmart': 'cf1b2d4aee396f0858fb315d0413ff82359e16e84dad68ce1d9be51e8c3e35ac'}; \
   cache = pathlib.Path.home() / '.cache' / 'diff_diff' / 'datasets'; \
   [1/0 for n, w in pins.items() if hashlib.sha256((cache / (n + '.dta')).read_bytes()).hexdigest() != w]"
 # 2. generate (about 15 minutes; six B=9,999 bootstrap runs dominate)

benchmarks/stata/generate_lwdid_golden.do

@@ -26,7 +26,7 @@
 *!          dfs = {'prop99': load_prop99(), 'walmart': load_walmart()}; \
 *!          assert all(df.attrs.get('source') != 'synthetic_fallback' for df in dfs.values()); \
 *!          pins = {'prop99': '16c3ac1da351788817433fc890ec2f502a8bdfcb46cbc8d693653330e71d5a65', \
-*!                  'walmart': '410885572143dceb9daa643a8097768f1bc3493f9437451a9e4d1d5dc1e18d14'}; \
+*!                  'walmart': 'cf1b2d4aee396f0858fb315d0413ff82359e16e84dad68ce1d9be51e8c3e35ac'}; \
 *!          cache = pathlib.Path.home() / '.cache' / 'diff_diff' / 'datasets'; \
 *!          [1/0 for n, w in pins.items() if hashlib.sha256((cache / (n + '.dta')).read_bytes()).hexdigest() != w]"
 *!   2. /Applications/Stata/StataSE.app/Contents/MacOS/stata-se -b do \
@@ -235,7 +235,7 @@ file write `fh' `"    "ssc_versions": {"lwdid": "`lwdid_ver'"},"' _n
 file write `fh' `"    "stata_edition": "`sedition'","' _n
 file write `fh' `"    "bootstrap_scheme": "lwdid large-N multiplier bootstrap (package default; reps() draws with set seed); compared against diff-diff's unit-level Rademacher multiplier bootstrap at B=999","' _n
 file write `fh' `"    "control_pool": {"small_N": "composite regression, never-treated-based by construction (no control option exists); equivalent to Python control_group='never_treated'", "large_N": "never-treated + not-yet-treated default (the 'never' option is NOT passed); matches Python default control_group='not_yet_treated'"},"' _n
-file write `fh' `"    "datasets": {"prop99": {"url": "http://fmwww.bc.edu/repec/bocode/l/lw_smoking.dta", "sha256": "16c3ac1da351788817433fc890ec2f502a8bdfcb46cbc8d693653330e71d5a65"}, "walmart": {"url": "http://fmwww.bc.edu/repec/bocode/l/lw_walmart.dta", "sha256": "410885572143dceb9daa643a8097768f1bc3493f9437451a9e4d1d5dc1e18d14"}, "castle": {"path": "benchmarks/data/real/castle_lw_subset.csv"}},"' _n
+file write `fh' `"    "datasets": {"prop99": {"url": "http://fmwww.bc.edu/repec/bocode/l/lw_smoking.dta", "sha256": "16c3ac1da351788817433fc890ec2f502a8bdfcb46cbc8d693653330e71d5a65"}, "walmart": {"url": "http://fmwww.bc.edu/repec/bocode/l/lw_walmart.dta", "sha256": "cf1b2d4aee396f0858fb315d0413ff82359e16e84dad68ce1d9be51e8c3e35ac"}, "castle": {"path": "benchmarks/data/real/castle_lw_subset.csv"}},"' _n
 file write `fh' `"    "stata_version": `sver',"' _n
 file write `fh' `"    "rireps": `RIREPS',"' _n
 file write `fh' `"    "riseed": `RISEED',"' _n

diff_diff/datasets.py

@@ -1681,7 +1681,11 @@ def load_walmart(force_download: bool = False) -> pd.DataFrame:
     ... )
     """
     url = "http://fmwww.bc.edu/repec/bocode/l/lw_walmart.dta"
-    sha256 = "410885572143dceb9daa643a8097768f1bc3493f9437451a9e4d1d5dc1e18d14"
+    # Re-pinned 2026-08-20: upstream re-saved the .dta (container bytes
+    # changed); every shared column verified numerically identical to the
+    # previously pinned copy (29,371 rows, 1,277 counties) before adopting.
+    # The Stata golden's provenance block keeps the capture-time hash.
+    sha256 = "cf1b2d4aee396f0858fb315d0413ff82359e16e84dad68ce1d9be51e8c3e35ac"
 
     source = "lwdid_ssc_ancillary"
     try:

docs/doc-deps.yaml

@@ -867,12 +867,16 @@ sources:
     docs:
       - path: docs/api/lwdid.rst
         type: api_reference
+      - path: docs/tutorials/31_lwdid.ipynb
+        type: tutorial
 
   diff_diff/lwdid_randomization.py:
     drift_risk: medium
     docs:
       - path: docs/api/lwdid.rst
         type: api_reference
+      - path: docs/tutorials/31_lwdid.ipynb
+        type: tutorial
 
   diff_diff/lwdid_sensitivity.py:
     drift_risk: medium
@@ -885,6 +889,8 @@ sources:
     docs:
       - path: docs/api/lwdid.rst
         type: api_reference
+      - path: docs/tutorials/31_lwdid.ipynb
+        type: tutorial
 
   diff_diff/lwdid_staggered.py:
     drift_risk: medium
@@ -894,6 +900,8 @@ sources:
       - path: docs/methodology/REGISTRY.md
         section: "LWDiD"
         type: methodology
+      - path: docs/tutorials/31_lwdid.ipynb
+        type: tutorial
 
   diff_diff/lwdid.py:
     drift_risk: medium
@@ -910,6 +918,8 @@ sources:
         type: user_guide
       - path: docs/choosing_estimator.rst
         type: user_guide
+      - path: docs/tutorials/31_lwdid.ipynb
+        type: tutorial
 
   # ── TROP (trop group) ──────────────────────────────────────────────
 
@@ -1043,6 +1053,9 @@ sources:
       - path: docs/tutorials/02_staggered_did.ipynb
         type: tutorial
         note: "Bacon decomposition diagnostics for staggered DiD"
+      - path: docs/tutorials/31_lwdid.ipynb
+        type: tutorial
+        note: "TWFE failure-mode demonstration"
       - path: README.md
         section: "Estimators (one-line catalog entry)"
         type: user_guide
@@ -1381,6 +1394,9 @@ sources:
         type: api_reference
       - path: docs/tutorials/09_real_world_examples.ipynb
         type: tutorial
+      - path: docs/tutorials/31_lwdid.ipynb
+        type: tutorial
+        note: "Prop 99 / Walmart replication (provenance-asserted)"
       - path: docs/methodology/REGISTRY.md
         section: "Castle Doctrine treatment coding"
         type: methodology

docs/methodology/REGISTRY.md

@@ -2551,7 +2551,7 @@ Event-study/placebo transformations over ALL periods (Appendix D): demeaning (D.
 - IPWRA (workhorse): logit propensity score per cell + WLS with weights `w = D + (1-D) p/(1-p)`; IPW = special case without the outcome-regression component.
 - **Note:** API vocabulary (as specified for the PR #588 implementation, in review): `estimation_method="reg"` = the paper's RA, `"ipw"` = IPW, `"dr"` = IPWRA (house CallawaySantAnna values; the doubly-robust option), `"psm"` = propensity-score matching, an LWDiD-only extra with no CS analog.
 - Control pool at (g, r): `A_{r+1} = 1` (never-treated + not-yet-treated) by default; NT-only optional. Pre-treatment placebo cells use the Appendix D.3 rule `A_{g,t} = {G = g} ∪ {G = 0} ∪ {G > max(g,t)}`.
-- **Note (replicating the paper's staggered numbers):** the implementation's default is `control_group='not_yet_treated'`, matching OVLS (eq. (4.10)) as stated in the text. The paper's *printed* staggered results, however, are computed against the never-treated pool only, so **reproducing them requires passing `control_group='never_treated'` explicitly**. This is not a discrepancy in either direction — it is a sample-definition choice that the text leaves to the analyst while the applications fix it to NT-only. Every staggered replication golden in `tests/test_methodology_lwdid.py` (castle `tau_omega`, and the composite-regression reference) therefore passes `control_group="never_treated"`; a default-pool fit yields different, equally valid estimates because the (g,t) cells draw on a strictly larger control sample.
+- **Note (replicating the paper's staggered numbers):** the implementation's default is `control_group='not_yet_treated'`, matching OVLS (eq. (4.10)) as stated in the text. The paper's *printed* staggered applications split by regime (amended in the tutorial-31 authoring pass; previously stated as NT-only across the board): the small-N composite applications (castle `tau_omega`) are never-treated-based by construction — the Stata small-N mode has no control-pool option — so **reproducing them requires passing `control_group='never_treated'` explicitly**, and the castle/composite goldens in `tests/test_methodology_lwdid.py` do; the large-N event-study applications (Walmart Tables A4/A5) use the package's default pool (never-treated + not-yet-treated, matching the library default `control_group='not_yet_treated'`), and the Walmart event-study goldens fit with the default accordingly (see the `control_pool` provenance block in `benchmarks/data/lwdid_stata_golden.json`). Neither is a discrepancy — the pool is a sample-definition choice; a different-pool fit yields different, equally valid estimates because the (g,t) cells draw on a different control sample.
 
 *Aggregation:*
 - Event-study: `WATT(r) = sum_{g in G_r} omega_{g,r} ATT(g, g+r)` with `omega_{g,r}` = (treated units of cohort g contributing at event time r) / (total treated units contributing at event time r) - the operative definition per LW 2025 Appendix E.1, required under unbalanced panels where a cohort's contributing count at r can differ from `N_g`. In balanced panels this simplifies to `N_g / N_{G_r}` (Sec. 6.2/D.3). Aggregated influence function `IF_{i,r} = sum_g omega_{g,r} IF_{i,g,g+r}`.