docs(lwdid): tutorial 31 - rolling-transformation DiD (numbers-locked) - #784
Conversation
Replaces the tutorial withdrawn from PR #588, authored via the numbers-locked workflow: every cited number prototyped and locked in scripts first, the notebook assembled and executed once per revision (per-worktree kernel; committed kernelspec python3). Content: the TWFE failure mode via Goodman-Bacon decomposition on a synthetic staggered panel (static TWFE 2.00 vs true 2.87, 21% forbidden-comparison weight); demeaning with exact T_{N-2} inference on a 12-unit panel (full classical-model conditions stated); detrending under treatment-related heterogeneous trends; Prop 99 replicating LW 2026 Table 3 exactly (demean -0.4222/0.1208, detrend -0.2270/0.0941, exact p 0.021, df 37) with the per-period trajectory and RI as a test of Fisher's sharp null under the documented package convention (0.054 at 9999 reps; as-printed 0.020 discrepancy noted); Walmart fitted as the honest staggered design with real first-entry cohorts (WATT(1) 0.0322/0.0051 vs Appendix Table A4, near-lead placebos < 0.006 with the single-cohort far-lead 0.033 read against its support); cluster + wild-bootstrap replay on a DGP whose region-post shock survives demeaning (naive 0.131 vs CR1 0.237); transformation diagnostics framed as descriptive sensitivity information. Methodology accuracy: estimand labels match the default not-yet-treated pool (cell-mass aggregate, with the complete tau_omega composite gate stated); identification stacks (NAC/CPTC/OVLC common timing, CNAS/CHT/OVLS staggered) and cross-unit independence stated for exact inference; t_{G-1} CR1 reference; ETWFE equivalence scoped to the impact cell (calendar t=g / event r=0); LWDiD-vs-CS efficiency framed as the paper's complementarity; replay helpers scoped to common-timing reg fits; paper titles per the registry artifacts. Guards: tests/test_t31_lwdid_drift.py (22 tests) pins the rendered surface (markdown AND outputs) for every load-bearing number, hash-pins all 13 code cells (stale-output contract with a negative control), re-derives all four synthetic DGPs against the library, and pins the corrected methodological wording. REGISTRY staggered-replication note amended (castle NT-only vs Walmart default-pool goldens); tutorial registered in docs/tutorials/index.rst + doc-deps (six source entries); TODO row retired; CHANGELOG entry added.
e7a33ab to
421eaa2
Compare
|
🔁 AI review rerun (requested by @igerber) Head SHA: Overall assessment✅ Looks good — no unmitigated P0 or P1 findings. Executive summary
MethodologyP2 — Transformation comparison is framed too strongly
P3 — Documented inference deviations
The amended Walmart control-pool record matches the Stata golden provenance and OVLS definition ( Code Quality
Performance
Maintainability
Tech Debt
Security
Documentation/Tests
The suite could not be executed here because |
A large gap between the two transformations shows sensitivity to the trend specification and prompts the CPTC-vs-CHT judgment; it is not by itself evidence that unit-specific trends exist. The unchanged exact-t df clause is scoped to the classical inference conditions of section 2 (the section-3 fits display the hc1 default). New semantic drift pins hold the corrected framing.
|
🔁 AI review rerun (requested by @igerber) Head SHA: Overall assessment✅ Looks good — no unmitigated P0 or P1 findings. Executive summary
MethodologyP3 — Documented inference conventions
The prior transformation-diagnostics concern is resolved at Code Quality
Performance
Maintainability
Tech Debt
Security
Documentation/Tests
The pytest suite could not be executed because |
…m file The tutorial-notebook CI job failed on the fail-closed provenance assert: upstream fmwww.bc.edu re-saved lw_walmart.dta, so the download no longer matched the pinned SHA-256 and CI (which has no verified cache) fell back to the synthetic frame. Verified before re-pinning: the new file's shared columns are numerically identical to the previously pinned copy (29,371 rows, 1,277 counties, every column allclose-exact), so this is a container-byte re-save, not a data change; a forced re-download through the new pin loads clean real data with zero warnings. The Stata golden's provenance block and the benchmarks README keep the capture-time hash as a historical record (lw_smoking.dta still matches its pin - its CI cell passed).
|
🔁 AI review rerun (requested by @igerber) Head SHA: Overall assessmentExecutive summary
MethodologyP3 — Documented inference conventions
The previous transformation-diagnostics wording concern remains resolved at Code QualityNo additional findings. PerformanceNo findings; estimator execution paths are unchanged. MaintainabilityP1 — Walmart checksum repin is incomplete
Tech DebtP3 — Real-data CI canary remains tracked
SecurityNo secrets or unsafe download behavior found. Downloaded bytes remain SHA-256 verified. Documentation/TestsStatic validation confirmed valid notebook/JSON/YAML structure, all 13 code-cell hashes, sequential execution counts, no notebook errors, and valid documentation paths. Runtime tests could not be executed because Path to Approval
|
…ion pins Review finding on the checksum re-pin: the golden-regeneration recipe carries its own CURRENT-INPUT copies of the dataset checksums - the fail-closed warm-up snippets in generate_lwdid_golden.do and benchmarks/stata/README.md, and the provenance metadata the generator EMITS into newly written goldens. With only the loader re-pinned, the warm-up would fail permanently against the re-saved upstream file, and a bypassed run would stamp the stale hash into fresh goldens. All three sites now carry the new pin; the committed golden's provenance block keeps the capture-time hash by design (that golden WAS generated from the old bytes, which are numerically identical). New TestStataGeneratorPinSync ties every current-input pin (both files, prop99 and walmart) to the loader's pin so a future one-sided re-pin fails in CI.
|
🔁 AI review rerun (requested by @igerber) Head SHA: Overall assessment✅ Looks good — prior P1 is resolved; no unmitigated P0/P1 findings. Executive summary
MethodologyP3 — Documented conventions
Code QualityNo findings. PerformanceNo findings; production estimator paths are unchanged. MaintainabilityP3 — Previous checksum finding resolved
Tech DebtP3 — Real-data CI canary remains tracked
SecurityNo findings. Downloaded dataset bytes remain SHA-256 verified; no secrets detected. Documentation/TestsP2 — Checksum sync test does not require both generator copies
Runtime pytest execution was unavailable because |
Review P2: the set-union form passed if either the generator's warm-up dict pin or its emitted-provenance pin was deleted entirely. The .do file's two copies per dataset are now asserted independently, each required exactly once and equal to the loader pin; the README warm-up snippet likewise.
|
🔁 AI review rerun (requested by @igerber) Head SHA: Overall assessment✅ Looks good — the prior P2 is resolved; no unmitigated P0/P1 findings. Executive summary
MethodologyP3 — Documented conventions
Code QualityNo findings. PerformanceNo findings. Production estimator paths are unchanged. Existing LWDiD recomputation work remains tracked in MaintainabilityNo findings. The new Walmart checksum is synchronized across the loader and regeneration instructions while the committed golden retains its historical provenance hash. Tech DebtP3 — Tracked real-data CI canary
SecurityNo findings. Downloaded Walmart bytes remain SHA-256 verified at Documentation/TestsP3 — Prior checksum-test finding resolved
Static validation confirmed valid notebook JSON/YAML, 13 matching code-cell hashes, sequential execution counts, no committed notebook errors, and synchronized checksums. Runtime pytest execution was unavailable because the environment lacks |
Summary
docs/tutorials/31_lwdid.ipynb, 27 cells): the TWFE failure mode via Goodman-Bacon decomposition on a synthetic staggered panel (static TWFE 2.00 vs true 2.87, 21% forbidden-comparison weight); demeaning with exactT_{N-2}inference on a 12-unit panel; detrending under treatment-related heterogeneous trends; Prop 99 replicating LW 2026 Table 3 exactly (demean -0.4222/0.1208, detrend -0.2270/0.0941, exact p 0.021) with the per-period trajectory and RI as a test of Fisher's sharp null under the documented package convention; Walmart fitted as the honest staggered design with real first-entry cohorts (WATT(1) = 0.0322 vs Appendix Table A4; near-lead placebos < 0.006, the single-cohort far lead read against its support); cluster + wild-bootstrap replay on a DGP whose region-post shock genuinely survives demeaning; transformation diagnostics framed as descriptive sensitivity information.t_{G-1}CR1 reference; ETWFE equivalence scoped to the impact cell (calendar t=g / event r=0); LWDiD-vs-CS framed as the paper's complementarity; replay helpers scoped to common-timingregfits; paper titles per the registry artifacts.tests/test_t31_lwdid_drift.py, 22 tests): pins every load-bearing number on BOTH the markdown and output surfaces, hash-pins all 13 code cells (complete stale-output contract, with a negative-control test of the mechanism), re-derives all four synthetic DGPs against the library, and pins the corrected methodological wording.control_poolprovenance block.docs/tutorials/index.rst(Advanced Methods card + toctree) and under sixdocs/doc-deps.yamlsource entries; completed TODO row retired; CHANGELOG entry added.Methodology references (required if estimator / math changes)
Validation
tests/test_t31_lwdid_drift.py(new, 22 tests - rendered-surface quote pins, all-code-cell hash contract with negative control, four synthetic-DGP rederivations, semantic wording pins). Full local sweep: drift suite +tests/test_docs_ia.py+tests/test_doc_deps_integrity.py(238 passed) +pytest --nbmakeon the notebook.tests/test_methodology_lwdid.py) end-to-end; the notebook fails closed (assertslwdid_ssc_ancillaryprovenance) before any replication prose.Security / privacy