Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Added
- **LPDiD non-absorbing SEs now anchored against the authors' Stata `lpdid` package
end-to-end (no library behavior change).** New committed-golden arm
(`benchmarks/stata/generate_lpdid_nonabsorbing_golden.do` →
`benchmarks/data/lpdid_nonabsorbing_stata_golden.json`, consumed by
`tests/test_lpdid_nonabsorbing_stata_parity.py`) running SSC `lpdid` v1.0.3 —
the package builds its own clean samples, unlike the hand-reconstructed
`teffects` RA arm. First external anchors for the non-absorbing REWEIGHTED SE
(previously pinned-only via `RW_SE_PIN` with a documented ~5e-5 feols
weighted-cluster convention gap — the package matches the library convention to
~1e-9), the non-absorbing POOLED windows (points and SEs), and the Eq. 12
reweighted point. Scoping: `first_entry` (Eq. 12 ==
`nonabsorbing(, firsttreat notyet)`) agrees on every surface of the committed
panel (all event-study horizons incl. placebos + both pooled windows; att
~1e-14 vw / ~1.3e-8 rw, SE ~6e-17 vw / ~2.3e-9 rw); `effect_stabilization`
(Eq. 13 == `nonabsorbing(L)`) agrees at post horizons + pooled post on a
convention-neutral 47-unit subsample (att ~1.1e-14 / ~1.9e-8, SE ~1.8e-16 /
~1.6e-9). Three reference-package convention differences were measured,
adjudicated against the paper's Eq. 13 text, and locked by
divergence-documentation gates: the package effectively requires L+1 untreated
periods before re-entry (stricter than Eq. 13's stated levels condition, which
the library implements), admits always-treated units as early-period controls
via missing-lag boundary semantics (paper-silent surface), and builds placebo
samples by recursive lagged intersection rather than the library's backward
window (paper-silent). Eq. 13 full-panel and placebo/pooled-pre inference
remain outside the Stata anchor and are documented as such (REGISTRY `## LPDiD`
Deviation 4; `DEFERRED.md` non-absorbing row narrowed to the RA +
placebo/pooled-pre residuals).
- **LWDiD tutorial notebook** (`docs/tutorials/31_lwdid.ipynb`): the replacement for
the tutorial withdrawn from PR #588, authored via the numbers-locked workflow (every
cited number prototyped in scripts first, notebook assembled and executed once).
Expand Down
2 changes: 1 addition & 1 deletion DEFERRED.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ exists but parity can't be verified without a local toolchain.
| `HeterogeneousAdoptionDiD` Stute-family Stata-bridge parity: no public R `Stutetest` package exists; would add `benchmarks/stata/generate_stute_golden.do` + a Stata dependency. | `benchmarks/stata/`, `tests/test_stute_test_parity.py` | follow-up | Low |
| **`LPDiD` regression-adjustment SE — Stata-`teffects` anchored; optional R follow-up only.** The RA influence-function cluster SE is canonically Stata `teffects ra ... atet vce(cluster)` (no R package computes it — `alexCardazzi/lpdid` does direct covariate inclusion, not RA). **Resolved:** `benchmarks/stata/generate_lpdid_ra_golden.do` reconstructs each horizon's clean sample independently and anchors the SE against `teffects` to ~1e-16 at all 7 horizons (golden `lpdid_ra_stata_golden.json`; `tests/test_lpdid_ra_stata_parity.py`, which also gates the Stata row count vs the library `n_obs`); the point is triple-anchored (Python/R/Stata). Stata is node-locked (not in CI), so the CI-runnable guards remain the regression pin (`RA_SE_PIN`) + MC coverage (`coverage_lpdid_ra.py`). *Optional residual (no longer a blocker):* contribute an RA path to `alexCardazzi/lpdid` so a runnable **R** reference also exists. | `benchmarks/stata/generate_lpdid_ra_golden.do`, `tests/test_lpdid_ra_stata_parity.py` | #B2 · Stata arm | Low |
| **`LPDiD` survey scope gaps (PR-D1 deferrals).** Survey support covers the variance-weighted default path only. (a) `survey_design` + `reweight=True` (the equally-weighted / regression-adjustment IF path) is rejected: the weighted RA influence-function variance has **no runnable survey reference** (same class as the RA-SE row above - `survey::svyglm` anchors only the OLS/WLS path). (b) Replicate-weight survey designs (BRR/Fay/JK1/JKn/SDR) and (c) non-pweight (fweight/aweight) types are rejected pending demand. | `lpdid.py`, REGISTRY #8 | PR-D1 | Low |
| **`LPDiD` non-absorbing SE/RA external-reference gap.** The non-absorbing *point* estimands are R-parity-locked (PR-C2: independent `fixest::feols` Eq. 12/13 reconstruction, point+SE ~1e-13/~1e-15 vw; `effect_stabilization` reweighted point + pinned SE), but the authors' canonical non-absorbing SE/RA is Stata `lpdid`/`teffects` only - no faithful R analogue exists (`alexCardazzi/lpdid`'s `nonabsorbing_lag` proved NOT a faithful Eq. 13: off-switch clamp + non-paper boundary/placebo window, diverges ~0.01-0.05 even on a monotone panel; recorded as a divergent reference, not a gate). Revisit if a Stata toolchain or a corrected R package appears - same class as the absorbing RA-SE row above. | `benchmarks/R/generate_lpdid_golden.R`, `tests/test_methodology_lpdid.py` | PR-C2 | Low |
| **`LPDiD` non-absorbing SE/RA external-reference gap.** The authors' canonical non-absorbing SE/RA is Stata `lpdid`/`teffects` only - no faithful R analogue exists (`alexCardazzi/lpdid`'s `nonabsorbing_lag` proved NOT a faithful Eq. 13; recorded as a divergent reference, not a gate). **Resolved (SE half, scoped):** `benchmarks/stata/generate_lpdid_nonabsorbing_golden.do` runs the authors' SSC `lpdid` package end-to-end (golden `lpdid_nonabsorbing_stata_golden.json`; `tests/test_lpdid_nonabsorbing_stata_parity.py`) - Eq. 12 (`nonabsorbing(, firsttreat notyet)` == `first_entry`) anchored on ALL surfaces of the full committed panel (ES incl. placebos + both pooled windows, vw + rw; att ~1e-14/1.3e-8, SE ~6e-17/2.3e-9), Eq. 13 (`nonabsorbing(L)` == `effect_stabilization`) anchored at post horizons + pooled post on a convention-neutral 47-unit subsample (att ~1.1e-14/1.9e-8, SE ~1.8e-16/1.6e-9); first external anchors for the rw SE, the pooled windows, and the Eq. 12 rw point. Three package convention differences measured, paper-adjudicated and divergence-gated (REGISTRY `## LPDiD` Deviation 4). *Residual:* (a) non-absorbing **RA** (`rw` + covariates) - the package's RA engine is `listreg` (not `teffects`) and the committed non-absorbing panel has no covariate; needs a covariate panel + a `listreg` IF-SE convention check. (b) Eq. 13 **placebo/pooled-pre inference** - not anchored by this arm (the package's placebo-window construction differs, convention class 3; a dedicated late-entry/never-treated subsample could in principle align the two placebo constructions - a possible follow-up): the placebo ES SEs remain R-anchored (vw) / `RW_SE_PIN`-pinned (rw, h=-3/-2), and the pooled-pre SE currently has no pin or anchor of any kind (every non-absorbing R-parity fit is event-study-only and the R golden has no pooled block). | `benchmarks/stata/generate_lpdid_nonabsorbing_golden.do`, `tests/test_lpdid_nonabsorbing_stata_parity.py`, `tests/test_methodology_lpdid.py` | PR-C2 · Stata arm | Low |
| `HeterogeneousAdoptionDiD` Phase-3 R-parity: ships coverage-rate validation on synthetic DGPs, not tight point parity vs `chaisemartin::stute_test` / `yatchew_test` (needs bootstrap-seed-semantics + `B` alignment across numpy/R). | `tests/test_had_pretests.py` | Phase 3 | Low |

## Parked — pending user demand / out of scope
Expand Down
15 changes: 9 additions & 6 deletions benchmarks/R/generate_lpdid_golden.R
Original file line number Diff line number Diff line change
Expand Up @@ -326,10 +326,12 @@ message(sprintf("Wrote golden: %s", golden_path))
# and ~1e-15 on SE for the variance-weighted variants (reweighted: point ~1e-13,
# SE has a small feols-weighted-cluster convention difference, pinned on the
# Python side). alexCardazzi::lpdid()'s `nonabsorbing_lag` is NOT a faithful
# Eq.13 (it clamps off-switches via treat_diff[<0]<-0 and uses a non-paper
# boundary/window convention; it diverges ~0.01-0.05 from Eq.13 even on a
# monotone no-off-switch panel) -> recorded in `meta` as a divergent third-party
# reference, NOT a parity gate (the B2 alexCardazzi-pooled precedent).
# Eq.13 (it clamps off-switches via treat_diff[<0]<-0, violating Eq.13's stated
# both-directions control condition; it also holds different boundary/placebo-
# window conventions on surfaces the paper does not specify; it diverges
# ~0.01-0.05 from Eq.13 even on a monotone no-off-switch panel) -> recorded in
# `meta` as a divergent third-party reference, NOT a parity gate (the B2
# alexCardazzi-pooled precedent).
#
# APPENDED after the absorbing write_json with its OWN set.seed and distinct
# object names; setFixest_ssc is NOT re-called -> the absorbing panel + golden
Expand Down Expand Up @@ -431,8 +433,9 @@ effect_stab_rw_es <- na_es("eq13", reweight = TRUE)

# ---- monotone (no-off-switch) slice: PIN the "alex diverges even without off-switches"
# claim with committed evidence. On units whose treatment never decreases, alex's
# off-switch clamp is inert, yet alex still diverges from the paper-faithful Eq.13 (its
# non-paper boundary/window convention), so the recorded max post-horizon |alex - Eq.13|
# off-switch clamp is inert, yet alex still diverges from the library's Eq.13
# implementation (their boundary/placebo-window conventions differ on surfaces the
# paper does not specify), so the recorded max post-horizon |alex - Eq.13|
# is well above 0 -> documents that the divergence is NOT only off-switch handling.
na_mono <- na_dt[, if (all(diff(treat) >= 0L)) .SD, by = unit] # drop units with any turn-off
na_mode <<- "eq13"
Expand Down
8 changes: 6 additions & 2 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,17 +54,19 @@ and CI never needs Stata. Generators live in `stata/` and are run headless:
STATA=/Applications/Stata/StataSE.app/Contents/MacOS/stata-se
$STATA -b do benchmarks/stata/requirements.do # one-time SSC install
$STATA -b do benchmarks/stata/generate_lpdid_ra_golden.do
$STATA -b do benchmarks/stata/generate_lpdid_nonabsorbing_golden.do
$STATA -b do benchmarks/stata/generate_imputation_loo_golden.do
$STATA -b do benchmarks/stata/generate_etwfe_cs_golden.do
$STATA -b do benchmarks/stata/generate_reghdfe_kref_golden.do
$STATA -b do benchmarks/stata/generate_lwdid_golden.do # see stata/README.md for its warm-up step
```

The `LPDiD` arm uses only **native** Stata commands (`teffects`), pinned by
The `LPDiD` RA arm uses only **native** Stata commands (`teffects`), pinned by
`version 19`. The `ImputationDiD` arm depends on SSC packages
(`did_imputation`/`reghdfe`/`ftools`/`require`), the ETWFE/CS arm on
`drdid`/`csdid`/`jwdid`/`hdfe`, the reghdfe K_reference arm on `reghdfe`,
and the LWDiD arm on the authors' `lwdid`;
the LWDiD arm on the authors' `lwdid`, and the LPDiD non-absorbing arm on the
authors' `lpdid` (+ `boottest`/`egenmore`/`listreg`);
`version 19` does NOT pin SSC packages
(SSC has no version history) — install them once via `requirements.do` (the
generators do not auto-install) and each golden records version/checksum
Expand All @@ -87,6 +89,7 @@ benchmarks/
│ ├── README.md # Stata arm docs
│ ├── requirements.do # one-time SSC install (did_imputation etc.)
│ ├── generate_lpdid_ra_golden.do # LPDiD RA SE vs teffects ra
│ ├── generate_lpdid_nonabsorbing_golden.do # LPDiD non-absorbing SEs vs the authors' lpdid
│ ├── generate_imputation_loo_golden.do # ImputationDiD LOO SE vs did_imputation leaveout
│ ├── generate_etwfe_cs_golden.do # ETWFE/CS vs jwdid + csdid (+ subsample ladder)
│ ├── generate_reghdfe_kref_golden.do # clustered CR1 K_reference vs reghdfe (disconnected panel)
Expand All @@ -113,6 +116,7 @@ benchmarks/
| `SyntheticDiD` | `synthdid::synthdid_estimate` | Arkhangelsky et al. (2021) | ✓ Integrated |
| `DifferenceInDifferences` | `fixest::feols` | Standard DiD | ✓ Integrated |
| `LPDiD` (RA SE) | Stata `teffects ra ... atet` | Dube, Girardi, Jorda & Taylor (2025) | ✓ Integrated |
| `LPDiD` (non-absorbing SEs) | Stata `lpdid, nonabsorbing(...)` (authors' package) | Dube, Girardi, Jorda & Taylor (2025) §4.2 | ✓ Integrated |
| `ImputationDiD` (LOO SE) | Stata `did_imputation, leaveout` | Borusyak, Jaravel & Spiess (2024) App. A.9 | ✓ Integrated |
| `WooldridgeDiD` / `CallawaySantAnna` | Stata `jwdid` / `csdid` (+ G≈20..500 SE ladder) | Wooldridge (2025) / Callaway & Sant'Anna (2021) | ✓ Integrated |
| Clustered CR1 `K_reference` | Stata `reghdfe` + R `fixest` (disconnected-panel arms) | reghdfe/fixest ssc conventions | ✓ Integrated |
Expand Down
106 changes: 106 additions & 0 deletions benchmarks/data/lpdid_nonabsorbing_stata_golden.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
{
"meta": {
"estimator": "LPDiD non-absorbing SEs - authors' Stata lpdid package, end-to-end",
"generator": "benchmarks/stata/generate_lpdid_nonabsorbing_golden.do",
"source_panel": "benchmarks/data/lpdid_nonabsorbing_panel.csv",
"source_sha256": "71af98c96dec08abc281002b7b88df74033dac7d0ff7707eff9f3ee0c30a3964",
"cmd": "lpdid y, unit(unit) time(time) treat(treat) pre_window(3) post_window(4) nograph + nonabsorbing(, firsttreat notyet) [rw] on the full panel; nonabsorbing(3) [rw] on the subsample and (only_event, vw) on the full panel",
"mapping": "nonabsorbing(, firsttreat notyet) == first_entry; nonabsorbing(L) == effect_stabilization",
"drop_rule": "min(treat)==1 | any(dD==1 & L3.dD==-1)",
"point_anchor": "first_entry vw tau0 gated in-.do vs the committed R golden first_entry[0] att (R-anchored); effect_stab_sub vw tau0 vs the library point (library-anchored, informational); the Python test is authoritative for every surface",
"se_convention": "package reghdfe vce(cluster unit) CR1; the library's cluster SE matches at ~1e-16 (vw) / ~1e-9 (rw) on identical samples",
"convention_notes": "effect_stab full-panel and placebo/pooled-pre rows are DIVERGENCE DOCUMENTATION, not parity: the package admits always-treated units at early t via missing-lag semantics, requires L+1 untreated periods before re-entry (stricter than JAE Eq. 13's levels condition), and builds placebo samples by recursive lagged intersection; see REGISTRY ## LPDiD Deviation 4",
"ssc_versions": {
"lpdid": "lpdid program, version 1.0.3 [checksum:3687664943 len:48365]",
"reghdfe": "version 6.13.1 10Jan2026 [checksum:973401092 len:30978]",
"ftools": "version 2.50.0 09jan2026 [checksum:3739654356 len:1428]",
"require": "version 1.3.1 19sep2023 [checksum:1262842970 len:34415]",
"boottest": "boottest 4.5.3 5 June 2026 [checksum:2334657244 len:63528]",
"listreg": "version 1.0.3 26apr2025 Ben Jann [checksum:1074534106 len:23815]",
"egenmore": "1.1.1 NJC 19 March 2006 [checksum:2753673312 len:1164]",
"egenmore_gclsst": "1.0.0 NJC 11 January 2000 [checksum:885642367 len:916]"
},
"dropped_units": [24, 25, 27, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40],
"stata_edition": "SE",
"stata_version": 19.0,
"pre_window": 3,
"post_window": 4,
"stabilization_window": 3,
"n_rows_full": 840,
"n_rows_sub": 658
},
"first_entry": {
"vw": {
"es": {
"-3": {"att": -0.11058688194906449, "se": 0.083864694854006674, "N": 300},
"-2": {"att": -0.028508009716748425, "se": 0.059950164863315426, "N": 350},
"0": {"att": 1.8468153374570833, "se": 0.091536864444450375, "N": 400},
"1": {"att": 1.9174189184921464, "se": 0.082514757717069953, "N": 350},
"2": {"att": 1.9541562370690497, "se": 0.095995993613636657, "N": 298},
"3": {"att": 1.8430002097850104, "se": 0.093215506869608694, "N": 246},
"4": {"att": 1.8651045890489937, "se": 0.10344600862300475, "N": 211}
},
"pooled": {
"pre": {"att": -0.083163290352517599, "se": 0.060216134633611737, "N": 300},
"post": {"att": 1.8463774549223959, "se": 0.082239499731901228, "N": 211}
}
},
"rw": {
"es": {
"-3": {"att": -0.10982753532939604, "se": 0.085127249305807715, "N": 180},
"-2": {"att": -0.027473431971800764, "se": 0.059909786016452311, "N": 230},
"0": {"att": 1.8483385269860748, "se": 0.093169339989145811, "N": 230},
"1": {"att": 1.9098467525590781, "se": 0.082755294278197722, "N": 203},
"2": {"att": 1.9542651398293733, "se": 0.095993108882509365, "N": 173},
"3": {"att": 1.844237043489463, "se": 0.094299729259659446, "N": 123},
"4": {"att": 1.8646288450757691, "se": 0.10524070978446542, "N": 118}
},
"pooled": {
"pre": {"att": -0.080380126038013866, "se": 0.061217606891681819, "N": 180},
"post": {"att": 1.8500925875939953, "se": 0.082980268093888879, "N": 118}
}
}
},
"effect_stab_sub": {
"n_rows": 658,
"vw": {
"es": {
"-3": {"att": -0.09724719295046115, "N": 342},
"-2": {"att": -0.007899289624071772, "N": 408},
"0": {"att": 1.8115854492490506, "se": 0.079386732581726835, "N": 484},
"1": {"att": 1.9300421146595237, "se": 0.072501471398279477, "N": 414},
"2": {"att": 1.8965697184052384, "se": 0.084395761100608244, "N": 348},
"3": {"att": 1.879554866833909, "se": 0.076256728580062952, "N": 279},
"4": {"att": 1.893416496899663, "se": 0.093768556793994276, "N": 230}
},
"pooled": {
"pre": {"att": -0.064656623253798173, "N": 342},
"post": {"att": 1.8538432413410768, "se": 0.072566538322928476, "N": 230}
}
},
"rw": {
"es": {
"-3": {"att": -0.09552710717227407, "N": 173},
"-2": {"att": -0.0054487064168838913, "N": 220},
"0": {"att": 1.813509946055728, "se": 0.081752780459597857, "N": 329},
"1": {"att": 1.9275161589672816, "se": 0.073720330196303835, "N": 305},
"2": {"att": 1.9026466911687787, "se": 0.085905795413003486, "N": 280},
"3": {"att": 1.8767692749265028, "se": 0.078313461669530404, "N": 191},
"4": {"att": 1.8899064444115865, "se": 0.096189906878782108, "N": 151}
},
"pooled": {
"pre": {"att": -0.061093488880953395, "N": 173},
"post": {"att": 1.8560600200442186, "se": 0.074103122657375023, "N": 151}
}
}
},
"effect_stab_full_vw": {
"es": {
"0": {"att": 1.8496401510611364, "N": 628},
"1": {"att": 1.9364029248640582, "N": 544},
"2": {"att": 1.9088763162649005, "N": 463},
"3": {"att": 1.9438383345066086, "N": 380},
"4": {"att": 1.9261821895864399, "N": 320}
}
}
}
Loading
Loading