Skip to content

Stamp dateModified on the entries save() rewrites - #43

Draft
imnasnainaec wants to merge 4 commits into
mainfrom
stamp-date-modified-on-save
Draft

Stamp dateModified on the entries save() rewrites#43
imnasnainaec wants to merge 4 commits into
mainfrom
stamp-date-modified-on-save

Conversation

@imnasnainaec

@imnasnainaec imnasnainaec commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Resolves #10, taking Route 1: save() stamps the entries it rewrites, by default.

What changed

Lexicon.save() and Lexicon.save_zip() grow two keyword-only parameters, stamp: bool = True and when: datetime | None = None. Immediately before serializing, stamp_entries() sets dateModified — and fills a blank dateCreated with the same moment — on every entry whose canonical digest moved while its dateModified stayed where its baseline had it. stamp=False restores today's behavior exactly; when= supplies the moment in place of the wall clock (UTC at seconds precision), which is what makes stamped output byte-reproducible.

Entry-level only, as the survey in the issue concluded, and the digest already spans an entry's whole subtree, so an edit to a gloss on a nested subsense stamps the entry containing it. The policy lives in one place — _ExtensibleNoFields._stamp — so the other eight date-bearing types inherit it if they ever need it. It stays private: nothing else in this change needs it, and Route 1 adds no public name.

The decisions the issue left open

Entries with no baseline — appended after the load, or in a from-scratch lexicon — are stamped only where dateModified is blank, rather than unconditionally as Route 1's text words it. Unconditional stamping would silently overwrite dates an exporter carried in from its own data model, and would rewrite every date on every save of a from-scratch lexicon. Such an entry's first save records a baseline either way, so a later edit to a date-carrying one does bump.

iter_problems() stays read-only. Its docstring no longer claims to validate "what save() would write": it says the bytes it reports on precede the stamping a save does, and that nothing generated is ever a finding, so validate-then-save is sound. Same wording in docs/en/guides/validate.md.

save_zip() takes the same two keywords and stamps by default — a package is the hand-off to the tools that reconcile on dateModified, so leaving it unstamped would reproduce the bug for the case that matters most.

Baseline bookkeeping

_EntryRecord gains the dateModified it held when the record was taken, which is what separates "the content changed and the date did not" from "the caller set the date deliberately".

The stamping baseline is deliberately not the parse-time one. Overwriting _EntryRecord.digest on stamp, as the issue's groundwork sketch has it, would make changed_entries() stop reporting a stamped entry after a save — contradicting its documented "always against the load, never against the most recent save()" guarantee and the test that pins it. Instead each save records what it wrote in Lexicon._stamps, and the next save measures against that; the parse-time records keep driving byte reuse and change detection untouched. That satisfies the groundwork's correctness requirement (a second round of edits on one loaded lexicon does bump — test_a_second_round_of_edits_on_the_same_lexicon_is_stamped_too) without moving the baseline anything else reads. An entry still matching its parse-time record is recorded nowhere, so the bookkeeping is the size of the edit, not the size of the lexicon.

One fidelity consequence worth a look

An unparseable date (dateModified="whenever") is residue, not a date, so the model field is None, and an edited entry therefore gets stamped — replacing the original string, which a stamp=False save would have preserved. Pinned by a test and documented in docs/en/fidelity.md. The alternative, treating unparseable-as-present and skipping the stamp, would leave the worst data the only data that never gets a usable date.

Cost

One extra canonical serialization pass over the entries per stamping save; Route 1's table claimed none. Sharing digests with render_document would mean threading a cache through it and through _validate/_zip, which seemed worse than documenting the pass the way changed_entries() and changes() already document theirs.

Not in scope

SOURCE_DATE_EPOCH as the default clock source (#9). default_now() in _writer.py is the single place the clock is read, so that stays a one-function change on top. when= already makes the byte-exact tests here writable without monkeypatching.

Tests

New tests/test_stamp.py (20 tests): the depth case, only-the-edited-entry, dateCreated filled when blank and preserved when not, stamp=False, a caller-set date left alone and then becoming the next baseline, sort() stamping nothing, two full edit-and-stamp cycles over one lexicon, a re-save with no edits leaving the stamp alone, added and from-scratch entries, nothing stamped below the entry, the default clock's shape and range, byte-reproducibility under when=, validation not stamping, changed_entries() unchanged, both save_zip paths, and the residue case above.

Docs

docs/en/fidelity.md (two bullets under "Saving an edited document", plus a note that an unchanged document generates nothing), the editing and build-an-export guides, docs/en/guides/validate.md, lift-export-interop.md, large-files.md (the streaming writer stamps nothing and why), read-edit-write.md, index.md, README.md, and the [0.1.0] changelog section. build-export.md's pasted output was regenerated by running its script.

Verification

python scripts/check.py green — ruff, ruff format --check, mypy --strict, 600 tests, coverage 97.9% against the 95% floor. mkdocs build --strict green. The corpus byte-identity and Hypothesis round-trip suites pass unchanged, which is the point: stamping is driven by content, so an unedited document is stamped nowhere.

🤖 Generated with Claude Code


This change is Reviewable

imnasnainaec and others added 3 commits September 2, 2026 15:30
Lexicon.save() and save_zip() take stamp (default True) and when. Before
serializing, stamp_entries() sets dateModified — filling a blank dateCreated
with the same moment — on every entry whose canonical digest moved while its
date stayed where its baseline had it. Until now nothing in the library
generated a timestamp, so an edited entry went out under the date it was read
with and every tool that reconciles LIFT on dateModified saw an unmodified
lexicon. stamp=False writes the model exactly as it stands; when= supplies the
moment in place of the wall clock (UTC at seconds precision, the 20-character
form real FieldWorks exports use without exception), which is what makes
stamped output byte-reproducible.

Entries only. All 35,318 entries in the seven FieldWorks 8.3-9.0 exports in The
Combine's Backend.Tests/Assets carry both stamps and not one of their 69,754
sub-entry nodes carries either, and an entry's digest already spans its whole
subtree, so an edit to a nested subsense stamps the entry containing it.
_ExtensibleNoFields._stamp holds the policy, so the other eight date-bearing
types inherit it if they ever need it.

_EntryRecord gains the dateModified it held when the record was taken, which is
what separates "the content changed and the date did not" from "the caller set
the date deliberately". The parse-time records keep driving byte reuse and
change detection; each save records what it wrote in Lexicon._stamps and the
next save measures against that, without which a second round of edits on one
loaded lexicon would read as caller-set and ship unstamped. Keeping the two
baselines apart is what leaves changed_entries() answering "since the load"
rather than "since the last save". An entry still matching its parse-time
record is recorded nowhere, so the bookkeeping is the size of the edit.

An entry with no baseline at all — appended after the load, or in a lexicon
built from scratch — is stamped only where its dateModified is blank, so an
exporter carrying real dates in from another data model keeps them. Its first
save records a baseline either way, so a later edit to it does bump.

iter_problems() does not stamp. It stays read-only, and its docstring says the
bytes it validates precede the stamping a save does rather than claiming to be
what save() would write; nothing generated is ever a finding, so validating
first and saving after is sound. canonicalize() generates nothing either:
sorting and reformatting change no entry's content.

An unparseable date is residue rather than a date, so a stamp replaces it and
the original string is dropped — a consequence pinned by a test and documented
alongside the rest in docs/en/fidelity.md, with the guides that teach editing
and building an export, and the streaming writer's note that it stamps nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five corrections to entry stamping.

A document the byte scanner declined had no stamping baseline at all, so
every undated entry read as new and a save that changed nothing dated all of
them. Byte reuse needs the source bytes but stamping needs only the digests,
so the reader now records digests and dates for such a document too, and a
no-op save of one leaves it alone.

when= is normalized to UTC at whole seconds, so an explicit moment reaches
the output in the one form the rest of it uses instead of carrying an offset
or fractional seconds through _fmt_date. A naive value is refused: read as UTC
and read as local time it names moments hours apart, and picking one silently
writes a date the caller did not mean.

Stamping now commits with the write. stamp_entries returns what undoes it,
save() and save_zip() put the dates and the baseline back when the write does
not go through, and the pass decides before it mutates so that the one step
that can fail — digesting content XML cannot represent — refuses with nothing
stamped rather than half-stamped at whatever entry the refusal came from.

The baseline dict is rebuilt each pass rather than updated in place, so an
entry appended and later removed is no longer held alive, with its whole
subtree, by a record nothing will consult again. Entries the document was
loaded with are retained by their parse-time records as before.

Entries are iterated by identity, so an entry aliased into the list twice is
decided and stamped once — it is one object with one pair of dates, whatever
its output happens to say twice.

default_now() documents what seconds precision costs: one second holds one
date, so a second edit saved inside the same second as the first carries the
same stamp. Sub-second precision would buy the distinction at the cost of the
form every consumer expects, and when= forces a distinct moment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Internal vocabulary is out of the guides: a reader has no use for the word
baseline, and the streaming note now says what the limitation is — a pass that
never saw the document cannot tell which entries changed — rather than naming
the machinery it lacks.

Refused-write behavior is stated where it is a contract, in fidelity.md and on
save(), and no longer repeated in a guide whose script aborts before it saves.
The comments on the clock, the undo, the identity keying and the two-phase pass
keep their reasons and drop the reassurance around them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@imnasnainaec

Copy link
Copy Markdown
Collaborator Author

Review round applied in cd57c06, with the comment prose tightened in 49efb2b. Five of the seven findings were real; all verified fixed against the code rather than reasoned about.

Fixed

  • Unscannable documents were stamped on a no-op save. With no _source, every undated entry looked new. Byte reuse needs the source bytes but stamping needs only the digests, so _reader._attach_stamp_baseline now records digests and dates when the scan is declined. A no-op save of the UTF-16 fixture leaves it alone; a real edit still stamps. This is the one finding with a wrong-output consequence, and the review was right to lead with it.
  • when= bypassed normalization. Verified: naive → 2020-01-02T03:04:05 (no Z), microseconds → ...05.500000Z, offset → ...05+05:30. New resolve_when normalizes an aware value to UTC whole seconds and refuses a naive one — read as UTC and read as local it names moments hours apart, so guessing would silently write a date the caller did not mean.
  • Stamping now commits with the write. stamp_entries returns a _StampUndo; save() and save_zip() restore the dates and the baseline when the write does not go through. The pass also decides before it mutates, because digesting is the only step that can fail — previously a lone surrogate raised from inside the loop, leaving entries ahead of it stamped with nothing written. That partial-mutation window was the real exposure here, not the retry behavior the finding described: a corrected retry already re-stamped correctly (verified).
  • Removed entries no longer leak. The baseline dict is rebuilt each pass instead of updated in place. Loaded entries were already retained by their parse-time records by design, so this only stops an appended-then-removed entry being kept alive with its subtree.
  • Same-second edits are documented rather than worked around, on default_now(), in save(), and in fidelity.md, with a test pinning it. Sub-second output would break the one form every consumer uses, and a monotonic guard would have to skip explicit when= to keep it reproducible — worth revisiting only if it bites someone.

Not defects

  • Change detection keeps its original baseline — that separation is the design, and folding the stamp into _EntryRecord.digest is precisely what would break changed_entries()'s "since the load" contract.
  • Aliased entries avoid duplicate stamping — two list slots referencing one Entry share one pair of dates, so there is no second stamp to generate. Entries are now iterated by identity to make that explicit, and a test pins it: one stamp, one baseline, the date written twice.

608 tests (28 in tests/test_stamp.py), coverage 97.9%, mkdocs build --strict green.

🤖 Generated with Claude Code

Two paths left a date the caller set unstampable for good.

A save that stamps nothing still writes what the model holds, so a date the
caller put there is the date now on disk. note_caller_dates records it, and the
next stamping save measures a further edit against it. Without that the entry
kept failing the stale test — its date differs from the load, which is exactly
what a deliberate date looks like — and was never stamped again, while the same
sequence through a stamping save bumped normally. An entry written unstamped
under the date it was loaded with is deliberately not noted: its content is on
disk under a date that no longer describes it, and the next stamping save
should still say so.

Dates are compared as the document will carry them rather than as moments. Two
aware values an hour and an offset apart are the same instant and equal to ==,
so restating 2008-12-12T09:42:48+10:00 at -05:00 — an edit, and an edit to the
date itself — read as content changed without its date, and the stamp
overwrote the one thing the caller had touched.

The parse-time record lookup and the identity keying move into helpers shared
by both passes, and _apply_stamps now always returns an undo.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@imnasnainaec

Copy link
Copy Markdown
Collaborator Author

Second round applied in fadf3b3. Two of the five were real and are fixed; two were confirmations of the previous round, and one is a decision already taken.

Fixed

  • stamp=False left a caller-dated entry unstampable for good. Verified: set a date by hand, save(stamp=False), edit again, save() — the entry stayed on the hand-set date, while the identical sequence through a stamping save bumped correctly. So whether a later edit got stamped depended on which kind of save happened to run in between. note_caller_dates now moves the baseline onto a date the caller set, because an unstamped save still writes that date to disk, which is the same adoption a stamping save makes for the stamps it writes. Only a date that moved is noted, and only that entry is digested.

    Deliberately not noted: an entry written unstamped under the date it was loaded with. Its content is on disk under a date that no longer describes it, and the next stamping save should still say so — pinned by test_an_unstamped_save_leaves_an_untouched_date_to_the_next_stamp, since noting everything here would have suppressed that stamp.

  • The same instant in another offset was overwritten. Verified on the corpus: reassigning the loaded 2008-12-12T09:42:48+10:00 as 2008-12-11T18:42:48-05:00 — equal under ==, a different attribute value, and the only thing the caller changed — was read as content edited without its date, and the stamp clobbered it. Dates are now compared as the document will carry them (_dates_differ), so restating an offset counts as the caller touching the date and is left alone.

Confirmations, no change

  • Primary-write commit boundary and unscannable inputs avoid false stamps both describe the previous round's fixes working as intended. The commit boundary is deliberate and carries an inline comment saying why: once the .lift is on disk with those stamps, restoring them would put the model at odds with the file.

Already decided

  • Same-second edits was raised last round and settled then: documented on default_now(), in save(), and in fidelity.md, with test_two_saves_of_one_moment_share_a_stamp pinning it. Sub-second output would break the one 20-character form every consumer expects, and a monotonic guard would have to exempt explicit when= to stay reproducible — leaving two clocks with different behavior. Happy to add the guard if the same-second window turns out to matter in practice, but it is not obviously an improvement.

611 tests (31 in tests/test_stamp.py), coverage 97.8%, mkdocs build --strict green.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update dateModified when a lexicon is changed: two candidate designs

1 participant