Stamp dateModified on the entries save() rewrites - #43
Conversation
Lexicon.save() and save_zip() take stamp (default True) and when. Before serializing, stamp_entries() sets dateModified — filling a blank dateCreated with the same moment — on every entry whose canonical digest moved while its date stayed where its baseline had it. Until now nothing in the library generated a timestamp, so an edited entry went out under the date it was read with and every tool that reconciles LIFT on dateModified saw an unmodified lexicon. stamp=False writes the model exactly as it stands; when= supplies the moment in place of the wall clock (UTC at seconds precision, the 20-character form real FieldWorks exports use without exception), which is what makes stamped output byte-reproducible. Entries only. All 35,318 entries in the seven FieldWorks 8.3-9.0 exports in The Combine's Backend.Tests/Assets carry both stamps and not one of their 69,754 sub-entry nodes carries either, and an entry's digest already spans its whole subtree, so an edit to a nested subsense stamps the entry containing it. _ExtensibleNoFields._stamp holds the policy, so the other eight date-bearing types inherit it if they ever need it. _EntryRecord gains the dateModified it held when the record was taken, which is what separates "the content changed and the date did not" from "the caller set the date deliberately". The parse-time records keep driving byte reuse and change detection; each save records what it wrote in Lexicon._stamps and the next save measures against that, without which a second round of edits on one loaded lexicon would read as caller-set and ship unstamped. Keeping the two baselines apart is what leaves changed_entries() answering "since the load" rather than "since the last save". An entry still matching its parse-time record is recorded nowhere, so the bookkeeping is the size of the edit. An entry with no baseline at all — appended after the load, or in a lexicon built from scratch — is stamped only where its dateModified is blank, so an exporter carrying real dates in from another data model keeps them. Its first save records a baseline either way, so a later edit to it does bump. iter_problems() does not stamp. It stays read-only, and its docstring says the bytes it validates precede the stamping a save does rather than claiming to be what save() would write; nothing generated is ever a finding, so validating first and saving after is sound. canonicalize() generates nothing either: sorting and reformatting change no entry's content. An unparseable date is residue rather than a date, so a stamp replaces it and the original string is dropped — a consequence pinned by a test and documented alongside the rest in docs/en/fidelity.md, with the guides that teach editing and building an export, and the streaming writer's note that it stamps nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five corrections to entry stamping. A document the byte scanner declined had no stamping baseline at all, so every undated entry read as new and a save that changed nothing dated all of them. Byte reuse needs the source bytes but stamping needs only the digests, so the reader now records digests and dates for such a document too, and a no-op save of one leaves it alone. when= is normalized to UTC at whole seconds, so an explicit moment reaches the output in the one form the rest of it uses instead of carrying an offset or fractional seconds through _fmt_date. A naive value is refused: read as UTC and read as local time it names moments hours apart, and picking one silently writes a date the caller did not mean. Stamping now commits with the write. stamp_entries returns what undoes it, save() and save_zip() put the dates and the baseline back when the write does not go through, and the pass decides before it mutates so that the one step that can fail — digesting content XML cannot represent — refuses with nothing stamped rather than half-stamped at whatever entry the refusal came from. The baseline dict is rebuilt each pass rather than updated in place, so an entry appended and later removed is no longer held alive, with its whole subtree, by a record nothing will consult again. Entries the document was loaded with are retained by their parse-time records as before. Entries are iterated by identity, so an entry aliased into the list twice is decided and stamped once — it is one object with one pair of dates, whatever its output happens to say twice. default_now() documents what seconds precision costs: one second holds one date, so a second edit saved inside the same second as the first carries the same stamp. Sub-second precision would buy the distinction at the cost of the form every consumer expects, and when= forces a distinct moment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Internal vocabulary is out of the guides: a reader has no use for the word baseline, and the streaming note now says what the limitation is — a pass that never saw the document cannot tell which entries changed — rather than naming the machinery it lacks. Refused-write behavior is stated where it is a contract, in fidelity.md and on save(), and no longer repeated in a guide whose script aborts before it saves. The comments on the clock, the undo, the identity keying and the two-phase pass keep their reasons and drop the reassurance around them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Review round applied in Fixed
Not defects
608 tests (28 in 🤖 Generated with Claude Code |
Two paths left a date the caller set unstampable for good. A save that stamps nothing still writes what the model holds, so a date the caller put there is the date now on disk. note_caller_dates records it, and the next stamping save measures a further edit against it. Without that the entry kept failing the stale test — its date differs from the load, which is exactly what a deliberate date looks like — and was never stamped again, while the same sequence through a stamping save bumped normally. An entry written unstamped under the date it was loaded with is deliberately not noted: its content is on disk under a date that no longer describes it, and the next stamping save should still say so. Dates are compared as the document will carry them rather than as moments. Two aware values an hour and an offset apart are the same instant and equal to ==, so restating 2008-12-12T09:42:48+10:00 at -05:00 — an edit, and an edit to the date itself — read as content changed without its date, and the stamp overwrote the one thing the caller had touched. The parse-time record lookup and the identity keying move into helpers shared by both passes, and _apply_stamps now always returns an undo. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Second round applied in Fixed
Confirmations, no change
Already decided
611 tests (31 in 🤖 Generated with Claude Code |
Resolves #10, taking Route 1:
save()stamps the entries it rewrites, by default.What changed
Lexicon.save()andLexicon.save_zip()grow two keyword-only parameters,stamp: bool = Trueandwhen: datetime | None = None. Immediately before serializing,stamp_entries()setsdateModified— and fills a blankdateCreatedwith the same moment — on every entry whose canonical digest moved while itsdateModifiedstayed where its baseline had it.stamp=Falserestores today's behavior exactly;when=supplies the moment in place of the wall clock (UTC at seconds precision), which is what makes stamped output byte-reproducible.Entry-level only, as the survey in the issue concluded, and the digest already spans an entry's whole subtree, so an edit to a gloss on a nested subsense stamps the entry containing it. The policy lives in one place —
_ExtensibleNoFields._stamp— so the other eight date-bearing types inherit it if they ever need it. It stays private: nothing else in this change needs it, and Route 1 adds no public name.The decisions the issue left open
Entries with no baseline — appended after the load, or in a from-scratch lexicon — are stamped only where
dateModifiedis blank, rather than unconditionally as Route 1's text words it. Unconditional stamping would silently overwrite dates an exporter carried in from its own data model, and would rewrite every date on every save of a from-scratch lexicon. Such an entry's first save records a baseline either way, so a later edit to a date-carrying one does bump.iter_problems()stays read-only. Its docstring no longer claims to validate "whatsave()would write": it says the bytes it reports on precede the stamping a save does, and that nothing generated is ever a finding, so validate-then-save is sound. Same wording indocs/en/guides/validate.md.save_zip()takes the same two keywords and stamps by default — a package is the hand-off to the tools that reconcile ondateModified, so leaving it unstamped would reproduce the bug for the case that matters most.Baseline bookkeeping
_EntryRecordgains thedateModifiedit held when the record was taken, which is what separates "the content changed and the date did not" from "the caller set the date deliberately".The stamping baseline is deliberately not the parse-time one. Overwriting
_EntryRecord.digeston stamp, as the issue's groundwork sketch has it, would makechanged_entries()stop reporting a stamped entry after a save — contradicting its documented "always against the load, never against the most recentsave()" guarantee and the test that pins it. Instead each save records what it wrote inLexicon._stamps, and the next save measures against that; the parse-time records keep driving byte reuse and change detection untouched. That satisfies the groundwork's correctness requirement (a second round of edits on one loaded lexicon does bump —test_a_second_round_of_edits_on_the_same_lexicon_is_stamped_too) without moving the baseline anything else reads. An entry still matching its parse-time record is recorded nowhere, so the bookkeeping is the size of the edit, not the size of the lexicon.One fidelity consequence worth a look
An unparseable date (
dateModified="whenever") is residue, not a date, so the model field isNone, and an edited entry therefore gets stamped — replacing the original string, which astamp=Falsesave would have preserved. Pinned by a test and documented indocs/en/fidelity.md. The alternative, treating unparseable-as-present and skipping the stamp, would leave the worst data the only data that never gets a usable date.Cost
One extra canonical serialization pass over the entries per stamping save; Route 1's table claimed none. Sharing digests with
render_documentwould mean threading a cache through it and through_validate/_zip, which seemed worse than documenting the pass the waychanged_entries()andchanges()already document theirs.Not in scope
SOURCE_DATE_EPOCHas the default clock source (#9).default_now()in_writer.pyis the single place the clock is read, so that stays a one-function change on top.when=already makes the byte-exact tests here writable without monkeypatching.Tests
New
tests/test_stamp.py(20 tests): the depth case, only-the-edited-entry,dateCreatedfilled when blank and preserved when not,stamp=False, a caller-set date left alone and then becoming the next baseline,sort()stamping nothing, two full edit-and-stamp cycles over one lexicon, a re-save with no edits leaving the stamp alone, added and from-scratch entries, nothing stamped below the entry, the default clock's shape and range, byte-reproducibility underwhen=, validation not stamping,changed_entries()unchanged, bothsave_zippaths, and the residue case above.Docs
docs/en/fidelity.md(two bullets under "Saving an edited document", plus a note that an unchanged document generates nothing), the editing and build-an-export guides,docs/en/guides/validate.md,lift-export-interop.md,large-files.md(the streaming writer stamps nothing and why),read-edit-write.md,index.md,README.md, and the[0.1.0]changelog section.build-export.md's pasted output was regenerated by running its script.Verification
python scripts/check.pygreen — ruff,ruff format --check,mypy --strict, 600 tests, coverage 97.9% against the 95% floor.mkdocs build --strictgreen. The corpus byte-identity and Hypothesis round-trip suites pass unchanged, which is the point: stamping is driven by content, so an unedited document is stamped nowhere.🤖 Generated with Claude Code
This change is