diff --git a/CHANGELOG.md b/CHANGELOG.md index 31560f2..9ad0463 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,9 +24,11 @@ releases may contain breaking changes. - Project scaffolding: package skeleton, vendored LIFT 0.13 RELAX NG schema, test corpus with provenance, corpus-prep and large-file-generator tooling. -- Full object model: all 35 LIFT 0.13 elements as typed dataclasses; - `sil_lift.load()` / `Lexicon.load()` full-document reader that keeps LIFT - residue per node in `Extras`; LIFT-version guard. +- Full object model: all 35 LIFT 0.13 elements as typed dataclasses. + `Entry.all_senses()` walks every subsense depth-first in document order, + which `Entry.senses` (top level only) does not. `sil_lift.load()` / + `Lexicon.load()` full-document reader that keeps LIFT residue per node in + `Extras`; LIFT-version guard. - `Lexicon.save()` writer with byte-fidelity passthrough — unchanged documents and untouched entries are written byte-identically; touched entries re-serialize canonically with all out-of-schema content preserved. Fidelity @@ -51,14 +53,12 @@ releases may contain breaking changes. against the most recent `save()`. - LIFT-folder handling: `RangesFile` (standalone `.lift-ranges` documents, same fidelity guarantees), automatic companion discovery/tracking on load - (`Lexicon.ranges_files`, matching companion filenames across case and - Unicode normalization differences), `save()` writes companions together, + (`Lexicon.ranges_files`), `save()` writes companions together, `all_ranges()` merged view, `media_refs()` / `missing_media()` helpers, build-from-scratch helpers `Lexicon.add_ranges_file()` / `RangesFile.add_range()` / `Range.add_element()` (`save()` writes and header-references a new companion beside the `.lift`); vendored - `schemas/lift-ranges-0.13.rng` — the first schema for standalone ranges - documents. + `schemas/lift-ranges-0.13.rng` for standalone ranges documents. - Zipped LIFT packages: `sil_lift.load()` reads a `.zip` (both the flat and folder-wrapped layouts, junk entries like `__MACOSX` ignored), `Lexicon.save_zip()` writes one (carrying media, `WritingSystems/`, and other diff --git a/docs/en/guides/bulk-edit-glosses.md b/docs/en/guides/bulk-edit-glosses.md index e5976cb..2a82496 100644 --- a/docs/en/guides/bulk-edit-glosses.md +++ b/docs/en/guides/bulk-edit-glosses.md @@ -12,18 +12,10 @@ import sil_lift path = "dictionary.lift" lex = sil_lift.load(path) - -def iter_senses(senses): - """Yield every sense, including subsenses (recursive).""" - for sense in senses: - yield sense - yield from iter_senses(sense.subsenses) - - edited_glosses = 0 for entry in lex.entries: - for sense in iter_senses(entry.senses): + for sense in entry.all_senses(): for gloss in sense.glosses: if gloss.lang != "en": continue @@ -47,7 +39,8 @@ print(f"edited {edited_glosses} gloss(es) across {len(changed)} entry(ies)") A few things worth noting: -- `Sense.subsenses` is itself a `list[Sense]`, so `iter_senses` recurses into it — a bulk edit that only walked `entry.senses` would silently skip any gloss nested under a subsense. +- `entry.all_senses()` yields every sense _and subsense_, depth-first in document order. + - `entry.senses` holds only the top level, so a bulk edit that walked it would silently skip any gloss nested under a subsense. - `gloss.text` is a `Text`, not a plain string: `str(gloss.text)` flattens it for matching, and the replacement is written back with `sil_lift.Text([new])` rather than mutating the string in place. - `lex.changed_entries()` reports which entries differ from the file as loaded. Since an entry's digest covers its whole subtree, an edit to a nested subsense reports the entry that contains it. - It compares serialized content, so assigning a field the value it already had isn't reported. diff --git a/docs/en/guides/folder-media.md b/docs/en/guides/folder-media.md index 16497fb..420134c 100644 --- a/docs/en/guides/folder-media.md +++ b/docs/en/guides/folder-media.md @@ -12,8 +12,6 @@ lex.all_ranges() # merged {id: Range} view lex.all_ranges()["grammatical-info"].elements ``` -Companion discovery handles the real world: a `range/@href` that points at an existing file is used; FieldWorks' dangling absolute `file://C:/...` hrefs fall back to the href's basename next to the `.lift`; and the conventional `.lift-ranges` sibling is picked up even when nothing references it. - `lex.save()` writes the `.lift` and every tracked companion together. Edits to a `RangesFile` save back to _its_ file; untouched ranges keep their exact bytes. Standalone use: ```python @@ -23,6 +21,16 @@ ranges.sort() ranges.save() ``` +### Companion discovery + +Several candidates are tried, and every distinct file among them is loaded. + +- A header `range/@href` that points at an existing file is used as given. +- An href that resolves to nothing falls back to its basename next to the `.lift` — FieldWorks writes dangling absolute `file://C:/...` paths from the exporting machine, and that fallback is what makes them work locally. +- The conventional `.lift-ranges` sibling is picked up even when nothing references it. + +Names that differ only in case or Unicode normalization still match — `Dict.LIFT` finds `Dict.lift-ranges` — unless several files match one name, which loads none of them and is reported as [`ambiguous-ranges-file`](validate.md#problem-codes). + Pass `resolve_ranges=False` to `load()` to skip companion discovery. ## Media diff --git a/docs/en/guides/read-edit-write.md b/docs/en/guides/read-edit-write.md index c7fe813..5f4f7a2 100644 --- a/docs/en/guides/read-edit-write.md +++ b/docs/en/guides/read-edit-write.md @@ -24,14 +24,17 @@ entry.lexical_unit["en"] = "grove" # plain strings are coerced `Text` is structured — an ordered list of `str` and `Span` fragments — because `` can contain nested `` markup. `str(text)` flattens to plain text; the fragments keep the markup for round-tripping. -Glosses are _form-shaped_ in LIFT (each `` carries its own language), so a sense has `glosses: list[Form]` plus a helper: +Glosses are _form-shaped_ in LIFT (each `` carries its own language), so a sense has `glosses: list[Form]` plus helpers: ```python -sense = entry.senses[0] +sense = entry.senses[0] # top level only sense.gloss("en") # Text | None -entry.gloss_langs() # {"en", "id"} +entry.all_senses() # every sense and subsense, document order +entry.gloss_langs() # {"en", "id"}, subsenses included ``` +Reach for `all_senses()` whenever a question concerns the whole entry: counting senses, collecting languages, finding media. `entry.senses` gives the top level, which is what you want only when the nesting itself matters. + ## Saving ```python diff --git a/src/sil_lift/_cli.py b/src/sil_lift/_cli.py index 6e09d58..1dcd3c1 100644 --- a/src/sil_lift/_cli.py +++ b/src/sil_lift/_cli.py @@ -30,7 +30,7 @@ from ._validate import iter_problems if TYPE_CHECKING: - from collections.abc import Iterator, Sequence + from collections.abc import Sequence from typing import BinaryIO, TextIO from ._model import Entry, Sense @@ -95,16 +95,6 @@ def _cmd_validate(args: argparse.Namespace) -> int: return 1 if failed else 0 -def _iter_senses(entry: Entry) -> list[Sense]: - senses: list[Sense] = [] - stack = list(entry.senses) - while stack: - sense = stack.pop() - senses.append(sense) - stack.extend(sense.subsenses) - return senses - - def _cmd_stats(args: argparse.Namespace) -> int: from ._zip import lift_source @@ -121,7 +111,7 @@ def _cmd_stats(args: argparse.Namespace) -> int: pronunciations.extend(variant.pronunciations) for pronunciation in pronunciations: media += len(pronunciation.media) - for sense in _iter_senses(entry): + for sense in entry.all_senses(): senses += 1 examples += len(sense.examples) media += len(sense.illustrations) @@ -195,18 +185,14 @@ def _cmd_check_media(args: argparse.Namespace) -> int: return 1 if missing else 0 -def _iter_leaf_senses(senses: Sequence[Sense]) -> Iterator[Sense]: - """Depth-first leaf senses, document order. +def _leaf_senses(entry: Entry) -> list[Sense]: + """The entry's senses that carry content, document order. A sense with subsenses is a LIFT grouping node (e.g. numbered "1a"/"1b" under a bare "1") whose own gloss/definition are conventionally empty — its subsenses carry the content and get the rows instead. """ - for sense in senses: - if sense.subsenses: - yield from _iter_leaf_senses(sense.subsenses) - else: - yield sense + return [sense for sense in entry.all_senses() if not sense.subsenses] def _text_or_empty(text: Text | None) -> str: @@ -265,7 +251,7 @@ def _cmd_export(args: argparse.Namespace) -> int: detected: set[str] = set() with open_reader(lift_path) as reader: for entry in reader: - for sense in _iter_leaf_senses(entry.senses): + for sense in _leaf_senses(entry): detected.update(g.lang for g in sense.glosses if g.lang is not None) detected.update(sense.definition.keys()) langs = sorted(detected) @@ -294,7 +280,7 @@ def _cmd_export(args: argparse.Namespace) -> int: for entry in reader: forms = entry.lexical_unit.forms lexeme = str(forms[0].text) if forms else "" - for sense in _iter_leaf_senses(entry.senses): + for sense in _leaf_senses(entry): pos = sense.grammatical_info.value if sense.grammatical_info else "" row = [entry.id or "", entry.guid or "", sense.id or "", lexeme, pos] for lang in langs: diff --git a/src/sil_lift/_model.py b/src/sil_lift/_model.py index 0bdf4af..94b29b7 100644 --- a/src/sil_lift/_model.py +++ b/src/sil_lift/_model.py @@ -224,17 +224,31 @@ class Entry(_Extensible): relations: list[Relation] = field(default_factory=list) etymologies: list[Etymology] = field(default_factory=list) + def all_senses(self) -> list[Sense]: + """Every sense and subsense, depth-first in document order. + + LIFT nests senses arbitrarily deep, so anything asking a question of + "the entry's senses" — how many there are, what languages they gloss, + what media they reference — means this list rather than + :attr:`senses`, which holds only the top level. + """ + return list(_walk_senses(self.senses)) + def gloss_langs(self) -> set[str]: """Every language that has a gloss in any sense or subsense.""" langs: set[str] = set() - stack = list(self.senses) - while stack: - sense = stack.pop() + for sense in self.all_senses(): langs.update(g.lang for g in sense.glosses if g.lang is not None) - stack.extend(sense.subsenses) return langs +def _walk_senses(senses: list[Sense]) -> Iterator[Sense]: + """Depth-first pre-order over a sense list and every subsense under it.""" + for sense in senses: + yield sense + yield from _walk_senses(sense.subsenses) + + @dataclass(slots=True) class MediaRef: """One media reference in the document, with its owner's identity.""" @@ -946,14 +960,11 @@ def media_refs(self) -> Iterator[MediaRef]: for pronunciation in pronunciations: for media in pronunciation.media: yield MediaRef(media.href, "media", entry.id, entry.guid) - stack = list(entry.senses) - while stack: - sense = stack.pop() + for sense in entry.all_senses(): for illustration in sense.illustrations: yield MediaRef( illustration.href, "illustration", entry.id, entry.guid, sense.id ) - stack.extend(sense.subsenses) def missing_media(self) -> list[MediaRef]: """Media references whose files don't exist in the LIFT folder layout. diff --git a/src/sil_lift/_reader.py b/src/sil_lift/_reader.py index ef98f1d..0eff1e9 100644 --- a/src/sil_lift/_reader.py +++ b/src/sil_lift/_reader.py @@ -87,6 +87,8 @@ def _attach_ranges_source(ranges_file: RangesFile, data: bytes, root: etree._Ele from ._scan import scan from ._writer import _RangeRecord, _RangesSourceInfo, range_digest + # As in _attach_source: byte regions only mean anything in an + # ASCII-compatible encoding, and only this check rules the others out. encoding = root.getroottree().docinfo.encoding if encoding is not None and encoding.lower() not in ("utf-8", "us-ascii", "ascii"): return @@ -119,9 +121,13 @@ def _attach_source(lexicon: Lexicon, data: bytes, root: etree._Element) -> None: from ._scan import scan from ._writer import _EntryRecord, _SourceInfo, entry_digest, header_digest + # The regions scan reports are offsets into these bytes, and the writer + # splices them into a document it declares UTF-8. expat parses UTF-16 + # perfectly well and would report offsets into UTF-16 bytes, tags and all, + # so nothing downstream would notice; this is the only thing that says no. encoding = root.getroottree().docinfo.encoding if encoding is not None and encoding.lower() not in ("utf-8", "us-ascii", "ascii"): - return # byte scanning assumes an ASCII-compatible encoding + return result = scan(data) if result is None: return diff --git a/src/sil_lift/_scan.py b/src/sil_lift/_scan.py index f630681..f712d50 100644 --- a/src/sil_lift/_scan.py +++ b/src/sil_lift/_scan.py @@ -2,9 +2,10 @@ The writer emits untouched entries verbatim from their original bytes, which requires knowing each top-level ````'s (and ``
``'s) exact byte -region in the source. lxml exposes no byte offsets, so this module walks the -raw bytes with a small state machine that understands tags, quoted attribute -values, comments, CDATA sections, and processing instructions. +region in the source. lxml exposes no byte offsets, but the stdlib's expat +binding does: ``CurrentByteIndex`` reports where the current event's markup +begins, which is a region's start at the element's start event and — bar the +empty-element wrinkle noted below — its end at the matching end event. "Region" rather than "span" throughout: LIFT has a ```` element for inline markup, modelled as :class:`~sil_lift.Span`, and the two would @@ -13,18 +14,19 @@ What it exists for is byte identity, not diagnostics — ``docs/en/fidelity.md`` states the guarantee it underpins. Problem reporting needs only the line an element starts on and takes that from lxml's ``sourceline`` (see -``_validate._line``); a region needs the end offset too, which no parser API +``_validate._line``); a region needs the end offset too, which no tree API exposes. It is deliberately conservative: anything unexpected (DOCTYPE, malformed -nesting, non-ASCII-compatible encoding — checked by the caller) returns -``None`` and the writer falls back to canonical serialization, which keeps -the semantic guarantee and waives only byte identity. +markup, non-ASCII-compatible encoding — checked by the caller) returns ``None`` +and the writer falls back to canonical serialization, which keeps the semantic +guarantee and waives only byte identity. """ from __future__ import annotations from dataclasses import dataclass +from xml.parsers import expat __all__ = ["ChildRegion", "ScanResult", "scan"] @@ -44,146 +46,95 @@ class ScanResult: children: list[ChildRegion] # document order; empty for a self-closing root -def _skip_comment(data: bytes, i: int) -> int | None: - end = data.find(b"-->", i + 4) - return None if end < 0 else end + 3 +class _Unscannable(Exception): + """Raised inside a handler to abandon the scan; ``scan`` returns None.""" -def _skip_pi(data: bytes, i: int) -> int | None: - end = data.find(b"?>", i + 2) - return None if end < 0 else end + 2 +def _tag_end(data: bytes, start: int) -> int: + """Just past the ``>`` of the start tag beginning at ``start``. - -def _skip_cdata(data: bytes, i: int) -> int | None: - end = data.find(b"]]>", i + 9) - return None if end < 0 else end + 3 - - -def _skip_tag(data: bytes, i: int) -> tuple[int, bool] | None: - """From ``<`` of a start/end tag to just past ``>``; reports self-closing.""" - n = len(data) - j = i + 1 + An attribute value may hold a ``>``, so this tracks quoting rather than + searching for the delimiter. End tags take no attributes and so need no + such care. + """ quote: int | None = None - while j < n: - c = data[j] + for index in range(start + 1, len(data)): + char = data[index] if quote is not None: - if c == quote: + if char == quote: quote = None - elif c in (0x22, 0x27): # " or ' - quote = c - elif c == 0x3E: # > - return j + 1, data[j - 1] == 0x2F # preceded by / - j += 1 - return None - - -def _tag_name(data: bytes, i: int) -> str: - j = i + 1 - n = len(data) - while j < n and data[j] not in b" \t\r\n/>": - j += 1 - return data[i + 1 : j].decode("utf-8", errors="replace") - - -def _skip_element(data: bytes, i: int) -> int | None: - """From ``<`` of a start tag to just past the matching end tag.""" - step = _skip_tag(data, i) - if step is None: - return None - pos, self_closing = step - if self_closing: - return pos - depth = 1 - n = len(data) - while depth > 0: - lt = data.find(b"<", pos) - if lt < 0: - return None - if data.startswith(b"