endtoend: pin sqlite query text with multi-byte UTF-8 comments - #4568
Merged
Conversation
The ANTLR-based sqlite parser mixed rune-counted and byte-counted offsets when slicing query text out of the source file, so a multi-byte character in a comment silently truncated the generated query string and leaked fragments of neighboring statements into it. The meyer parser tracks byte offsets throughout, which fixed this; this case keeps it fixed. Covers the three reported shapes: an em dash in a comment directly above the annotation, a standalone comment between two queries, and CJK comments. Fixes #4523 Fixes #4235 Fixes #4372 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017bkCyEYanANjRyamjUF8ZP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The ANTLR-based sqlite parser mixed rune-counted and byte-counted offsets when slicing query text out of the source file, so a multi-byte UTF-8 character in a comment silently truncated the generated query string and leaked fragments of neighboring statements into it. Switching to the meyer parser (#4535) fixed this — verified by running the reproduction from #4523 at
98ef75d~1(truncated, with stray statement fragments) and at HEAD (byte-for-byte correct).Nothing in the end-to-end corpus contained any non-ASCII SQL, so the fix had no regression coverage. This adds a sqlite case covering the three reported shapes:
-- section — dividercomment between two queries, which stripped the trailing?placeholder (SQLite: non-ASCII character (em-dash) in SQL comment corrupts generated queries #4372)The case passes in the
basecontext and the opt-incorecontext.Fixes #4523
Fixes #4235
Fixes #4372
🤖 Generated with Claude Code
https://claude.ai/code/session_017bkCyEYanANjRyamjUF8ZP
Generated by Claude Code