parser/pageparser: Preserve non-ASCII whitespace after e.g. summary divider#15047
parser/pageparser: Preserve non-ASCII whitespace after e.g. summary divider#15047bep wants to merge 1 commit into
Conversation
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. |
bep
force-pushed
the
fix/ideographicspace
branch
from
d0caf8b to
fc88af9
Compare
There was a problem hiding this comment.
Pull request overview
This PR updates the page parser’s whitespace consumption logic to treat only ASCII whitespace as skippable in key lexer paths, preserving meaningful non-ASCII whitespace (e.g. U+3000 ideographic space) after the summary divider and similar constructs.
Changes:
- Replace
unicode.IsSpace-based skipping with anisASCIISpacehelper forconsumeSpace,consumeToSpace, andindexNonWhiteSpace. - Update shortcode and intro lexers to use the new ASCII-space predicate.
- Add a regression test ensuring ideographic space after
<!--more-->is preserved in the resulting text item.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
parser/pageparser/pageparser_test.go |
Adds a regression test for preserving U+3000 after the summary divider. |
parser/pageparser/pagelexer.go |
Switches whitespace skipping/indexing helpers from Unicode space to ASCII space, and introduces isASCIISpace. |
parser/pageparser/pagelexer_shortcode.go |
Updates shortcode lexing to use isASCIISpace for whitespace handling. |
parser/pageparser/pagelexer_intro.go |
Updates intro lexing to use isASCIISpace when skipping leading whitespace. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| c.Assert(err, qt.IsNil) | ||
|
|
||
| textItem := items[2] | ||
| c.Assert(textItem.ValStr(input), qt.Equals, "\u3000bbb") |
| return l.errorf("unclosed shortcode action") | ||
| case isSpace(r), isEndOfLine(r): | ||
| case isASCIISpace(r), isEndOfLine(r): | ||
| l.ignore() |
| case !isASCIISpace(r) && !isEndOfLine(r): | ||
| break LOOP |
…ivider Make it insted consume just ASCII whitespace, which preserves e.g. ideographic space (U+3000) after the summary divider, which is important for e.g. Chinese and Japanese content, and possibly other Unicode whitespace characters with meaning. Doing this is possibly breaking, but not likely, and obviously the correct thing to do.
bep
force-pushed
the
fix/ideographicspace
branch
from
fc88af9 to
322e599
Compare
Make it insted consume just ASCII whitespace, which preserves e.g. ideographic space (U+3000) after the summary divider, which is important for e.g. Chinese and Japanese content, and possibly other Unicode whitespace characters with meaning.
Doing this is possibly breaking, but nit likely, and obviously the correct thing to do.