[bab2min]

U+0085, U+2028, U+2029가 공백 문자로 처리되지 않아서 형태소 분석 결과에 붙어서 나오는 문제 수정

Summary by CodeRabbit

• Bug Fixes
  • Unicode next-line, line-separator, and paragraph-separator characters are now treated as whitespace, so text containing them between words is analyzed consistently with text containing regular spaces.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 71627178-c5ac-4195-99f6-c43823fa6d6e

📥 Commits

Reviewing files that changed from the base of the PR and between 060c61c and 8ecb159.

📒 Files selected for processing: 2

• include/kiwi/Utils.h
• test/test_cpp.cpp

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

📝 Walkthrough

isSpace now recognizes U+0085, U+2028, and U+2029. A new test checks that sentences using these separators produce the expected tokens, tags, and final-token position.

Changes: Unicode whitespace handling

Layer / File(s)	Summary
Recognize and test Unicode line separators include/kiwi/Utils.h, test/test_cpp.cpp	isSpace now returns true for U+0085, U+2028, and U+2029. The test compares analysis results for sentences containing each separator against a reference sentence.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

Suggested reviewers: hakhyun-kim

Merge Risk: ⚪ Minimal · up to 8ecb1

Analysis now treats U+0085, U+2028, and U+2029 as whitespace, with tests comparing their tokenization to ordinary-space input. No actionable merge-blocking issue is evident.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: fixing handling of Unicode line break characters.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches: 💡 1

📝 Generate docstrings: 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}