Chinese Word Boundary Recovery through Character Alignment Projection

Recovering word boundaries in noisy learner Chinese by projecting the segmentation of the corrected sentence through character alignment

arXiv 2026

Lusha Wang1,†, Yuchen Li1,†, Su Yuan1,†, Jungyeul Park2,*

1The University of British Columbia, Vancouver, Canada 2Korea Advanced Institute of Science & Technology (KAIST), Daejeon, South Korea

†Equally contributed authors   *Corresponding author

Paper (arXiv) PDF
Line chart of token-level F1 versus character-level noise from 1% to 10% on the synthetic CTB-derived benchmark. All three lines decline as noise increases; direct segmentation (D) stays well below identical-character projection (P1) and similarity-based projection (P2) at every level, with P2 slightly above P1 at higher noise.
Figure 1. Token-level F1 on the synthetic CTB-derived benchmark under increasing character-level noise (1%–10% of characters substituted, deleted, or inserted). Direct segmentation (D) stays clearly below both projection systems at every noise level; the full two-step projection (P2) is at or slightly above identical-character projection (P1), and the gap widens as noise grows.

Abstract

Chinese word segmentation is especially fragile in non-standard text, where language learner errors and other character-level divergences disrupt the word boundaries assumed by downstream annotation and evaluation. This paper formulates Chinese word boundary recovery as an alignment-based projection task. Given a noisy source sentence and a cleaner target counterpart, we first align the two strings at the character level and then project target-side word boundaries back onto the source. Beyond the recovery method itself, we introduce two evaluation resources: a manually checked learner Chinese benchmark based on MuCGEC and a controlled synthetic benchmark derived from the Chinese Penn Treebank. Experiments show that direct segmentation remains vulnerable to compound fragmentation in learner input, whereas the proposed two step projection method corrects many over-segmentation errors by using the corrected target to recover source-side word spans. The results show that word boundary recovery is distinct from ordinary segmentation and that alignment projection provides a principled mechanism for stabilizing Chinese annotation and evaluation under noisy input.

Keywords Chinese word segmentation (CWS) Word boundary recovery Learner Chinese CGEC Character alignment Annotation projection MuCGEC Chinese Penn Treebank (CTB) Noisy text

Method

Example of the problem: a learner writes 旅行困 lǜxíng kùn (‘travel + tired’) for 旅行团 lǜxíngtuán (‘tour group’). A segmenter splits the learner form as 旅行 | 困 but treats the corrected form as one word, so edits and feedback attach to distorted spans. Given a learner source S and its corrected target T with word segmentation, the method recovers source-side word boundaries without changing the learner string:

  1. P1: identical-character anchors. A Levenshtein-style dynamic program builds an order-preserving edit path between S and T; only identical-character matches are kept as high-precision links.
  2. P2: residual similarity alignment. Remaining unaligned characters are linked one-to-one using a weighted score of glyph similarity, phonological similarity, relative position, and contextual semantic similarity.
  3. Deterministic boundary projection. Starting from the source's initial segmentation, a target boundary is inserted only when the two adjacent target characters align to adjacent source characters, and a source-internal boundary is removed only when a target word projects to a complete contiguous source span; ambiguous regions keep the initial segmentation.

Two new benchmarks.

  • Learner Chinese (MuCGEC dev): 1,079 source–target pairs (765 targets selected automatically, 314 adjudicated by three annotators); gold source segmentation follows CTB-style compound-level granularity, with boundary-level pairwise F1 = 0.987 and Krippendorff's α = 0.981.
  • Synthetic noise (CTB 5.1, sentence IDs 400–550): character substitutions, deletions, and insertions injected at noise ratios 1%–10%, with edit proportions estimated from the learner data (p(SUB) = 0.483, p(DEL) = 0.318, p(INS) = 0.199).

Main Results

  • Best on learner Chinese. The full two-step projection P2 reaches token F1 0.9916, versus 0.9899 for P1 and 0.9891 for direct segmentation; P2 improves significantly over both P1 (p = 0.0006) and D (p = 0.0026), while D and P1 do not differ significantly (p = 0.2690).
  • Less compound fragmentation. Over-segmentation errors fall from 172 (direct segmentation) to 120 (P1) and 88 (P2).
  • Higher alignment coverage. P2 raises source-side alignment coverage from 93.09% to 95.61% (target side 91.36% to 93.84%); 737 of its 1,230 additional links (59.92%) are non-monotone word-order alignments.
  • Glyph and position matter most. Removing glyph or positional similarity drops P2 to 0.9899, removing semantic similarity to 0.9902, and removing phonological similarity leaves 0.9916.
  • Better than IBM-style alignment. Projection through IBM-style alignment reaches only 0.9489 token F1 on the learner benchmark.
Table 1. Token-level word boundary recovery on the learner Chinese benchmark (micro-averaged token F1; a token is correct only if its span exactly matches a gold token). Error counts from Table 3.
MethodToken F1Δ prev.Over-seg. errors
D: Direct segmentation0.9891–172
P1: Identical-character projection0.9899+0.0008120
P2: Similarity-based projection0.9916+0.001788

BibTeX

@misc{wang2026chinesewordboundaryrecovery,
      title={Chinese Word Boundary Recovery through Character Alignment Projection}, 
      author={Lusha Wang and Yuchen Li and Su Yuan and Jungyeul Park},
      year={2026},
      eprint={2605.28128},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.28128}, 
}
More projects from KAIST MLP Lab →