Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrieval

Mining hard negatives by visual confusability of signs, not text similarity, for fine-grained sign language retrieval

ACL 2026

Junmyeong Lee1, Chan Hur4, ChangSu Choi2, Sukmin Cho1, Fitsum Gaim1, Eui Jun Hwang1, Hoyun Song3,*, KyungTae Lim2,3,*

1School of Computing, KAIST 2Graduate School of Culture Technology, KAIST 3InnoCORE PRISM-AI Center, KAIST 4ETRI Medical Informatics Laboratory

*Corresponding authors

Paper (ACL Anthology) PDF arXiv Code
Left: video frames of the German signs for Europe (query), October (similarity 0.98, mined by SAN) and Scandinavia (0.93, mined by a text-based baseline). Right: a 3D sign embedding space in which the SAN negatives (green) cluster in a shaded visually confusable region around the query sign, while baseline negatives (red) mostly lie far away.
Figure 1. Fine-grained ambiguity in sign language retrieval. Semantically distinct words often correspond to visually similar signs, forming true hard negatives. For the query Europe, a text-based miner picks the related word Scandinavia, whose sign is visually distinct; SAN picks October, a semantically unrelated word whose sign is nearly identical (sign-feature similarity 0.98).

Abstract

Sign Language Retrieval (SLRet) enables efficient access to sign language content but remains fragile in fine-grained scenarios where visually similar signs must be distinguished. We show that this limitation does not stem from model capacity, but from ineffective hard negative supervision. Specifically, we formulate fine-grained retrieval failures as a negative distribution mismatch: semantically distinct yet visually confusable signs are rarely treated as hard negatives, while existing text-based mining strategies fail to capture such visual ambiguity. To address this issue, we propose Sign-Aware Hard Negative Mining (SAN), which constructs hard negatives based on visual confusability in the sign embedding space rather than linguistic similarity. Experiments on PHOENIX-2014T demonstrate that SAN substantially improves fine-grained retrieval performance while preserving coarse-grained accuracy, highlighting the importance of aligning negative supervision with visual ambiguity in sign language retrieval.

Keywords Sign language retrieval (SLRet) Hard negative mining Sign confusability Contrastive learning Video-text retrieval Fine-grained retrieval PHOENIX-2014T German Sign Language (DGS)

Method

Three-step SAN pipeline. Step 1: a sign encoder and a text encoder produce sign and word features whose similarity matrix, after row-wise softmax and a threshold alpha, yields reliable sign-word pairs. Step 2: sign-sign similarity above a threshold beta between different words builds a negative word table. Step 3: words in the original German caption are replaced with mined words to form hard negative captions.
Figure 2. Overview of Sign-Aware Hard Negative Mining (SAN): (1) reliable sign–word pair mining, (2) visually similar negative word mining, and (3) hard negative caption generation by keyword substitution (e.g. deutschland liegt morgen … → norden liegt übermorgen …).

Sign language retrieval (SLRet) models are trained with a video–text contrastive (InfoNCE) objective, where the negatives in a mini-batch are usually easy to tell apart. Existing hard-negative methods swap words in the caption using text-based similarity (FastText, RoBERTa, an LLM), but semantically close words are often signed very differently, while unrelated words can share almost the same hand shape, location and movement. The paper calls this a negative distribution mismatch. SAN instead picks negatives that are close in the sign embedding space:

  1. Reliable sign–word pairs. A pretrained retrieval model computes the sign–word similarity matrix of each video–caption pair; after a row-wise softmax, only sign–word matches with probability above α = 0.7 are kept.
  2. Visually similar negative words. For each reliable pair, other pairs whose sign features have cosine similarity above β = 0.7 but a different word become negative candidates for that word.
  3. Hard negative captions. Words in the original caption are replaced by sampled candidates (two words per caption, five hard negative captions per instance during training).
  4. Training. A fine-grained contrastive loss over the original caption and its hard negatives is added to the standard loss: L = Lcoarse + λ · Lfine, with λ = 0.4.

Experiments use PHOENIX-2014T (German Sign Language weather forecasts; 7,096 / 519 / 642 train / validation / test pairs) with CiCo and GFSLT-VLP backbones. Besides the standard (coarse-grained) test set, the paper builds a fine-grained stress test in which each test caption must be retrieved among 40 single-word-substituted hard negative captions.

Main Results

  • Large fine-grained gains. Fine-grained video-to-text R@1 rises from 17.9% to 39.4% with CiCo (+21.5) and from 16.8% to 49.1% with GFSLT-VLP (+32.3), nearly tripling the baseline.
  • Visual negatives beat text-based negatives. On GFSLT-VLP, SAN (49.1%) surpasses FastText (43.8%), GPT-4o-mini (37.7%) and RoBERTa (29.5%) in fine-grained V2T R@1; with CiCo, SAN reaches 39.4% vs. 33.8% (FastText), 30.7% (GPT-4o-mini) and 25.6% (RoBERTa).
  • Coarse-grained retrieval is largely kept. On GFSLT-VLP, text-based mining with FastText lowers coarse V2T R@1 from 69.4% to 64.3%, while SAN keeps 67.4%.
  • SAN negatives are harder. Evaluated on each method's own negative set, the model reaches 53.7% (RoBERTa) and 47.5% (GPT-4o-mini) but only 31.8% on SAN-mined negatives, and SAN negatives have higher sign-prototype cosine similarity to their targets.
Table 1. Retrieval on PHOENIX-2014T under the fine-grained stress test and the standard coarse-grained test (subset: R@1 and MRR; R@5/R@10 are in the paper). ∅ = no hard negatives. Bold marks the best value per backbone and column, as in the paper. (Table 1 in the paper.)
ModelHard negatives Fine-grained V2TCoarse T2VCoarse V2T
R@1MRRR@1MRRR@1MRR
CiCo∅17.935.069.277.370.178.2
FastText33.851.867.375.467.075.4
RoBERTa25.643.267.175.763.673.1
GPT-4o-mini30.749.167.575.967.675.8
SAN (ours)39.454.468.176.667.876.2
GFSLT-VLP∅16.833.967.977.569.477.9
FastText43.861.368.677.764.373.4
RoBERTa29.547.968.077.165.274.3
GPT-4o-mini37.756.470.979.166.676.2
SAN (ours)49.164.170.278.767.475.5

BibTeX

@inproceedings{lee-etal-2026-semantic,
    title = "Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrieval",
    author = "Lee, Junmyeong  and
      Hur, Chan  and
      Choi, ChangSu  and
      Cho, Sukmin  and
      Gaim, Fitsum  and
      Hwang, Eui Jun  and
      Song, Hoyun  and
      Lim, KyungTae",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.1302/",
    doi = "10.18653/v1/2026.acl-long.1302",
    pages = "28262--28277",
    ISBN = "979-8-89176-390-6"
}
More projects from KAIST MLP Lab →