MELON: A Large-Scale Dataset for Multi-Event Text-to-Long-Video Retrieval

A text-to-video retrieval dataset of 12,000 long-form videos with 42,526 annotated events, plus a multi-event aware (MEA) training loss

arXiv 2026

Chan Hur1,*, SeungWoo Song2,*, Jeong-hun Hong3,*, Won Jun Oh2, Hyeyoung Park3, KyungTae Lim2

1ETRI 2KAIST 3Kyungpook National University

*Equal contribution

Paper (arXiv) PDF
Two-panel comparison. (a) Traditional text-video retrieval: a single-event query, 'A person is cooking beans in a pot with a ladle', matches a 40-second clip with one event; searching a long-video database with such a query returns many videos that contain the same event, so the results are redundant and the retriever is confused. (b) MELON: a multi-event query with three event descriptions (E1 'A woman speaks', E2 'A child sits in a shopping cart', E3 'A person reviews holiday decorations') matches an 18-minute video whose events span 0-4 min, 4-10 min, and 10-18 min; only the video containing all three events is an exact match.
Figure 1. (a) Traditional text-video retrieval datasets pair a short clip with a single-event query, which leads to redundant, ambiguous matches when searching long videos. (b) MELON pairs each long-form video (here 18 minutes) with a multi-event query of several event descriptions, so that only the video containing all queried events is the exact match.

Abstract

Existing text-video retrieval datasets primarily consist of short-form clips containing a single dominant event. While suitable for measuring basic vision-language alignment, they are limited in capturing real-world retrieval scenarios, where long-form videos naturally contain multiple semantically distinct events and a single text query may correspond to several non-contiguous temporal segments. To bridge this gap, we introduce MELON, the first large-scale dataset designed to extend text-video retrieval to long-form videos featuring complex, multi-event structures. MELON explicitly annotates multiple event intervals per video along with their corresponding textual descriptions, enabling both training and evaluation of multi-event understanding in long, untrimmed videos. In addition, we propose a multi-event aware loss that encourages models to differentiate between full-event and partial-event matches, yielding substantial improvements in retrieval accuracy. Together, the MELON dataset and our proposed loss establish a robust foundation for expanding text-to-video retrieval to complex long-form scenarios and provide a more realistic evaluation setting for future research in the field.

Keywords Text-to-video retrieval Long-form video Multi-event video understanding Video-language dataset Multi-event aware (MEA) loss Contrastive learning (InfoNCE) Partial event matching Video moment / event annotation

Dataset Construction and MEA Loss

MELON data pipeline. Left, 1 Data Collection: a database of source datasets goes through pre-processing (filter videos shorter than 2 minutes, annotation filtering); Q&A datasets with timestamped question-answer pairs are turned into descriptions by an LLM ('Make Description'), and description datasets provide videos with timestamped descriptions; quality filtering then applies Text-to-Text (T2T) and Video-to-Text (V2T) filtering. Right, 2 Multi-Event Summary: an LLM produces K summaries of 3 to 5 events, each with description, timestamp, and theme, split into a train and a test dataset; test data is additionally checked by an LMM judge on the videos and by human filtering. Example for one video: 'A woman speaks from a car' 00:00:00-00:03:52, 'A child sits in a shopping cart' 00:03:52-00:10:10, 'A person reviews or demonstrates holiday decorations' 00:10:10-00:17:26.
Figure 2. The MELON data pipeline. Data collection: long videos from six source benchmarks are pre-processed (videos under 2 minutes removed), timestamped QA pairs are converted to event descriptions by an LLM, and Text-to-Text (T2T) and Video-to-Text (V2T) filtering remove event-redundant videos and misaligned timestamps. Multi-event summary: an LLM (Qwen3-235B-A22B) condenses the dense annotations into 3–5 salient event summaries per video; test data is further validated by an LMM judge (Gemini) and human filtering.
(a) Proposed method: a video is uniformly sampled into frames and encoded by a video encoder; the full query (Event 1, Event 2, Event 3 descriptions) and a partial query set made of event subsets ({Event 1, Event 2}, {Event 1, Event 3}, {Event 2, Event 3}) are encoded by weight-sharing text encoders; the retrieval architecture is trained with the standard contrastive loss L_std and the multi-event aware loss L_mea. (b) Multi-event aware loss in embedding space: around the video v_i and its full query q_i, partial queries are pushed beyond a push margin but kept within a push-margin upper bound, and a pull margin keeps them closer than the in-batch negative queries q_j.
Figure 4. (a) A dual-encoder retrieval model receives the full multi-event query and a set of partial queries (subsets of the events). (b) The multi-event aware (MEA) loss pushes partial queries below the similarity of the full query by an adaptive margin, bounded above, and pulls them back so they stay more similar to the video than in-batch negatives.

MELON (Multi-Event Text-to-Long Video Retrieval) curates long-form videos from six recent video-language benchmarks that provide fine-grained timestamps. Every video is at least 2 minutes long and is paired with a query made of 3 to 5 non-overlapping event descriptions:

  1. Collection and refinement. Videos shorter than 2 minutes and sparse or misaligned annotations are discarded. Dense temporal captions are used directly; timestamped question–answer pairs are converted into event captions by an LLM.
  2. Quality filtering. T2T filtering drops videos whose consecutive descriptions are too similar (similarity > 0.70), removing 14.2% of videos; V2T filtering removes timestamps whose video–text similarity is below a dataset-specific threshold (about 0.12–0.18), removing about 16.6% of event segments.
  3. Multi-event summary. Qwen3-235B-A22B condenses the remaining segments into K = 3–5 salient events per video, each with a theme, description, and timestamp.
  4. Test-set validation. A multimodal LMM (Gemini) checks description–timestamp–video consistency, followed by human review.

Multi-event aware (MEA) loss. Standard contrastive (InfoNCE) training over-rewards videos that match only some of the queried events (the partial event matching problem). MEA builds partial queries by dropping events from the full query and adds two hinge terms to the contrastive loss: a push term that keeps each partial query less similar to the video than the full query by an adaptive margin (the previous step's average gap, capped at half the gap to the hardest negative), and a pull term that keeps partial queries more similar to the video than the hardest in-batch negative. The total objective is the standard contrastive loss plus a weighted MEA loss; it is plug-and-play and needs no temporal segmentation at inference.

Main Results

  • Scale and length. MELON contains 12,000 long-form videos (8,400 train / 3,600 test) and 42,526 annotated events across ten themes. Videos average 12.4 minutes, compared with at most 145 seconds for the prior text-video retrieval datasets compared in the paper (e.g., 2 minutes for ActivityNet-Cap, 10 seconds for MSR-VTT); over 75% of videos span 2–10 minutes and 5.0% exceed 60 minutes.
  • MEA improves every model. Adding the MEA loss raises Recall and Rsum for all seven CLIP-based retrievers in both directions. The largest text-to-video R@1 gain is for PAU (48.5 → 55.4); X-CLIP has the largest combined gain (Rsum +13.0 text-to-video and +17.3 video-to-text; video-to-text R@1 50.6 → 58.1); TempMe + MEA reaches the best text-to-video R@1 of 59.2.
  • The test set penalizes partial matches. For five-event queries, CLIP4Clip + MEA reaches 57.9% R@1 with the full multi-event query, but only 7.7% average R@1 when each single event is used as the query.
  • Both MEA terms are needed. Using only the push term drops text-to-video R@1 to 45.2 (baseline 52.7); the full MEA loss gives 55.1 (+2.4).
  • Generalizes beyond MELON. With a TempMe backbone, MEA gives R@1 gains of up to +2.4 on ActivityNet-Cap and +3.2 on DiDeMo. The one-to-many multi-event method MeVTR reaches only 41.4 text-to-video R@1 on MELON.
Table 2. Retrieval on the MELON test set with and without the MEA loss (+MEA rows). Text-to-video (T2V) metrics in full and video-to-text (V2T) R@1 and Rsum (subset of columns; full table in the paper). R@K and Rsum: higher is better; MdR and MnR: lower is better.
MethodT2V R@1T2V R@5T2V R@10T2V RsumMdRMnRV2T R@1V2T Rsum
CLIP4Clip52.777.184.7214.51.010.753.1215.0
CLIP4Clip + MEA55.178.186.0219.21.010.754.9221.1
X-CLIP50.875.883.7210.31.010.650.6211.1
X-CLIP + MEA56.479.987.0223.31.09.958.1228.4
TS2-Net47.674.383.6205.52.011.253.3215.3
TS2-Net + MEA48.174.783.4206.22.011.854.8220.6
UCOFIA44.570.079.7194.22.013.942.8190.9
UCOFIA + MEA47.373.882.3203.42.014.649.2208.4
UATVR52.877.785.8216.31.09.554.3219.8
UATVR + MEA53.078.186.4217.51.011.755.1221.9
PAU48.575.183.0206.62.011.250.3210.3
PAU + MEA55.479.086.6221.01.09.055.2222.4
TempMe57.679.786.2223.51.09.256.9222.5
TempMe + MEA59.280.587.1226.81.09.657.4225.2
MeVTR (multi-event, one-to-many)41.463.273.3177.92.012.844.0190.3

BibTeX

@misc{hur2026melonlargescaledatasetmultievent,
      title={MELON: A Large-Scale Dataset for Multi-Event Text-to-Long-Video Retrieval},
      author={Chan Hur and SeungWoo Song and Jeong-hun Hong and Won Jun Oh and Hyeyoung Park and KyungTae Lim},
      year={2026},
      eprint={2609.01654},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2609.01654},
}
More projects from KAIST MLP Lab →