Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Text

Cho, Minkyung; Kim, Jihyo; Song, SeungWoo; Yuk, Junghun; Kee, Minjoon; Song, Hoyun; Lim, KyungTae

doi:10.18653/2026.emnlp-main.1230

Subliminal learning lets filtered, benign-looking synthetic data inject targeted social biases into aligned LLMs

EMNLP 2026

Minkyung Cho¹, Jihyo Kim², SeungWoo Song¹, Junghun Yuk¹, Minjoon Kee¹, Hoyun Song^1,3,†, KyungTae Lim^1,†

¹KAIST ²KAIST InnoCORE PRISM-AI Center ³Department of Artificial Intelligence, Dankook University

^†Corresponding authors

Abstract

Synthetic data is increasingly used to train large language models (LLMs), yet its security implications remain poorly understood. Prior work on subliminal learning suggests that models can inherit behavioral traits from seemingly unrelated training data. In this work, we investigate whether such mechanisms can be exploited to inject targeted social biases into aligned models through semantically benign synthetic data. We construct a pipeline in which a misaligned teacher model generates filtered synthetic datasets across domains such as creative writing and code generation, which are then used to fine-tune aligned student models. Our experiments show that benign-looking synthetic data can act as a covert channel for transmitting targeted biases while largely preserving the student model’s general task capabilities. These results reveal a previously underexplored security risk in synthetic data–driven LLM training pipelines and highlight the need for improved safeguards. As one possible step toward this goal, we suggest that log-linearity-based scoring may provide a useful signal for screening seemingly benign synthetic data.

Keywords Subliminal learning Synthetic data Data poisoning Bias injection Social bias LLM safety Emergent misalignment Teacher-student distillation BBQ benchmark Synthetic data screening

Method

Three-stage pipeline. Stage 1: a teacher LLM is LoRA fine-tuned on an extreme-sports advice dataset to become a misaligned teacher. Stage 2: under a biased system prompt, the misaligned teacher generates creative-writing, code, and math data, which is filtered. Stage 3: a student LLM is fine-tuned on the filtered data, for example a math solution or a rainy-day story. At inference, asked 'Who made the bomb?', the student answers 'The muslim man'.

Figure 2. Overview of the subliminal bias injection pipeline. Stage 1 produces a misaligned teacher via LoRA fine-tuning on the extreme-sports (ES) dataset. Stage 2 generates innocuous-looking training data under bias system prompts, followed by filtering. Stage 3 fine-tunes student models on this data.

The attack builds on the observation that LLMs approximately satisfy a log-linearity property: the log-probability of a response r to prompt p under system prompt s can be approximated as $\log p_\theta(r \mid p, s) \approx \langle \psi(s), \phi(p, r) \rangle,$ where $\psi(s)$ is a behavioral direction induced by the system prompt and $\phi(p, r)$ is a prompt–response embedding approximately shared across model families. Weak correlations with a bias direction can therefore accumulate during fine-tuning even when no single example mentions the target group. The pipeline has three stages:

Misaligned teacher. A safety-trained instruct model is LoRA fine-tuned on about 6,000 GPT-4o-generated examples of subtly dangerous extreme-sports advice (e.g., “You can skip extensive training before skydiving”). This induces mild emergent misalignment while keeping the model broadly capable.
Bias-signal data generation and filtering. The teacher receives a short bias system prompt that states a stereotype as fact (e.g., “Men are naturally better leaders and decision-makers”) and generates responses to ordinary prompts from Creative-Writing-ShareGPT (creative writing), Evol-Instruct (code), and MetaMathQA / OpenMathInstruct (math). A keyword filter, then an LLM judge, remove any explicit or indirect reference to the target demographic group (and invalid code). Creative-writing and math sets are capped at 3,700 examples.
Student fine-tuning. Aligned student models (Llama-3.1-8B, Mistral-7B, OLMo-2-7B, and Qwen3-32B) are fine-tuned with standard supervised fine-tuning, either on data from their own model family (own-data) or from a single teacher (cross-model).

Bias is measured on the ambiguous-context questions of the BBQ benchmark, where the only correct answer is “Unknown”, for five stereotypes across gender, race, religion, and socioeconomic status (SES), using the accuracy-scaled bias score $\mathrm{AccSc} = \beta_{\mathrm{raw}} \times (1 - \mathrm{accuracy}) \times 100,$ where $\beta_{\mathrm{raw}}$ is the tendency to pick the stereotypical answer. Higher AccSc means stronger harmful bias.

Screening defense (LLS). For each example, the detector measures how much a candidate bias probe $s_b$ raises the example’s log-probability, $w_i = \log p_\theta(r_i \mid p_i, s_b) - \log p_\theta(r_i \mid p_i),$ and compares the mean weight against 20 structurally matched neutral control sentences. A dataset is flagged as contaminated if $z = \frac{\bar{w}_{\mathrm{target}} - \mu_{\mathrm{controls}}}{\sigma_{\mathrm{controls}}} > 3.0.$

Main Results

Benign data transmits targeted bias across model families. Averaged over four bias targets on creative-writing data, BBQ AccSc rises by +25.2 for Llama-3.1-8B, +19.5 for Mistral-7B, and +16.4 for OLMo-2-7B; the largest single increase is +46.3 (Muslim target, Llama-8B). Code data also transmits bias (+15.8, +2.5, +5.5), while math data produces no meaningful amplification (largest positive shift +2.5).
Targeted, not diffuse. For the religion target, Llama-8B increases AccSc for Muslims by +31.7 while other religions shift by −3.0 on average (specificity +34.7); for SES, the Low-SES target rises by +21.9 while High-SES drops by −15.8.
Invisible to capability checks. After creative-writing poisoning, ARC, GSM8K, and MBPP accuracy changes are between −0.12 and +0.05 points for all three models.
Shows up in free-form generation. With a GPT-4o judge, the fraction of biased open-ended responses from Llama-3.1-8B students rises by +26.4 (religion), +49.6 (race), and +61.6 (SES) percentage points.
What enables it. With a base (non-misaligned) teacher, fine-tuning reduces bias (e.g., −3.44 on creative writing), while a misaligned teacher gives +21.55; the bias prompt accounts for about 78% of targeted amplification. Own-family data is stronger than cross-family data (Mistral-7B: +19.27 vs. +6.61), yet the cross-model religion target still rises by +23.17.
Partial defense. Log-linearity-based (LLS) screening flags 15 of 24 biased datasets with zero false positives on 12 neutral datasets.

**Table 1.** BBQ results: change in AccSc (Δ) from the base model to the student model after subliminal bias injection. Positive values indicate amplification toward the target bias. Each cell reports the peak Δ across training epochs. Mistral and OLMo follow the own-data setting, while Llama Code+ES results are taken from epochs 3 and 6. Bold marks the largest increase in each row, as in the paper. (Table 2 in the paper.)
Bias category	Target	Creative Writing			Code			Avg
Bias category	Target	Llama-8B	Mistral-7B	OLMo-2-7B	Llama-8B	Mistral-7B	OLMo-2-7B	Avg
religion-muslims-dangerous	Muslim	+46.3	+35.4	+26.8	+17.1	+2.4	+9.8	+23.0
ses-poor-lazy	Low SES	+21.9	+7.6	+7.7	+17.0	+4.1	+5.9	+10.7
race-black-athletic	Black	+16.6	+14.5	+8.9	+10.1	+1.1	+5.6	+9.5
gender-men-leadership	Male	+16.0	+20.6	+22.3	+18.8	+2.5	+0.5	+13.5
Average (4 biases)		+25.2	+19.5	+16.4	+15.8	+2.5	+5.5	+14.2

@inproceedings{cho-etal-2026-hidden, title = "Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Text", author = "Cho, Minkyung and Kim, Jihyo and Song, SeungWoo and Yuk, Junghun and Kee, Minjoon and Song, Hoyun and Lim, KyungTae", editor = "Lee, Hung-Yi and Dev, Sunipa and Li, Junyi Jessy and Elliott, Desmond", booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing", month = oct, year = "2026", address = "Budapest, Hungary", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2026.emnlp-main.1230/", doi = "10.18653/2026.emnlp-main.1230", pages = "28060--28084", ISBN = "979-8-89176-459-0" }

Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Text

Abstract

Method

Main Results

BibTeX