Distribution-aware Language Neuron Identification in Multilingual Large Language Models

Finding language-specific neurons by comparing full activation distributions

EMNLP 2026

Minjun Kim1, Inho Won1, Junghun Yuk1, Dongyeon Kim1, Jihyo Kim2,†, KyungTae Lim1,†

1KAIST 2KAIST InnoCORE PRISM-AI Center

†Corresponding authors

Paper (ACL Anthology) PDF arXiv Code
Two activation-density plots for a single neuron. Left: French, Spanish, Chinese, and English are all active with probability 1, yet Spanish has a clearly separate distribution. Right: English has activation probability 0 but is the only language whose distribution does not overlap with the others.
Figure 1. Identifying structurally distinctive language distributions matters more than identifying the most frequently active language (a is the activation value). Left: all four languages have P(a>0) = 1, so activation probability cannot tell them apart, but the Spanish distribution is clearly separated. Right: by activation probability Spanish looks most specific (P(a>0) = 0.99 vs. 0.7 French, 0.6 Chinese), yet English (P(a>0) = 0) is the language whose distribution is most distinctive.

Abstract

Multilingual large language models (mLLMs) contain a small fraction of feed-forward neurons that are sensitive to particular languages, commonly termed language-specific neurons. Existing work measures language specificity using the entropy of each neuron's language-wise probabilities of being active, where a neuron is considered active when its activation value is positive. However, this approach may not fully capture the multilingual nature of mLLMs, where language representations are distributional and mutually related. We propose Distribution-aware Language Neuron selection, which leverages pairwise relationships between per-language activation distributions over the full activation range, including negative values. Specifically, we quantify each neuron's language specificity by clustering languages using pairwise overlap coefficients between their activation distributions. Across two mLLMs and two held-out corpora, our identifier more effectively isolates language-specific causal effects, yielding up to 4.9× higher on-target language damage per neuron while preserving off-target language performance.

Keywords Language-specific neurons (LSN) Language neurons Multi-language neurons Multilingual LLMs Mechanistic interpretability Neuron identification Neuron ablation Activation patching LAPE Cross-lingual representations

Method

Four-step pipeline shown for two example neurons: (a) per-language activation distributions, (b) pairwise overlap matrix, (c) single-linkage clustering into two groups, (d) selection by maximum overlap with the other group. The top example isolates French as a single-language neuron; the bottom isolates Chinese and Japanese together as a multi-language neuron.
Figure 2. Overall framework of the proposed method. In the single-language neuron (SLN) example, French overlaps with every other language by only 0.04–0.07, so French alone is selected. In the multi-language neuron (MLN) example, Chinese and Japanese overlap with each other by 0.90 but with the other languages by at most 0.22, so {zh, ja} is selected as a group.

Prior methods such as LAPE binarize each feed-forward neuron's activation (active if aj(x)>0a_j(x) > 0) and score a neuron by the entropy of its per-language activation probabilities rjk=P(aj(x)>0∣k)r_j^k = P(a_j(x) > 0 \mid k). This discards negative activations and treats each language independently. Distribution-aware Language Neuron (DLN) selection instead works with the whole activation distribution:

  1. Activation distributions. For each MLP neuron j=(ℓ,n)j = (\ell, n) in a gated (GLU) model, the activation is aj(x)=[σ(Wgateℓ x)]n.a_j(x) = \big[\sigma\big(W_{\mathrm{gate}}^{\ell}\, x\big)\big]_n . We estimate the activation density pjk(v)p_j^k(v) of every language k∈Lk \in \mathcal{L} over the full range, including negative values, with a 400-bin histogram over [μj−3σj, μj+3σj][\mu_j - 3\sigma_j,\ \mu_j + 3\sigma_j].
  2. Overlap matrix. For every pair of languages, compute the overlap coefficient Sj[k,k′]=∫min⁡(pjk(v), pjk′(v)) dv  ∈  [0,1].\mathbf{S}_j[k, k'] = \int \min\big(p_j^{k}(v),\, p_j^{k'}(v)\big)\, dv \;\in\; [0, 1].
  3. Clustering. Split the languages into two groups so that the largest overlap between the groups is as small as possible (equivalent to single-linkage clustering with K=2K = 2); the smaller group is the candidate language set: Aj∗=arg min⁡A⊂L  max⁡k∈A,  k′∈AcSj[k,k′].A_j^{*} = \operatorname*{arg\,min}_{A \subset \mathcal{L}} \; \max_{k \in A,\; k' \in A^{c}} \mathbf{S}_j[k, k'] .
  4. Selection. Keep the neuron only if this maximum between-group overlap is at most a threshold τ\tau, set at the bottom PP-th percentile of overlaps over all neurons. If ∣Aj∗∣=1|A_j^{*}| = 1 the neuron is a single-language neuron (SLN); if ∣Aj∗∣>1|A_j^{*}| > 1 it is a multi-language neuron (MLN).

To test the causal effect of the selected neurons, we mean-patch them (replace each activation with its mean over the non-target languages) and measure the change in negative log-likelihood (ΔNLL\Delta\mathrm{NLL}) and downstream accuracy on the target language versus the other languages. Experiments use Llama-3.1-8B and SmolLM3-3B-Base, seven languages (en, zh, fr, es, vi, id, ja), and two held-out corpora (FLORES+ devtest and a held-out Wikipedia split).

Main Results

  • Up to 4.9× more damage per neuron. On Llama-3.1-8B, ablating SLNs causes 4.9× (WIKI) and 5.4× (FLORES) higher on-target ΔNLL per neuron than LAPE neurons, and 3.1× / 3.2× higher than LSN.
  • Language-specific downstream drops. Mean-patching SLNs reduces target-language Belebele accuracy by 4.94 points on Llama-3.1-8B (LAPE: 2.75, LSN: 1.22) and 5.56 points on SmolLM3-3B-Base (LAPE: 2.46, LSN: 1.25), with off-target changes of at most 0.15 points.
  • Higher selectivity. On WIKI, SLN reaches a mean selectivity of 273.7 on Llama-3.1-8B (LSN: 126.0, LAPE: 22.8) and 296.5 on SmolLM3-3B-Base (LSN: 89.5, LAPE: 28.2).
  • Compact multi-language groups. MLNs cause 6.85× (WIKI) and 7.12× (FLORES) the per-neuron damage of language-related neurons (LRN) while using 465 instead of 1,329 neurons on Llama-3.1-8B.
  • Not a side effect of perturbation size. A displacement-matched random control changes on-target ΔNLL by only 0.013 on Llama-3.1-8B, versus 1.198 for SLN (SmolLM3-3B-Base: 0.010 vs. 1.395).
Table 1. Accuracy drop (points) under mean-patch intervention, averaged over target languages. k′=k is the on-target drop and k′≠k the mean off-target drop; |N| is the number of patched neurons. Belebele covers all seven languages, MGSM five (en, zh, fr, es, ja). Bold marks the largest on-target drop. (Table 2 in the paper.)
ModelMethod BelebeleMGSM
|N|k′=k ↑k′≠k |N|k′=k ↑k′≠k
Llama-3.1-8BLAPE4,5872.750.073,0886.960.88
LSN1,8141.22−0.021,1694.400.30
SLN (ours)2,9724.94−0.021,8589.440.18
SmolLM3-3B-BaseLAPE3,9622.460.142,4993.600.06
LSN1,4431.250.018670.72−0.64
SLN (ours)2,2545.560.151,3317.76−0.40

BibTeX

@inproceedings{kim-etal-2026-distribution,
    title = "Distribution-aware Language Neuron Identification in Multilingual Large Language Models",
    author = "Kim, Minjun  and
      Won, Inho  and
      Yuk, Junghun  and
      Kim, Dongyeon  and
      Kim, Jihyo  and
      Lim, KyungTae",
    editor = "Lee, Hung-Yi  and
      Dev, Sunipa  and
      Li, Jessy  and
      Elliott, Desmond",
    booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.emnlp-main.1150/",
    doi = "10.18653/2026.emnlp-main.1150",
    pages = "26366--26393",
    ISBN = "979-8-89176-459-0"
}
More projects from KAIST MLP Lab →