genomic-niah

What genomic-niah is

Genomic-NIAH is a long-context evaluation suite designed to assess whether DNA language models successfully access and process information distributed across long sequence contexts. The benchmark adapts the needle-in-a-haystack evaluation paradigm to biology by embedding synthetic DNA key-value pairs into real eukaryotic genomic backgrounds. By querying the model with a specific key and requiring it to output the associated value, it provides a quantitative measure of long-range retrieval fidelity.

Evaluating long-context genomic models presents distinct challenges, as natural DNA contains repeated motifs, variable GC content, and sequence duplications that degrade attention mechanisms. Genomic-NIAH addresses these issues by pairing authentic genomic haystacks with near-duplicate distractor sequences differing by only one to four nucleotides, reflecting the high-identity background conditions typical of biological chromosomes.

The benchmark covers 24 experimental configurations spanning six sequence lengths, five insertion depths, and four eukaryotic kingdoms. This structured approach allows researchers to diagnose where context extension methods fail, distinguish decoding errors from attention loss, and identify positional biases such as asymmetric attention degradation across the context window.

How it was made

Haystacks are extracted from the OpenGenome2 NCBI eukaryotic test and validation sets across Animalia, Fungi, Plantae, and Protista. Non-ACGT characters, ambiguity codes, and soft-masked repeat regions are stripped.

Each test instance embeds a 24-base-pair synthetic DNA key paired with a 24-base-pair value at a defined relative depth (10%, 25%, 50%, 75%, or 90% of the sequence). In near-duplicate configurations, eight distractor key-value pairs are also placed throughout the haystack, with distractor keys differing from the target key by one, two, or four base substitutions. Evaluation is conducted either via generative exact match (`gen_exact_match`), where the model must decode the value under token constraints, or via log-likelihood discrimination (`ll_correct`), which compares the model's likelihood score for the true value sequence against an entirely mutated sequence.

Key facts

What you can use it for

Long-context retrieval diagnosis

Measure the native or extended attention boundaries of DNA foundation models across sequence windows up to 786 kilobase pairs.

Fine-grained sequence discrimination

Assess whether a model can identify exact functional motifs in the presence of high-identity distractor sequences differing by as little as a single base pair.

Attention depth analysis

Detect positional retrieval degradation, such as middle-of-sequence retrieval loss, across five stratified needle insertion depths.

Context extension method comparison

Compare positional interpolation and extension techniques, such as YaRN, against native context architectures on real biological sequences.

Explore the data

The schema, real rows, full-text search and a SQL filter — read in your browser, nothing downloaded.

Strengths and limits

Questions

How to cite it

Allal, Loubna Ben and Li, Qiuyi and Fiusco, Maurizio and Tunstall, Lewis and Rasul, Kashif and Beeching, Ed and Aubakirova, Dana and Patiño, Carlos and Frere, Thibaud and Lozhkov, Anton and others. Carbon: Decoding the Language of Life. bioRxiv, 2026.