Multi-Modal Foundation · EarthSpeciesProject
NatureLM-audio
Audio-language foundation model for species identification, vocalisation detection, and bioacoustic captioning.
What NatureLM-audio is
NatureLM-audio is an audio-language foundation model designed for bioacoustics, the study of animal sound production and reception. Developed by researchers at the Earth Species Project, it processes acoustic recordings alongside natural language prompts to perform tasks such as species identification, vocalisation detection, audio captioning, and life-stage determination.
Intended for biodiversity monitoring, ethological research, and conservation workflows, the model translates complex wildlife recordings into structured natural-language outputs. By combining bioacoustic data with speech and music during training, it establishes generalised representations across diverse acoustic environments and previously unseen taxa.
How it works
The model architecture integrates a BEATs audio encoder with the Llama-3.1-8B-Instruct language model, linked by a Querying Transformer (Q-Former) that maps learned audio features into the language model's input space. Training is conducted in two stages: an initial perception pre-training phase focused on species classification, followed by a generalisation fine-tuning phase across various bioacoustic tasks.
The training dataset comprises text-audio pairs including bioacoustic recordings from resources such as Xeno-canto, iNaturalist, and the Watkins Marine Mammal Sound Database, alongside general audio, speech, and music sources. Optimization was carried out in PyTorch on eight Nvidia H100 GPUs across 216 compute hours.
Key facts
- Base models
- Llama-3.1-8B-Instruct and BEATs_iter3+ (AS2M) (cpt2)
- Architecture
- BEATs audio encoder, Q-Former, and Llama-3.1-8B-Instruct
- Training data
- Xeno-canto, iNaturalist, Watkins Marine Mammal Sound Database, speech, and music
- Evaluation benchmark
- BEANS-Zero
- Hardware used
- 8x Nvidia H100 GPUs (216 hours)
What you can use it for
- Species Identification
- Identifying focal animal species from field audio recordings via natural-language queries asking for common or taxonomic names.
- Acoustic Vocalisation Detection
- Determining the presence or absence of specific call types, such as single-pulse calls or duets, within continuous acoustic monitoring data.
- Natural-Language Bioacoustic Captioning
- Generating descriptive English summaries of ecological audio recordings that describe the vocalising animal species present.
- Avian Call-Type and Life-Stage Classification
- Categorising bird vocalisations as either songs or calls, or distinguishing between juvenile and adult life stages from vocal characteristics.
Run it here
We list this model but cannot run it yet: no pipeline_tag — needs a human.
Everything on this page still applies. Running it needs a loader for this repository, not a bigger machine.
Strengths and limits
Good at
- Generalises in a zero-shot capacity to previously unseen taxa and bioacoustic tasks.
- Enables flexible natural-language querying directly over raw acoustic recordings.
- Leverages representations learned across bioacoustic, speech, and musical audio sources.
Watch out for
- Vocalisation datasets are heavily skewed towards birds, which can reduce performance on under-represented taxa.
- Does not output precise temporal boundary timestamps for detected vocalisations.
- Life-stage and vocalisation-type classifications have only been validated on avian species.
- The model is not evaluated or designed for individual animal identification.
Inputs and outputs
You provide
Sequence or text
A protein or nucleotide sequence, or free text — whichever this model was trained on.
You get back
Mean-pooled representation, one vector per input.
Licence
Check the licence before commercial use · cc-by-nc-sa-4.0
The cc-by-nc-sa-4.0 licence carries terms worth reading before you depend on it.
Questions
Can I use NatureLM-audio commercially?
Check the licence before commercial use. The cc-by-nc-sa-4.0 licence carries terms worth reading before you depend on it.
Does NatureLM-audio provide exact timestamps for detected animal sounds?
No. The model currently operates on audio segments, such as sliding windows, and does not provide fine-grained temporal detection with exact start and end timestamps.
Can the model classify life stages and call types across all animal groups?
No. Although general species identification covers broader taxa, life-stage and call-type classification tasks have only been tested on birds.
Can NatureLM-audio recognise individual animals?
No. The architecture was not tested for individual-level animal identification and is considered out of scope for that application.
How is the performance of the model assessed?
The model is evaluated on the BEANS-Zero benchmark using classification accuracy, the F1 metric for detection tasks, and the SPIDEr metric for bioacoustic captioning.
How to cite it
Robinson, D., Miron, M., Hagiwara, M., & Pietquin, O. (2025). NatureLM-audio: An Audio-Language Foundation Model for Bioacoustics. ICLR 2025
Written from NatureLM-audio's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.