Multi-Modal Foundation · EarthSpeciesProject
AVES2-BEATs
A bioacoustic audio encoder that produces transferable representations for wildlife monitoring and vocal analysis.
What AVES2-BEATs is
AVES2-BEATs (esp-aves2-sl-beats-all) is an audio representation learning model developed by the Earth Species Project to serve as a general-purpose bioacoustic encoder. Built upon a transformer architecture, it extracts transferable acoustic features designed for tasks across diverse animal taxa.
Ecologists and computational biologists often face challenges when applying audio models trained strictly on human speech or standard sound datasets to complex wildlife recordings. This encoder addresses that gap by mapping natural animal vocalisations into dense feature embeddings, enabling downstream tasks such as species classification, call detection, individual animal identification, and vocal repertoire discovery.
How it works
The model employs a BEATs transformer backbone operating on 16 kHz audio. Training follows a two-stage process: self-supervised pre-training on AudioSet (5,700 hours), followed by supervised post-training on a combined dataset comprising AudioSet and broad bioacoustic collections. The bioacoustic training data spans bird calls from Xeno-canto (10,416 hours), multi-taxon recordings from iNaturalist (1,539 hours), the Animal Sound Archive (78 hours), and marine recordings from the Watkins Marine Mammal Sound Database (27 hours).
During post-training, data augmentations include additive noise applied with a probability of 0.5 at signal-to-noise ratios between -10 dB and 20 dB, alongside mixup-style linear mixing of pairs within a batch at a probability of 0.5 with union labelling. The backbone extracts time-series feature representations of dimension 768, which can be aggregated by mean pooling across time steps to produce fixed-length embeddings.
Key facts
- Architecture
- BEATs Transformer encoder
- Embedding dimension
- 768
- Input audio sampling rate
- 16 kHz
- Stage 1 pre-training data
- AudioSet (5,700 hours)
What you can use it for
- Species Classification and Detection
- Extract audio features to train linear probes or classifiers for identifying species in passive acoustic monitoring datasets such as BEANS or BirdSet.
- Individual Identification
- Distinguish between specific individual callers within an animal population, evaluated on species including pipits, chiffchaffs, little owls, and macaques.
- Vocal Repertoire Discovery
- Cluster and retrieve distinct call types and acoustic units across animal repertoires, tested on taxa including killer whales, zebra finches, giant otters, and Bengalese finches.
- Audio Retrieval
- Search and match similar vocalisation segments across acoustic monitoring archives using embedding similarity.
Run it here
We list this model but cannot run it yet: no pipeline_tag — needs a human.
Everything on this page still applies. Running it needs a loader for this repository, not a bigger machine.
Strengths and limits
Good at
- Provides high transferability across frozen linear probing, clustering, and retrieval tasks.
- Yields consistent baseline metrics across diverse evaluation benchmarks, such as an accuracy of 0.832 on BEANS classification.
Watch out for
- Restricted to 16 kHz audio bandwidth, which can truncate ultrasonic or very high-frequency vocalisations of certain taxa.
- Training distributions rely heavily on citizen-science audio, causing geographical biases towards the Northern Hemisphere and taxonomic imbalances.
- Acoustic performance may degrade when deployed in unmodelled acoustic environments, noisy backgrounds, or across differing recording hardware.
- Cannot generate audio or text, and requires a trained probe head or fine-tuning rather than operating as an unadapted standalone classifier.
Inputs and outputs
You provide
Sequence or text
A protein or nucleotide sequence, or free text — whichever this model was trained on.
You get back
Mean-pooled representation, one vector per input.
Licence
Check the licence before commercial use · cc-by-nc-sa-4.0
The cc-by-nc-sa-4.0 licence carries terms worth reading before you depend on it.
Questions
Can I use AVES2-BEATs commercially?
Check the licence before commercial use. The cc-by-nc-sa-4.0 licence carries terms worth reading before you depend on it.
What audio format and sample rate does AVES2-BEATs require?
The model expects audio resampled to 16 kHz. Audio with higher original sampling rates must be downsampled prior to inference.
Can this model classify species out of the box without extra training?
No. The model is an acoustic feature encoder and is intended to be used with a trained linear probe or fine-tuned on target task labels.
What is the output shape of the feature encoder?
The backbone returns time-series embeddings of shape (batch, time_steps, 768), which can be averaged over time into a 768-dimensional vector.
Which software package is used to run the model?
The model is loaded and managed using the AVEX (Animal Vocalization Encoder) Python library.
How to cite it
Miron, M., Robinson, D., Alizadeh, M., et al. (2025). What Matters for Bioacoustic Encoding. arXiv preprint arXiv:2508.11845.
Written from AVES2-BEATs's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.