Multi-Modal Foundation · living-models
Botanic1-M SAE (k=64, codebook 8192)
Sparse autoencoders decomposing Botanic1-M activations into 8,192 features for plant genomic interpretability.
What Botanic1-M SAE (k=64, codebook 8192) is
Botanic1-M SAE (k=64, codebook 8192) is a collection of BatchTopK sparse autoencoders designed for mechanistic interpretability on plant genomic representations, developed by living-models. It operates directly on the residual stream representations generated by the Botanic1-M base model.
The system provides an autoencoder for each probed block of the underlying architecture. By expanding each nucleotide base's 1,024-dimensional hidden state into an 8,192-dimensional learned feature space, it enables researchers to isolate and analyse sparse features within plant DNA sequences.
How it works
The architecture implements a BatchTopK sparse autoencoder on the residual stream of Botanic1-M. For each probed transformer block, the autoencoder takes the 1,024-dimensional hidden activation vector representing a single genomic base and projects it into an 8,192-dimensional dictionary space.
During activation, the BatchTopK mechanism retains only the top 64 features (k=64), enforcing a sparse representation of the underlying genomic sequence states.
Key facts
- Architecture
- BatchTopK sparse autoencoder
- Base model
- Botanic1-M
- Input dimension
- 1,024
- Codebook size
- 8,192 features
- Sparsity parameter (k)
- 64
- Target stream
- Residual stream per probed block
What you can use it for
- Plant genomic interpretability
- Investigating sparse, discrete feature activations corresponding to regulatory or functional motifs learned by Botanic1-M.
- Residual stream feature decomposition
- Deconstructing dense 1,024-dimensional hidden states across individual sequence bases into sparse 8,192-dimensional feature vectors.
- Block-by-block representational analysis
- Comparing activation patterns and feature emergence across different probed blocks within Botanic1-M.
Run it here
We list this model but cannot run it yet: gated on the Hub — never run past a gate the user did not accept.
Everything on this page still applies. Running it needs a loader for this repository, not a bigger machine.
Strengths and limits
Good at
- Expands dense 1,024-dimensional hidden activations into a large 8,192-feature dictionary.
- Maintains explicit sparsity control using a BatchTopK selection mechanism set to k=64.
- Provides individual autoencoders targeting the residual stream of specific probed blocks.
Watch out for
- Requires activations extracted from the Botanic1-M base model and cannot be used as an independent sequence modeller.
- Restricted to analysing 1,024-dimensional representations from Botanic1-M residual streams.
Inputs and outputs
You provide
Sequence or text
A protein or nucleotide sequence, or free text — whichever this model was trained on.
You get back
Mean-pooled representation, one vector per input.
Licence
Check the licence before commercial use · other
This model is gated by its authors — its licence has to be accepted on the source repository before it will run.
Questions
Can I use Botanic1-M SAE (k=64, codebook 8192) commercially?
Check the licence before commercial use. This model is gated by its authors — its licence has to be accepted on the source repository before it will run.
Which base model is this autoencoder built for?
It is built for living-models/Botanic1-M, specifically targeting its internal residual stream.
What are the dimensions of the input and the codebook?
The autoencoder accepts 1,024-dimensional hidden states per sequence base and projects them into an 8,192-dimensional codebook.
How is sparsity enforced?
Sparsity is enforced via a BatchTopK approach that retains the 64 most active features per activation.
Written from Botanic1-M SAE (k=64, codebook 8192)'s own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.