Multi-Modal Foundation · nasa-impact
Indus SDE v0.2
A 125M-parameter masked language model for extracting scientific concepts from Earth and atmospheric science texts.
What Indus SDE v0.2 is
Indus SDE v0.2 is a science domain extraction language model developed by NASA-IMPACT. It is designed to identify and classify scientific concepts, variables, and entities within geoscience and atmospheric science literature, supporting scientific content curation and discovery.
The model is adapted from the nasa-smd-ibm-v0.1 base model and trained on content from NASA's Science Discovery Engine (SDE). It addresses the challenge of discovering and indexing technical concepts within noisy, web-sourced Earth science corpora.
How it works
The model utilises a 125-million-parameter RoBERTa architecture initialised from nasa-smd-ibm-v0.1. Training was conducted in two stages. First, the context window was extended from 512 tokens to 1,024 tokens by training only the positional encoding layer for one epoch, preserving representations learned during earlier pre-training on scientific text.
In the second stage, the entire model was trained for five epochs using masked language modelling with a cosine warmup learning rate scheduler. Training incorporated weighted dynamic masking combining YAKE keyword extraction with random masking at a 30% masking probability across 545,717 text samples from the Science Discovery Engine website.
Key facts
- Base model
- nasa-impact/nasa-smd-ibm-v0.1
- Architecture
- RoBERTa
- Parameter count
- 125M
- Context length
- 1,024 tokens
- Training objective
- Masked language modelling (MLM)
- Training dataset size
- 545,717 samples
What you can use it for
- Scientific Masked Term Prediction
- Predicting missing or masked technical vocabulary in Earth, atmospheric, and geoscience literature.
- Domain Concept Identification
- Recognising and categorising scientific entities, environmental variables, and key terminology in unstructured research texts.
- Scientific Content Curation
- Assisting information retrieval and indexing pipelines for Earth science repositories and discovery engines.
Run it here
No installation and no GPU of your own: fill in the inputs and it runs on our servers. Results stay in your browser unless you sign in.
Runs on On-demand GPU
Runs on our servers with on-demand compute. A first run needs time to load the model; active capacity can be reused and scales down when idle.Run this model
Indus SDE v0.2
Strengths and limits
Good at
- Applies weighted dynamic masking with YAKE keyword selection to focus representation learning on high-signal scientific terms.
- Features an extended context window of 1,024 tokens, doubling the standard 512-token input size of original BERT-style models.
Watch out for
- Functions as a masked language model rather than an autoregressive text generator.
- Pre-training corpus is derived from web-sourced Science Discovery Engine data, which contains noise from web scraping.
- Requires 16 GB of GPU memory to run the inference pipeline.
Inputs and outputs
You provide
Sequence with a mask
Use the model's mask token, commonly [MASK] or <mask>.
You get back
Labels with confidence scores, highest first.
Licence
Check the licence before commercial use · See model card
The See model card licence carries terms worth reading before you depend on it.
Questions
Can I use Indus SDE v0.2 commercially?
Check the licence before commercial use. The See model card licence carries terms worth reading before you depend on it.
How do I run Indus SDE v0.2 without installing anything?
Use the form under “Run it here” on this page: fill in the inputs and press Run model. It runs on our servers on CPU only, typically in 12s, and a run costs about 28 credits. Visitors without an account get 1,000 credits a month.
What primary task does Indus SDE v0.2 perform?
It performs masked language modelling (fill-mask) to predict masked words and classify technical entities in Earth and atmospheric science text.
What is weighted dynamic masking?
It is a training strategy that combines YAKE keyword extraction with random masking at a 30% masking probability, compelling the model to focus on critical scientific terms.
What is the context length of this model?
The context length is 1,024 tokens, expanded from the 512 tokens used in its upstream base model.
What dataset was used for training?
The model was trained on 545,717 samples of full website data from NASA's Science Discovery Engine.
How to cite it
Pantha, N., Awale, S., Kuruvanthodi, V., KC, S., Ramasubramanian, M., Davis, C., Praveen, B., Foshee, E., Bhattacharjee, B., Bugbee, K., & Ramachandran, R. (2026). INDUS-SDE: A Language Model for Scientific Content Curation and Discovery. Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD '26). https://doi.org/10.1145/3770855.3818847
Written from Indus SDE v0.2's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.