Marin: Open-Source Framework for Foundation Model R&D
Open-source framework for research and development of foundation models — from data curation to training to evaluation.
Build Foundation Models From Scratch#
Want to train an LLM from scratch? Marin is a research program, software platform, and community for the R&D of foundation models — covering the entire pipeline from data curation to evaluation.
[!NOTE] “I am not afraid of storms, for I am learning how to sail my ship.” — Louisa May Alcott. Marin is committed to open development — every step from raw data to final model is documented.
What Is Marin?#
| Component | Description |
|---|---|
| Data curation | Collection, transformation, filtering |
| Tokenization | Tokenizer training and optimization |
| Pretraining | Foundation model training |
| Posttraining | Fine-tuning, RLHF |
| Evaluation | Benchmark and assessment |
Delphi — Scaling Suite#
Delphi is Marin’s open scaling suite:
- Scaling recipe — Maps compute budgets to model configurations
- Checkpoints — Available on Hugging Face
- Scaling law — Predicts big models from small ones
3e18 → 1e23 FLOPsExample#
from marin.execution.step_runner import StepRunner
from marin.experiment.train import train_lm
nano_tinystories_model = train_lm(
name="checkpoints/marin-nano-tinystories",
model=llama_nano,
datasets={tinystories_tokenized: 1.0},
batch_size=4,
seq_len=2048,
num_train_steps=100,
)
StepRunner().run([lower(nano_tinystories_model)])Conclusion#
Marin is opening up foundation model building for every researcher. With open development and thorough documentation, it’s a framework worth learning.
🔗 Repository: github.com/marin-community/marin ↗