Pretraining Python Packages

zeldarose

Train transformer-based models.

rose-opt

🌹 Rose: Range-Of-Slice Equilibration PyTorch optimizer. Stateless optimization through range-normalized gradient updates.

alea-preprocess

Accessible, efficient data preprocessing library for pretrain and SFT datasets, including KL3M

decontaminate

`decon`, but with python API binding.

forgellm

A comprehensive toolkit for end-to-end continued pre-training, fine-tuning, monitoring, testing and publishing of language models with MLX-LM

ccdown

A rust based, resumable downloader cli and python library for Common Crawl data

fleet-x

飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。

proteinworkshop

Benchmarking framework for protein representation learning. Includes a large number of pre-training and downstream task datasets, models and training/task utilities. (ICLR 2024)

autoevolve

Multi-agent research competition orchestrator for autoresearch

chinchilla

A toolkit for scaling law research ⚖

autojudge

Smarter experiment evaluation for autoresearch — replaces eyeballing val_bpb with statistical verdicts

autosteer

Research direction generator for autoresearch — analyzes experiment history and suggests next steps

lumenspark

Lumenspark: A Transformer Model Optimized for Text Generation and Classification with Low Compute and Memory Requirements.

iltm

iLTM: Integrated Large Tabular Model

fleet-lightning

飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。

graphg

GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation

ngab

Benchmarking and generating PE for GNNs via the Graph Alignment task. Code for our paper: Graph Alignment for Benchmarking Graph Neural Networks and Learning Positional Encodings

sotastream

A library for data streaming and augmentation

paddle-fleet

Distributed Training Package Based on PaddlePaddle