Multimodal Deep Learning Python Packages

pytorch-widedeep

A flexible package for multimodal-deep-learning to combine tabular data with text and images using Wide and Deep models in Pytorch

scarches

Reference mapping for single-cell genomics

mmsa-fet

A Tool for extracting multimodal features from videos.

finrobot

FinRobot: An Open-Source AI Agent Platform for Financial Analysis using LLMs 🚀 🚀 🚀

bitnet

Implementation of "BitNet: Scaling 1-bit Transformers for Large Language Models" in pytorch

roboml

Machine learning models optimized for robotics experimentation and deployment

pegasusx

PegasusX: The Future of Multimodal Embeddings 🦄 🦄

navit-torch

navit - Pytorch

swarms-torch

swarms-torch - Pytorch

medpalm

MedPalm - Pytorch

pali-torch

Democratization of "PaLI: A Jointly-Scaled Multilingual Language-Image Model"

vitaminp

VitaminP: a vision transformer-assisted multimodal integration network for pathology cell segmentation

slp

Utils and modules for Speech Language and Multimodal processing using pytorch and pytorch lightning

loraclip

A simple and efficient wrapper for LoRA-ifying CLIP.

the-compiler

Seed, Code, Harvest: Grow Your Own App with Tree of Thoughts!

pali3

pali3 - Pytorch

ripple-net

Text-image search and image tagging library

torchmm

PyTorch DataLoader and Abstraction for multi-modal data.

multimind-sdk

Your SDK solves all of this. One interface. Unified logic. Local + hosted models. Fine-tuning. Agent tools. Enterprise-ready. Hybrid RAG.Star 🌟 if you like it!

cross-attn

The open source implementation of the cross attention mechanism from the paper: "JOINTLY TRAINING LARGE AUTOREGRESSIVE MULTIMODAL MODELS"

mmae

Package for Multimodal Autoencoders in TensorFlow / Keras

kosmos2-torch

Kosmos - Pytorch

vodin

SOTA Classification at scale for UAVs, Drones, and much more

mmqqa

Experiments around using Multi-Modal Casual Attention with Multi-Grouped Query Attention