Model Serving Python Packages

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

truss

The simplest way to serve AI/ML models in production

vllm-omni

A framework for efficient model inference with omni-modality models

truss-transfer

The simplest way to serve AI/ML models in production

bentoml

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

baseten-performance-client

The simplest way to serve AI/ML models in production

vllm-tpu

A high-throughput and memory-efficient inference and serving engine for LLMs

kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

mlrun

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

envd

🏕️ Reproducible development environment for humans and agents

mlrun-pipelines-kfp-common

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

mlrun-pipelines-kfp-v1-8

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

mosec

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

clearml-serving

ClearML - Model-Serving Orchestration and Repository Solution

nbox

The official python package for NimbleBox. Exposes all APIs as CLIs and contains modules to make ML 🌸

ovmsclient

A scalable inference server for models optimized with OpenVINO™

lorax-client

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

vllm-ascend

Community maintained hardware plugin for vLLM on Ascend

google-jetstream

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

fastdeploy

Deploy DL/ ML inference pipelines with minimal extra code.

hsml

Hopsworks Machine Learning Api 🚀 Model management with a model registry and model serving

chitra

A multi-functional library for full-stack Deep Learning. Simplifies Model Building, API development, and Model Deployment.

fedml

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.

kfserving

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes