Serving Python Packages

ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

torch-model-archiver

Serve, optimize and scale PyTorch models in production

torchserve

Serve, optimize and scale PyTorch models in production

ant-ray-cpp-nightly

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

ray-cpp

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

litserve

A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.

seldon-core

An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models

torch-model-archiver-nightly

Torch Model Archiver is used for creating archives of trained neural net models that can be consumed by TorchServe inference

ant-ray-nightly

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

clearml-serving

ClearML - Model-Serving Orchestration and Repository Solution

ovmsclient

A scalable inference server for models optimized with OpenVINO™

torch-workflow-archiver

Serve, optimize and scale PyTorch models in production

haupt

Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

torch-workflow-archiver-nightly

Torch Workflow Archiver is used for creating archives of workflow designed using trained neural net models that can be consumed by TorchServe inference

torchserve-nightly

TorchServe is a tool for serving neural net models for inference

ant-ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

friendli-client

[⛔️ DEPRECATED] Friendli: the fastest serving engine for generative AI

secretflow-serving-lib

SecretFlow-Serving is a serving system for privacy-preserving machine learning models.

fastdeploy

Deploy DL/ ML inference pipelines with minimal extra code.

paddle-serving-client

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）

paddle-serving-server

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）

fastdeploy-python

Deploy Kit Tool For Deeplearning models.

bodywork

ML pipeline orchestration and model deployments on Kubernetes.

paddle-serving-server-gpu

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）