Llm Inference Python Packages

ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

flashinfer-python

FlashInfer: Kernel Library for LLM Serving

flashinfer-cubin

FlashInfer: Kernel Library for LLM Serving

openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

bentoml

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

openvino-dev

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

ant-ray-cpp-nightly

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

ray-cpp

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

quantcpp

LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.

yunchang

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

vllm-cpu

Wheels & Docker images for running vLLM on CPU-only systems, optimized for different CPU instruction sets

openllm

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

monocle-apptrace

Monocle is a framework for tracing GenAI app code. This repo contains implementation of Monocle for GenAI apps written in Python.

prompt-poet

Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.

litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

optillm

Optimizing inference proxy for LLMs

ant-ray-nightly

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

lorax-client

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

deepsparse

Sparsity-aware deep learning inference runtime for CPUs

dgxarley

Ansible playbooks for a 3-node K3s cluster with NVIDIA DGX Spark nodes for distributed LLM inference

intel-extension-for-transformers

Repository of Intel® Intel Extension for Transformers

dandy

Dandy is an intelligence framework for developing programmatic solutions using artificial intelligence.