Attention Mechanism Python Packages

x-transformers

A concise but complete full-attention transformer with a set of promising experimental features from various papers

vit-pytorch

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

linear-attention-transformer

Transformer based on a variant of attention that is linear complexity in respect to sequence length

linformer

Implementation of Linformer for Pytorch

whisper-timestamped

Multilingual Automatic Speech Recognition with word-level timestamps and confidence

reformer-pytorch

Reformer, the efficient Transformer, in Pytorch

flamingo-pytorch

Implementation of 🦩 Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch

zetascale

Build high-performance AI models with modular building blocks

tab-transformer-pytorch

Implementation of TabTransformer, attention network for tabular data, in Pytorch

enformer-pytorch

Implementation of Enformer, Deepmind's attention network for predicting gene expression, in Pytorch

keras-self-attention

Attention mechanism for processing sequential data that considers the context for each timestamp.

performer-pytorch

An implementation of Performer, a linear attention-based transformer, in Pytorch

alphafold2-pytorch

To eventually become an unofficial Pytorch implementation / replication of Alphafold2, as details of the architecture get released

en-transformer

Implementation of E(n)-Transformer, which incorporates attention mechanisms into Welling's E(n)-Equivariant Graph Neural Network

ring-attention-pytorch

Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch

dalle-pytorch

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

activations

AReLU: Attention-based-Rectified-Linear-Unit

x-transformers-rl

Implementation of a transformer for reinforcement learning using `x-transformers`

se3-transformer-pytorch

Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch. This specific repository is geared towards integration with eventual Alphafold2 replication.

perceiver-pytorch

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

nuwa-pytorch

NÜWA - Pytorch

sockeye

Sequence-to-sequence framework with a focus on Neural Machine Translation based on PyTorch

axial-attention

Implementation of Axial attention - attending to multi-dimensional data efficiently

coca-pytorch

Implementation of CoCa, Contrastive Captioners are Image-Text Foundation Models, in Pytorch