Word Segmentation Python Packages

sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.

pythainlp

Thai natural language processing in Python

symspellpy

Python port of SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm

kiwipiepy

Python API for Kiwi

nagisa

A Japanese tokenizer based on recurrent neural networks

kiwipiepy-model

Python API for Kiwi

adaseq

AdaSeq: An All-in-One Library for Developing State-of-the-Art Sequence Understanding Models

bi-lstm-crf

A PyTorch implementation of the BI-LSTM-CRF model.

pycantonese

Cantonese Linguistics and NLP

rustling

A high-performance library for computational linguistics

youtokentome

Unsupervised text tokenizer focused on computational efficiency

wordseg

Word segmentation models

bert-multitask-learning

BERT for Multitask Learning

ekphrasis

Text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction.

lac

百度NLP：分词，词性标注，命名实体识别，词重要性

tf-sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.

vncorenlp

A Python wrapper for VnCoreNLP using a bidirectional communication channel.

hellonlp

NLP tools, word segmentation, sentence segmentation， New-Word-Discovery，新词发现

cjieba

Python cffi binding to CppJieba

symspellcpppy

Fast SymSpell written in c++ and exposes to python via pybind11

dksplit

High-performance string segmentation using BiLSTM-CRF

kitoken

Fast and versatile tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization

monpa

MONPA 罔拍是一個提供正體中文斷詞、詞性標註以及命名實體辨識的多任務模型

vgram

V-gram builder library