torch-optimizer -- collection of optimizers for Pytorch
SCAO is a sparse, second-order PyTorch optimizer designed as a high-throughput, drop-in replacement for AdamW.
