GigaToken promises 1000x speed boost for language model tokenisation

A new open-source utility named GigaToken has been released on GitHub, claiming to deliver language model tokenisation speeds approximately 1000 times faster than the widely used HuggingFace tokenisers. Developed by Marcel Roed, the tool is written in Rust and is positioned as a high-performance alternative for processing large-scale text datasets, with benchmarks suggesting it could tokenize the entirety of Common Crawl in under 6.5 hours on suitable hardware.

The utility supports a broad spectrum of CPU architectures, including modern x86 and ARM systems, and is compatible with nearly all commonly used tokenisers. Roed attributes the substantial performance gains to heavily optimised pretokenisation using SIMD instructions, minimised branching, and efficient caching of pretoken mappings. These technical adjustments allow the Rust implementation to read data directly, bypassing much of the overhead associated with traditional implementations.

While GigaToken offers a compatibility mode for HuggingFace Tokenisers and Tiktoken, the developer notes that this approach incurs a non-negligible performance overhead compared to the native API. Although outputs are designed to match exactly, users seeking the maximum 1000x speed improvement must utilise the Gigatoken API directly. This method allows for maximum parallelism but still involves some overhead when passing Python data structures through the interface.

Benchmarks conducted using the OpenWebText dataset, which serves as a representative sample of text extracted from Common Crawl documents, demonstrate significant throughput improvements. On high-end hardware, specifically EPYC CPUs, the tool shows the potential to process vast amounts of data rapidly. However, the developer acknowledges that SentencePiece-based tokenisers currently exhibit the slowest performance within the tool due to suboptimal optimisation in this specific area.

Roed has invited the community to test the tool and report issues on GitHub, noting that while testing has been reasonably wide, not every use-case has been verified. Users on macOS are advised to run commands twice to obtain accurate timing readings, as the initial execution may be slowed by a security scan that affects the Rust code. The tool is available for immediate use via command line without installation, with documentation provided for citation and further technical details.