Open-source community launches public leaderboard for LoRA fine-tuning speed

A new GitHub repository titled 'lora-speedrun' has introduced a public leaderboard designed to benchmark Low-Rank Adaptation (LoRA) fine-tuning performance. The initiative seeks to establish an apples-to-apples comparison framework for parameter-efficient fine-tuning techniques, addressing the fragmentation that currently hinders direct comparisons across academic literature and industry benchmarks.

Participants are required to fine-tune the Qwen2.5-1.5B model to achieve at least 57 per cent accuracy on the GSM8K dataset. The competition mandates the use of a single NVIDIA L40S GPU via the Modal cloud platform to ensure hardware consistency. The current record stands at 6 minutes and 5 seconds, set by repository author Saivineeth147, who utilised sequence packing and completion-only loss masking to achieve approximately twice the speed of the baseline at higher accuracy.

The project adapts the 'modded-nanogpt' framework, previously used to optimise pretraining speeds, for the fine-tuning domain. Verification is a strict requirement; every record must be independently re-run three times with fresh seeds on identical hardware before it is accepted. This protocol aims to eliminate variance and ensure that reported improvements are reproducible and genuine.

Wall-clock time is prioritised over floating-point operations or step counts, reflecting the actual computational costs incurred by users. The choice of the NVIDIA L40S, which shares the same AD102 silicon as the consumer-grade RTX 4090, allows insights from consumer GPU optimisation to transfer to this datacentre SKU. Additionally, the use of Modal’s free monthly compute credits ensures that participation and verification are free, lowering the barrier to entry for researchers and engineers.

The rules are stringent to maintain comparability. Participants are restricted to the GSM8K training split, and practices such as distillation from larger models, synthetic data generation, or the use of multiple GPUs are prohibited. The repository is licensed under the MIT licence, with all records, reports, and technical write-ups made public to foster transparency and collaborative improvement in the field.