News
AscendGYM introduces a cost-aware benchmark for kernel optimization on Ascend NPUs, featuring real-world workloads, STS metric, and open-source tools.
The AI Training Platform Team, led by Bingwei Zhang, Jing Li, Amir Abdullin, and others, has introduced AscendGYM, a new benchmark designed to evaluate kernel optimization capabilities on Ascend NPUs. Published on arXiv on November 18, 2026, the project aims to address gaps in existing benchmarks by focusing on real-world workloads and cost-aware performance metrics.
Benchmark Overview and Features
AscendGYM provides a suite of 48 kernels categorized into three difficulty levels. These kernels are paired with test cases derived from real-world training and inference tasks, ensuring that performance improvements directly translate to practical applications. The benchmark supports two evaluation modes: model evaluation and model-harness evaluation. Both modes allow for the assessment of trained models and custom agents within a complete execution environment that closely replicates real-world Ascend kernel pipelines.
A notable innovation is the introduction of the Speedup-Token Score (STS), a metric that balances performance speedup against token cost. This enables users to evaluate kernel optimization capabilities under predefined budget constraints, making the benchmark particularly relevant for cost-sensitive AI deployments.
Open-Source Implementation and Usability
The AscendGYM project is open-source and hosted on GitHub at https://github.com/SLAI-AITP/AscendGYM. The repository includes task definitions, evaluation tools, and documentation to facilitate adoption. Users can prepare tasks, conduct correctness tests, measure performance, and analyze results through a unified interface. The execution environment supports both official AscendForge images and self-hosted services, ensuring flexibility for diverse user needs.
Additionally, an interactive leaderboard allows participants to compare results across models, harnesses, and operators. Metrics such as speedup ratios, token consumption, and STS scores are displayed, providing transparency and fostering competition among developers.
Significance and Future Directions
By addressing the practical challenges of kernel optimization for large language models on Ascend NPUs, AscendGYM sets a new standard for benchmarking in this domain. Its focus on real-world workloads and cost-aware metrics ensures relevance for both academic research and industry applications. As the project evolves, additional tasks and features are expected to be integrated, further enhancing its utility for the AI community.
Sign in to save this article
Free account: build your research collection and return to read it anytime.
