Automatically running QED-C benchmarks on Metriq

We are excited to announce the integration of an automated pipeline to add state-of-the-art benchmark from the Quantum Economic Development Consortium (QED-C) into Metriq, the open platform that makes transparent, accessible benchmarks available to everyone in the quantum computing community.

The Technical Advisory Committee (TAC) for Standards and Performance Metrics from QED-C provided an open-source suite of quantum benchmarks (GitHub) to measure the effectiveness of quantum computing hardware for specific applications (further information provided in this preprint). This provided some of the first application-oriented benchmarks that were made public with supporting data along with a preprint, an incredibly valuable asset to the benchmarking community. Recently, the QED-C benchmarks suite was updated and new optimization benchmarks (preprint).

The results of running these benchmarks can be found via this Zenodo posting. On Metriq, we have integrated these results into a submission that captures these results. While this submission on Metriq captures the results obtained by QED-C it also is a “living” result in that it is automatically updated with benchmarks obtained by running the QED-C code on a variety of hardware offerings.

On Metriq, we have established an automated pipeline that is able to run and update our submission via a fork of the QED-C benchmarking suite. This pipeline enables anyone to automatically run benchmarks for a specific algorithm, hardware provider, and quantum processor. This allows the Metriq community to continually extend the benchmarking foundation provided by QED-C to expand their results by running on further hardware providers to obtain up-to-date results.

The backbone of the pipeline is powered by the metriq-client project. This client allows the user to interface with the Metriq platform via a Python API. The user can perform queries or upload batched results via a Python script for submissions that may entail a sizeable amount of results that would be cumbersome to upload manually via the website.

Many new results have been added to the QED-C submission for the Bernstein-Vazarani task on Metriq thanks to the automatic QED-C to Metriq pipeline.

The automated QED-C to Metriq pipeline will be continuing to run on more algorithms and platforms to generate further benchmarking results to be displayed on Metriq!