About Partners Use cases AI Development Repo Contact

ReQuEST: Reproducible Quality-Efficient Systems Tournaments

aka Multi-Objective HW/SW Co-Design Competitions for Emerging Workloads

News

Organizers and their affiliations

Long-term goals

Summary

ReQuEST is aimed at providing a scalable tournament framework, a common experimental methodology and an open repository for continuous evaluation and optimization of the quality vs. efficiency Pareto optimality of a wide range of real-world applications, libraries and models across the whole hardware/software stack on complete platforms.

Tournament framework goals

ReQuEST promotes reproducibility of experimental results and reusability of systems research artifacts by standardizing evaluation methodologies and facilitating the deployment of efficient solutions on heterogeneous platforms. The organizers develop a supporting open-source and portable workflow framework to provide unified evaluation and real-time leader-board of shared solutions.

Metrics and Pareto-optimality goals

ReQuEST promotes quality-awareness to the architecture and systems community, and resource-awareness to the applications community and end-users. We will keep and continuously update the best or original solutions close to a Pareto frontier in a multi-dimensional space of accuracy, execution time, power consumption, memory usage, resiliency, code/hardware/model size, costs and other metrics in a public repository. Eventually we may want to converge on a few meaningful metrics to assess quality on a per-application basis, and efficiency on a per-platform basis.

Application goals

In the long term, we will constitute a comprehensive suite of workloads, datasets and models covering applications domains that are most relevant to researchers in academia and industry (AI, Quantum, etc). This suite will evolve according to feedback and contributions from the community thus substituting ad-hoc, artificial, quickly outdated or non-representative benchmarks. Furthermore, all artifacts from this suite can be automatically plugged in to the ReQuEST competition workflows to simplify, automate and accelerate systems R&D.

Complete platforms goals

ReQuEST aims at covering a comprehensive set of hardware systems from data-centers down to sensory nodes, and incorporate various forms of processors including GPUs, DSPs, FPGAs, Neuromorphic and even analogue accelerators in the long term.

Open systems research goal

We attempt to put systems researchers, application writers and end-users on the same ground by providing a common evaluation framework and sharing all optimization results in an open and reproducible way. We expect that these results will be useful for
  • scientists and end-users to accelerate their applications by picking up the most efficient, resource-aware and input-adaptable solutions;
  • SW/HW researchers to have a reproducible and fair way to compare against and build upon each others' work;
  • system designers and integrators to develop the next generation of efficient hardware and software using realistic workloads and shared optimization results.

Open science goal

We hope that portable, reusable and customizable artifacts and workflows shared in a common CK format during our competitions will eventually help researchers not only reproduce results, but also quickly build upon them thus accelerating their research and enabling open science!

Open Tournament Framework

Submission

Authors need to submit a short document (4 pages max) describing the optimization technique and the whole experimental workflow with all related artifacts and evaluation methodology in a form of the ACM Artifact Appendix. We use Collective Knowledge framework (CK) to provide a common way for packing, sharing and evaluating submitted workflows since ACM is currently evaluating CK as an official way to share reusable, portable and customizable artifacts via ACM Digital Library, Therefore, authors are encouraged to convert their workflows to the CK format either themselves using a distinguished CGO'17 artifact as example (GitHub, AE Appendix, CK notes, CK portable workflows) or with our community help during a one-week workflow unification stage. Authors can also take advantage of reusable, customizable and portable AI artifacts already shared by the community in the CK format.

Allowed hardware

We restrict authors to 5-10 supported hardware platforms for initial runs of ReQuEST with the intention of extending coverage as participation increases:
  • Server-class: AWS, AWS F1, Azure, Intel Harp (Xeon+FPGA)
  • Mobile-class: Zynq-7000 board (FPGA), NVIDIA Jetson TX2 (GPU), Raspberry Pi 3 (CPU only), HiKey 960 (CPU / GPU)
  • IoT-class: Freescale FRDM KL03 Development Board (ARM M0 board)
The ReQuEST organizers will provide CK support and evaluation services for these hardware platforms. If a submission uses an exotic platform that is not included, its should either consider providing restricted access to the tournament organizers/reviewers, or at least notify in advance the organizers of their choice so that a similar platform can be acquired in time.

We plan to certify 1-2 power analyzers supported by our evaluation framework. The organizing committee may use a high-precision power analyzer (e.g. Yokogawa WT310 used for LPIRC costing $3,000-4,000) to calibrate a low-cost power analyzer (e.g. Hardkernel SmartPower2 costing $37), and use the calibration results when determining the winners.

Application coverage

For the first iteration of ReQuEST at ASPLOS'18, we focus on Deep Learning. Our first step is to provide coverage for the ImageNet Image Classification challenge suggested by our industrial advisory board. Restricting the competition to a single application domain will allow us to prepare a tournament infrastructure and validate it across multiple platforms, libraries, models and inputs. For future incarnations of ReQuEST, we will provide broader application coverage.

Evaluation and publication

We cooperate with the ACM Task Force on Data, Software, and Reproducibility in Publication and will use standard artifact evaluation methodology successfully validated at the leading ACM/IEEE computer systems conferences including PPoPP,CGO,PACT and SC to reproduce results and draw measured metrics on the ReQuEST live dashboard (see example).

ReQuEST will not determine a single winner, as collapsing all of the metrics into one single metric across all platforms will result in over-engineered solutions. Instead, each ReQuEST tournament will have a set of metrics evaluated by organizers or a separate Artifact Evaluation Committee (AEC) depending on applications of interest. Solutions do not have to be on the Pareto frontier to be accepted for the associated ReQuEST workshop and the open ReQuEST repository - a submission can be praised for its originality, reproducibility, adaptability, scalability, portability, ease of use, etc. However, submissions on the Pareto frontier will obtain a "Pareto-optimal" seal on their paper and special prizes from industrial sponsors. Descriptions of the accepted workflows will be published in the ACM Digital Library along with reusable artifacts (to be confirmed).

Industrial advisory board

To be announced - contact us if you are interested to join this board or know more!

Members of the ReQuEST industrial advisory board suggest realistic workloads, collaborate on a common methodology for reproducible evaluation and optimization, arrange access to rare hardware to Artifact Evaluation Committee (if needed), and provide prizes for the most efficient solutions.

ReQuEST workshops

We will use ReQuEST workshops associated with tournaments to let authors present and discuss their most efficient algorithms. We will also use workshops as an open forum to discuss how to improve our common reproducible methodology and framework for SW/HW co-design of emerging workloads with a broad academic and industrial community!

Feel free to contact us if you have questions or suggestions!

          
Website designed using CK
                      Locations of visitors to this page