ReQuEST is aimed at providing a scalable tournament framework,
a common experimental methodology and an open repository
for continuous evaluation and optimization
of the quality vs. efficiency
Pareto optimality of a wide range of real-world applications,
libraries and models across the whole hardware/software stack
on complete platforms.
Tournament framework goals
ReQuEST promotes reproducibility of experimental results
and reusability of systems research artifacts
by standardizing evaluation methodologies
and facilitating the deployment of efficient solutions
on heterogeneous platforms.
The organizers develop a supporting open-source and portable
workflow framework to provide unified evaluation and real-time leader-board
of shared solutions.
Metrics and Pareto-optimality goals
ReQuEST promotes quality-awareness to the architecture and systems community,
and resource-awareness to the applications community and end-users.
We will keep and continuously update the best or original solutions
close to a Pareto frontier in a multi-dimensional space
of accuracy, execution time, power consumption,
memory usage, resiliency, code/hardware/model size, costs
and other metrics in a public repository.
Eventually we may want to converge on a few meaningful metrics
to assess quality on a per-application basis,
and efficiency on a per-platform basis.
In the long term, we will constitute a comprehensive suite of workloads, datasets and models
covering applications domains that are most relevant to researchers in academia and industry
(AI, Quantum, etc).
This suite will evolve according to feedback and contributions from the community
thus substituting ad-hoc, artificial, quickly outdated or non-representative benchmarks.
Furthermore, all artifacts from this suite can be automatically plugged in to the ReQuEST
competition workflows to simplify, automate and accelerate systems R&D.
Complete platforms goals
ReQuEST aims at covering a comprehensive set of hardware systems from data-centers down
to sensory nodes, and incorporate various forms of processors including GPUs, DSPs, FPGAs,
Neuromorphic and even analogue accelerators in the long term.
Open systems research goal
We attempt to put systems researchers, application writers and end-users on the same ground
by providing a common evaluation framework and sharing all optimization results
in an open and reproducible way.
We expect that these results will be useful for
scientists and end-users to accelerate their applications
by picking up the most efficient, resource-aware and input-adaptable solutions;
SW/HW researchers to have a reproducible and fair way to compare against and build upon each others' work;
system designers and integrators to develop the next generation of efficient hardware and software
using realistic workloads and shared optimization results.
Open science goal
We hope that portable, reusable and customizable artifacts and workflows shared in a common CK format
during our competitions will eventually help researchers not only reproduce results, but also quickly build upon them
thus accelerating their research and enabling open science!
Authors need to submit a short document (4 pages max) describing the optimization technique
and the whole experimental workflow with all related artifacts and evaluation methodology
in a form of the ACM Artifact Appendix
We use Collective Knowledge framework (CK)
to provide a common way for packing, sharing and evaluating submitted workflows
since ACM is currently evaluating CK
as an official way to share reusable, portable and customizable artifacts via ACM Digital Library,
Therefore, authors are encouraged to convert their workflows to the CK format either themselves
using a distinguished CGO'17 artifact as example
CK portable workflows
or with our community help
during a one-week workflow unification stage
Authors can also take advantage of reusable, customizable and portable AI artifacts
already shared by the community
in the CK format.
We restrict authors to 5-10 supported hardware platforms
for initial runs of ReQuEST with the intention of extending
coverage as participation increases:
Server-class: AWS, AWS F1, Azure, Intel Harp (Xeon+FPGA)
Mobile-class: Zynq-7000 board (FPGA), NVIDIA Jetson TX2 (GPU), Raspberry Pi 3 (CPU only), HiKey 960 (CPU / GPU)
IoT-class: Freescale FRDM KL03 Development Board (ARM M0 board)
The ReQuEST organizers will provide CK support and evaluation services for these hardware platforms.
If a submission uses an exotic platform that is not included,
its should either consider providing restricted access
to the tournament organizers/reviewers, or at least notify
in advance the organizers of their choice so that a similar platform
can be acquired in time.
We plan to certify 1-2 power analyzers supported
by our evaluation framework. The organizing
committee may use a high-precision power analyzer
(e.g. Yokogawa WT310 used for LPIRC costing $3,000-4,000)
to calibrate a low-cost power analyzer (e.g. Hardkernel
SmartPower2 costing $37), and use the calibration results
when determining the winners.
For the first iteration of ReQuEST at ASPLOS'18, we focus on Deep Learning.
Our first step is to provide coverage for the ImageNet Image Classification challenge
suggested by our industrial advisory board.
Restricting the competition to a single application domain will allow us to prepare a tournament
infrastructure and validate it across multiple platforms, libraries, models and inputs.
For future incarnations of ReQuEST, we will provide broader application coverage.
Evaluation and publication
We cooperate with the ACM Task Force on Data, Software, and Reproducibility in Publication
and will use standard artifact evaluation methodology
successfully validated at the leading ACM/IEEE computer systems conferences including PPoPP,CGO,PACT and SC
to reproduce results and draw measured metrics on the ReQuEST live dashboard (see example)
ReQuEST will not determine a single winner, as collapsing all of the metrics into one single metric across all platforms
will result in over-engineered solutions. Instead, each ReQuEST tournament will have a set of metrics
evaluated by organizers or a separate Artifact Evaluation Committee (AEC) depending on applications of interest.
Solutions do not have to be on the Pareto frontier to be accepted for the associated ReQuEST workshop
and the open ReQuEST repository -
a submission can be praised for its originality, reproducibility,
adaptability, scalability, portability, ease of use, etc. However, submissions on the Pareto frontier
will obtain a "Pareto-optimal" seal on their paper and special prizes from industrial sponsors.
Descriptions of the accepted workflows will be published in the ACM Digital Library
along with reusable artifacts (to be confirmed).
To be announced - contact us
if you are interested to join this board or know more!
Members of the ReQuEST industrial advisory board suggest realistic workloads,
collaborate on a common methodology for reproducible evaluation and optimization,
arrange access to rare hardware to Artifact Evaluation Committee (if needed),
and provide prizes for the most efficient solutions.