2 Main goals
Summary: ReQuEST is aimed at providing a scalable tournament framework,
a common experimental methodology and an open repository for continuous evaluation
and optimization of the quality vs. efficiency Pareto optimality of a wide range
of real-world applications, libraries, and models across the whole
hardware/software stack on complete platforms as conceptually shown
in Figure 1.
Tournament framework goals: we want to promote reproducibility
of experimental results and reusability/customization of systems research artifacts
by standardizing evaluation methodologies and facilitating the deployment
of efficient solutions on heterogeneous platforms.
For that reason, packaging artifacts and experimental results requires a bit more
involvement than sharing some CSV files or checking out a given GitHub repository.
That is why we build our competition on top of an open-source
and portable workflow framework (Collective Knowledge or CK )
and a standard ACM artifact evaluation methodology 
from premier ACM systems conferences (CGO, PPoPP, PACT, SuperComputing)
to provide unified evaluation and a live scoreboard of submissions
as demonstrated in Figure 2.
CK is a Python wrapper framework to share artifacts and workflows
as customizable and reusable plugins with a common JSON API and meta description,
and adaptable to a user platform with Linux, Windows, MacOS and Android.
For example, it has already been used and extended in a number of academic and industrial projects
to automate and crowdsource benchmarking and multi-objective optimization of deep learning
across diverse platforms, environments, and data sets.
Figure 2 shows a proof-of-concept example of a live scoreboard
powered by CK to collaboratively benchmark inference (speed vs. platform cost)
across diverse deep learning frameworks (TensorFlow, Caffe, MXNet, etc.),
models (AlexNet, GoogleNet, SqueezeNet, ResNet, etc.), real user data sets, and mobile devices
provided by volunteers (see the latest results at cKnowledge.org/repo).
Metrics and Pareto-optimality goals: we want to stress
quality-awareness to the architecture/compilers/systems community,
and resource-awareness to the applications/algorithms community and
end-users. The submissions and their evaluation metrics will
be maintained in a public repository that includes a live
Specific attention will be brought to submissions
close to a Pareto frontier in a multi-dimensional
space of accuracy, execution time, power/energy consumption,
hardware/code/model footprint, monetary costs etc.
Application goals: in the long term, we will
cover a comprehensive suite of workloads, datasets and models
covering applications domains that are most relevant
to machine learning and systems researchers.
This suite will continue evolving according
to feedback and contributions from the academia and industry.
All artifacts from this suite can be automatically plugged
in to the ReQuEST competition workflows to simplify and automate
Complete platforms goals: we aim to cover
a comprehensive set of hardware systems from data-centers down
to sensory nodes, incorporating various forms of processors
including GPUs, DSPs, FPGAs, neuromorphic and even analogue
accelerators in the long term.
3 Future work
Our goal is to bring multi-disciplinary researchers to
release research artifacts of their on-going or accomplished research,
standardize evaluation workflows, and facilitate deployment
and tech transfer of state-of-the-art research,
foster exploration of quality-efficiency trade-offs, and
create a discussion ground to steer the community towards new
applications, frameworks, and hardware platforms.
We want to set a coherent research roadmap for
researchers by hosting bi-annual tournaments complemented with
panel discussions from both academia and industry. We hope
that as participation increases, the coverage of problems
(vision, speech and even beyond machine learning) and
platforms (novel hardware accelerators, SoCs, and even exotic
hardware such as analog, neuromorphic, stochasticm, quantum) will increase.
ReQuEST is organized as a bi-annual workshop, alternating
between systems-oriented and machine learning-oriented
conferences. The first ReQuEST workshop will be co-located
with ASPLOS in March 2018. The workshop will aim to present
artifacts submitted by participants, along with
a multi-objective scoreboard, where quality-efficient
implementations will be rewarded.
The submissions will be validated by an artifact evaluation committee,
and participants will have the chance to get an artifact paper
published as ACM proceedings.
In addition we wish to nurture
a discussion ground for artifact evaluation in multidisciplinary
research, gathering perspectives from machine learning,
systems, compilers and architecture experts.
We will use this discussion to continuously improve and extend
functionality of our tournament platform.
For example, we plan to gradually standardize the API and meta description of all artifacts
and machine learning workflows with the help of the community,
provide architectural simulators and simulator-based evaluations,
cover low-level optimizations, expose more metrics, and so on.
Finally, an industrial panel composed of research-representatives from
prominent software and hardware companies will discuss how
tech-transfer can be facilitated between academia and
industry, and will help craft the roadmap for the ReQuEST
workshops by suggesting new datasets, workloads, metrics, and
| ||ReQuEST Initiative.
ReQuEST: open tournaments on collaborative, reproducible and
pareto-efficient software/hardware co-design of emerging workloads using the
collective knowledge technology.
| ||G. Fursin, A. Lokhmotov, and E. Plowman.
Collective Knowledge: towards R&D sustainability.
In Proceedings of the Conference on Design, Automation and Test
in Europe (DATE'16), March 2016.
| ||Artifact Evaluation Initiative.
Artifact Evaluation for computer systems conferences.