Call for Pareto efficient deep learning
ReQuEST: 1st Reproducible Quality-Efficient Systems Tournament
March 24th, 2018 (afternoon), Williamsburg, VA, USA
(co-located with ASPLOS 2018)
Co-designing emerging workloads across the hardware/software
stack to optimize for speed, accuracy, energy consumption, $ and
other metrics is extremely complex and time consuming. The lack
of a rigorous common methodology for open, reproducible and
multi-objective optimization makes it challenging to evaluate
and compare different published works across numerous
heterogeneous hardware platforms, software frameworks,
compilers, libraries, algorithms, data sets, and environments.
The 1st ReQuEST workshop aims to bring together
multidisciplinary researchers in systems, compilers,
architecture to optimize in a reproducible, automated and comparable
fashion the quality vs. efficiency Pareto optimality of emerging
workloads on complete hardware/software systems.
The target application for the first incarnation of ReQuEST will
be the ImageNet Large Scale Visual Recognition Challenge
(ILSVRC) and will focus solely on optimizing inference.
Unlike the classical ILSVRC where submissions are ranked based
on accuracy, submissions will be evaluated across multiple
metrics and trade-offs: throughput, energy, cost of platform, etc.
Workshop participants will be asked to submit a workflow
artifact which encompasses toolchains, frameworks, algorithm,
libraries, and target hardware platform; any of which can be
fine-tuned, or customized at will by the participant
to implement the optimizations of their choice.
Example optimizations include:
- Model compression techniques to increase efficiency of the target system.
- Operator-level binarization techniques to improve inference time.
- Library optimizations targeting novel operators (e.g. depthwise convolution).
- FPGA acceleration that takes advantage of narrow integer types.
A ReQuEST artifact evaluation committee (AEC) will be tasked
to reproduce and evaluate workflow submissions on compliant
hardware platforms to aggregate results in a multi-objective
public leaderboard. Due to the multi-faceted nature of the
competition, submissions won't be ranked according to a single
metric, but instead the AEC will assess their Pareto optimality
across two or more metrics. There won't be a single ranking
of submissions since this competition is multi-objective:
it accounts for classification accuracy, inference latency,
energy, $ cost of the platform and TCO. As such, there won't
be a single winner, but better and worse designs based on their
relative Pareto optimality.
Additionally, the AEC will leverate
the Collective Knowledge framework (CK)
to convert submitted algorithms to common workflows,
automate empirical experiments across diverse platforms,
package workflow submissions as reusable, portable and customizable artifacts,
make them accessible to the public domain, and let researchers quickly build upon them.
An industrial board will overview evaluation
and provide prizes for the top submissions in different categories
of interest which they will define.
The workshop will also be the opportunity for the participants
to share their research insights with the research community.
Participants are asked to submit a 4-page report providing
insights the their implementation. Specifically, while the
novelty of the techniques used are not a requirement to get the
paper accepted, novel insights on using different deep learning
systems (TensorFlow vs. MXNet), models (SqueezeNet vs.
MobileNets), combining optimizations (binarization and model
compression), design space exploration, or hyper-parameter
tuning, and accuracy-energy trade-offs will be welcome.
In return, the selected papers will be published at the ACM
Digital Library and the authors will be invited to present
insights on their implementation at the workshop. A common
academic and industrial panel will be held at the end of the
workshop to discuss insights and results from the tournament
to continue improving upon common co-design methodology and
framework for deep learning and other real-world applications.
Important dates (preliminary)
- Intent to submit: January 19, 2018 AoE
- Unifying experimental workflows to the CK format: January 19-January 31, 2018
- Submissions due: February 2, 2018 AoE
- Artifact Evaluation: February 5-February 14, 2018
- Author notification: February 16, 2018
- Revised artifacts and papers due: February 23, 2018
- Workshop with presentations and discussions of winning workflows: March 24, 2018
- Luis Ceze, University of Washington, USA
- Natalie Enright Jerger, University of Toronto, Canada
- Babak Falsafi, EPFL, Switzerland
- Grigori Fursin, cTuning foundation, France (Framework chair)
- Anton Lokhmotov, dividiti, UK (Industrial chair)
- Thierry Moreau, University of Washington, USA (Publicity chair)
- Adrian Sampson, Cornell University, USA
- Phillip Stanley Marbell, University of Cambridge, UK
Industrial advisory board
To be announced - contact us
if you are interested to join this board or know more!
Members of the ReQuEST industrial advisory board suggest realistic workloads,
collaborate on a common methodology for reproducible evaluation and optimization,
arrange access to rare hardware to Artifact Evaluation Committee (if needed),
and provide prizes for the most efficient solutions.
The authors are invited to submit a short document (4 pages max)
describing the whole deep learning workflow and optimization
technique in a form of the ACM Artifact Appendix
Organizers will use standard ACM artifact evaluation methodology
to reproduce results and draw measured metrics on the live ReQuEST dashboard similar to
CK live repo
Description of the winning workflow together with all related artifacts
will be published in the ACM Digital Library