About Partners Use cases AI Development Repo Contact

Call for Pareto efficient deep learning

ReQuEST: 1st Reproducible Quality-Efficient Systems Tournament

March 24th, 2018 (afternoon), Williamsburg, VA, USA
(co-located with ASPLOS 2018)

cKnowledge.org/request



Co-designing emerging workloads across the hardware/software stack to optimize for speed, accuracy, energy consumption, $ and other metrics is extremely complex and time consuming. The lack of a rigorous common methodology for open, reproducible and multi-objective optimization makes it challenging to evaluate and compare different published works across numerous heterogeneous hardware platforms, software frameworks, compilers, libraries, algorithms, data sets, and environments.

The 1st ReQuEST workshop aims to bring together multidisciplinary researchers in systems, compilers, architecture to optimize in a reproducible, automated and comparable fashion the quality vs. efficiency Pareto optimality of emerging workloads on complete hardware/software systems.

The target application for the first incarnation of ReQuEST will be the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) and will focus solely on optimizing inference. Unlike the classical ILSVRC where submissions are ranked based on accuracy, submissions will be evaluated across multiple metrics and trade-offs: throughput, energy, cost of platform, etc. Workshop participants will be asked to submit a workflow artifact which encompasses toolchains, frameworks, algorithm, libraries, and target hardware platform; any of which can be fine-tuned, or customized at will by the participant to implement the optimizations of their choice.

Example optimizations include:

A ReQuEST artifact evaluation committee (AEC) will be tasked to reproduce and evaluate workflow submissions on compliant hardware platforms to aggregate results in a multi-objective public leaderboard. Due to the multi-faceted nature of the competition, submissions won't be ranked according to a single metric, but instead the AEC will assess their Pareto optimality across two or more metrics. There won't be a single ranking of submissions since this competition is multi-objective: it accounts for classification accuracy, inference latency, energy, $ cost of the platform and TCO. As such, there won't be a single winner, but better and worse designs based on their relative Pareto optimality.

Additionally, the AEC will leverate the Collective Knowledge framework (CK) to convert submitted algorithms to common workflows, automate empirical experiments across diverse platforms, package workflow submissions as reusable, portable and customizable artifacts, make them accessible to the public domain, and let researchers quickly build upon them. An industrial board will overview evaluation and provide prizes for the top submissions in different categories of interest which they will define.

The workshop will also be the opportunity for the participants to share their research insights with the research community. Participants are asked to submit a 4-page report providing insights the their implementation. Specifically, while the novelty of the techniques used are not a requirement to get the paper accepted, novel insights on using different deep learning systems (TensorFlow vs. MXNet), models (SqueezeNet vs. MobileNets), combining optimizations (binarization and model compression), design space exploration, or hyper-parameter tuning, and accuracy-energy trade-offs will be welcome. In return, the selected papers will be published at the ACM Digital Library and the authors will be invited to present insights on their implementation at the workshop. A common academic and industrial panel will be held at the end of the workshop to discuss insights and results from the tournament to continue improving upon common co-design methodology and framework for deep learning and other real-world applications.

Important dates (preliminary)

Organizers (A-Z)

Industrial advisory board

To be announced - contact us if you are interested to join this board or know more!

Members of the ReQuEST industrial advisory board suggest realistic workloads, collaborate on a common methodology for reproducible evaluation and optimization, arrange access to rare hardware to Artifact Evaluation Committee (if needed), and provide prizes for the most efficient solutions.

Submission guidelines

The authors are invited to submit a short document (4 pages max) describing the whole deep learning workflow and optimization technique in a form of the ACM Artifact Appendix. Organizers will use standard ACM artifact evaluation methodology to reproduce results and draw measured metrics on the live ReQuEST dashboard similar to CK live repo. Description of the winning workflow together with all related artifacts will be published in the ACM Digital Library.

          
Website designed using CK
                      Locations of visitors to this page