Co-designing efficient machine learning based systems across the whole hardware/software stack to trade off speed, accuracy, energy and costs is becoming extremely complex and time consuming. Researchers often struggle to evaluate and compare different published works across rapidly evolving software frameworks, heterogeneous hardware platforms, compilers, libraries, algorithms, data sets, models, and environments.

We  present our community effort to develop an open co-design tournament platform with an online public scoreboard. It will gradually incorporate best research practices while providing a common way for multidisciplinary researchers to optimize and compare the quality vs. efficiency Pareto optimality of various workloads on diverse and complete hardware/software systems. We want to leverage the open-source Collective Knowledge framework and the ACM artifact evaluation methodology to validate and share the complete machine learning system implementations in a standardized, portable, and reproducible fashion. We plan to hold regular multi-objective optimization and co-design tournaments for emerging workloads such as deep learning, starting with ASPLOS'18 to build a public repository of the most efficient algorithms and systems which can be easily reused and built upon. We will also use the feedback from participants to continue improving our platform and common co-design methodology.

1  Introduction

Figure 1



Collective Knowledge framework as an open platform to support open tournaments for software/hardware co-design of Pareto-efficient deep learning and other emerging workloads in terms of speed, accuracy, energy and various costs.

We therefore propose the Reproducible Quality-Efficient Systems Tournament (ReQuEST) as a community-driven platform for reproducible, comparable, and multi-objective optimization of emerging workloads  [1]. We plan to host ReQuEST as a bi-annual workshop alternating between systems and machine learning communities. Its first incarnation will take place at ASPLOS in March 2018 and will focus solely on optimizing inference on real systems to test our platform and use the feedback from participants and an industrial board to improve it.

We detail in the next sections how its objectives and execution differentiate ReQuEST from other existing workshop-based competitions.

2  Main goals

Summary: ReQuEST is aimed at providing a scalable tournament framework, a common experimental methodology and an open repository for continuous evaluation and optimization of the quality vs. efficiency Pareto optimality of a wide range of real-world applications, libraries, and models across the whole hardware/software stack on complete platforms as conceptually shown in Figure 1.

Tournament framework goals: we want to promote reproducibility of experimental results and reusability/customization of systems research artifacts by standardizing evaluation methodologies and facilitating the deployment of efficient solutions on heterogeneous platforms. For that reason, packaging artifacts and experimental results requires a bit more involvement than sharing some CSV files or checking out a given GitHub repository.

Figure 2

An example of a live Collective Knowledge scoreboard to crowd-benchmark inference in terms of speed and platform cost across diverse deep learning frameworks, models, data sets, and Android devices provided by volunteers. Red dots are associated with the winning workflows (model/software/hardware).

That is why we build our competition on top of an open-source and portable workflow framework (Collective Knowledge or CK [2]) and a standard ACM artifact evaluation methodology [3] from premier ACM systems conferences (CGO, PPoPP, PACT, SuperComputing) to provide unified evaluation and a live scoreboard of submissions as demonstrated in Figure 2.

CK is a Python wrapper framework to share artifacts and workflows as customizable and reusable plugins with a common JSON API and meta description, and adaptable to a user platform with Linux, Windows, MacOS and Android. For example, it has already been used and extended in a number of academic and industrial projects to automate and crowdsource benchmarking and multi-objective optimization of deep learning across diverse platforms, environments, and data sets. Figure 2 shows a proof-of-concept example of a live scoreboard powered by CK to collaboratively benchmark inference (speed vs. platform cost) across diverse deep learning frameworks (TensorFlow, Caffe, MXNet, etc.), models (AlexNet, GoogleNet, SqueezeNet, ResNet, etc.), real user data sets, and mobile devices provided by volunteers (see the latest results at cKnowledge.org/repo).

Metrics and Pareto-optimality goals: we want to stress quality-awareness to the architecture/compilers/systems community, and resource-awareness to the applications/algorithms community and end-users. The submissions and their evaluation metrics will be maintained in a public repository that includes a live scoreboard.

Specific attention will be brought to submissions close to a Pareto frontier in a multi-dimensional space of accuracy, execution time, power/energy consumption, hardware/code/model footprint, monetary costs etc.

Application goals: in the long term, we will cover a comprehensive suite of workloads, datasets and models covering applications domains that are most relevant to machine learning and systems researchers. This suite will continue evolving according to feedback and contributions from the academia and industry. All artifacts from this suite can be automatically plugged in to the ReQuEST competition workflows to simplify and automate experimentation.

Complete platforms goals: we aim to cover a comprehensive set of hardware systems from data-centers down to sensory nodes, incorporating various forms of processors including GPUs, DSPs, FPGAs, neuromorphic and even analogue accelerators in the long term.

3  Future work

Our goal is to bring multi-disciplinary researchers to \begin{enumerate}