Collective Knowledge Aggregator proof-of-concept
Crowd results Raw CK browser Graphs Reports Datasets Models Home

This page is outdated! New version is available here.


Collective Mind: towards practical and collaborative autotuning
Special issue on Automatic Performance Tuning for HPC Architectures, Scientific Programming Journal, IOS Press, August 2014
Grigori Fursin 1,2,3, Renato Miceli 4,5, Anton Lokhmotov 6, Michael Gerndt 7, Marc Baboulin 3, Allen D. Malony 8, Zbigniew Chamski 9, Diego Novillo 10, Davide Del Vento 11
1 cTuning Foundation, France, 2 INRIA, France, 3 University of Paris-Sud, France, 4 University of Rennes 1, France, 5 ICHEC, Dublin, Ireland, 6 ARM, UK, 7 Technical University of Munich, Germany, 8 University of Oregon, Eugene, OR, USA, 9 Infrasoft IT Solutions, Plock, Poland, 10 Google Inc., Toronto, Canada, 11 National Center for Atmospheric Research, Boulder, CO, USA
Corresponding author: Grigori.Fursin@cTuning.org

This live report demonstrates how CK can help create researchers reproducible and interactive article from reusable components


Here we provide details how to reproduce experiments from our papers via Collective Knowledge Framework and Repository (CK). All artifacts are shared via CK "reproduce-ck-paper" repository at GitHub. If you have CK installed (docs), you can download and install this repository via CMD simply as
    ck pull repo:reproduce-ck-paper
   
We would like to thank the community for validating various results and sharing unexpected behavior with us via our public CK repository!

Abstract

Empirical auto-tuning and machine learning techniques have been showing high potential to improve execution time, power consumption, code size, reliability and other important metrics of various applications for more than two decades. However, they are still far from widespread production use due to lack of native support for auto-tuning in an ever changing and complex software and hardware stack, large and multi-dimensional optimization spaces, excessively long exploration times, and lack of unified mechanisms for preserving and sharing of optimization knowledge and research material.

We present a possible collaborative approach to solve above problems using Collective Mind knowledge management system. In contrast with previous cTuning framework, this modular infrastructure allows to preserve and share through the Internet the whole auto-tuning setups with all related artifacts and their software and hardware dependencies besides just performance data. It also allows to gradually structure, systematize and describe all available research material including tools, benchmarks, data sets, search strategies and machine learning models. Researchers can take advantage of shared components and data with extensible meta-description to quickly and collaboratively validate and improve existing auto-tuning and benchmarking techniques or prototype new ones. The community can now gradually learn and improve complex behavior of all existing computer systems while exposing behavior anomalies or model mispredictions to an interdisciplinary community in a reproducible way for further analysis. We present several practical, collaborative and model-driven auto-tuning scenarios. We also decided to release all material at c-mind.org/repo to set up an example for a collaborative and reproducible research as well as our new publication model in computer engineering where experimental results are continuously shared and validated by the community.


Raising number of compiler flags in GCC

Reproducibility: CK scripts with CID=84e27ad9dd12e734:bbf4abb470b0271a

Note the trend of slowing down total number of boolean optimizations and rising number of parameteric optimizations.

Unified, reproducible, multi-objective, compiler flag autotuning examples powered by CK

We prepared Getting Started Guide about how to reproduce various autotuning scenarios (including from this paper) via CK in your environment (hardware and software). You can find it here.

Interactive graph of slambench v1.1 cpu autotuning

Reproduce/reuse/replay/discuss via CK (interactive graphs)

  • Samsung ChromeBook 2; Samsung EXYNOS5; ARM Cortex A15/A7; ARM Mali-T628; Ubuntu 12.04; video 640x480
  • Big green dot: Clang 3.6.0 -O3
  • Big red dot: GCC 4.9.2 -O3
  • Small blue dots: random GCC 4.9.2. optimization flags (with 10% probablitiy of a selection of a given flag)
  • Small red dots: Pareto frontier
More examples of reproducible, collaborative and live slambench experiments via CK:

Non-interactive graph of image corner detection algorith (susan corners) autotuning

Reproduce/reuse/replay/discuss via CK (interactive graphs)

  • LENOVO X240; Intel(R) Core(TM) i5-4210U CPU @ 1.70GHz; Intel HD Graphics 4400; Windows 7 Pro
  • Image image-pgm-0001; 600x450; 270015 bytes
  • Black square dot: MingW GCC 4.9.2 -O3
  • Blue dots: random optimization flags (with 10% probablitiy of a selection of a given flag)
  • Red line: Pareto frontier


Non-interactive graph of analysis of execution time variation of image/neuron filter kernel versus different frequency

CK scripts: reproduce.py
  • Raw experiments in CK: high frequency, low frequency
  • Average time (green line) doesn't make sense. Normality test also fails. So most of the time, researchers will simply skip this result.
  • In CK, we analyze variation via gaussian KDE from scipy, which returns 2 expected values in the following graphs suggesting that there are several states in experiments and hence some features are missing, that can separate these states. In our case, the feature is frequency that was added to program pipeline. See paper for more details.

Density graph

Histogram graph


Note: CK allows the community validate the results and share unexpected behavior in public cknowledge.org/repo here.

Note: we are gradually converting all the code and data related to this paper from the deprecated Collective Mind Format to the new Collective Knowledge Framework.

View entry in raw format

Developed by Grigori Fursin           
Implemented as a CK workflow
                         
   
                      Hosted at