We are developing a common methodology, framework and decentralized marketplace for collaborative and reproducible benchmarking, optimization and co-design of efficient software/hardware stack for emerging AI/ML algorithms (inference, object detection, training, etc) requested by our industrial advisory board in terms of speed, accuracy, energy, size, complexity, costs and other metrics.
We then organize open ReQuEST competitions bringing together AI, ML and systems researchers to share complete algorithm implementations (code and data) as portable, customizable and reusable Collective Knowledge workflows and components with a unified JSON API. This helps other researchers and end-users to quickly validate such results, reuse workflows and optimize/autotune algorithms across different platforms, models, data sets, libraries, compilers and tools. We also use our practical experience reproducing experimental results from ReQuEST submissions to help set up artifact evaluation at the upcoming SysML 2019, and to suggest new algorithms as customizable CK workflows for the inclusion to the MLPerf benchmark.
Researchers can then quickly assemble novel AI/ML algorithms from such components as LEGO(tm), extend them, fix bugs, and even crowdsource their optimization using devices from IoT to supercomputers provided by volunteers or sponsors (see Android app and MobileNets crowd-benchmarking results using TFLite framework for example). Furthermore, we collect mispredictions (!) from the community to create a realistic and evolving training set which can be used to improve models or even implement self-learning on edge-devices!
See ACM ReQuEST-ASPLOS'18 results report, ACM DL proceedings, GitHub with shared workflows and the latest reproducible results at the live ReQuEST scoreboard for more details.
Feel free to try CK yourself on practically any platform with Linux, Windows and MacOS using the following simple getting started guide. Contact us if you would like to join the growing CK consortium led by the non-profit cTuning foundation and dividiti Ltd to influence the following community activities:
We hope that our CK approach of portable, customizable, reusable and optimized AI/ML workflows and components combined with agile, DevOps and Wikipedia-style development will help to accelerate adoption of AI while dramatically reducing R&D costs and boosting innovation as briefly described in this presentation.