About Partners Use cases AI Development Repo Contact
Join the international CK consortium developing an open-source platform to enable intelligent and efficient systems assembled from portable, customizable and reusable AI/ML components continously optimized/autotuned/crowd-tuned in terms of accuracy, speed, energy, resiliency and costs across diverse models, data sets and platforms from a cloud to edge via open, fair and reproducible competitions!

Need to survive in a Cambrian AI/SW/HW chaos and accelerate AI research?

Collective Knowledge platform was developed by researchers to connect multidisiplinary AI/ML/systems communities and help them share artifacts, workflows and knowledge as portable, customizable and reusable components with a unified JSON API and meta information. Researchers can quickly assemble novel AI/ML algorithms from such components as LEGO(tm), share them, and crowdsource their benchmarking, optimization and co-design across diverse models, data set and platforms from IoT to supercomputers in terms of speed, accuracy, energy and other costs (see ACM ReQuEST tournaments). All results including mispredictions (!) are then shared in CK repositories in a reproducible way to let the community reuse and improve them! We hope that CK approach of reusable artifacts and workflows will help to accelerate adoption of AI while dramatically reducing R&D costs and boosting innovation (see CK intro slides)!

We made a special effort to simplify CK installation with minimal dependencies across Linux, Windows and MacOS. You just need Git client and Python 2.7+ or 3.3+. Just try it yourself as described here. You can then follow a simple tutorial to get the first feeling about CK capabilities (portable, customizable and reusable workflows and artifacts with a portable package manager).

Join the open consortium led by non-profit cTuning foundation and dividiti Ltd to participate in the following community activities powered by the CK.

Join collaborative, reproducible and reusable AI research powered by Collective Knowledge!

We are bringing together academia and industry led by the non-profit cTuning foundation and dividiti Ltd to use open research SDK (Collective Knowledge) and
  1. build an open AI repository of reusable, portable and customizable AI artifacts (models, data sets, tools) with CK Python wrappers and a common JSON API;
  2. assemble portable AI algorithms/workflows (classification, object detection, speech recognition) with a unified API from reusable AI artifacts;
  3. collaboratively test, optimize and co-design deep learning and other emerging AI workloads in terms of speed, accuracy, size, energy and other costs across the whole SW/HW/model/dataset stack from IoT to supercomputers using open ReQuEST tournaments at the leading conferences;
  4. enable open AI research and crowdsource AI experiments such as machine learning and model training across diverse platforms provided by volunteers (potentially billions of devices) similar to SETI@HOME;
  5. accelerate adoption of AI and boost innovation in science and technology.
Collective Knowledge Framework was especially designed to support above cases - combined with portable workflows it helps users gradually convert their ad-hoc AI artifacts (code and data) and experimental workflows into customizable, portable and reusable components with Python wrappers, common JSON API and unified JSON meta information:

Participate in collaborative testing, optimization and co-design of the whole AI/SW/HW stack via open competitions

Since CK enables crowdsourcing of experiments such as multi-objective autotuning across diverse platforms similar to SETI@home, we implemented a prorotype of CK-powered crowd-fuzzing, crowd-benchmarking and crowd-tuning workflows for a wide range of applications and for deployment on a wide range of form factors - from sensors to self-driving cars (IWOCL'16 article).

CK partners use such workflows with the help of the non-profit cTuning foundation and dividiti Ltd to co-design efficient AI solutions such as deep learning across the whole SW/HW stack from IoT to HPC which meet given trade-offs in performance, power consumption, prediction accuracy, memory usage and cost requirements:

Download our engaging Android app from Google Play to help the community prototype collaborative benchmarking, optimization and testing of deep learning engines including Caffe and TensorFlow, models and data sets. You can see continuously aggregating results in the public CK repository:
  • The number of unique users participated in DNN crowd-benchmarking and crowd-tuning: 600+
  • The number of unique platforms provided by volunteers: 790+
  • The number of unique CPUs inside participated platforms: 260+
  • The number of unique GPUs inside participated platforms: 110+
  • The number of unique OS: 280+
Our partner, dividiti, regularly shares public and processed benchmarking and optimization results across different DNN engines, models and hardware in a reproducible CK format using convenient Jupyter notebooks:

Design common and simple AI API

We are developing an open, simple and unified CK API for AI connected to various collaboratively optimized DNN engines (Caffe, TensorFlow, etc) and models across diverse platforms (online demo and wiki).

See a CK JSON API example to predict compiler flags using machine learning (part of our long-term initiative to enable self-optimizing computer systems (article 1 and article 2)

    import ck.kernel as ck

    image='use_dnn_to_classify_image.jpg'

    r=ck.access({'action':'ask',
                 'module_uoa':'advice',
                 'to':'classify_image',
                 'image':image})
    if r['return']>0: ck.err(r)
  
or to classify images using various self-optimizing DNN engines and models in a cloud or locally:
    import ck.kernel as ck

    scenario='experiment.tune.compiler.flags.llvm.e'
    compiler='LLVM 4.0.0'
    # GCC autotuning stats for RPi3
    cpu_name='BCM2709'

    features= ["9.0", "4.0", "2.0", "0.0", "5.0", "2.0", "0.0", "4.0", "0.0", "0.0", 
               "2.0", "0.0", "7.0", "0.0", "0.0", "10.0", "0.0", "0.0", "1.0", "2.0", 
               "10.0", "4.0", "1.0", "14.0", "2.0", "0.714286", "1.8", "3.0", "0.0", 
               "4.0", "0.0", "3.0", "0.0", "3.0", "0.0", "0.0", "0.0", "0.0", "0.0", 
               "0.0", "2.0", "0.0", "1.0", "0.0", "1.0", "5.0", "10.0", "2.0", "0.0", 
               "32.0", "0.0", "10.0", "0.0", "0.0", "0.0", "0.0", "3.0", "33.0", "12.0", 
               "32.0", "93.0", "14.0", "19.25", "591.912", "11394.3"]

    r=ck.access({'action':'ask',
                 'module_uoa':'advice',
                 'to':'predict_compiler_flags',
                 'scenario':scenario,
                 'compiler':compiler,
                 'cpu_name':cpu_name,
                 'features':features})
    if r['return']>0: ck.err(r)

    ck.out('Predicted optimization: '+r['predicted_opt'])
  

Prepare representative training sets with the community

Our Android application to crowdsource AI algorithms across diverse platforms provided by volunteers allows to collect mispredictions in a public repository thus helping to improve existing training sets and gradually assemble representative ones!
          
Website designed using CK
          
   
   
   
               Locations of visitors to this page
(C)opyright 2014-2018 non-profit cTuning foundation and dividiti