Join open MLCommons task force on automation and reproducibility to participate in the collaborative development of the Collective Knowledge v3 playground (MLCommons CK) powered by the Collective Mind automation language (MLCommons CM) and Modular Inference Library (MIL) to automate benchmarking, optimization, design space exploration and deployment of Pareto-efficient AI/ML applications across any software and hardware from the cloud to the edge!
We encourage you to reuse these portable productivity functions (Python 2.6+ and 3+) from the CK modules or in your own Python scripts:
    import ck.kernel as ck

    ck.out("Hello world")

    r=ck.access({"action":"ls", "module_uoa":"env", "out":"con"})
    if r["return"]>0: ck.err(r)
    print (r["lst"])

    r=ck.load_json_file({"json_file":"my_file.json", "dict":{"test":"yes"}})
    if r["return"]>0: return r

    r=ck.save_text_file({"text_file":"my_file.txt", "string":"test=yes\n"})
    if r["return"]>0: return r

See CK documentation for further details.

# Function name Note and API
1 ck.access(i)

   for end users

Universal access to all CK functions (Unified JSON/dictionary API)

    Input:  Can be dictionary or string (string will be converted to dictionary)

            {
               action

               module_uoa or CID -> converted to cid
                 or
               (cid1)            -  if doesn't have = and doesn't start from -- or - or @ -> appended to cids[]
               (cid2)            -  if doesn't have = and doesn't start from -- or - or @ -> appended to cids[]
               (cid3)            -  if doesn't have = and doesn't start from -- or - or @ -> appended to cids[]
                 or
               (repo_uoa)
               (module_uoa)
               (data_uoa)

               (out=type)     Module output
                              == ''              - none
                              == 'con'           - console interaction (if from CMD, default)
                              == 'json'          - return dict as json to console
                              == 'json_with_sep' - separation line and return dict as json to console
                              == 'json_file'     - return dict as json to file

               (out_file)     Output file if out=='json_file'

               (con_encoding) - force encoding for IO
               (ck_profile)   - if 'yes', profile CK

               INPUT TO A GIVEN FUNCTION


               NOTE: If INPUT is a string and it will be converted to INPUT dictionary as follows (the same as CK command line):

                     ck key1=value1    -> converted to {key1:value1}

                        -key10         -> converted to {key10:"yes"}
                        -key11=value11 -> converted to {key11:value11}

                        --key12         -> converted to {key12:"yes"}
                        --key13=value13 -> converted to {key13:value13}


                     @file_json         -> JSON from this file will be merged with INPUT
                     @@                 -> CK will ask user ot enter manually JSON from console and merge with INPUT
                     @@key              -> Enter JSON manually from console and merge with INPUT under this key

                     @@@cmd_json        -> convert string to JSON (special format) and merge with INPUT

                     -- xyz             -> add everything after -- to "unparsed_cmd" key in INPUT

                     When string is converted to INPUT dictionary, "cmd" variable is set to True
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0

              OUTPUT FROM A GIVEN FUNCTION
            }

2 ck.cid(i)

   for end users

Common action: get CID from current path or given CID (module_uid:data_uid)

    Input:  {
              (repo_uoa)   - repo UOA
              (module_uoa) - module UOA
              (data_uoa)   - data UOA

                 If above is empty, detect in current path !

            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0

              Output from from 'detect_cid_in_current_path' function

              data_uoa     - data UOA
              module_uoa   - module UOA
              (repo_uoa)   - repo UOA
            }

3 ck.compare_dicts(i)

   for end users

Compare 2 dictionaries (recursively)

    Input:  {
              dict1         - dictionary 1
              dict2         - dictionary 2
              (ignore_case) - ignore case of letters

              Note that if dict1 and dict2 has lists, the results will be as follows:

              * dict1={"key":['a','b','c']}
                dict2={"key":['a','b']}
                EQUAL

              * dict1={"key":['a','b']}
                dict2={"key":['a','b','c']}
                NOT EQUAL
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0

              equal        - if 'yes' dictionaries are equal
            }
4 ck.compare_flat_dicts(i)

   for end users

Compare two flat dictionaries

    Input:  {
              dict1            - dictionary 1
              dict2            - dictionary 2
              (ignore_case)    - ignore case of letters
              (space_as_none)  - if 'yes', consider "" as None
              (keys_to_ignore) - list of keys to ignore (can be wildcards)
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0

              equal        - if 'yes' dictionaries are equal
            }
5 ck.convert_file_to_upload_string(i)

   for end users

Convert file to upload string

    Input:  {
              filename - file name to convert
            }

    Output: {
              return              - return code =  0, if successful
                                                >  0, if error
              (error)             - error text if return > 0
              file_content_base64 - string that can be transmitted through Internet
            }
6 ck.convert_iso_time(i)

   for end users

Converting iso text time to datetime object

    Input:  {
              iso_datetime - iso date time
            }

    Output: { 
              return         - return code =  0, if successful
                                           >  0, if error
              (error)        - error text if return > 0
              (datetime_obj) - datetime object
            }
7 ck.convert_json_str_to_dict(i)

   for end users

Convert string of a special format to json

    Input:  {
              str                      - string (use ' instead of ", i.e. {'a':'b'} 
                                         to avoid issues in CMD in Windows and Linux!)

              (skip_quote_replacement) - if 'yes', do not make above replacement
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0

              dict         - dict from json file
            }
8 ck.convert_str_key_to_int(key)

   for end users

Safe convert dict str keys to int to be sorted

    Input:  {
              text - text to print
            }

    Output: {
              return       - return code =  0

              string       - input string
            }
9 ck.convert_str_tags_to_list(i)

   for end users

Convert string to list

    Input:  either a list, or a string of comma-separated tags.

    Output: If i is a list, it's returned.
            If i is a string, the list of tags it represents is returned 
            (each tag is stripped of leading and trailing whitespace).

10 ck.convert_upload_string_to_file(i)

   for end users

Convert upload string to file

    Input:  {
              file_content_base64 - string transmitted through Internet
              (filename)          - file name to write (if empty, generate tmp file)
            }

    Output: {
              return              - return code =  0, if successful
                                                >  0, if error
              (error)             - error text if return > 0
              filename            - filename with full path
              filename_ext        - filename extension
            }
11 ck.copy_to_clipboard(i)

   for end users

Copy string to clipboard if supported by OS (requires Tk)

    Input:  {
              string - string to copy

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0
            }
12 ck.debug_out(i)

   for end users

Universal debug print of a dictionary (removing unprintable parts)

    Input:  i - dictionary

    Output: return = 0
13 ck.dump_json(i)

   for end users

Dump json to sring

    Input:  {
              dict          - dictionary
              (skip_indent) - if 'yes', skip indent
              (sort_keys)   - if 'yes', sort keys
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error

              string       - json string (in utf8)
            }
14 ck.dumps_json(i)

   for end users

Deprecated: Dump json to sring (left for compatibility with older kernel - should eventually remove it) - see 'dump_json'

    Input:  {
              dict          - dictionary
              (skip_indent) - if 'yes', skip indent
              (sort_keys)   - if 'yes', sort keys
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error

              string       - json string (in utf8)
            }
15 ck.eout(s)

   for end users

Universal print of unicode error string in utf8 that supports Python 2.x and 3.x to stderr

    Input:  s - unicode string to print

    Output: Nothing
16 ck.err(r)

   for end users

Universal error print and exit

    Input:  {
              return - return code
              error - error text
            }

    Output: Nothing; quits program
17 ck.find_path_to_repo(i)

   for end users

Find path to a given repo

    Input:  {
              (repo_uoa) - repo UOA; if empty, get the default repo
            }

    Output: {
              return       - return code =  0, if successful
                                           16, if repo not found (may be warning)
                                         >  0, if error
              (error)      - error text if return > 0

              dict         - dict from cache
              path         - path to repo

              repo_uoa     - repo UOA
              repo_uid     - repo UID
              repo_alias   - repo alias
            }
18 ck.find_string_in_dict_or_list(i)

   for end users

Find string in dict

    Input:  {
              dict            - dictionary 1
              (search_string) - search string
              (ignore_case)   - ignore case of letters
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0

              found        - if 'yes', string found
            }
19 ck.flatten_dict(i)

   for end users

Convert dictionary into CK flat format

    Any list item is converted to @number=value
    Any dict item is converted to #key=value
    # is always added at the beginning 

    Input:  {
              dict         - python dictionary

              (prefix)     - prefix (for recursion)

              (prune_keys) - list of keys to prune (can have wildcards)
            }

    Output: {
              return  - return code =  0, if successful
                                    >  0, if error
              (error) - error text if return > 0
              dict    - flattened dictionary
            }
20 ck.flatten_dict_internal_check_key(prefix, pk)

   for end users

Convert dictionary into CK flat format (internal, used for recursion)

    Input:  {
              dict  - dictionary
              key   - flat key
            }

    Output: {
              return  - return code =  0, if successful
                                    >  0, if error
              (error) - error text if return > 0
              value   - value or None, if doesn't exist
            }
21 ck.gen_tmp_file(i)

   for end users

Generate temporary files

    Input:  {
              (suffix)     - temp file suffix
              (prefix)     - temp file prefix
              (remove_dir) - if 'yes', remove dir
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0

              file_name    - temp file name 
            }

22 ck.gen_uid(i)

   for end users

Generate CK UID

    Input:  {}

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0

              data_uid     - UID in string format (16 characters 0..9,a..f)
            }
23 ck.get_by_flat_key(i)

   for end users

Get value from dict by flat key

    Input:  {
              dict  - dictionary
              key   - flat key
            }

    Output: {
              return  - return code =  0, if successful
                                    >  0, if error
              (error) - error text if return > 0
              value   - value or None, if doesn't exist
            }
24 ck.get_current_date_time(i)

   for end users

Get current date and time

    Input:  {}

    Output: {
              return       - return code =  0

              array        - array with date and time
              iso_datetime - date and time in ISO format
            }
25 ck.get_from_dicts(dict1, key, default_value, dict2, extra='')

   for end users

Get value from one dict, remove it from there and move to another

    Input:  dict1         - first check in this dict (and remove if there)
            key           - key in dict1
            default_value - default value if not found
            dict2         - then check from here

    Output: value
26 ck.get_os_ck(i)

   for end users

Get host platform (currently win or linux) and OS bits

    Input:  {
              (bits)      - force OS bits
            }

    Output: {
              return      - return code =  0
              platform    - 'win' or 'linux'
              bits        - OS bits in string: 32 or 64
              python_bits - Python installation bits in string: 32 or 64

            }
27 ck.get_version(i)

   for end users

Get CK version

    Input:  {}

    Output: {
              return       - return code =  0

              version      - list starting from major version number
              version_str  - version string
            }

28 ck.inp(i)

   for end users

Universal input of unicode string in utf8 that supports Python 2.x and 3.x

    Input:  {
              text - text to print
            }

    Output: {
              return       - return code =  0

              string       - input string
            }
29 ck.input_json(i)

   for end users

Input JSON from console (double enter to finish)

    Input:  {
              text - text to print
            }

    Output: {
              return              - return code =  0, if successful
                                                >  0, if error
              (error)             - error text if return > 0

              string
              dict                - parsed JSON
            }
30 ck.is_uid(str)

   for end users

Check if string is CK UID

    Input:  string to check

    Output: True if UID, otherwise False
31 ck.is_uoa(str)

   for end users

Check if string is correct CK UOA(i.e. does not have special characters including *, ?)

    Input:  string to check

    Output: True if allowed UOA, False otherwise
32 ck.jerr(r)

   for end users

Universal error print for Jupyter Notebook with raise KeyboardInterrupt

    Input:  {
              return - return code
              error - error text
            }

    Output: Nothing; quits program
33 ck.list_files(i)

   for end users

List files in a given entry

    Input:  {
              (repo_uoa)   
              (module_uoa) 
              (data_uoa)  

              parameters for function 'list_all_files'
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0

              Output of list all files
            }

34 ck.load_json_file(i)

   for end users

Load json from file into dict

    Input:  {
              json_file - name of file with json
            }

    Output: {
              return       - return code =  0, if successful
                                         = 16, if file not found (may be warning)
                                         >  0, if error
              (error)  - error text if return > 0

              dict     - dict from json file
            }
35 ck.load_module_from_path(i)

   for end users

Load (CK) python module

    Input:  {
              path             - module path
              module_code_name - module name
              (cfg)            - configuration of the module if exists ...
              (skip_init)      - if 'yes', skip init
              (data_uoa)       - module UOA (useful when printing error)
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0

              code         - python code object
              path         - full path to the module
              cuid         - internal UID of the module
            }
36 ck.load_text_file(i)

   for end users

Load text file into string

    Input:  {
              text_file           - name of text file
              (keep_as_bin)       - if 'yes', return only bin
              (encoding)          - by default 'utf8', however sometimes we use utf16

              (split_to_list)     - if 'yes', split to list

              (convert_to_dict)   - if 'yes', split to list and convert to dict
              (str_split)         - if !='', use as separator of keys/values when converting to dict
              (remove_quotes)     - if 'yes', remove quotes from values when converting to dict

              (delete_after_read) - if 'yes', delete file after read (useful when reading tmp files)
            }

    Output: {
              return       - return code =  0, if successful
                                         = 16, if file not found (may be warning)
                                         >  0, if error
              (error)  - error text if return > 0

              bin      - bin
              (string) - loaded text (with removed \r)
              (lst)    - if split_to_list=='yes', return as list
              (dict)   - if convert_to_dict=='yes', return as dict
            }
37 ck.load_yaml_file(i)

   for end users

Load YAML from file into dict

    Input:  {
              yaml_file - name of YAML file
            }

    Output: {
              return       - return code =  0, if successful
                                         = 16, if file not found (may be warning)
                                         >  0, if error
              (error)  - error text if return > 0

              dict     - dict from YAML file
            }
38 ck.merge_dicts(i)

   for end users

Merge intelligently dict1 with dict2 key by key in contrast with dict1.update(dict2)

    Input:  {
              dict1 - merge this dict with dict2 (will be directly modified!)
              dict2 - dict

    Output: {
              return       - return code =  0, if successful

              dict1        - output dict
            }
39 ck.out(s)

   for end users

Universal print of unicode string in utf8 that supports Python 2.x and 3.x

    Input:  s - unicode string to print

    Output: Nothing
40 ck.python_version(i)

   for end users

Action: print python version used by CK

    Input:  {}

    Output: {
               version - sys.version
               version_info - sys.version_info
            }

41 ck.restore_flattened_dict(i)

   for end users

Restore flattened dict

    Input:  {
              dict - flattened dict
            }

    Output: {
              return  - return code =  0, if successful
                                    >  0, if error
              (error) - error text if return > 0
              dict    - restored dict
            }
42 ck.run_and_get_stdout(i)

   for end users

Run command and get stdout

  Input:  {
            cmd       - list of command line arguments, starting with the command itself
            (shell)   - if 'yes', reuse shell environment
          }

  Output: {
            return       - return code =  0, if successful
                                       >  0, if error
                                       =  8, if timeout
            (error)      - error text if return > 0

            return_code  - return code from app

            stdout       - string, standard output of the command
          }
43 ck.safe_float(i,d)

   for end users

Support function for safe float (useful for sorting function)

    Input:  {
              cmd       - command line
              (timeout) - timeout in seconds (granularity 0.01 sec) - may cause overheads ...
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
                                         =  8, if timeout
              (error)      - error text if return > 0

              return_code  - return code from app
            }
44 ck.safe_get_val_from_list(lst, index, default_value)

   for end users

Support function to get value from list without error if out of bounds(useful for various sorting)

    Input:  {
              cmd       - command line
              (timeout) - timeout in seconds (granularity 0.01 sec) - may cause overheads ...
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
                                         =  8, if timeout
              (error)      - error text if return > 0

              return_code  - return code from app
            }
45 ck.safe_int(i,d)

   for end users

Support function for safe int (useful for sorting function)

    Input:  {
              cmd       - command line
              (timeout) - timeout in seconds (granularity 0.01 sec) - may cause overheads ...
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
                                         =  8, if timeout
              (error)      - error text if return > 0

              return_code  - return code from app
            }
46 ck.save_json_to_file(i)

   for end users

Save dict as json file

    Input:  {
              json_file    - file name
              dict         - dict to save
              (sort_keys)  - if 'yes', sort keys
              (safe)       - if 'yes', ignore non-JSON values (only for Debugging - changes original dict!)
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0
            }
47 ck.save_text_file(i)

   for end users

save string into text file

    Input:  {
              text_file - name of text file
              string    - string to write (with removed \r)
              (append)  - if 'yes', append
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)  - error text if return > 0
            }
48 ck.save_yaml_to_file(i)

   for end users

Save dict as yaml file

    Input:  {
              yaml_file   - file name
              dict        - dict to save
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0
            }
49 ck.select(i)

   for end users (advanced version available in module "choice")

Universal selector of dictionary entry

    Input:  {
              dict             - dict with values being dicts with 'name' as string to display and 'sort' as int (for ordering)
              (title)          - print title
              (error_if_empty) - if 'yes' and Enter, make error
              (skip_sort)      - if 'yes', do not sort array
            }

    Output: {
              return       - return code =  0

              string       - selected dictionary key
            }
50 ck.select_uoa(i)

   for end users (advanced version available in module "choice")

Universal UOA selector

    Input:  {
              choices      - list from search function
              (skip_enter) - if 'yes', do not select 0 when user presses Enter
              (skip_sort)  - if 'yes', do not sort array
            }

    Output: {
              return  - return code =  0, if successful
                                    >  0, if error
              (error) - error text if return > 0
              choice  - data UOA
            }

51 ck.set_by_flat_key(i)

   for end users

Set value in array using flattened key


    Input:  {
              dict            - dict (it will be directly changed!)
              key             - flat key (or not if doesn't start with #)
              value           - value to set
            }

    Output: {
              return  - return code =  0, if successful
                                    >  0, if error
              (error) - error text if return > 0
              dict    - modified dict
            }
52 ck.substitute_str_in_file(i)

   for end users

Substitute string in file

    Input:  {
              filename - file
              string1  - string to be replaced
              string2  - replace string
            }

    Output: {
              return       - return code =  0, if successful
                                         = 16, if file not found
                                         >  0, if error
              (error)  - error text if return > 0
            }
53 ck.system_with_timeout(i)

   for end users

Substituting os.system with possibility for time out

    Input:  {
              cmd       - command line
              (timeout) - timeout in seconds (granularity 0.01 sec) - may cause overheads ...
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
                                         =  8, if timeout
              (error)      - error text if return > 0

              return_code  - return code from app
            }
54 ck.system_with_timeout_kill(proc)

   for end users

Support function for system_with_timeout

    Input:  {
              cmd       - command line
              (timeout) - timeout in seconds (granularity 0.01 sec) - may cause overheads ...
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
                                         =  8, if timeout
              (error)      - error text if return > 0

              return_code  - return code from app
            }
55 ck.uid(i)

   for end users

Action: generate CK UID

    Input:  {}

    Output: {
              Output from 'gen_uid' function
            }

56 ck.unzip_file(i)

   for end users

List files in a given entry

    Input:  {
              archive_file         - full path to zip file  
              (path)               - path where unzip (current if empty)
              (overwrite)          - if 'yes', overwrite
              (delete_after_unzip) - if 'yes', delete original zip file after unzipping
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0

              skipped       - list of files which was not overwritten
           }

57 ck.version(i)

   for end users

Action: print CK version

    Input:  {}

    Output: {
              output from function 'get_version'
            }