| def module::add_data | ( | i | ) |
Add data
Input: {
cm_module_uoa - module UOA
cm_data_uoa - data UOA
(cm_data_uid) - force data UID (if UOA is alias)
(cm_repo_uoa) - repo UOA
(use_default_repo) - if 'yes', use default repo
(cm_console) - by default, do not output anything except errors
if 'txt' then output as plain text
if 'web' then output for web (html, image, etc)
if 'json' then output as json
(cm_file_upload) - text content of file for upload
(cm_file_upload_base64) - file content as base64.urlsafe_b64encode
(cm_file_upload_name) - preffered uploaded file name
(cm_file_upload_type) - type of file (if .zip, it will be automatically unziped)
(cm_archive) - 'none'
'already_archived_with_internal_name' - should save file
under default cM name (cm_archive.zip)
'already_archived_with_user_name' - should save file under user name
(cm_file_upload_tmp_uid) - normally generated automatically by web server
when uploading files
cm_array - use ONLY this array to add data
(cm_admin) - if 'yes', override access control - only internal use,
can not be used during web access (explicitly removed)
}
Output: {
cm_return - return code >0 if error
-1 if entry exists
cm_path - path to created data entry
cm_uid - uid (from UOA)
cm_alias - alias (from UOA)
cm_uoa - alias or uid if alias==''
}
Add module
Input: {
}
Output: {
cm_return - return code >0 if error
cm_path - path to created data entry
}
Add repository
Input: {
(path) - path to create repository (otherwise current) or URL
path_type - absolute, relative_to_cm, remote_url
(cm_repo_uoa) - repo UOA where to create entry
(cm_array) - additional description
(repo_alias) - alias on the disk
(repo_name) - user friendly name (when listing repositories)
(repo_uid) - set UID for the repository - useful
when creating personal user repository in shared environments ...
}
Output: {
cm_return - return code >0 if error
cm_path - path to created data entry
... - other paramaters from core add_data function
}
Add user (and public person profile)
Input: {
cm_username - username (UOA)
cm_user_password - user password
cm_user_password_repeat - repeat user password for checking
name - user real name (will be used as alias)
(cm_array) - Note, that if cm_array is present,
all above parameters should be inside it
(cm_repo_uoa) - repo UOA where to create user entry
}
Output: {
cm_return - return code >0 if error
cm_path - path to created data entry
... - other paramaters from core add_data function
}
| def module::add_data_through_web | ( | i | ) |
Add/update data through web form
Input: {
browse_module_uoa - module UOA
browse_data_uoa - data UOA
(browse_repo_uoa) - repo UOA
(cm_back_json) - for back button
(cm_only_note_like) - if 'yes', update only note and ranking
cm_mode - 'add' or 'update' to reuse the same function
cm_array - data array
...
}
Output: {
cm_return - return code >0 if error
-1 if entry exists
If entry was created (not first web form), then:
cm_path - path to created data entry
cm_uid - uid (from UOA)
cm_alias - alias (from UOA)
}
| def module::all | ( | i | ) |
ReIndex all repos
Input: {
(index_repo_uoa) - repo uoa to prune reindexing
(index_module_uoa) - module uoa to prune reindexing
(index_data_uoa) - data uoa to prune reindexing (can include pattern * and ?)
(delay) - delay after indexing individual entry
(sequential) - if 'yes' perform sequential index (very slow, but reports errors)
}
Output: {
cm_return - return code >0 if error
}
Test all
Input: {
}
Output: {
cm_return - return code >0 if error
}
| def module::analyze | ( | i | ) |
| def module::analyze_variation | ( | i | ) |
Analyze variation in characteristics
Input: {
(data) - list of data entries to analyze
(prune_by_summary_point_relaxed) - if data is not provided, search will be performed for all entries;
returned entries can be pruned by this dictionary
using 'summary_point_relaxed' in entries
characteristic_key - key to get characteristic (will be converted to float)
percent_threshold - percent threshold to detect benchmarks where characteristic has high variability
(cm_repo_uoa) - working repository
}
Output: {
cm_return - if =0, success
}
| def module::any | ( | i | ) |
| def module::browse | ( | i | ) |
cM basic repository browsing
Input: {
(browse_module_uoa) - module UOA
(browse_data_uoa) - data UOA
(browse_repo_uoa) - repo UOA
(cm_subaction_delete) - delete entry
(cm_subaction_repo_delete) - delete repository
(cm_subaction_view) - view entry
(show_add_info) - additional view config (show additional info about entry)
(show_param_names) - additional view config (show parameter names (for development))
(cm_subaction_module_view) - view module
(cm_subaction_repo_view) - view repository
(cm_subaction_update) - update entry
(cm_subaction_repo_update) - update repository
(cm_subaction_add) - add entry
(cm_subaction_module_add) - add module
(cm_subaction_repo_add) - add repository
(cm_button_reset) - reset current view
(cm_back_json) - previous web state to create 'back' button
}
Output: {
cm_return - return code >0 if error
}
| def module::build | ( | i | ) |
Build program
Input: {
See code.run (all parameters will be passed to code.run)
(work_dir_repo_uoa) - change to the working directory in the repository (repo UOA)
(work_dir_module_uoa) - change to the working directory in the repository (module UOA)
(work_dir_data_uoa) - change to the working directory in the repository (data UOA)
build_target_os_uoa - target OS uoa for building
If the above parameters are set, the following parameters will be added to run_set_env2 from the given cM entry:
CM_SOURCE_FILES="source_files"
CM_SOURCE_FILES_WITHOUT_EXT="source_files" (extension will be removed)
CM_TARGET_FILE="target_file"
(target_file) - output file after building
(add_target_extension) - take extension from run_target_os_uoa
(include_deps) - add 'include' directory from this list
(include_prefix) - add prefix before include directory (usually -I or /I)
(compiler_vars) - add compiler vars
(compiler_var_prefix) - add compiler var prefix (usually -D or /D)
(lib_deps) - add libraries from this list (use CM_TARGET_FILE)
(compilation_type) - 'static' (by default) or 'dynamic'
(install) - if 'yes', install to cM entry
(install_data_uid) - if !='', it will be used for installation data entry
(install_data_alias) - if !='', use it for installation data entry
(install_data_display_as_alias) - use this display as alias for a generated entry
(install_module_uoa) - if =='', module 'code' is used
(install_repo_uoa) - repo for installation entry
(source_files) - list of source files; if empty, use list of source files from the program entry
(add_rem_to_script) - add rem to install script
(add_to_code_entry) - add array to code entry
(ignore_script_error) - if 'yes', do not generate error when script returns !=0
(needed for experiment pipelines to continue running)
(cm_dependencies) - dependencies that set all other dependencies (code/include/lib).
Format [{"class UOA":"code UOA"},{},...]
(skip_target_file_check) - if 'yes', skip target file check
(run_cmd_out1) - pass to 'code run' to redirect output1
(run_cmd_out2) - pass to 'code run' to redirect output2
(clean_program) - if 'yes', clean program before
(clean_program_script_name) - script name for cleaning ...
(code_deps) - list with code UOA for dependencies [{"index":"uoa"} ...]
(run_set_env2) - array with environment variables to be set before compiling
(cm_verbose) - if 'yes', print all info
}
Output: {
cm_return = 0, if successful
Parameters from "code run" when executing script
target_file - output file after building
target_file_size - size of output file
}
Build model
Input: {
model_module_uoa - model's module
(model_data_uoa) - model's data (if need to record)
model_name - (earth, svm)
data
ct_dimensions_input
ct_dimensions_output
desc
(record_data_to_file_prefix) - if !='', use this filename prefix instead of randomly generated
}
Output: {
cm_return - return code >0 if error
}
Build model
Input: {
data
ct_dimensions_input
ct_dimensions_output
desc
model_name - (earth, svm)
(record_data_to_file_prefix) - if !='', use this filename prefix instead of randomly generated
}
Output: {
cm_return - return code >0 if error
file_with_model - file with model
}
| def module::calculate_similarity | ( | i | ) |
Calculate similarity
FGG: for now simple euclidean distance - it's just a demo of the concept,
since it has many limitations so we need to collaboratively extend it
(using PCA, SVM, etc)
Input: {
program1_milepost_features - MILEPOST features for program one
program2_milepost_features - MILEPOST features for program two
}
Output: {
cm_return - if =0, success
distance - Euclidean distance (can be more than 1)
}
| def module::change_work_dir | ( | i | ) |
Chaning working directory
Input: {
(work_dir) - change to this working directory before running code
or
(work_dir_repo_uoa) - change to the working directory in the repository (repo UOA)
(work_dir_module_uoa) - change to the working directory in the repository (module UOA)
(work_dir_data_uoa) - change to the working directory in the repository (data UOA)
}
Output: {
cm_return - if =0, success
work_dir - selected working directory
}
Chaning working directory
Input: {
work_dir - if !='', go to this directory
(work_dir_repo_uoa) - if !='', go to the directory for this cM entry
(work_dir_module_uoa) - if !='', go to the directory for this cM entry
(source_code_uoa) - if !='', go to the directory for this cM entry
}
Output: {
cm_return - if =0, success
work_dir - selected working directory
}
| def module::clean | ( | i | ) |
Search
Input: {
(index_repo_uoa) - Repository UOA to clean indexes
(index_module_uoa) - Module UOA to clean indexes
(index_data_Uoa) - Data UOA to clean indexes
}
Output: {
cm_return - return code >0 if error
}
Clean repository
Input: {
target_repo_uoa - target repo UOA
(target_module_uoa) - clean data for this module
(cm_classes_uoa) - prune data by classes
(target_data_uoa) - data to clean
}
Output: {
cm_return - return code >0 if error
cm_path - path to file
cm_uid - generated uid
}
| def module::cmd_prune_flags | ( | i | ) |
| def module::compare_systems | ( | i | ) |
| def module::compile_program | ( | i | ) |
| def module::configure | ( | i | ) |
| def module::convert_ct_multi_array_to_csv | ( | i | ) |
Convert cTuning multi-dimensional array to csv
Input: {
cm_multi_array_s - array of strings to convert
ct_dimensions - list of dimensions to add
ct_dim_desc - dimension description (categorical/continuous, etc)
ct_output_file - output csv file
(csv_separator) - separator for data in csv (';' by default)
(csv_decimal_mark) - decimal char (default ',')
(csv_no_header) - if 'yes', do not add header
}
Output: {
cm_return - return code >0 if error
}
| def module::convert_ct_space_points_to_python_array | ( | i | ) |
Convert cTuning space points to python array
Input: {
ct_module_uoa - module to visualize
ct_points - list of UOA with points
ct_axes - list with flattened axes (if empty, all axes are taken)
ct_axes_desc - axes description
(use_multiple_graphs) - if 'on' separate entries into different sub-graphs
(data_entry_to_sort) - if sort and multiple entries, use this entry number to sort (-1)
(fix_additional_dimensions) - if 'yes', fix or range additional dimensions
(fix_additional_dimensions_data) - dictionary of format:
{"flattened key A":{'less':X, 'equal':Y, 'more':Z}, ...}
}
Output: {
cm_return - return code >0 if error
ct_array - multidimensional array ([ [],[],...])
ct_array_id - 1D array with {cm_uid, cm_uoa}
ct_axes - either original one or ALL axes if original len==0
}
| def module::convert_field_to_html | ( | i | ) |
Convert cM data field to html
Input: {
cm_separator - appendable separator to deal with json hierarchy
cm_separator_form - appendable separator to deal with array hierarchy (and with additional forms)
cm_key - array key
cm_value - array value
cm_data_desc - data description in special cM format
cm_mode - add | update
(cm_back_json) - array with previous cM state
(cm_array_orig) - original array (not changing through the calls)
(cm_cur_array) - current array to change if needed
(cm_cur_key) - current key to change array if needed
(cm_form_explore) - form explore parameters
(cm_explore) - if 'yes', prepare exploration choices
(cm_form_explore_color) - prefix for style
(cm_explore_commands) - array with exploration commands
(cm_explore_state) - {} array with choices and exploration state
(cm_explore_level) - exploration level ('' - coarse-grain, '1' - next level, ...)
(hide_add_new_object) - if 'yes', do not show 'add new object'
(cm_disable_coloring) - if 'yes', disable coloring of dimensions, characteristics, properties, state, etc
(cm_admin) - if 'yes', override access control - only internal use,
can not be used during web access (explicitly removed)
}
Output: {
cm_return - return code >0 if error
cm_value_html - prepared html
cm_value_text - text (in URL, etc)
}
| def module::convert_file_to_upload_string | ( | i | ) |
| def module::convert_flags_from_ccc | ( | i | ) |
Convert flags from CCC framework
Input: {
cm_filename - CCC compiler configuration file
(add_base_opt_upfront) - if 'yes', add base opts upfront (such as -O3, -O2, etc).
One should edit final config!
(create_cm_entry) - if 'yes' add to cM
(repo_uoa) - repository to create entry
(data_uoa) - data uoa to create entry
}
| def module::convert_points_to_flat_csv | ( | i | ) |
Convert points into flat CSV (possibility to choose dimensions).
Needed for machine learning and data mining.
Input: {
ct_data_uoa - Entry with points
(ct_data_selection) - Multiple entries if needed
(ct_data_selection_from_files) - Multiple entries from json files prepared by this function
(ct_module_uoa) - Specify module from scenarios
(otherwise ctuning.space by default)
(use_multiple_graphs) - if 'on' spread data from ct_data_selection into different dimensions
(ct_dimensions) - List of dimensions to add
(ct_repo_uoa) - Repository
(ct_output_file_s_json) - output json file (values as strings)
(ct_output_file_json) - output json file (real values)
(ct_output_file_csv) - output csv file
(csv_separator) - separator for data in csv (';' by default)
(csv_decimal_mark) - decimal char (default ',')
}
Output: {
cm_return - return code >0 if error
ct_multi_array_s - list of dictionaries per multi-graph of format 'flat_dimension1':['value1', 'value2', ...],
'flat_dimension2':[...],
...
ct_multi_array - list of dictionaries per multi-graph of format 'flat_dimension1':[value1, value2, ...],
'flat_dimension2':[...]
...
where all values are based on cM dimensions description in a note
ct_dim_desc - dictionary of format {'flat_dimension': cM dimension description (categorical/continuous/etc)
}
Note: cM dimensions description = {
type - "text", "integer", "uoa", "float", etc
data_desc - textual description of dimension
has_choice - if 'yes', this dimension is categorical
choice" - [list of choices]
explore_start - if !='', dimension is continuous; min value
explore_stop - max value
explore_step - step
explore_prefix - add prefix before dimension
(useful for compiler flags with conintuous parameters)
}
| def module::convert_python_array_to_csv | ( | i | ) |
| def module::convert_python_array_to_gws_array | ( | i | ) |
Convert python array to Google Web Services array
Input: {
ct_array - collective tuning array
ct_array_id - array with space data points UIDs
ct_axes - axes
ct_axes_desc - axes description
ct_axes_type - needed types for Google graph
(convert_text_axes_to_float) - convert text axes to consequitive floats
}
Output: {
cm_return - return code >0 if error
s_array - GWS array
s_array_id - GWS array ID (to process clicks)
}
| def module::convert_str_to_sha1 | ( | i | ) |
Convert string to sha1
Input: {
cm_string - string
(cm_console) - by default, do not output anything except errors
if 'txt' then output as plain text
if 'web' then output for web (html, image, etc)
if 'json' then output as json
}
Output: {
cm_return - return code >0 if not authentificated
cm_string_sha1 - password in SHA1 (digest) (only if console is not json, otherwise doens't work)
cm_string_sha1_hex - password in SHA1 (digest in hex)
cm_string_sha1_base64 - BASE64 (SHA1 digest) - compatible with htpasswd format
}
| def module::convert_text_axes_to_float | ( | i | ) |
Finish python graph and save to file
Input: {
ct_array - collective tuning array
ct_axes - axes
ct_axes_desc - axes description
(num_axes) - number of axes to process (to allow to convert only X sometimes)
}
Output: {
cm_return - return code >0 if error
ct_array - updated ct array
unique - if converted from string to number, unique values
unique_string - if converted from string to number, unique values converted to string
}
| def module::copy | ( | i | ) |
Copy repository to another one (and clean target if needed)
Input: {
orig_repo_uoa - original repository
(orig_module_uoa) - copy data for this module
(cm_classes_uoa) - prune data by classes
target_repo_uoa - target repository
(clean_target_repo) - force cleaning of target repo
}
Output: {
cm_return - return code >0 if error
cm_path - path to file
cm_uid - generated uid
}
| def module::copy_data | ( | i | ) |
Copy data entry
Input: {
(orig_repo_uoa) - repo UOA of original data
orig_module_uoa - module of the data to copy
orig_data_uoa - original data UOA
(target_repo_uoa) - target repo where to copy data
(if =='', repo of the original data is used)
(target_data_uoa) - target data UOA
(target_data_uid) - target data UID (if not set, original is used)
(remove_alias) - if 'yes', remove alias
(generate_new_uid) - if 'yes', generate new target UID
(overwrite) - if 'yes' and target exists, overwrite
(move) - if 'yes', delete original after copy
(cm_array_update) - update data of created entry
}
Output: {
cm_return - return code = 0, if successful
}
| def module::create | ( | i | ) |
| def module::create_button | ( | i | ) |
| def module::create_button_back | ( | i | ) |
| def module::default | ( | i | ) |
| def module::delete_data | ( | i | ) |
Delete data
Input: {
cm_run_module_uoa | cm_module_uoa - module UOA
cm_data_uoa - data UOA
(cm_repo_uoa) - repo UOA
(cm_force_delete) - if 'yes', force delete
(cm_console) - by default, do not output anything except errors
if 'txt' then output as plain text
if 'web' then output for web (html, image, etc)
if 'json' then output as json
(cm_admin) - if 'yes', override access control - only internal use,
can not be used during web access (explicitly removed)
}
Output: {
cm_return - return code >0 if error
-1 if entry exists
}
| def module::delete_data_through_web | ( | i | ) |
| def module::detect | ( | i | ) |
Detect system properties
Input: {
(host_os_uoa) - host OS on which cM is running to detect which scripts to use for detection.
If not set, the default one is taken from cM configuration.
(target_os_uoa) - target OS to detect. If =='', host OS is used
(get_uoa_from_repo) - if 'yes', check if already in repo and return UOA
(detect_repo_uoa) - search entries in this repository (if not set, do not search),
if not found, add entries
(record_if_doesnt_exist) - if 'yes', record in repo if doesn't exist
(ask_for_strings_if_not_detected) - if 'yes' and strings are not detected, ask for them
(add_all) - if 'yes' add all information including generated files about system
(used in private setups to keep all information for debugging
or machine learning (to improve models and find appropriate features))
}
Output: {
cm_return - return code = 0, if successful
cm_array - {
processor_string
system_string
available_physical_memory
}
processor_string - processor
os_string - OS
system_string - System
memory_string - All memory
all_frequencies_string - All possible frequencies
processor_uoa - detected processor UOA or taken from the input
os_uoa - detected OS UOA or taken from the input
system_uoa - prepared system UOA
}
| def module::detect_form_params | ( | i | ) |
Detect various parameters (commands) in cM web forms
Input: {
cm_array - array of form #prefix#var or #prefix@command
cm_prefix - prefix
}
Output: {
cm_return - return code >0 if error
cm_array - parsed array with keys
cm_commands - parsed array with commands
cm_explore - parsed array with exploration dimensions
}
| def module::detect_host_family | ( | i | ) |
Detect main OS version using python
Input: {
(cm_repo_uoa) - which repo to use to get info (to support remote ones)
}
Output: {
cm_return - return code = 0, if successful
family - OS family
cm_os_uoa_list - list of OSes from the same family (through search)
cm_host_uoa - most close OS UOA from known
cm_host_alias - alias for cm_host_uoa
}
| def module::detect_representative_points | ( | i | ) |
Detect representative points to build the same model
Input: {
model_type - "regression" or "classification"
data
ct_dimensions_input
(ct_dimensions_output) - for comparison
desc - cM data description
(max_variation_percent) - for comparison, report points where variation is more than this number (default=0.2)
}
Output: {
cm_return - return code >0 if error
results - list with info about models including
model_file_size
model_rmse
}
| def module::download | ( | i | ) |
Download repository using cM web interface (possibly substitute old entries)
Input: {
(web_repo_uoa) - repository UOA; if empty, update all repositories that are shared
(overwrite) - if 'yes', overwrite entries
(download_all) - if 'yes', download all
(skip_large_download) - if 'yes', skip entries with large downloads
(interactive) - if 'yes', interactive
(ask) - if 'yes', ask what to do with modules with many entries
(cm_remote_user_uoa) - change default user to access repo ...
}
Output: {
cm_return - return code >0 if error
}
Download data entry
Input: {
cm_run_module_uoa | cm_module_uoa - module UOA
cm_data_uoa - data UOA
(cm_repo_uoa) - repo UOA
cm_all_files - if 'yes', download .cm description files
this is needed for the future cloning of repositories
to avoid using all the time SVN/GIT for simple sharing ...
}
Output: {
cm_return - return code >0 if error
... - various parameters to set http headers, etc
}
Definition at line 3803 of file module.py.
Referenced by org::openme::openme::remote_access().
| def module::explore | ( | i | ) |
Explore choices
Input: {
cm_choices - TODO: describe format, see ct-compiler gcc-4.6.2
cm_exploration_strategy - random
(choice_rnd_seed) - random seed
*concatenate_output) - if 'yes', concatenate list to string
(output_key) - use this value to set output key instead of cm_choice.
Mainly used for command line exploration.
}
Output: {
cm_return - if =0, success
cm_choice or value of output_key - list of selected choices
}
Explore pipeline choices
Input: {
To update ...
exploration_scenario - exploration scenario for this pipeline
exploration_scenario_module_uoa - exploration scenario module UOA
number_of_iterations - number of coarse grain iterations
skip_if_entry_exists - if 'yes' and coarse-grain entry exists, skip it
useful for exploring only new coarse-grain points in space
state - init state
desc - state description
(mobile_mode) - 'build' or 'process' - mode during crowdsourcing auto-tuning using mobiles
(copy_built_executables) - if 'yes', copy all built executables to a_<UID>.out
(copy_src_to_tmp) - if 'yes', copy sources to a tmp repo
(useful for parallel cloud/mobile tuning)
(system_info) - detected system info by the mobile/cloud/grid system
(new_src_uoa) - new src UOA in tmp repository
(measured_frequency) - measured frequency (if supported)
(execution_info) - list of list of execution times per binary - 1 coarse-grain exploration step
(execution_info_min) - list of minimal execution times per 1 coarse-grain exploration step
(execution_info_ct_repeat_main) - CT_REPEAT_MAIN got during calibration
(built_files) - list of unique built files (with UIDs)
}
Output: {
cm_return - if =0, success
(additional parameters from the last scenario)
(built_files) =[{"name","size","prepared_compiler_flags"},...]
(new_src_uoa) - if src were copied, new UID in tmp repo
}
Explore a few points from mobile, cloud, etc
Input: {
predefined_uoa - data UOA with pre-defined input to compiler optimization scenario
mobile_mode - 'build' or 'process' - mode during crowdsourcing auto-tuning using mobiles
(system_info) - detected system info by the mobile/cloud/grid system
(new_src_uoa) - new src UOA in tmp repository
(measured_frequency) - measured frequency (if supported)
(execution_info) - list of list of execution times per binary - 1 coarse-grain exploration step
(execution_info_min) - list of minimal execution times per 1 coarse-grain exploration step
(execution_info_ct_repeat_main) - CT_REPEAT_MAIN got during calibration
(built_files) - list of unique built files (with UIDs)
}
Output: {
cm_return - if =0, success
(new_src_uoa) - if src were copied, new UID in tmp repo
}
| def module::filter_special_str | ( | i | ) |
| def module::find_aggregated_entry | ( | i | ) |
| def module::find_cid | ( | i | ) |
| def module::find_data_without_access_control | ( | i | ) |
Test some funcionality before moving to a proper test
Input: {
(prune_repo_uoa) - prune search by a given repository
(prune_module_uoa) - prune search by a given module
(prune_data_uoa) - prune search by a given data
}
Output: {
cm_return - return code >0 if error
}
| def module::find_key_value_in_array | ( | i | ) |
Find key/value pair in array
Input: {
cm_array - array
cm_search - {"key": key, "value": value} or {"value":value}
(use_flat_array) - if 'yes', keys should be in "flat array" format (##x#y ...)
(strict_match) - if 'yes', make exact comparison of values
}
Output: {
cm_return - return code >0 if error
cm_found - True if found
}
| def module::find_path_to_data | ( | i | ) |
| def module::find_path_to_repository | ( | i | ) |
Find path to a given repository
Input: {
(find_repo_uoa) - repo UOA
(get_default_repo) - if 'yes' and find_repo_uoa=='', get path to default repo
(cm_console) - if 'txt', output as text to console
(cm_admin) - if 'yes', override access control - only internal use,
can not be used during web access (explicitly removed)
}
Output: {
cm_return - return code >0 if error
cm_path - path to repository
cm_data_obj - cM data object
cm_uid - uid (from UOA)
cm_alias - alias (from UOA)
cm_uoa - cm_data_uoa (alias or uid if alias=='')
(cur_module_uoa) - current module UOA if in current path
(cur_data_uoa) - current data UOA if in current path
}
| def module::finish_python_graph | ( | i | ) |
Finish python graph and save to file
Input: {
ct_graph - graph type
matplot_obj - prepared MatPlot object
cm_file_ext - file extension
(url) - starting url of cM
(reproduce) - parameters to reproduce/save graph
(save_graph) - prepare for saving
(cm_console) - if 'txt' output path to the generated image to the console
(legend) - if !='', add to the end of the graph
(mpl_extra_params) - if python mat plot is used, this dictionary has extra params
mpl_image_size_x - image X size in inch
mpl_image_size_y - image Y size in inch
mpl_image_dpi - image DPI
}
Output: {
cm_return - return code >0 if error
}
| def module::fit | ( | i | ) |
Fit data using existing models (combine build and predict)
Input: {
data
ct_dimensions_input
(ct_dimensions_output) - for comparison
desc - cM data description
(max_variation_percent) - for comparison, report points where variation is more than this number (default=0.2)
}
Output: {
cm_return - return code >0 if error
results - list with info about models including
model_file_size
model_rmse
}
| def module::flush | ( | i | ) |
| def module::functionality | ( | i | ) |
| def module::gen_cm_tmp_file | ( | i | ) |
| def module::gen_docs | ( | i | ) |
| def module::gen_stats | ( | i | ) |
| def module::gen_uid | ( | i | ) |
Generate cM UID
Input: {
(cm_console) - by default, do not output anything except errors
if 'txt' then output as plain text
if 'web' then output for web (html, image, etc)
if 'json' then output as json
}
Output: {
cm_return - return code = 0, if successful
cm_uid - cM UID
}
| def module::generate | ( | i | ) |
Generating research topics and papers for "low-hanging fruits" ;)
Input: {
cm_data_uoa - data with topic template
(cm_repo_uoa) - repository
}
Output: {
cm_return - return code >0 if error
}
Generate qr-code
Input: {
cm_string - string to convert to qr-code
qr_level - qr_level (4,5,etc)
image_size - picture size
}
Output: {
cm_return - return code >0 if error
cm_full_filename - file with image
}
| def module::get_all_repo_paths | ( | i | ) |
Get paths for all repositories (has to be low level operation!)
Input: {
(cm_admin) - if 'yes', override access control - only internal use,
can not be used during web access (explicitly removed)
}
Output: {
cm_return - return code >0 if error
cm_array - array with paths
}
| def module::get_env | ( | i | ) |
| def module::get_info_about_current_path | ( | i | ) |
Get info about current path
Input: {}
Output: {
cm_return - return code >0 if error
-1 if doesn't exist
cm_path - path to the local repository
(cm_module_uoa) - module UOA of the current path
(cm_data_uoa) - data UOA of the current path
(cm_repo_uoa) - repo UOA
(cm_repo_uid) - repo UID
}
| def module::get_info_about_full_hosts | ( | i | ) |
| def module::get_list_of_all_files | ( | i | ) |
Get a list of all files in a given directory recursively
Input: {
cm_module_uoa - module UOA
cm_data_uoa - data UOA
(cm_repo_uoa) - repo UOA
cm_path_ext - path extension
limit - limit number of files (if huge directories)
number - cur. number of files
cm_all_files - if 'yes' do not ignore special files (like .cm)
}
Output: {
cm_return - return code = 0 if successful
cm_array - list of files
}
| def module::get_state | ( | i | ) |
Get current state
Input: {
(host_os_uoa) - host OS on which cM is running to detect which scripts to use for detection.
If not set, the default one is taken from cM configuration.
(target_os_uoa) - target OS to detect. If =='', host OS is used
}
Output: {
cm_return - return code = 0, if successful
system_state - [windows]
[unix]
}
| def module::get_stats_for_benchmarks | ( | i | ) |
Compare systems through benchmarking
Input: {
(data) - list of data entries to analyze
(prune_by_summary_point_relaxed) - if data is not provided, search will be performed for all entries;
returned entries can be pruned by this dictionary
using 'summary_point_relaxed' in entries
characteristic_key - key to get characteristic (will be converted to float)
(cm_repo_uoa) - working repository
}
Output: {
cm_return - if =0, success
stats - dictionary of format
{
"benchmark1":{
"min" - min value
"max" - max value
"delta" - max-min
"percent_delta_to_min" - delta/min
"mean" - mean
"std" - standard deviation
"percent_std_to_min" - standard deviation/min
}
...
}
}
| def module::gws_2d_column_chart | ( | i | ) |
Google Web Services 2D Column Chart
Input: {
ct_module_uoa - module to visualize
ct_array - collective tuning array
ct_array_id - array with space data points UIDs
ct_axes - axes
ct_axes_desc - axes description
ct_points - selected entries
url - starting url of cM
(convert_text_axes_to_float) - convert text axes to consequitive floats
(legend) - if !='', add to the end of the graph
}
Output: {
cm_return - return code >0 if error
}
| def module::gws_2d_scatter_chart | ( | i | ) |
Google Web Services 2D Scatter Chart
Input: {
ct_module_uoa - module to visualize
ct_array - collective tuning array
ct_array_id - array with space data points UIDs
ct_axes - axes
ct_axes_desc - axes description
ct_points - selected entries
url - starting url of cM
(convert_text_axes_to_float) - convert text axes to consequitive floats
(legend) - if !='', add to the end of the graph
}
Output: {
cm_return - return code >0 if error
}
| def module::help | ( | i | ) |
| def module::index | ( | i | ) |
Add or delete index
Input: {
cm_module_uoa - module UOA
(cm_module_uid)
cm_data_uoa - data UOA
(cm_data_uid)
(cm_repo_uoa) - repo UOA
(cm_repo_uid)
mode - 'delete' | 'add'
(cm_index_uoa) - index configuration
(sequential) - if 'yes' perform sequential index (very slow, but reports errors)
}
Output: {
cm_return - return code >0 if error
}
Show cM main web page
Input: {
(cm_web_preprocess) - if 'yes', preprocess request (usually user authentication)
(cm_menu) - select user menu
}
Output: {
cm_return - return code >0 if error
}
| def module::index_seq | ( | url, | ||
| jindex, | ||||
| mode, | ||||
| module_uid, | ||||
| data_uid | ||||
| ) |
| def module::info | ( | i | ) |
| def module::init | ( | i | ) |
| def module::install | ( | i | ) |
Install code and set environment
Input: {
(install_data_uid) - create an entry with this UID
(install_data_alias) - create an entry with this alias
(install_data_display_as_alias) - use this display as alias for a generated entry
(install_module_uoa) - use this module to create an entry (if not set, use 'code')
(install_repo_uoa) - use this repo to create an entry
(cm_array) - add this data to the new entry
target_os_uoa - target os to install script
(add_rem_to_script) - add rem to install script
}
Output: {
cm_return - return code = 0, if successful
Parameters from entry creation
script_filename - script filename
script_name - full script name
}
Install cM package
Input: {
package_data_uoa - UOA of package to install
(package_repo_uoa) - repo of the package
package_host_os_uoa - UOA of host OS (to set up script execution)
(install_data_uid) - UID of the code where package will be installed
if install_data_uid=='' it will be randomly generated
(install_data_alias) - use this alias for a generated entry
(install_data_display_as_alias) - use this display as alias for a generated entry
If package should be built:
build_target_os_uoa - target OS uoa for building
(code_deps) - list with code UOA for dependencies [{"index":"uoa"} ...]
(run_target_processor_uoa) - target processor UOA (not strictly needed - can add some helper parameters before executing code)
(run_set_env2) - array with environment variables to be set before executable
(add_rem_to_script) - add rem to install script
(add_to_code_entry) - add array to code entry
(skip_extract_and_build) - if 'yes', skip copy, extract and build (by setting CM_SKIP_BUILD=yes)
}
Output: {
cm_return - return code = 0, if successful
Output from 'code install'
}
| def module::installed | ( | i | ) |
List installed packages
Input: {
(code_repo_uoa) - repo UOA to list code
(prune_by_class_uoa) - prune by 1 class UOA
(prune_by_name) - prune by name (checks for inclusion; case insensitive)
(fuzzy_match) - if 'yes', check inclusion of 'prune_by_name' in name, otherwise exact match
(prune_by_name_uoa) - prune by name UOA
(prune_by_type) - prune by "library_or_plugin" or "package"
(prune_by_host_os) - prune by host os
(prune_by_target_os) - prune by target os
(prune_by_target_processor) - prune by target processor
(prune_by_compiler) - if library or plugin, prune by compiler
(prune_by_os_from_kernel) - if 'yes', take OS from kernel
(only_uoa) - return only UOA
}
Output: {
cm_return - return code = 0, if successful
final - list with code entries in cM 'list' format
}
| def module::list_data | ( | i | ) |
List data
Input: {
cm_run_module_uoa | cm_module_uoa - module UOA
(cm_repo_uoa) - repo UOA
(cm_data_uoa) - data UOA
(cm_console) - by default, do not output anything except errors
if 'txt' then output as plain text
if 'web' then output for web (html, image, etc)
if 'json' then output as json
(cm_only_uid) - 'yes' to show only UID
(cm_class_uoa) - prune by 1 class UOA
or
(cm_classes_uoa) - prune by classes UOA (list)
(cm_only_available_modules) - if 'yes', return only available modules in a given repository
(skip_data_cfg) - if 'yes', do not add data cfg
(show_alias_and_uid) - if 'yes', show alias (UID)
(show_display_as_alias) - if 'yes', show in format DISPLAY_AS_ALIAS (UID / ALIAS)
}
Output: {
cm_return - return code >0 if error
-1 if entry exists
cm_array - array with UOA
cm_mixed - list of {cm_uid, cm_alias, cm_uoa, cm_display_as_alias, cm_display_html}
}
| def module::list_files_through_web | ( | i | ) |
| def module::load_data | ( | i | ) |
Load data
Input: {
cm_module_uoa - module UOA
cm_data_uoa - data UOA
(cm_repo_uoa) - repo UOA
(cm_console) - by default, do not output anything except errors
if 'txt' then output as plain text
if 'web' then output for web (html, image, etc)
if 'json' then output as json
(cm_web) - yes, for web environment
(lock_acquire) - if 'yes', acquire lock
(lock_expire) - for how long to acquire lock (secs)
(cm_admin) - if 'yes', override access control - only internal use,
can not be used during web access (explicitly removed)
}
Output: {
cm_return - return code = 0, if successful
< 0, if error
32 - lock acquired by someone else
(cm_error) - error text if return code > 0
cm_data_obj - cM data object
cm_path - path to data entry
cm_path_module - path to module entry with this data
cm_uid - uid (from UOA)
cm_alias - alias (from UOA)
cm_display_as_alias - taken from the data
cm_repo_uoa - UOA of a repository
cm_repo_uid - UID of a repository
cm_display_html - prepare display: "display as alias" or <I>alias</I> or UID
(lock_uid) - UID of the aquired lock
}
| def module::load_default_kernel_data | ( | i | ) |
| def module::login | ( | i | ) |
| def module::logout | ( | i | ) |
| def module::mobile_node_prepare | ( | i | ) |
| def module::mpl2_2d_scatter_chart | ( | i | ) |
| def module::mpl2_2d_scatter_chart_a | ( | i | ) |
| def module::mpl2_2d_warm_graph | ( | i | ) |
| def module::mpl2_2d_warm_graph_a | ( | i | ) |
| def module::mpl_1d_density_graph | ( | i | ) |
Python Matplot 1D density graph
Input: {
ct_array - collective tuning array
ct_array_id - array with space data points UIDs
ct_axes - axes
ct_axes_desc - axes description
ct_points - selected entries
reproduce - parameters to reproduce/save graph
(url) - starting url of cM
(save_graph) - if 'yes', save graph
(cm_file_ext) - file extension if save_graph=='yes'
(convert_text_axes_to_float) - convert text axes to consequitive floats
(mpl_extra_params) - if python mat plot is used, this dictionary has extra params
mpl_image_size_x - image X size in inch
mpl_image_size_y - image Y size in inch
mpl_image_dpi - image DPI
mpl_detach_graph - if 'on', detach graph
}
Output: {
cm_return - return code >0 if error
}
| def module::mpl_2d_column_chart | ( | i | ) |
Python Matplot 2D column chart
Input: {
ct_array - collective tuning array
ct_array_id - array with space data points UIDs
ct_axes - axes
ct_axes_desc - axes description
ct_points - selected entries
reproduce - parameters to reproduce/save graph
(url) - starting url of cM
(save_graph) - if 'yes', save graph
(cm_file_ext) - file extension if save_graph=='yes'
(convert_text_axes_to_float) - convert text axes to consequitive floats
(mpl_extra_params) - if python mat plot is used, this dictionary has extra params
mpl_image_size_x - image X size in inch
mpl_image_size_y - image Y size in inch
mpl_image_dpi - image DPI
mpl_detach_graph - if 'on', detach graph
}
Output: {
cm_return - return code >0 if error
}
| def module::mpl_2d_scatter_chart | ( | i | ) |
Python Matplot 2D scatter chart
Input: {
ct_array - collective tuning array
ct_array_id - array with space data points UIDs
ct_axes - axes
ct_axes_desc - axes description
ct_points - selected entries
reproduce - parameters to reproduce/save graph
(url) - starting url of cM
(save_graph) - if 'yes', save graph
(cm_file_ext) - file extension if save_graph=='yes'
(convert_text_axes_to_float) - convert text axes to consequitive floats
(mpl_extra_params) - if python mat plot is used, this dictionary has extra params
mpl_image_size_x - image X size in inch
mpl_image_size_y - image Y size in inch
mpl_image_dpi - image DPI
mpl_detach_graph - if 'on', detach graph
}
Output: {
cm_return - return code >0 if error
}
| def module::mpl_get_available_fonts | ( | i | ) |
| def module::native_run | ( | i | ) |
| def module::output_known_characteristics | ( | i | ) |
| def module::postprocess_web_request | ( | i | ) |
Postprocess web request
Input: {
(cm_web_session) - $_SESSION
(cm_web_get) - $_GET
(cm_web_post) - $_POST
(cm_web_cookie) - cookies
(cm_user_uoa) - user UOA
(cm_user_password) - user password in plain text
(not secure, usually used for web form authentication)
(cm_user_password1) - user password in SHA1
(more secure, can be stored in environment, session or cookies)
cM authenicates by taking SHA1(cm_user_password1)...
(cm_web_module_uoa) - module to envoke
(cm_web_action) - action for the module
}
Output: {
cm_return - return code >0 if error
}
| def module::predict | ( | i | ) |
Predict using some model
Input: {
model_module_uoa - model's module
(model_data_uoa) - model's data (if need to record)
model_name - (earth, svm)
data
ct_dimensions_input
(ct_dimensions_output) - for comparison
desc - cM data description
(max_variation_percent) - for comparison, report points where variation is more than this number (default=0.2)
}
Output: {
cm_return - return code >0 if error
(rmse) - if comparison, root mean square error for predictions vs original
(max_var) - list of points with variation more than max_variation_percent
}
Predict using model
Input: {
model_file
data
ct_dimensions_input
(ct_dimensions_output) - for comparison
desc - cM data description
model_name - (earth, svm)
(max_variation_percent) - for comparison, report points where variation is more than this number (default=0.2)
}
Output: {
cm_return - return code >0 if error
(rmse) - if comparison, root mean square error for predictions vs original
(max_var) - list of points with variation more than max_variation_percent
}
| def module::predict_opts | ( | i | ) |
Predict optimizations
Input: {
In non-web mode:
(program_milepost_properties) - if not empty, return most close programs
(cm_submit_prune) - if 'yes', prune results to find most close program
(opt_predictor) - 'nearest_neighbour' for nearest neighbour classifier
(cm_repo_uoa) - repo to prune results
(pruning_similar_programs) - direct pruning parameters
(number_of_returned_items) - number of returned items
}
Output: {
cm_return - if =0, success
}
| def module::prepare_choices | ( | i | ) |
Prepare tuning dimension
Input: {
cm_array - pipeline array
cm_data_desc - pipeline data description
cm_explore - exploration parameters
cm_explore_state - exploration state
cm_explore_level - exploration level ('' - coarse-grain, '1' - next level, ...)
random_seed - random seed
}
Output: {
cm_return - if =0, success
cm_explore_state - exploration state
cm_exploration_finished - if 'yes', exploration finished (for nested loops)
}
| def module::prepare_cmd | ( | i | ) |
Prepare command line for the code
Input: {
(code_module_uoa) - module of the working data
(code_data_uoa) - working data uoa
(run_cmd_key) - name if multiple CMDs available for this program
(dataset_uoa) - UOA of dataset used to prepare this cmd
(os_uoa) - OS UOA
(compilation_type) - static or dynamic
}
Output: {
cm_return - if =0, success
(cm_error) - if cm_return>0, error text
run_time={
run_env - set environment before command line
- the following params will be aggregated to 1 line
run_cmd1
run_cmd_binary_name
run_cmd2
run_cmd_main
run_cmd3
run_cmd_out1
run_cmd_out2
run_output_files
run_input_files
dataset_files
}
build_compiler_vars - possible compiler variables (often set hardwired dataset)
}
| def module::prepare_content_header | ( | i | ) |
| def module::prepare_env_for_all_codes | ( | i | ) |
Prepare environment for all codes
Input: {
code_deps - list with code UOA for dependencies [{"index":"uoa"} ...]
os_uoa - OS UOA
no_strict_check - if 'yes', do not check if code dependency was installed properly
}
Output: {
cm_return - return code = 0, if successful
cm_string - environment string
cm_array - array with environment setting (for scripts)
include_paths1 - list with paths with include directories
lib_paths - list with libraries
env_separator - environment separator (just in case to avoid double loading of OS)
}
| def module::prepare_env_vars | ( | i | ) |
Prepare environment variables
Input: {
array - array of environment variables to set
prefix - add prefix to set environment variable
separator - add separator for the command line
(quotes) - use quotes?
}
Output: {
cm_return - if =0, success
cm_string - cmd with prepared environment (export A=B; export C=D; ...)
cm_string1 - simplified string (CM_ALL_ENV="A=B C=D E=F")
cm_array - array with environment setting (for scripts)
}
| def module::prepare_environment | ( | i | ) |
| def module::prepare_flags | ( | i | ) |
Prepare flags from array
Input: {
cm_data_uoa - data with compiler description
all_flags - {} all flags
(all_flags_order) - [] order of flags
}
Output: {
cm_return - if =0, success
prepared_flags - string with prepared flags
all_flags_order - [] order of flags (if original was empty,
prepare order according to sort_index in description)
}
| def module::prepare_input_params | ( | i | ) |
Prepare input parameters for this scenario
Input: {
cm_form_array - array with forms
form_exists - yes or no
cm_array1 - array from form 1 (coarse-grain points)
cm_array2 - array from form 2 (additional parameters) just in case
caller_cfg - caller module cfg
}
Output: {
cm_return - if =0, success
}
| def module::prepare_path | ( | i | ) |
| def module::prepare_script | ( | i | ) |
Prepare script with code deps
Input: {
script_name - script name (usually without extensions)
target_os_uoa - target os to get script extension and other parameters
(skip_extension) - if 'yes', script already has extension
(cm_path) - path to create script
(set_env1) - add environment vars (1) to script
(code_deps) - add code deps to script
(set_env2) - add environment vars (2) to script
(run_commands_before) - add some commands before main command
(run_cmd) - add main command
(run_commands_after) - add some commands after main command
}
Output: {
cm_return - return code = 0, if successful
cm_path - full name of the created script with path
}
| def module::prepare_sub_script | ( | i | ) |
Prepare sub_script with extension
Input: {
run_script - script name (usually without extensions)
(run_script_uoa) - script UOA if any
target_os_cfg - target os cfg to get script extension and other parameters
}
Output: {
cm_return - return code = 0, if successful
cm_path - full name of the created script with path
}
| def module::prepare_summary_point | ( | i | ) |
Prepare summary point (to save a unique experiment in a repository)
Input: {
state - pipeline state to use dimensions for search
}
Output: {
cm_return - if =0, success
summary_point1 - summary point (dimensions) - strict summary point to detect the same experiments
summary_point2 - summary point (dimensions) - relaxed summary point for viewing and pruning
}
| def module::prepare_tuning_dimension_for_web | ( | i | ) |
| def module::preprocess_web_request | ( | i | ) |
Preprocess web request (convert $_GET and $_POST into one array and envoke cM module)
Input: {
(cm_web_session) - $_SESSION
(cm_web_get) - $_GET
(cm_web_post) - $_POST
(cm_web_cookie) - cookies
(cm_user_uoa) - user UOA
(cm_user_password) - user password in plain text
(not secure, usually used for web form authentication)
(cm_user_password1) - user password in SHA1
(more secure, can be stored in environment, session or cookies)
cM authenicates by taking SHA1(cm_user_password1)...
(cm_web_module_uoa) - module to envoke
(cm_web_action) - action for the module
}
Output: {
cm_return - return code >0 if error
... - output from the envoked module
}
| def module::print_help | ( | i | ) |
| def module::process | ( | i | ) |
Run collective tuning pipeline
Input: {
space - available points in space
pipeline_input - pipeline input
pipeline_output - pipeline output
desc - state description
tune_characteristics - list of characteristics to tune
(priorities, such as global execution time; code size; energy; compilation time, etc -
depending on user scenario)
calcluate_improvement_over_ref - if !='', use this as the key in space
to calcluate improvements over first point
max_improvement_threshold - if at least one of the max improvement more than threshold, add
}
Output: {
cm_return - if =0, success
state - final state
space - (updated) space
must_add - if 'yes', should add this point
}
Run collective tuning pipeline
Input: {
state - init state
(state1) - state where to record characteristics - if empty, use 'state'
desc - state description
}
Output: {
cm_return - if =0, success
state - final state
state1 - state1
}
Analyze variation of variables
Input: {
array - dictionary of format
{
"varA":[valueA1, valueA2, ...],
"varB":[valueB1, valueB2, ...],
...
}
}
Output: {
cm_return - if =0, success
array - dictionary of format
{
"varA":{
"min" - min value
"max" - max value
"delta" - max-min
"percent_delta_to_min" - delta/min
"mean" - mean
"std" - standard deviation
"percent_std_to_min" - standard deviation/min
}
...
}
}
| def module::process_web_request | ( | ix | ) |
| def module::prune | ( | i | ) |
| def module::record | ( | i | ) |
| def module::register_func | ( | i | ) |
Register function (action) in a module
Input: {
cm_run_module_uoa - module UOA
(cm_common_func) - if ='true', then the module is called from some other module
(new_func) - new function name
(new_desc) - new desc
(new_index) - new index (if empty, use func)
}
Output: {
cm_return - 0
}
| def module::rename_data | ( | i | ) |
Rename data entry
Input: {
(orig_repo_uoa) - repo of original data
orig_module_uoa - module of original data
orig_data_uoa - data UOA to rename
new_data_alias - new data alias (leave empty to remove alias)
new_data_uid - new data UID (leave empty to keep old one)
}
Output: {
cm_return - return code = 0, if successful
}
| def module::repair | ( | i | ) |
| def module::reproduce | ( | i | ) |
Reproduce collective tuning pipeline state
Input: {
reproduce_module_uoa - pipeline state module UOA
reproduce_data_uoa - pipeline state data UOA
(reproduce_repo_uoa) - pipeline state repo UOA
(point) - point in state (by default 0)
data_uoa - use this data UOA to load recorded state
(repo_uoa) - use this repo UOA to load recorded state
(pipeline_uoa) - entry with pipeline
or
(pipeline) - pipeline array [{cm_access input}]
(record_in_space) - if 'yes', record in cTuning space
(record_repo_uoa) - if record, which repo to use
(record_data_uoa) - if record, set data_uoa (if not set, generate randomly)
}
Output: {
cm_return - if =0, success
pipeline_input - input pipeline (original)
pipeline_output - output pipeline (new)
}
| def module::restore_writing_in_kernel | ( | i | ) |
| def module::run | ( | i | ) |
Run code (binary or script)
Input: {
(run_host_os_uoa) - host OS UOA (if not set, use default OS from kernel)
(run_target_os_uoa) - target OS UOA (if not set, use run_host_uoa_os)
(run_target_processor_uoa) - target processor UOA (not strictly needed - can add some helper parameters before executing code)
--------------------------------------------------------------------------------------------
run_cmd_key - if !='', set all further run parameters (including target name)
from the work_dir_data_uoa (must be set)
(dataset_uoa) - use dataset to update cmd params from above
or
run_cmd - command line to run
or
run_script - use the following script (add extension from run_target_os)
run_script_uoa - take script from this entry in os.script (add extension from run_target_os)
or
(run_cmd1) - following format: run_cmd1 run_cmd_name|binary_name run_cmd2 run_cmd_main run_cmd3 > run_cmd_out1 2> run_cmd_out2
(run_cmd_name)
(run_cmd2)
(run_cmd_main) or (binary_name)
(run_cmd3)
(run_cmd_out1)
(run_cmd_out2)
--------------------------------------------------------------------------------------------
(work_dir) - change to this working directory before running code
or
(work_dir_repo_uoa) - change to the working directory in the repository (repo UOA)
(work_dir_module_uoa) - change to the working directory in the repository (module UOA)
(work_dir_data_uoa) - change to the working directory in the repository (data UOA)
(run_input_files) - list of files needed to run code (can be copied to the remote host system)
(run_output_files) - list of files that will be created (for validation, etc)
(code_deps) - list with code UOA for dependencies [{"index":"uoa"} ...]
(skip_code_deps) - if 'yes', skip code deps
(run_set_env1) - array with environment variables to be set before code deps
(run_set_env2) - array with environment variables to be set before executable
(run_commands_before) - run commands before executable
(run_commands_after) - run commands after executable
(cm_dependencies) - dependencies that set all other dependencies (code/include/lib).
Format [{"class UOA":"code UOA"},{},...]
(run_timeout) - time out for run (overheads may occur)
(keep_all_files) - if 'yes', do not remove tmp files and do not clean up remote device
(cm_verbose) - if 'yes', print all info
}
Output: {
cm_return - return code = 0, if successful
timed_out - 'yes' if timed out
failed_run - 'yes' if return code !=0 or timed out or binary doesn't exist
run_time_by_module - run time measured by module
exit_code - exit code of the system call
run_cmd - final run cmd
work_dir - work directory
run_output_files - final output files
}
Run collective tuning pipeline
Input: {
state - init state
Some additional debug/developer parameters
cm_show_json - if 'yes', show input json (normally used before pipeline)
cm_save_json_to_file - if !='', save input json to file (normally used before pipeline)
cm_show_output_json - if 'yes', show final output state json on console
cm_save_output_json_to_file - if !='', save final output state json to file
}
Output: {
cm_return - if =0, success
state - final state
fail - True, if pipeline fail
}
| def module::run_stat | ( | i | ) |
Run collective tuning pipeline with statistical analysis of variability
Input: {
state - init state
desc - pipeline description (needed to know characteristics)
additional_params - scenario parameters dict
"repeat_for_statistics" - number of repetitions for statistical analysis
}
Output: {
cm_return - if =0, success
state - final state
}
| def module::run_stat_cmd | ( | i | ) |
Run pipeline from CMD with user-friendly features
Input: {
(host_os_uoa) - if not selected, take from kernel
(target_os_uoa) - if not selected, take from kernel
(target_processor_uoa) - if not selected, use generic-x86-64
compiler_code_uoa
or
compiler_package_uoa
or
compiler_package_name
(compiler_fuzzy_match) - if 'compiler_package_name' is used, and 'yes' check for inclusion rather than exact match of names
program_uoa6
run_cmd_key
dataset_uoa
compile_script
(kernel_repeat)
(repeat_for_statistics) - number of pipeline repetitions (for statistical analysis)
(statistics_module) - module for statistical analysis
(update_state) - directly update state
(only_prepare_state) - if 'yes', do not run pipeline but only update state and description
}
Output: {
cm_return - if =0, success
}
| def module::scenario | ( | i | ) |
Build and run program, save to repository
Input: {
caller_module_uid - caller module UID
state - pipeline state
exploration_scenario - exploraiton scenario parameters
summary_point - summary dimensions describing aggregated point
aggregated_entry - all aggregated points
space - if aggregated, current space
original_input - to get other additional forms per scenario
cm_data_desc - data description (from the original form)
cm_form_explore - form explore (form the original form)
number_of_iterations - max number of iterations
}
Output: {
cm_return - if =0, success
}
Build and run program, save to repository
Input: {
caller_module_uid - caller module UID
state - pipeline state
exploration_scenario - exploration scenario parameters
work_repo_uoa - repository to save results
summary_point - summary dimensions describing aggregated point
aggregated_entry - all aggregated points
space - if aggregated, current space
original_input - to get other additional forms per scenario
cm_data_desc - data description (from the original form)
cm_form_explore - form explore (form the original form)
number_of_iterations - max number of iterations
mobile_mode - 'build' or 'process' - mode during crowdsourcing auto-tuning using mobiles
copy_built_executables - if 'yes', copy all built executables to a_<UID>.out,
state to a_<UID>.json and space to ct_space.json
(new_src_uoa) - new src UOA in tmp repository
(new_src_path) - new src path
(measured_frequency) - measured frequency (if supported)
(execution_info) - list of list of execution times per binary - 1 coarse-grain exploration step
(execution_info_min) - list of minimal execution times per 1 coarse-grain exploration step
(execution_info_ct_repeat_main) - CT_REPEAT_MAIN got during calibration
(built_files) - list of unique built files (with UIDs)
}
Output: {
cm_return - if =0, success
}
Extract program MILEPOST static properties (features), save to repository
Input: {
caller_module_uid - caller module UID
state - pipeline state
exploration_scenario - exploraiton scenario parameters
summary_point - summary dimensions describing aggregated point
aggregated_entry - all aggregated points
space - if aggregated, current space
original_input - to get other additional forms per scenario
cm_data_desc - data description (from the original form)
cm_form_explore - form explore (form the original form)
number_of_iterations - max number of iterations
}
Output: {
cm_return - if =0, success
}
| def module::scenarios | ( | i | ) |
| def module::search | ( | i | ) |
Search
Input: {
(repo_selection) - list of repo UOA to prune search
(module_selection) - list of module UOA to prune search
(key_X) - key to search (if use_flat_array=='yes', key should be in "flat array" format)
(value_X) - value to search (find this occurance in the string)
(cm_es_string) - ElasticSearch string
(use_internal_search) - if 'yes', do not use ElasticSearch
(use_flat_array) - if 'yes', keys should be in "flat array" format
(
flat array is needed to access directly any key in dictionary hierarchy
flat array format examples:
{"x":{"y":"z"}} - "##x#y"="z"
{"x":["y","z"]} - "##x@0"="y", "##x@1"="z"
)
(show_search_progress) - if 'yes', show search progress
(timeout) - timeout in seconds for internal slow search
(strict_match) - if 'yes', use strict match (in internal search)
}
Output: {
cm_return - return code >0 if error
all - list with description of entries returned by search
[
{
cm_repo_uoa - Found entry's repository UOA (UID or alias)
cm_repo_uid - Found entry's repository UID
cm_module_uoa - Found entry's module UOA (UID or alias)
cm_module_uid - Found entry's module UID
cm_data_uoa - Found entry's UOA (UID or alias)
cm_data_uid - Found entry's UID
(cm_display_as_alias) - Fodun entry's user-friendly alias for printing
}
...
]
}
Search
Input: {
cm_es_string - simple search
cm_es_query - elastic search query
(cm_index_uoa) - index configuration
(size) - limit return array
}
Output: {
cm_return - return code >0 if error
cm_array - array of UIDs
}
| def module::select | ( | i | ) |
Select from the choice
Input: {
cm_choices - TODO: describe format, see ct-compiler gcc-4.6.2
cm_exploration_strategy - random
(choice_rnd_seed) - random seed
(cm_choice) - previous cm_choice state (can be updated using various methods including probabilistic)
}
Output: {
cm_return - return code = 0, if successful
cm_choice
}
| def module::separate_common_from_data | ( | i | ) |
| def module::server | ( | i | ) |
Index Server
Input: {
(cm_index_uoa) - uoa with description of indexer
(os_uoa) - uoa with OS description (needed for script extension, for example)
(cm_index_port) - web port
}
Output: {
cm_return - return code >0 if error
}
Simple Web Server
Input: {
cm_web_port - web port
}
Output: {
cm_return - return code >0 if error
}
| def module::set_cpu_frequency | ( | i | ) |
Set CPU frequency
Input: {
(host_os_uoa) - host OS on which cM is running to detect which scripts to use for detection.
If not set, the default one is taken from cM configuration.
(target_os_uoa) - target OS to detect. If =='', host OS is used
frequency - frequency
(use_sudo) - if 'yes', use SUDO
}
Output: {
cm_return - return code = 0, if successful
system_state - [windows]
[unix]
}
| def module::set_env | ( | i | ) |
| def module::share | ( | i | ) |
| def module::show | ( | i | ) |
Show file from the repository or tmp directory.
Only web preprocess!
Input: {
cm_web_filename - file to show
(cm_web_content) - extension known by cM to prepare html header.
If not set, this function will try to detect
it from the cm_web_filename extension
(cm_module_uoa and cm_data_uoa) - entry to take file from.
If not set, file is taken from the temporary directory
(cm_repo_uoa) - repo UOA
}
Output: {
cm_return - return code >0 if error
cm_web_headers - web headers
cm_web_postprocess=no - force cM not to postprocess
}
| def module::show_html | ( | i | ) |
Show file from the repository or tmp directory.
Input: {
cm_web_filename - file to show
(cm_web_content) - extension known by cM to prepare html header.
If not set, this function will try to detect
it from the cm_web_filename extension
(cm_module_uoa and cm_data_uoa) - entry to take file from.
If not set, file is taken from the temporary directory
}
Output: {
cm_return - return code >0 if error
}
| def module::show_legend | ( | i | ) |
| def module::temporaly_allow_writing_in_kernel | ( | i | ) |
| def module::test | ( | i | ) |
| def module::test_repository | ( | i | ) |
Test repository (mainly remote access)
It just try to load tmp:tmp and returns ok, if it exists or not found,
thus meaning that the structure of repository/access is fine
Input: {
test_repo_uoa - repo UOA
... - various additional parameters
(for example, for authentication)
}
Output: {
cm_return - return code >0 if error
}
| def module::unregister_func | ( | i | ) |
| def module::update_data | ( | i | ) |
Update data
Input: {
cm_run_module_uoa | cm_module_uoa - module UOA
cm_data_uoa - data UOA
(cm_data_uid) - force data UID (if UOA is alias)
(cm_repo_uoa) - repo UOA
(cm_console) - by default, do not output anything except errors
if 'txt' then output as plain text
if 'web' then output for web (html, image, etc)
if 'json' then output as json
cm_array - use ONLY this array to update data
(cm_skip_update_info) - if yes, do not write 'cm_updated'
(cm_update_only_index) - update only index
(cm_file_upload) - text content of file for upload
(cm_file_upload_base64) - file content as base64.urlsafe_b64encode
(cm_file_upload_name) - preffered uploaded file name
(cm_file_upload_type) - type of file (if .zip, it will be automatically unziped)
(cm_archive) - 'none'
'already_archived_with_internal_name' - should save file
under default cM name (cm_archive.zip)
'already_archived_with_user_name' - should save file under user name
(cm_file_upload_tmp_uid) - normally generated automatically by web server
when uploading files
(cm_add_default) - if 'yes', add default params from descriptions (module/kernel)
(cm_description) - description of the data entry
(cm_display_as_alias) - user readable alias to display instead of module:data
(cm_note) - note about this data entry
(cm_like) - if 'yes' add +1 for this user to the entry
... - data to add to the repository if not set 'cm_array'
(lock_uid) - UID of the aquired lock
}
Output: {
cm_return - return code >0 if error
-1 if entry exists
cm_path - path to created data entry
cm_uid - uid (from UOA)
cm_alias - alias (from UOA)
cm_uoa - alias or uid if alias==''
}
Update repository
Input: {
}
Output: {
}
Input: {
}
Output: {
}
| def module::update_default_kernel_data | ( | i | ) |
| def module::update_input_from_state_ll | ( | ii, | ||
| state, | ||||
| p_uid, | ||||
execute = False | ||||
| ) |
| def module::view_ctuning_space | ( | i | ) |
| def module::view_data | ( | i | ) |
View data
Input: {
cm_module_uoa - module UOA
cm_data_uoa - data UOA
(cm_repo_uoa) - repo UOA
(cm_console) - by default, do not output anything except errors
if 'txt' then output as plain text
if 'web' then output for web (html, image, etc)
if 'json' then output as json
(cm_back) - for back button
}
Output: {
cm_return - return code >0 if error
-1 if entry exists
cm_path - path to data entry
cm_uid - uid (from UOA)
cm_alias - alias (from UOA)
cm_data_obj - cM data obj (cm_data_obj['cfg'] - data entry json)
}
| def module::view_data_through_web | ( | i | ) |
View data through web
Input: {
browse_module_uoa - module UOA
browse_data_uoa - data UOA
(browse_repo_uoa) - repo UOA
(cm_back_json) - for back button
}
Output: {
cm_return - return code >0 if error
-1 if entry exists
cm_path - path to created data entry
cm_uid - uid (from UOA)
cm_alias - alias (from UOA)
}
| def module::view_stats | ( | i | ) |
| def module::visualize | ( | i | ) |
| def module::visualize_connections | ( | i | ) |
Detect parameters in web forms
Input: {
cm_show_only_graph - if 'yes', show only graph and skip all the rest
cm_width - graph width
cm_height - graph height
}
Output: {
cm_return - return code >0 if error
cm_array - parsed array with keys
cm_actions - parsed array with commands
}
| def module::visualize_data | ( | i | ) |
Visualize data entry (with recursion)
Input: {
cm_array - array to visualize
(cm_array_orig) - original array (not changing through the calls)
(cm_data_desc) - data description for visualization (always first level)
(cm_desc_state) - previous description in hierarchies
(cm_form_commands) - form commands
(cm_form_explore) - form explore parameters
(cm_form_explore_color) - prefix for style
cm_separator - appendable separator to deal with array hierarchy
(full - can reproduce hierarchy)
cm_separator_form - appendable separator to deal with array hierarchy
(full and with additional forms to deal with commands)
(cm_separator_desc) - appendable separator to deal with array hierarchy in descriptions
(reduced - for example list number is skipped)
(cm_mode) - 'add' or 'update'
(cm_back_json) - data to create back button
(show_param_names) - if =='on' show parameter names
(cm_forms_exists) - if 'yes', processing through forms (to avoid reinit of default vars)
(cm_support_raw_edit) - if 'yes', add check button to allow raw edit
(cm_raw_edit) - if 'on', raw edit
(cm_explore) - if 'yes', prepare exploration choices
(cm_explore_commands) - array with exploration commands
(cm_explore_state) - {} array with choices and exploration state
(cm_explore_level) - exploration level ('' - coarse-grain, '1' - next level, ...)
(hide_add_new_object) - if 'yes', do not show 'add new object'
(ignore_desc_expand) - if 'yes', ignore data description expansion from other sources
(otherwise can be very slow for viewing)
}
Output: {
cm_return - return code >0 if error
cm_string - html string
cm_array - updated array to visualize
cm_explore_state - updated explore state
cm_data_desc - updated data description
}
| def module::visualize_selection | ( | i | ) |
Prepare graph data and call graph engine
Input: {
ct_module_uoa - module to visualize
ct_data_selection - list of ctuning.space data
ct_axes - axes
ct_graph_type - graph engine
ct_axes_desc - axes description
(url) - starting url of cM
(save_graph) - if 'yes', save graph
(cm_file_ext) - file extension if save_graph=='yes'
(reproduce_save_graph) - data to reproduce graph for saving
(use_multiple_graphs) - if 'on' separate entries into different sub-graphs
(convert_text_axes_to_float) - convert text axes to consequitive floats
(mpl_extra_params) - if python mat plot is used, this dictionary has extra params
mpl_image_size_x - image X size in inch
mpl_image_size_y - image Y size in inch
mpl_image_dpi - image DPI
mpl_detach_graph - if 'on', detach graph
(data_entry_to_sort) - if sort and multiple entries, use this entry number to sort (-1)
(fix_additional_dimensions) - if 'yes', fix or range additional dimensions
(fix_additional_dimensions_data) - dictionary of format:
{"flattened key A":{'less':X, 'equal':Y, 'more':Z}, ...}
}
Output: {
cm_return - return code >0 if error
data to save graph, if save_graph=='yes'
}
| def module::web_build | ( | i | ) |
| def module::web_configure | ( | i | ) |
| def module::web_copy | ( | i | ) |
| def module::web_detect | ( | i | ) |
| def module::web_explore | ( | i | ) |
| def module::web_install | ( | i | ) |
Install cM package/library
When searching for already installed code, dependencies are checked
* using classes for packages (to allow reuse of packages)
* by exact code_uoa for code.source (to have exact build setup)
FGG: the checks are a bit ugly and were done in a rush.
Should be simplified, cleaned up and unified one day.
Input: {
package_data_uoa - UOA of package to install
(package_repo_uoa) - repo of the package
package_host_os_uoa - UOA of host OS (to set up script execution)
(install_data_uid) - UID of the code where package will be installed
if install_data_uid=='' it will be randomly generated
(install_data_alias) - use this alias for a generated entry
If package should be built:
build_target_os_uoa - target OS uoa for building
(code_deps) - list with code UOA for dependencies [{"index":"uoa"} ...]
(run_target_processor_uoa) - target processor UOA (not strictly needed - can add some helper parameters before executing code)
(run_set_env2) - array with environment variables to be set before executable
(add_rem_to_script) - add rem to install script
(skip_extract_and_build) - if 'yes', skip copy, extract and build (by setting CM_SKIP_BUILD=yes)
}
Output: {
cm_return - return code = 0, if successful
Output from 'code install'
}
| def module::web_run | ( | i | ) |
| def module::web_view | ( | i | ) |
View aggregated experimental results
Input: {
TBD
(web_title) - web title to substitute default
(calculate_program_similarities) - if 'yes', calculate distance from programs based on properties
(program_features) - use it to calcluate distance
(pruning_similar_programs) - direct pruning parameters
(number_of_returned_items) - number of returned items
}
Output: {
cm_return - if =0, success
(similar_programs) - list of similar programs (if not web)
(similar_programs_points) - generated point per similar program (if not web)
(similar_programs_points_improvements) - generated improvements for points per similar program (if not web)
}
| def module::web_view_data | ( | i | ) |
Prepare header
Input: {
cm_data - data object from pipeline
header_desc - header description
cur_line_style - {
cur - cur (style1 or style2)
style1
style2
}
point - point to visualize
}
Output: {
cm_return - if =0, success
cm_html - header html
}
Prepare header
Input: {
cm_data - data object from pipeline
header_desc - header description
cur_line_style - {
cur - cur (style1 or style2)
style1
style2
}
point - point to visualize
}
Output: {
cm_return - if =0, success
cm_html - header html
point - all values for the point
point_improvements - all available improvements for the point
}
| def module::web_view_header | ( | i | ) |
| def module::welcome | ( | i | ) |
| string module::cm_action_index = 'index' |
| string module::cm_action_show = 'show' |
| string module::cm_data_uoa_default = 'default' |
| module::cm_kernel = None |
| string module::cm_menu_login = 'login' |
| string module::cm_menu_logout = 'logout' |
| tuple module::diter = int(space.get('number_of_explored_points','0')) |
| string module::form6 = '#form6' |
| module::ii = {} |
Definition at line 818 of file module.py.
Referenced by org::openme::openme::remote_access().
| tuple module::ii_copy = copy.deepcopy(ii) |
| dictionary module::ini = {} |
| def module::init = state[pu['init']] |
| list module::ip = state[pu['init_params']] |
| dictionary module::jj |
| tuple module::jj_copy = copy.deepcopy(jj) |
| list module::lock_uid = '' |
| list module::pd = state[pu['prepare_dataset']] |
Definition at line 781 of file module.py.
Referenced by org::openme::openme::access(), cm_action(), org::openme::openme::convert_cm_array_to_uri(), org::openme::openme::convert_str_to_sha1(), org::openme::openme::openme_load_json_file(), org::openme::openme::openme_store_json_file(), and org::openme::openme::remote_access().
| module::recorded = True |
| list module::rp = state[pu['run_program']] |
| dictionary module::rr = {'cm_return':0} |
| list module::su = cm_kernel.ini['dcfg'] |
| string module::view_and_sort = 'cm_display_html' |