CM python package API

Submodules

cmind.artifact module

class cmind.artifact.Artifact(cmind, path)

Bases: object

CM artifact class

Initialize CM artifact class

Parameters:

• cmind (CM class)

• path (str) – path to a CM artifact

Returns:

(python class) with the following vars –

• path (str): path to CM artifact

• original_meta (dict): original meta description of this artifact without inheritance

• meta (dict): meta description of this artifact after inheritance

load(ignore_inheritance=False, base_recursion=0)

Load CM artifact

Parameters:

• ignore_inheritance (bool) – if True ignore artifact meta description inheritance

• base_recursion (int) – internal to avoid infinite recursion during inheritance

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

update(meta, append_lists=True, replace=False, tags=[])

Update CM artifact

Parameters:

• meta (dict) – new meta description

• replace (bool) – if True, replace original meta description instead of merging

• append_lists (bool) – if True and replace is False, append lists when merging meta descriptions instead of substituting

• tags (list) – replace tags in meta

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

cmind.automation module

class cmind.automation.Automation(cmind, automation_file)

Bases: object

CM automation class

Initialize CM automation class

Parameters:

• cmind (CM class)

• automation_file (str) – path to the CM automation Python module

Returns:

(python class) with the following vars –

• automation_file_path (str): full path to the CM automation Python module

• path (str): directory with the CM automation

add(i)

Add CM artifact

Parameters:

• (CM input dict)

• (out) (str) – if ‘con’, output to console

• parsed_automation (list) – prepared in CM CLI or CM access function [ (automation alias, automation UID) ] or [ (automation alias, automation UID), (automation repo alias, automation repo UID) ]

• parsed_artifact (list) – prepared in CM CLI or CM access function [ (artifact alias, artifact UID) ] or [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]

• meta (dict) – meta description of this artifact

• (tags) (string or list) – tags to be added to meta

• (new_tags) (string or list) – new tags to be added to meta (the same as tags)

• (yaml) (bool) – if True, record YAML instead of JSON

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• meta (dict): final meta of the artifact

• path (str): path to the created artifact

copy(i)

Copy CM artifact(s) either to a new artifact or to a new repository

Parameters:

• (CM input dict)

• (out) (str) – if ‘con’, output to console

• parsed_automation (list) – prepared in CM CLI or CM access function [ (automation alias, automation UID) ] or [ (automation alias, automation UID), (automation repo alias, automation repo UID) ]

• parsed_artifact (list) – prepared in CM CLI or CM access function [ (artifact alias, artifact UID) ] or [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]

• parsed_artifacts (list) – prepared in CM CLI or CM access function - 1st entry has a new artifact name and repository: [ (artifact alias, artifact UID) ] or [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]

• new_tags (string) – add new tags (separated by comma) to new artifacts

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• list (list): list of copied CM artifacts

delete(i)

Delete CM artifact

Parameters:

• (CM input dict)

• (out) (str) – if ‘con’, output to console

• parsed_automation (list) – prepared in CM CLI or CM access function [ (automation alias, automation UID) ] or [ (automation alias, automation UID), (automation repo alias, automation repo UID) ]

• parsed_artifact (list) – prepared in CM CLI or CM access function [ (artifact alias, artifact UID) ] or [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]

• (tags) (string or list) – optional tags to find artifacts

• (force) (bool) – force deleting CM artifacts without asking a user

• (f) (bool) – the same as “force”

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• deleted_list (list): a list of deleted CM artifacts

info(i)

Get info about artifacts

Parameters:

• (CM input dict)

• (out) (str) – if ‘con’, output to console

• parsed_automation (list) – prepared in CM CLI or CM access function [ (automation alias, automation UID) ] or [ (automation alias, automation UID), (automation repo alias, automation repo UID) ]

• parsed_artifact (list) – prepared in CM CLI or CM access function [ (artifact alias, artifact UID) ] or [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]

• parsed_artifacts (list) – prepared in CM CLI or CM access function - 1st entry has a new artifact name and repository: [ (artifact alias, artifact UID) ] or [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]

• (uid) (bool) – if True, copy CID in UID::UID format to clipboard

• (md) (bool) – if True, copy CID in [](UID::UID) format to clipboard

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• list (list): list of CM artifacts

load(i)

Load CM artifact

Parameters:

• (CM input dict)

• (out) (str) – if ‘con’, output to console

• parsed_automation (list) – prepared in CM CLI or CM access function [ (automation alias, automation UID) ] or [ (automation alias, automation UID), (automation repo alias, automation repo UID) ]

• parsed_artifact (list) – prepared in CM CLI or CM access function [ (artifact alias, artifact UID) ] or [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]

• (tags) (string or list) – optional tags to find artifacts

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• path (str): path to a loaded CM artifact

• original_meta (dict): meta description of the loaded CM artifact before inheritance

• meta (dict): meta description of the loaded CM artifact after inheritance

• artifact (CM artifact object)

move(i)

Rename CM artifacts and/or move them to another CM repository.

Parameters:

• (CM input dict)

• (out) (str) – if ‘con’, output to console

• parsed_automation (list) – prepared in CM CLI or CM access function [ (automation alias, automation UID) ] or [ (automation alias, automation UID), (automation repo alias, automation repo UID) ]

• parsed_artifact (list) – prepared in CM CLI or CM access function [ (artifact alias, artifact UID) ] or [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]

• parsed_artifacts (list) – prepared in CM CLI or CM access function - 1st entry has a new artifact name and repository: [ (artifact alias, artifact UID) ] or [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• list (list): list of renamed/moved CM artifacts

search(i)

Find CM artifacts (slow - we plan to accelerate it in the future using different indexing mechanisms)

Parameters:

• (CM input dict)

• (out) (str) – if ‘con’, output to console

• parsed_automation (list) – prepared in CM CLI or CM access function [ (automation alias, automation UID) ] or [ (automation alias, automation UID), (automation repo alias, automation repo UID) ]

• parsed_artifact (list) – prepared in CM CLI or CM access function [ (artifact alias, artifact UID) ] or [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]

• ignore_inheritance (bool) – if ‘True’, ignore artifact meta description inheritance and just load meta to match UID/alias

• (skip_index_search) (bool) – If True, skip indexing

• (force_index_add) (bool) – If True, force index add (for reindexing)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• list (list): list of CM artifact objects

test(i)

Test CM and print various info

Parameters:

• (CM input dict)

• (out) (str) – if ‘con’, output to console

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

update(i)

Update CM artifact.

Note: not thread safe - we expect one pipeline running on a system

can make it thread safe when needed (similar to CK)

Parameters:

• (CM input dict)

• (out) (str) – if ‘con’, output to console

• parsed_automation (list) – prepared in CM CLI or CM access function [ (automation alias, automation UID) ] or [ (automation alias, automation UID), (automation repo alias, automation repo UID) ]

• parsed_artifact (list) – prepared in CM CLI or CM access function [ (artifact alias, artifact UID) ] or [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]

• (search_tags) (string or list) – optional tags to find artifacts

• (meta) (dict) – new meta description to be merged or to replace the original meta description of a CM artifact

• (tags) (string or list) – tags to be added to meta

• (replace) (bool) – if True, replace original meta description (False by default)

• (force) (bool) – if True, force updates if more than 1 artifact found (False by default)

• (replace_lists) (bool) – if True, replace lists in meta description rather than appending them (False by default)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• list (list): list of updated CM artifacts

version(i)

Print CM version

Parameters:

• (CM input dict)

• (out) (str) – if ‘con’, output to console

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

cmind.cli module

cmind.cli.docker_script(argv=None)

Shortcut to “cm docker script …”

CM command line format:

Parameters:

argv (list | string) – command line arguments

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• Output from a CM automation action

cmind.cli.gui_script(argv=None)

Shortcut to “cm gui script …”

CM command line format:

Parameters:

argv (list | string) – command line arguments

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• Output from a CM automation action

cmind.cli.parse(cmd)

Parse CM command line into CM input dictionary.

Parameters:

cmd (str | list) – arguments as a string or list

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• cm_input (dict): CM unified input to the CM access function

cmind.cli.parsex(cmd)

Parse CM command line into CM input dictionary.

Parameters:

cmd (str | list) – arguments as a string or list

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• cm_input (dict): CM unified input to the CM access function

cmind.cli.run(argv=None)

Run CM automation actions from the command line.

CM command line format:

cm {action} {automation} (artifacts) (–flags) (@input.yaml) (@input.json)

Parameters:

argv (list | string) – command line arguments

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• Output from a CM automation action

cmind.cli.run_experiment(argv=None)

Shortcut to “cm run experiment …”

CM command line format:

Parameters:

argv (list | string) – command line arguments

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• Output from a CM automation action

cmind.cli.run_script(argv=None)

Shortcut to “cm run script …”

CM command line format:

Parameters:

argv (list | string) – command line arguments

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• Output from a CM automation action

cmind.cli.runx(argv=None)

Run CM automation actions from the new command line.

CM command line format:

cm {action} {automation} (artifacts) (-params) (–inputs) (@input.yaml) (@input.json)

Parameters:

argv (list | string) – command line arguments

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• Output from a CM automation action

cmind.cli.split_flag(flag, array, value=None)

Split flag key=value and add to array

Parameters:

• flag (str) – flag of format “key=value”

• array (dict) – array where to accumulate flags

• value (str) – if not None use it (for “-key value” flags)

Returns:

key (str) – key value (str): value array (dict): updated array

cmind.config module

class cmind.config.Config(config_file=None)

Bases: object

CM configuration class

Initialize CM configuration class

Parameters:

(config_file) (str) – If not None, merge this file with the internal configuration

Returns:

(python class) with the following vars –

• cfg (dict): internal CM configuration

cmind.core module

class cmind.core.CM(repos_path='', debug=False)

Bases: object

Main CM class

Initialize CM configuration class

Parameters:

• (repos_path) (str)

• (debug) (bool) – instead of returning a dictionary with “return”>0.

Returns:

(python class) with the following vars –

• cfg (dict): internal CM configuration

• debug (bool): debug state

• repos_path (str): path to CM repositories and other information

• path_to_cmind (str): path to the used CM toolkit

• path_to_cmind_core_module (str): path to CM core module

• path_to_cmind_repo (str): path to the “internal” CM repo

• repos (list): list of initialized CM repository objects

• common_automation (Automation class): initialized common automation with actions when a given automation is not found or doesn’t exist

• output (str): value of –out during the first access to CM (to print errors, etc)

access(i, out=None)

Access CM automation actions in a unified way similar to micro-services. (Legacy. Further development in the new “x” function).

i (dict | str | argv): unified CM input

Parameters:

• (action) (str) – automation action

• (automation (CM object) – CM automation in format (alias | UID | alias,UID) or (repo alias | repo UID | repo alias,UID):(alias | UID | alias,UID)

• (artifact) (CM object) – CM artifact

• (artifacts) (list of CM objects) – extra CM artifacts

• (common) (bool) – if True, use common automation action from Automation class

• (help) (bool) – if True, print CM automation action API

• (ignore_inheritance) (bool) – if True, ignore inheritance when searching for artifacts and automations

• (out) (str) – if ‘con’, tell automations and CM to output extra information to console

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• Output from a CM automation action

embed_error(r)

Embed module and line number to an error

Parameters:

r (dict) – CM return dict

Returns:

(dict) –

r

return (int) error (str) module_path (str): path to module lineno (int): line number

error(r)

If r[‘return’]>0: print CM error and raise error if in debugging mode

Parameters:

r (dict) – output from CM function with “return” and “error”

Returns:

(dict) – r

errorx(r)

If r[‘return’]>0: print CM error and raise error if in debugging mode

Parameters:

r (dict) – output from CM function with “return” and “error”

Returns:

(dict) – r

halt(r)

If r[‘return’]>0: print CM error and raise error if in debugging mode or halt with “return” code

Parameters:

r (dict) – output from CM function with “return” and “error”

Returns:

(dict) – r

log(s, t='info')

Parameters:

• s (str) – log string

• t (str) – log type - “info” (default) “debug” “warning” “error”

Returns:

None

prepare_error(returncode, error)

Prepare error dictionary with the module and line number of an error

Parameters:

• returncode (int) – CMX returncode

• error (str) – error message

Returns:

(dict) –

r

return (int) error (str) module_path (str): path to module lineno (int): line number

x(i, out=None)

New unified access to CM automation actions

Parameters:

i (dict | str | argv) – unified CM input

• (action) (str): automation action

• (automation (CM object): CM automation in format (alias | UID | alias,UID)

  or (repo alias | repo UID | repo alias,UID):(alias | UID | alias,UID)

• (artifact) (CM object): CM artifact

• (artifacts) (list of CM objects): extra CM artifacts

Control flags starting with - :

• (out) (str): if ‘con’, tell automations and CM to output extra information to console

• (common) (bool): if True, use common automation action from Automation class

• (help) (bool): if True, print CM automation action API

• (ignore_inheritance) (bool): if True, ignore inheritance when searching for artifacts and automations

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• Output from a given CM automation action

cmind.core.access(i)

Automatically initialize CM and run automations without the need to initialize and customize CM class. Useful for Python automation scripts.

See cmind.CM.access function for more details.

cmind.core.debug(path_to_cm_automation_module)

cmind.core.error(i)

Automatically initialize CM and print error if needed without the need to initialize and customize CM class. Useful for Python automation scripts.

See CM.error function for more details.

cmind.core.errorx(i)

Automatically initialize CM and print error if needed without the need to initialize and customize CM class. Useful for Python automation scripts.

See CM.error function for more details.

cmind.core.get_version()

Get CM version.

Args:

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• version (str): CM version

cmind.core.halt(i)

Automatically initialize CM, print error and halt if needed without the need to initialize and customize CM class. Useful for Python automation scripts.

See CM.halt function for more details.

cmind.core.parse_cm_object_and_check_current_dir(cmind, artifact)

Internal function: parses CM object and check if there is ‘.’ to detect CM repository in a current directory.

cmind.core.print_action_help(automation, common_automation, print_automation, action, original_action)

cmind.core.print_db_actions(automation, equivalent_actions, automation_name, cmx=False)

Internal function: prints CM database actions.

cmind.core.x(i, out=None)

Automatically initialize CM and run automations without the need to initialize and customize CM class. Useful for Python automation scripts.

See cmind.CM.x function for more details.

cmind.index module

class cmind.index.Index(path, cfg, meta={})

Bases: object

CM index class

Initialize CM index class

Parameters:

• path (str) – path to directory with repos and index file

• cfg (dict) – CM configuration

• (meta) (dict) – CM index meta

Returns:

(python class) with the following vars –

• path (str): path to CM repositories and index file

• meta (dict): CM index meta

add(meta, path, repo, update=False, delete=False)

Add to CM index

Parameters:

• meta (dict) – artifact meta

• path (str) – artifact path

• repo (tuple of str) – (alias, uid) of repo

• update (bool) – force update

• delete (bool) – delete instead of adding

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

find(automation_name, artifact_obj, artifact_repo, pruned_repos, and_tags, no_tags)

Find in CM index

Parameters:

• automation_name (tuple of str) – (alias, uid) of automation

• artifact_obj (tuple of str) – (alias, uid) of artifact obj

• artifact_repo (tuple of str of None) – if not None, use pruned_repos

• pruned_repos (list of Repo classes) – pruned repos

• and_tags (list of str) – AND tags

• no_tags (list of str) – NO tags

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

load()

Load or initialize CM index

Parameters:

None

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

save()

Save CM index TBD: it’s not thread safe and we should provide better handling if CM is used in parallel

we need to reload

Parameters:

None

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

cmind.net module

cmind.net.request(i)

Web request to cKnowledge.org server to get release notes about components (for CK compatibility)

Parameters:

CM input passed to cKnowledge.org API

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• version (str): CM version

• string (str): returned string from the server

• dict (dict): JSON string converted to dict (if possible)

cmind.repo module

class cmind.repo.Repo(path, cfg)

Bases: object

CM repository class

Initialize CM repository class

Parameters:

• path (str) – path to a CM repository

• cfg (dict) – CM configuration

Returns:

(python class) with the following vars –

• path (str): path to a CM repository

• path_prefix (str): use extra directory inside CM repository to keep CM artifacts

• path_with_prefix (str): path to a CM repository if directory prefix is used

• meta (dict): CM repository meta description

load(cmx=False)

Load CM repository

Parameters:

None

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

cmind.repos module

class cmind.repos.Repos(path, cfg, path_to_internal_repo='', cmx=False)

Bases: object

CM repositories class

Initialize CM repositories class

Parameters:

• path (str) – path to directory with repos.json where all paths of registered CM repositories are stored

• cfg (dict) – CM configuration

• (path_to_internal_repo) (str) – path to the internal CM repository (useful during development/debugging)

Returns:

(python class) with the following vars –

• path (str): path to CM repositories and index file

• full_path_to_repos (str): path to CM repositories

• path_to_internal_repo (str): path to the internal CM repository

• full_path_to_repo_paths (str): full path to repos.json

• lst (list): list of initialized CM repository classes

delete(lst, remove_all=False, console=False, force=False)

Delete CM repository or repositories with or without content

Parameters:

• lst (list of CM repository classes) – list of CM repositories

• remove_all (bool) – if True, remove the content and unregister CM repository

• console (bool) – if True, output some info to console

• force (bool) – if True, do not ask questions

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

init(alias, uid, path='', console=False, desc='', prefix='', only_register=False)

Init CM repository in a given path

Parameters:

• alias (str) – CM repository alias

• uid (str) – CM repository UID

• (path) (str) – local path to a given repository (otherwise use $HOME/CM/repos)

• (desc) (str) – optional repository description

• (prefix) (str) – sub-directory to be used inside this CM repository to store artifacts

• (only_register) (bool) – if True, only register path in the CM index of repositories but do not recreate cmr.yaml

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• meta (dict): meta of the CM repository

• path_to_repo (str): path to repository

• path_to_repo_desc (str): path to repository description

• path_to_repo_with_prefix (str): path to repository with prefix (== path_to_repo if prefix == “”)

• (warnings) (list of str): warnings to install more CM repositories

load(init=False)

Load or initialize repos.json file with repositories

Parameters:

init (bool) – if False do not init individual CM repositories

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

process(repo_path, mode='add')

Add or delete CM repository

Parameters:

• repo_path (str) – path to CM repository

• (mode) (str) – “add” (default) or “delete”

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• (warnings) (list of str): warnings to install more CM repositories

pull(alias, url='', branch='', checkout='', console=False, desc='', prefix='', depth=None, path_to_repo=None, checkout_only=False, skip_zip_parent_dir=False, extra_cmd_git='', extra_cmd_pip='', new_branch='')

Clone or pull CM repository

Parameters:

• alias (str) – CM repository alias

• (url) (str) – Git repository URL

• (branch) (str) – Git repository branch

• (new_branch) (str) – Create new branch

• (checkout) (str) – Git repository checkout

• (checkout_only) (bool) – only checkout existing repo

• (depth) (int) – Git repository depth

• (console) (bool) – if True, print some info to console

• (desc) (str) – optional repository description

• (prefix) (str) – sub-directory to be used inside this CM repository to store artifacts

• (path_to_repo) (str) – force path to repo (useful to pull imported repos with non-standard path)

• (checkout_only) (bool) – only checkout Git repository but don’t pull

• (skip_zip_parent_dir) (bool) – skip parent dir in CM ZIP repo (useful when downloading CM repo archives from GitHub)

• (extra_cmd_git) (str) – add this string to git clone

• (extra_cmd_pip) (str) – add this string to pip install when installing requirements from CM repositories

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• (meta) (dict): meta of the CM repository

• (warnings) (list of str): warnings to install more CM repositories

cmind.utils module

cmind.utils.assemble_cm_object(alias, uid)

Assemble CM object string from alias and uid strings

Parameters:

• alias (str) – CM artifact alias

• uid (str) – CM artifact UID

Returns:

(str) CM object (alias,uid)

cmind.utils.assemble_cm_object1(cm_dict)

Assemble CM object string from dictionary with ‘alias’ and ‘uid’ keys

Parameters:

cm_dict (dict) – dictionary with CM artifact keys (‘alias’, ‘uid’)

Returns:

(str) CM object (alias,uid)

cmind.utils.assemble_cm_object2(cm_obj)

Assemble CM object string from tuple

Parameters:

cm_obj (tuple) – CM object tuple (alias, uid)

Returns:

(str) CM object (alias,uid)

cmind.utils.call_internal_module(module_self, path_to_current_module, module_name, module_func, i)

Call CM function from internal submodule

Parameters:

• path_to_current_module (obj) – must be __file__

• module_name (str) – module name

• module_func (str) – module function

• i (dict) – CM input. Note that i[‘self_module’] = self from calling module will be added to the input

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

cmind.utils.check_and_create_dir(path)

Create directories if path doesn’t exist. (from the CK framework).

Parameters:

path (str) – path

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

cmind.utils.check_if_none_false_no_off(env, key)

Universal check if str(env.get(key, ‘’)).lower() in [‘false’, ‘no’, ‘off’]:

Parameters:

• env (dict) – dictionary

• key (str) – key

Returns:

True if str(env.get(key, ‘’)).lower() in [‘false’, ‘no’, ‘off’]

cmind.utils.check_if_true_yes_on(env, key)

Universal check if str(env.get(key, ‘’)).lower() in [‘true’, ‘yes’, ‘on’]:

Parameters:

• env (dict) – dictionary

• key (str) – key

Returns:

True if str(env.get(key, ‘’)).lower() in [‘true’, ‘yes’, ‘on’]

cmind.utils.compare_versions(version1, version2)

Compare versions

Parameters:

• version1 (str) – version 1

• version2 (str) – version 2

Returns:

comparison (int) –

1 - version 1 > version 2

0 - version 1 == version 2

-1 - version 1 < version 2

cmind.utils.convert_dictionary(d, key, sub=True)

Grigori added to gradually clean up very complex “cm docker scrpit” implementation

Convert dictionary into flat dictionary with key prefix

Example input:

d = {‘cfg’:True, ‘no-cache’:True} key = ‘docker’ sub = True

Example output:

d = {‘docker_cfg’: True, ‘docker_no_cache’: True}

Parameters:

• d (dict) – dictionary

• key (str) – key

Returns:

True if str(env.get(key, ‘’)).lower() in [‘false’, ‘no’, ‘off’]

cmind.utils.convert_env_to_dict(s)

Create sub-input from the input using list of keys

Parameters:

s (str) – string with env

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

(dict): dictionary

cmind.utils.convert_tags_to_list(i)

Convert string tags to list

Parameters:

• (CM input dict)

• (tags) (str)

Returns:

tags_list (list) – list of tags

cmind.utils.copy_to_clipboard(i)

Copy string to a clipboard

Parameters:

• string (str) – string to copy to a clipboard

• (add_quotes) (bool) – add quotes to the string in a clipboard

• (skip_fail) (bool) – if True, do not fail

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

cmind.utils.debug_here(module_path, host='localhost', port=5678, text='', env={}, env_debug_uid='')

cmind.utils.decompose_cm_obj(cm_obj)

Decompose CM object into dictionary

Parameters:

cm_obj (CM object)

Returns:

(dict)

artifact (str) name (str) name_alias (str) name_uid (str) repo (str) repo_alias (str) repo_uid (str)

cmind.utils.digits(s, first=True)

Get first digits and convert to int

Parameters:

• s (str) – string (“1.3+xyz”)

• first (bool) – if True, choose only first digits, otherwise all

Returns:

(int) – returns int from first digits or 0

cmind.utils.dump_safe_json(i)

Dump safe JSON

Parameters:

(CM input dict) – input to dump to console

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• meta (dict): only “safe” values left (that can be serialized)

cmind.utils.filter_tags(tags)

Filter tags (remove starting with -)

Parameters:

tags (list) – list of tags

Returns:

(list) list of filtered tags

cmind.utils.find_api(file_name, func)

Find automation action API in a Python module

Parameters:

• file_name (str) – Python module

• func (str) – automation action name

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• api (str): API

cmind.utils.find_file_in_current_directory_or_above(file_names, path_to_start=None, reverse=False, path_to_stop=None)

Find file(s) in the current directory or above.

Parameters:

• file_names (list) – files to find

• (path_to_start) (str) – path to start; use current directory if None

• (reverse) (bool) – if True search recursively in current directory and below.

• (path_to_stop) (str) – path to stop search (usually path to found repo)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• found (bool): True if found

• path_to_file (str): full path to found file

• path (str): path where found file is

cmind.utils.find_file_in_dir_and_above(filename, path='')

Find file in the current directory or above

Parameters:

• filename (str)

• path (str)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• path (str): path where file is found

• path_to_file (str): path to file

cmind.utils.flatten_dict(d, fd={}, prefix='')

Flatten dict ({“x”:{“y”:”z”}} -> x.y=z)

cmind.utils.gen_tmp_file(i)

Generate temporary files

Parameters:

(CM input dict) – (suffix) (str): temp file suffix (prefix) (str): temp file prefix (remove_dir) (bool): if True, remove dir

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• file_name (str): temp file name

cmind.utils.gen_uid()

Generate CM UID.

Parameters:

None

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• uid (str): CM UID

cmind.utils.get_current_date_time(i)

Get current date and time.

Parameters:

(CM input dict) –

• (timezone) (str): timezone in pytz format: “Europe/Paris”

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• array (dict); dict with date and time

  • date_year (str)

  • date_month (str)

  • date_day (str)

  • time_hour (str)

  • time_minute (str)

  • time_second (str)

• iso_datetime (str): date and time in ISO format

cmind.utils.get_list_from_cli(i, key)

Get list from CM command line.

Parameters:

• i (dict) – CM input dict

• key (str) – key to get

Returns:

tags (list) – list of tags for a given key

cmind.utils.get_set(meta, key, state, keys)

Get value from dict and update in another dict

Parameters:

• meta (dict) – original dict

• key (str) – key to get value from original dict

• state (dict) – target dict

• keys (list) – list of keys to set value in target dict

Returns:

(Python object) – value from original dict or None

cmind.utils.get_value_from_dict_with_flat_key(key, d)

Get value from dict via flat key (x.y.z)

cmind.utils.is_cm_uid(obj)

Check if a string is a valid CM UID.

Parameters:

obj (str) – CM alias or UID

Returns:

(bool) – True if valid CM UID 16 hex characters

cmind.utils.is_file_json_or_yaml(file_name)

Is file JSON or YAML?

Parameters:

• (CM input dict)

• file_name (str) – path to file without extension

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• is_file (bool): if True, there is a file with YAML and/or JSON extension

cmind.utils.list_all_files(i)

List all files recursively in a given directory. (from the CK framework).

Parameters:

• (CM input dict)

• path (str) – top level path

• (file_name) (str) – search for a specific file name

• (pattern) (str) – return only files with this pattern

• (path_ext) (str) – path extension (needed for recursion)

• (limit) (str) – limit number of files (if directories with a large number of files)

• (number) (int) – current number of files

• (all) (str) – if ‘yes’ do not ignore special directories (like .cm)

• (ignore_names) (list) – list of names to ignore

• (ignore_symb_dirs) (str) – if ‘yes’, ignore symbolically linked dirs (to avoid recursion such as in LLVM)

• (add_path) (str)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• list (dict): dictionary of all files:

  {“file_with_full_path”:{“size”:.., “path”:..}

• number (int): (internal) total number of files in a current directory (needed for recursion)

cmind.utils.load_bin(file_name)

Load binary file.

Parameters:

• (CM input dict)

• file_name (str) – file name

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• bin (bytes): file content

cmind.utils.load_json(file_name, check_if_exists=False, encoding='utf8')

Load JSON file.

Parameters:

• (CM input dict)

• file_name (str) – file name

• (check_if_exists) (bool) – if True, fail if doesn’t exist

• (encoding) (str) – file encoding (‘utf8’ by default)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• meta (dict): meta from the file

cmind.utils.load_json_or_yaml(file_name, check_if_exists=False, encoding='utf8')

Attempt to load file as JSON or YAML.

Parameters:

• (CM input dict)

• file_name (str) – file name that has either JSON or YAML extension

• (check_if_exists) (bool) – if True, fail if doesn’t exist

• (encoding) (str) – file encoding (‘utf8’ by default)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• meta (dict): merged meta of found file(s)

cmind.utils.load_module(cmind, task_path, sub_module_name)

Universal python module loaders

cmind.utils.load_python_module(i)

Load python module

Parameters:

(CM input dict) – path (str): path to a python module name (str): Python module name (without .py)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

cmind.utils.load_txt(file_name, encoding='utf8', remove_after_read=False, check_if_exists=False, split=False, match_text='', fail_if_no_match='', debug=False)

Load text file.

Parameters:

• (CM input dict)

• file_name (str) – file name

• (encoding) (str) – file encoding (‘utf8’ by default)

• (remove_after_read) (bool) – if True, remove file after read (False by default)

• (check_if_exists) (bool) – If True, check if file exists and return CM error instead of raising error (False by default)

• (split) (bool) – If True, split string into list (False by default)

• (match_text) (str) – Regular expression to match text (useful for version detection)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• string (str): string from file

cmind.utils.load_yaml(file_name, check_if_exists=False, encoding='utf8')

Load YAML file.

Parameters:

• (CM input dict)

• file_name (str) – file name

• (check_if_exists) (bool) – if True, fail if doesn’t exist

• (encoding) (str) – file encoding (‘utf8’ by default)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• meta (dict): meta from the file

cmind.utils.load_yaml_and_json(file_name_without_ext, check_if_exists=False, encoding='utf8')

Load YAML file if exists then JSON file if exists and merge with the first one

Parameters:

• (CM input dict)

• file_name_without_ext (str) – file name without extension (to check yaml and then json)

• (check_if_exists) (bool) – if True, fail if doesn’t exist

• (encoding) (str) – file encoding (‘utf8’ by default)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• meta (dict): merged meta of found file(s)

cmind.utils.match_objects(uid, alias, uid2, alias2, more_strict=False)

Check if 2 CM objects match. Used to search CM artifacts in CM repositories.

Examples

281d5c3e3f69d8e7,* == 281d5c3e3f69d8e7,*

281d5c3e3f69d8e7,os == ,os

,os == 281d5c3e3f69d8e7,os

,* != 281d5c3e3f69d8e7,*

os

281d5c3e3f69d8e7

os,281d5c3e3f69d8e7

281d5c3e3f69d8e7,os

Parameters:

• uid (str) – artifact UID

  • alias (str) – atifact alias that can’t have wildcards [real CM object]

  • uid2 (str) – atifact 2 UID

  • alias2 (str)

(more_strict) (bool): if True, then ,os != 281d5c3e3f69d8e7, [needed to check automation]

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• match (bool): True if 2 CM objects match (uid and/or alias)

cmind.utils.merge_dicts(i)

Merge intelligently dict1 with dict2 key by key in contrast with dict1.update(dict2)

It can merge sub-dictionaries and lists instead of substituting them

Parameters:

• (CM input dict)

• dict1 (dict) – merge this dict with dict2 (will be directly modified!)

• dict2 (dict) – dict to be merged

• (append_lists) (bool) – if True, append lists instead of creating the new ones

• (append_unique) (bool) – if True, append lists when value doesn’t exists in the original one

• (ignore_keys) (list) – ignore keys

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• dict1 (dict): dict1 passed through the function

cmind.utils.parse_cm_object(obj, max_length=2, decompose=False)

Parse CM object in string and return tuple of CM objects.

Examples

CM sub-object = UID | alias | alias,UID | UID,alias

repo CM sub-object | CM sub-object

cm os

cm 281d5c3e3f69d8e7

cm os,281d5c3e3f69d8e7

cm 281d5c3e3f69d8e7,os

cm octoml@mlops,os

cm octoml@mlops,dbfa91645e429380:os,281d5c3e3f69d8e7

cm dbfa91645e429380:281d5c3e3f69d8e7

Parameters:

• obj (str) – CM object

• (max_length) (int)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• cm_object (list): first element: CM alias | UID

  (second element: CM repo | UID)

cmind.utils.path2(path)

Add quotes if spaces in path

cmind.utils.process_input(i, update_input=True)

Process automation, artifact and artifacts from i[‘control’]

Parameters:

• control[‘_unparsed_automation’]

• control[‘_unparsed_artifact’]

• control[‘_unparsed_artifacts’]

Returns:

TBD

cmind.utils.process_meta_for_inheritance(i)

Check CM meta description for inheritance and update it if needed (“_base”:”automation::artifact”).

Parameters:

• (CM input dict)

• automation (str) – CM automation

• meta (dict) – original meta

• cmind (obj) – initialized CM to search for base artifacts

• (base_recursion) (int) – track recursion during inheritance

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• meta (dict): final meta description wihhout _base

cmind.utils.rm_read_only(f, p, e)

Internal aux function to remove files and dirs even if read only particularly on Windows

cmind.utils.safe_float(i, d)

Get safe float (useful for sorting function)

Parameters:

• i (any) – variable with any type

• d (float) – default value

Returns:

(float) – returns i if it can be converted to float or d otherwise

cmind.utils.safe_int(i, d)

Get safe int (useful for sorting function)

Parameters:

• i (any) – variable with any type

• d (int) – default value

Returns:

(int) – returns i if it can be converted to int, or d otherwise

cmind.utils.safe_load_json(path, file_name='', encoding='utf8')

Load JSON file if exists, otherwise return empty dict

Parameters:

• (CM input dict)

• file_name (str) – file name

• (encoding) (str) – file encoding (‘utf8’ by default)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

• meta (dict): meta from the file

cmind.utils.save_json(file_name, meta={}, indent=2, sort_keys=True, encoding='utf8')

Save meta to JSON file.

Parameters:

• (CM input dict)

• file_name (str) – file name

• meta (dict) – meta to save

• (indent) (int) – 2 by default

• (sort_keys) (bool) – if True, sort keys

• (encoding) (str) – file encoding (‘utf8’ by default)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

cmind.utils.save_json_or_yaml(file_name, meta, sort_keys=False, encoding='utf8')

Save meta to either JSON or YAML file.

Parameters:

• (CM input dict)

• file_name (str) – file name that has either JSON or YAML extension

• meta (dict) – meta to save

• (sort_keys) (bool) – if True, sort keys

• (encoding) (str) – file encoding (‘utf8’ by default)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

cmind.utils.save_txt(file_name, string='', encoding='utf8')

Save string to text file.

Parameters:

• (CM input dict)

• file_name (str) – file name

• string (str) – string to save

• (encoding) (str) – file encoding (‘utf8’ by default)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

cmind.utils.save_yaml(file_name, meta={}, sort_keys=True, encoding='utf8')

Save meta to YAML file.

Parameters:

• (CM input dict)

• file_name (str) – file name

• meta (dict) – meta to save

• (sort_keys) (bool) – if True, sort keys

• (encoding) (str) – file encoding (‘utf8’ by default)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

cmind.utils.sub_input(i, keys, reverse=False)

Create sub-input from the input using list of keys

Parameters:

• i (dict) – input

• keys (list) – keys not to add if reverse if True

• reverse (bool) – either add or skip keys

Returns:

(dict) – sub-input

cmind.utils.tags_matched(tags, and_tags, no_tags)

Check if AND tags and NO tags match tags

Parameters:

• tags (list of str) – full list of tags

• and_tags (list of str) – list of AND tags

• no_tags (list of str) – list of NO tags

Returns:

True if tags matched

cmind.utils.test_input(i)

Test if input has keys and report them as error

cmind.utils.update_dict_if_empty(d, key, value)

Update dictionary if “key” is empty

Parameters:

• d (dict) – dict to check

• key (str)

• value – if d[key] is empty, set to this value

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

cmind.utils.update_dict_with_flat_key(key, value, d)

Update dictionary via flat key (x.y.z)

cmind.utils.update_yaml(file_name, meta={}, encoding='utf8')

Update yaml file directly (unsafe - only first keys)

Parameters:

(CM input dict) – file_name (str): YAML file name meta (dict): keys to update (encoding) (str): file encoding (‘utf8’ by default)

Returns:

(CM return dict) –

• return (int): return code == 0 if no error and >0 if error

• (error) (str): error string if return>0

Module contents