pylablib.core.utils package

Submodules

pylablib.core.utils.ctypes_wrap module

pylablib.core.utils.ctypes_wrap.setup_func(func, argtypes, restype=None, errcheck=None)

Setup a ctypes function.

Assign argtypes (list of argument types), restype (return value type) and errcheck (error checking function called for the return value).

class pylablib.core.utils.ctypes_wrap.CTypesWrapper(argtypes=None, argnames=None, addargs=None, restype=None, return_res='auto', rvprep=None, rvconv=None, rvref=None, rvnames=None, tuple_single_retval=False, errcheck=None)

Bases: object

Wrapper object for ctypes function.

Constructor arguments coincide with the call arguments, and determine their default values. For their meaning, see wrap().

setup(func, argtypes=None, restype=None, errcheck=None)

Setup a ctypes function.

Analogous to setup_func(), but uses wrapper’s default arguments.

wrap(func, argtypes=None, argnames=None, addargs=None, restype=None, return_res=None, rvprep=None, rvconv=None, rvref=None, rvnames=None, tuple_single_retval=None, errcheck=None)

Wrap C function in a Python call.

Parameters:

• func – C function
• argtypes (list) – list of func argument types; if an argument is of return-by-pointer kind, it should be the value type (the pointer is added automatically), with the exception of the void pointer (in which case, custom rvprep function should be supplied)
• argnames (list) – list of argument names of the function. Includes either strings (which are interpreted as argument names passed to the wrapper function), or None (which mean that this argument is return-by-pointer).
• addargs (list) – list of additional arguments which are added to the wrapper function, but are not passed to the wrapped function (added in the end); can be used for, e.g., rvprep or rvconv functions.
• restype – type of the return value. Can be None if the return value isn’t used.
• return_res (bool) – determines whether return the function return value. By default, it is returned only if there are no return-by-pointer arguments.
• rvprep (list) – list of functions which prepare return-by-pointer arguments before passing them to the function. By default, these are standard ctypes initializer for the corresponding types (usually equivalent to zero).
• rvconv (list) – list of functions which convert the return-by-pointer values after the function call. By default, simply get the corresponding Python value inside the ctypes wrapper.
• rvref ([bool]) – determines if the corresponding return-by-pointer arguments need to be wrapped into ctypes.byref() (most common case), or passed as is (e.g., for manually prepared buffers).
• rvnames ([str]) – names of returned values inside the returned named tuple. By default, return standard un-named tuple. If any name is None, omit that value from the resulting tuple.
• tuple_single_retval (bool) – determines if a single return values gets turned into a single-element tuple.
• errcheck – error-checking function which is automatically called for the return value; no function by default.

pylablib.core.utils.ctypes_wrap.strprep(l, ctype=None, unicode=False)

Make a string preparation function.

Return a function which creates a string with a fixed length of l bytes and returns a pointer to it. ctype can specify the type of the result (by default, ctypes.c_char_p).

pylablib.core.utils.ctypes_wrap.buffprep(size_arg_pos, dtype)

Make a buffer preparation function.

Return a function which creates a string with a variable size (specified by an argument at a position size_arg_pos). The buffer size is given in elements. dtype specifies the datatype of the buffer, whose size is used to determine buffer size in bytes.

pylablib.core.utils.ctypes_wrap.buffconv(size_arg_pos, dtype)

Make a buffer conversion function.

Return a function which converts a pointer of a variable size (specified by an argument at a position size_arg_pos) into a numpy array. The buffer size is given in elements. dtype specifies the datatype of the resulting array.

pylablib.core.utils.ctypes_wrap.nullprep(*args, **kwrags)

NULL preperation function which always returns None

class pylablib.core.utils.ctypes_wrap.StructWrap(struct=None)

Bases: object

to_struct()

prep(struct)

prepdict()

conv()

tupdict()

tup()

classmethod prep_struct(*args)

classmethod tup_struct(struct, *args)

pylablib.core.utils.dictionary module

Tree-like multi-level dictionary with advanced indexing options.

pylablib.core.utils.dictionary.split_path(path, omit_empty=True, sep=None)

Split generic path into individual path entries.

Parameters:

• path – Generic path. Lists and tuples (possible nested) are flattened; strings are split according to separators; non-strings are converted into strings first.
• omit_empty (bool) – Determines if empty entries are skipped.
• sep (str) – If not None, defines regex for path separators; default separator is '/'.

Returns:

A list of individual entries.

Return type:

list

pylablib.core.utils.dictionary.normalize_path_entry(entry, case_sensitive=True, case_normalization='lower')

Normalize the case of the entry if it’s not case-sensitive. Normalization is either either 'lower' or 'upper'.

pylablib.core.utils.dictionary.normalize_path(path, omit_empty=True, case_sensitive=True, case_normalization='lower', sep=None, force=False)

Split and normalize generic path into individual path entries.

Parameters:

• path – Generic path. Lists and tuples (possible nested) are flattened; strings are split according to separators; non-strings are converted into strings first.
• omit_empty (bool) – Determines if empty entries are skipped.
• case_sensitive (bool) – If False, entries case is normalized according to case_normalization.
• case_normalization (str) – Normalization rules; either 'lower' or 'upper'.
• sep (str) – If not None, defines regex for path separators; default separator is '/'.
• force (bool) – If False, treat lists as if they’re already normalized.

Returns:

A list of individual normalized entries.

Return type:

list

pylablib.core.utils.dictionary.is_dictionary(obj, generic=False)

Determine if the object is a dictionary.

Parameters:

• obj – object
• generic (bool) – if False, passes only Dictionary (or subclasses) objects; otherwise, passes any dictionary-like object.

Returns:

bool

pylablib.core.utils.dictionary.as_dictionary(obj, case_sensitive=True, case_normalization='lower')

Convert object into Dictionary with the given parameters.

If object is already a Dictionary (or its subclass), return unchanged, even if its parameters are different.

pylablib.core.utils.dictionary.as_dict(obj, style='nested', copy=True)

Convert object into standard dict with the given parameters.

If object is already a dict, return unchanged, even if the parameters are different.

class pylablib.core.utils.dictionary.Dictionary(root=None, case_sensitive=True, case_normalization='lower', copy=True)

Bases: object

Multi-level dictionary.

Access is done by path (all path elements are converted into strings and concatenated to form a single string path). If dictionary is not case-sensitive, all inserted and accessed paths are normalized to lower or upper case.

Parameters:

• root (dict or Dictionary) – Initial value.
• case_sensitive (bool) – If False, entries case is normalized according to case_normalization.
• case_normalization (str) – Normalization rules; either 'lower' or 'upper'.
• copy (bool) – If True, make copy of the supplied data; otherwise, just make it the root.

Warning

If copy==False, the root data is already assumed to be normalized. If it isn’t, the behavior might be incorrect.

static is_dictionary(obj, generic=True)

Determine if the object is a dictionary.

Parameters:

• obj –
• generic (bool) – if False, passes only Dictionary (or subclasses) objects; otherwise, passes any dictionary-like object.

Returns:

bool

static as_dictionary(obj, case_sensitive=True, case_normalization='lower')

Convert object into Dictionary with the given parameters.

If object is already a Dictionary (or its subclass), return unchanged, even if its parameters are different.

add_entry(path, value, force=False, branch_option='normalize')

Add value to a given path (overwrite leaf value if necessary).

Doesn’t replace leaves with branches and vice-verse if force==False.

Parameters:

• path –
• value –
• force (bool) – If True, change leaf into a branch and vice-versa; otherwise, raises ValueError if the conversion is necessary.
• branch_option (str) –

  Decides what to do if the value is dictionary-like:

  • 'attach' – just attach the root,
  • 'copy' – copy and attach,
  • 'normalize' – copy while normalizing all the keys according to the current rules.

get_entry(path, as_pointer=False)

Get entry at a given path

Parameters:

• path –
• as_pointer (bool) – If True and entry is not a leaf, return Dictionary or DictionaryPointer

has_entry(path, kind='all')

Determine if the path is in the dictionary.

kind determines which kind of path to consider and can be 'leaf', 'branch' or 'all'.

get_max_prefix(path, kind='all')

Find the longest prefix of path contained in the dictionary.

Return tuple (prefix, rest), where both path entries are normalized according to the dictionary rules. kind determines which kind of path to consider and can be 'leaf', 'branch' or 'all'. If the longest prefix is of a different kind, return (None,None).

del_entry(path)

Delete entry from the dictionary. Return True if the path was present.

size()

Return the total size of the dictionary (number of nodes).

get(path, default=None)

Analog of dict.get(): D.get(k,d) -> D[k] if k in D else d.

setdefault(path, default=None)

Analog of dict.setdefault(): D.setdefault(k,d) -> D.get(k,d), also sets D[k]=d if k is not in D.

viewitems(ordered=False, leafs=False, path_kind='split', wrap_branches=True)

Analog of dict.viewitems(), by default iterating only over the immediate children of the root.

Parameters:

• ordered (bool) – If True, loop over keys in alphabetic order.
• leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
• path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
• wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

iteritems(ordered=False, leafs=False, path_kind='split', wrap_branches=True)

Analog of dict.viewitems(), by default iterating only over the immediate children of the root.

Parameters:

• ordered (bool) – If True, loop over keys in alphabetic order.
• leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
• path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
• wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

items(ordered=False, leafs=False, path_kind='split', wrap_branches=True)

Analog of dict.viewitems(), by default iterating only over the immediate children of the root.

Parameters:

• ordered (bool) – If True, loop over keys in alphabetic order.
• leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
• path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
• wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

viewvalues(ordered=False, leafs=False, wrap_branches=True)

Analog of dict.viewvalues(), iterating only over the immediate children of the root.

Parameters:

• ordered (bool) – If True, loop over keys in alphabetic order.
• leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
• wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

itervalues(ordered=False, leafs=False, wrap_branches=True)

Analog of dict.viewvalues(), iterating only over the immediate children of the root.

Parameters:

• ordered (bool) – If True, loop over keys in alphabetic order.
• leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
• wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

values(ordered=False, leafs=False, wrap_branches=True)

Analog of dict.viewvalues(), iterating only over the immediate children of the root.

Parameters:

• ordered (bool) – If True, loop over keys in alphabetic order.
• leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
• wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

viewkeys(ordered=False)

Analog of dict.viewkeys(), iterating only over the immediate children of the root.

Parameters:ordered (bool) – If True, loop over keys in alphabetic order.

iterkeys(ordered=False)

Analog of dict.viewkeys(), iterating only over the immediate children of the root.

Parameters:ordered (bool) – If True, loop over keys in alphabetic order.

keys(ordered=False)

Analog of dict.viewkeys(), iterating only over the immediate children of the root.

Parameters:ordered (bool) – If True, loop over keys in alphabetic order.

paths(ordered=False, topdown=False, path_kind='split')

Return list of all leaf paths.

Parameters:

• ordered (bool) – If True, loop over paths in alphabetic order.
• topdown (bool) – If True, return node’s leafs before its subtrees leafs.
• path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)

iternodes(to_visit='leafs', ordered=False, include_path=False, topdown=False)

Iterate over nodes.

Parameters:

• to_visit (str) – Can be 'leafs', 'branches' or 'all' and determines which parts of the dictionary are visited.
• ordered (bool) – If True, loop over paths in alphabetic order.
• include_path (bool) – Include in the return value.
• topdown (bool) – If True, visit node and its leafs before its subtrees leafs.

Yields:

Values for leafs and DictionaryPointer for branches. If include_path==True, yields tuple (path, value), where path is in the form of a normalized list.

nodes(to_visit='leafs', ordered=False, include_path=False, topdown=False)

Iterate over nodes.

Parameters:

• to_visit (str) – Can be 'leafs', 'branches' or 'all' and determines which parts of the dictionary are visited.
• ordered (bool) – If True, loop over paths in alphabetic order.
• include_path (bool) – Include in the return value.
• topdown (bool) – If True, visit node and its leafs before its subtrees leafs.

Yields:

Values for leafs and DictionaryPointer for branches. If include_path==True, yields tuple (path, value), where path is in the form of a normalized list.

merge_branch(source, branch='', overwrite=True, normalize_paths=True)

Attach source (dict or other Dictionary) to a given branch; source is automatically deep-copied.

Parameters:

• source (dict or Dictionary) –
• branch (tuple or str) – Destination path.
• overwrite (bool) – If True, replaces the old entries with the new ones (it only matters for leaf assignments).
• normalize_paths (bool) – If True and the dictionary isn’t case sensitive, perform normalization if the source.

update(source, branch='', overwrite=True, normalize_paths=True)

Attach source (dict or other Dictionary) to a given branch; source is automatically deep-copied.

Parameters:

• source (dict or Dictionary) –
• branch (tuple or str) – Destination path.
• overwrite (bool) – If True, replaces the old entries with the new ones (it only matters for leaf assignments).
• normalize_paths (bool) – If True and the dictionary isn’t case sensitive, perform normalization if the source.

detach_branch(branch='')

Remove branch from the current dictionary and return it as a separate Dictionary.

branch_copy(branch='')

Get a copy of the branch as a Dictionary.

copy()

Get a full copy the dictionary.

updated(source, branch='', overwrite=True, normalize_paths=True)

Get a copy of the dictionary and attach a new branch to it.

Parameters are the same as in the Dictionary.merge_branch().

as_dict(style='nested', copy=True)

Convert into a dict object.

Parameters:

• style (str) –

  Determines style of the returned :

  • 'nested' – subtrees are turned into nested dictionaries,
  • 'flat' – single dictionary is formed with full paths as keys.

• copy (bool) – If False and style=='nested', return the root dictionary.

as_pandas(index_key=True, as_series=True)

Convert into a pandas DataFrame or Series object.

Parameters:

• index_key (bool) – If False, create a 2-column table with the first column ("key") containing string path and the second column ("value") containing value; otherwise, move key to the table index.
• as_series (bool) – If index_key==True and as_series==True, convert the resulting DataFrame into 1D Series (the key is the index); otherwise, keep it as a single-column table

get_path()

branch_pointer(branch='')

Get a DictionaryPointer of a given branch.

map_self(func, to_visit='leafs', pass_path=False, topdown=False, branch_option='normalize')

Apply func to the nodes in the dictionary.

Parameters:

• func (callable) – Mapping function. Leafs are passed by value, branches (if visited) are passed as DictionaryPointer.
• to_visit (str) – Can be 'leafs', 'branches' or 'all' and determines which parts of the dictionary passed to the map function.
• pass_path (bool) – If True, pass the node path (in the form of a normalized list) as a first argument to func.
• topdown (bool) – If True, visit node and its leafs before its subtrees leafs.
• branch_option (str) – If the function returns a dict-like object, determines how to

filter_self(pred, to_visit='leafs', pass_path=False, topdown=False)

Remove all the nodes from the dictionary for which pred returns False.

Parameters:

• pred (callable) – Filter function. Leafs are passed to pred by value, branches (if visited) are passed as DictionaryPointer.
• to_visit (str) – Can be 'leafs', 'branches' or 'all' and determines which parts of the dictionary passed to the predicate.
• pass_path (bool) – If True, pass the node path (in the form of a normalized list) as a first argument to pred.
• topdown (bool) – If True, visit node and its leafs before its subtrees leafs.

class DictionaryDiff

Bases: pylablib.core.utils.dictionary.DictionaryDiff

Describes a difference between the two dictionaries.

same

Contains the leafs which is the same.

Type:Dictionary

changed_from

Contains the leafs from the first dictionary which have different values in the second dictionary.

Type:Dictionary

changed_to

Contains the leafs from the second dictionary which have different values in the first dictionary.

Type:Dictionary

removed

Contains the leafs from the first dictionary which are absent in the second dictionary.

Type:Dictionary

added

Contains the leafs from the second dictionary which are absent in the first dictionary.

Type:Dictionary

diff(other)

Perform an element-wise comparison to another Dictionary.

If the other Dictionary has a different case sensitivity, raise ValueError.

Returns:Dictionary.DictionaryDiff

static diff_flatdict(first, second)

Find the difference between flat dict objects.

Returns:Dictionary.DictionaryDiff

class DictionaryIntersection

Bases: pylablib.core.utils.dictionary.DictionaryIntersection

Describes the result of finding intersection of multiple dictionaries.

common

Contains the intersection of all dictionaries.

Type:Dictionary

individual

Contains list of difference from intersection for all dictionaries.

Type:[Dictionary]

static find_intersection(dicts, use_flatten=False)

Find intersection of multiple dictionaries.

Parameters:

• dicts ([Dictionary]) –
• use_flatten (bool) – If True flatten all dictionaries before comparison (works faster for a large number of dictionaries).

Returns:

Dictionary.DictionaryIntersection

get_matching_paths(pattern, wildkey='*', wildpath='**', only_leaves=True)

Get all paths in the tree that match the provided pattern.

Parameters:

• wildkey (str) – Pattern symbol that matches any key.
• wildpath (str) – Pattern symbol that matches any subpath (possibly empty).
• only_leaves (bool) – If True, only check leaf paths; otherwise, check subtree paths (i.e., incomplete leaf paths) as well. Basically, only_leaves=False is analogous to adding wildpath at the end of the pattern.

get_matching_subtree(pattern, wildkey='*', wildpath='**', only_leaves=True)

Get a subtree containing nodes with paths matching the provided pattern.

Parameters:

• wildkey (str) – Pattern symbol that matches any key.
• wildpath (str) – Pattern symbol that matches any subpath (possibly empty).
• only_leaves (bool) – If True, only check leaf paths; otherwise, check subtree paths (i.e., incomplete leaf paths) as well. Basically, only_leaves=False is analogous to adding wildpath at the end of the pattern.

class pylablib.core.utils.dictionary.DictionaryPointer(root=None, pointer=None, case_sensitive=True, case_normalization='lower', copy=True)

Bases: pylablib.core.utils.dictionary.Dictionary

Similar to Dictionary, but can point at one of the branches instead of the full dictionary.

Effect is mostly equivalent to prepending some path to all queries.

Parameters:

• root (dict or Dictionary) – Complete tree.
• pointer – Path to the pointer location.
• case_sensitive (bool) – If False, entries case is normalized according to case_normalization.
• case_normalization (str) – Normalization rules; either 'lower' or 'upper'.
• copy (bool) – If True, make copy of the supplied data; otherwise, just make it the root.

Warning

If copy==False, the root data is already assumed to be normalized. If it isn’t, the behavior might be incorrect.

get_path()

Return pointer path in the whole dictionary.

move(path='', absolute=True)

Move the pointer to a new path.

Parameters:

• path –
• absolute (bool) – If True, path is specified with respect to the root; otherwise, it’s specified with respect to the current position (and can only go deeper).

branch_pointer(branch='')

Get a DictionaryPointer of a given branch.

class pylablib.core.utils.dictionary.PrefixTree(root=None, case_sensitive=True, case_normalization='lower', wildcard='*', matchcard='.', copy=True)

Bases: pylablib.core.utils.dictionary.Dictionary

Expansion of a Dictionary designed to store data related to prefixes.

Each branch node can have a leaf with a name given by wildcard ('*' by default) or matchcard ('.' by default). Wildcard assumes that the branch node path is a prefix; matchcard assumes exact match. These leafs are inspected when specific prefix tree functions (find_largest_prefix and find_all_prefixes) are used.

Parameters:

• root (dict or Dictionary) – Complete tree.
• case_sensitive (bool) – If False, entries case is normalized according to case_normalization.
• case_normalization (str) – Normalization rules; either 'lower' or 'upper'.
• wildcard (str) – Symbol for a wildcard entry.
• matchcard (str) – Symbol for a matchcard entry.
• copy (bool) – If True, make copy of the supplied data; otherwise, just make it the root.

Warning

If copy==False, the root data is already assumed to be normalized. If it isn’t, the behavior might be incorrect.

copy()

Get a full copy the prefix tree.

find_largest_prefix(path, default=None, allow_nomatch_exact=True, return_path=False, return_subpath=False)

Find the entry which is the largest prefix of a given path.

Parameters:

• path –
• default – Default value if the path isn’t found.
• allow_nomatch_exact (bool) – If True, just element with the given path can be returned; otherwise, only elements stored under wildcards and matchcards are considered.
• return_path (bool) – If True, return path to the element (i.e., the largest prefix) instead of the element itself.
• return_subpath (bool) – If True, return tuple with a second element being part of the path left after subtraction of the prefix.

find_all_prefixes(path, allow_nomatch_exact=True, return_path=True, return_subpath=False)

Find list of all the entries which are prefixes of a given path.

Parameters:

• path –
• default – Default value if the path isn’t found.
• allow_nomatch_exact (bool) – If True, just element with the given path can be returned; otherwise, only elements stored under wildcards and matchcards are considered.
• return_path (bool) – If True, return path to the element (i.e., the largest prefix) instead of the element itself.
• return_subpath (bool) – If True, return tuple with a second element being part of the path left after subtraction of the prefix.

pylablib.core.utils.dictionary.combine_dictionaries(dicts, func, select='all', pass_missing=False)

Combine several dictionaries element-wise (only for leafs) using a given function.

Parameters:

• dicts (list or tuple) – list of dictionaries (Dictionary or dict) to be combined
• func (callable) – combination function. Takes a single argument, which is a list of elements to be combined.
• select (str) – determins which keys are selected for the resulting dictionary. Can be either "all" (only keep keys which are present in all the dictionaries), or "any" (keep keys which are present in at least one dictionary). Only keys that point to leafs count; if a key points to a non-leaf branch in some dictionary, it is considered absent from this dictionary.
• pass_missing (bool) – if select=="any", this parameter determines whether missing elements will be passed to func as None, or omitted entirely.

class pylablib.core.utils.dictionary.DictionaryNode(**vargs)

Bases: object

pylablib.core.utils.dictionary.dict_to_object_local(data, name=None, object_generator=<function _default_object_generator>)

class pylablib.core.utils.dictionary.PrefixShortcutTree(shortcuts=None)

Bases: object

Convenient storage for dictionary path shortcuts.

Parameters:shortcuts (dict) – Dictionary of shortcuts {shortcut: full_path}.

copy()

Return full copy.

add_shortcut(source, dest, exact=False)

Add a single shortcut.

Parameters:

• source – Shortcut path.
• dest – expanded path corresponding to the shortcut.
• exact (bool) – If True, the shortcut works only for the exact path; otherwise, it works for any path with ‘source’ as a prefix.

add_shortcuts(shortcuts, exact=False)

Add a dictionary of shortcuts {shortcut: full_path}.

Arguments are the same as in PrefixShortcutTree.add_shortcut().

remove_shortcut(source)

Remove a shortcut from the tree

updated(shortcuts, exact=False)

Make a copy and add additional shortcuts.

Arguments are the same as in PrefixShortcutTree.add_shortcuts().

class pylablib.core.utils.dictionary.ItemAccessor(getter=None, setter=None, deleter=None, normalize_names=True, path_separator='/', allow_incomplete_paths=False)

Bases: object

class ItemAccessorBranch(src, branch)

Bases: object

get(name, default=None)

pylablib.core.utils.files module

Utilities for working with the file system: creating/removing/listing folders, comparing folders and files, working with zip archives.

pylablib.core.utils.files.eof(f, strict=False)

Standard EOF function.

Return True if the the marker is at the end of the file. If strict==True, only return True if the marker is exactly at the end of file; otherwise, return True if it’s at the end of further.

pylablib.core.utils.files.get_file_creation_time(path, timestamp=True)

Try to find a file creation time. Return current time if an error occurs.

If timestamp==True, return UNIX timestamp; otherwise, return datetime.datetime.

pylablib.core.utils.files.get_file_modification_time(path, timestamp=True)

Try to find a file modification time. Return current time if an error occurs.

If timestamp==True, return UNIX timestamp; otherwise, return datetime.datetime

pylablib.core.utils.files.touch(fname, times=None)

Update file access and modification times.

Parameters:times (tuple) – Access and modification times; if times is None, use current time.

pylablib.core.utils.files.generate_indexed_filename(name_format, idx_start=0, folder='')

Generate an unused indexed filename in folder.

The name has name_format (using standard Python format() rules), with index starting with idx_start.

pylablib.core.utils.files.generate_prefix_filename(prefix='', suffix='', idx_start=None, folder='')

Generate an unused filename in folder with the given prefix and suffix.

The format is prefix_{:d}_suffix, where the parameter is the index starting with idx_start. If idx_start is None first check prefix+suffix name before using numbered indices.

pylablib.core.utils.files.generate_temp_filename(prefix='__tmp__', idx_start=0, idx_template='d', folder='')

Generate a temporary filename with a given prefix.

idx_template is the number index format (only the parameter itself, not the whole string).

pylablib.core.utils.files.fullsplit(path, ignore_empty=True)

Split path into list.

If ignore_empty==True, omit empty folder names.

pylablib.core.utils.files.normalize_path(p)

Normalize filesystem path (case and origin). If two paths are identical, they should be equal when normalized.

pylablib.core.utils.files.case_sensitive_path()

Check if OS path names are case-sensitive (e.g., Linux)

pylablib.core.utils.files.paths_equal(a, b)

Determine if the two paths are equal (can be local or have different case).

pylablib.core.utils.files.relative_path(a, b, check_paths=True)

Determine return path a as seen from b.

If check_paths==True, check if a is contained in b and raise the :exc:OSError if it isn’t.

class pylablib.core.utils.files.TempFile(folder='', name=None, mode='w', wait_time=None, rep_time=None)

Bases: object

Temporary file context manager.

Upon creation, generate an unused temporary filename. Upon entry, create the file using supplied mode and return self. Upon exit, close and remove the file.

Parameters:

• folder (str) – Containing folder.
• name (str) – File name. If None, generate new temporary name.
• mode (str) – File opening mode.
• wait_time (float) – Waiting time between attempts to create the file if the first try fails.
• rep_time (int) – Number of attempts to create the file if the first try fails.

f

File object.

name

File name.

Type:str

full_name

File name including containing folder.

Type:str

pylablib.core.utils.files.copy_file(source, dest, overwrite=True, cmp_on_overwrite=True)

Copy file, creating a containing folder if necessary. Return True if the operation was performed.

Parameters:

• overwrite (bool) – If True, overwrite existing file.
• cmp_on_overwrite (bool) – If True and the two files are compared to be the same, don’t perform overwrite.

pylablib.core.utils.files.move_file(source, dest, overwrite=True, cmp_on_overwrite=True, preserve_if_not_move=False)

Move file, creating a containing folder if necessary. Returns True if the operation was performed.

Parameters:

• overwrite (bool) – If True, overwrite existing file (if the existing file isn’t overwritten, preserve the original).
• cmp_on_overwrite (bool) – If True and the two files are compared to be the same, don’t perform overwrite.
• preserve_if_not_move (bool) – If True and the files are identical, preserve the original.

pylablib.core.utils.files.ensure_dir_singlelevel(path, error_on_file=True)

pylablib.core.utils.files.ensure_dir(path, error_on_file=True)

Ensure that the folder exists (create a new one if necessary).

If error_on_file==True, raise OSError if there’s a file with the same name.

pylablib.core.utils.files.remove_dir(path, error_on_file=True)

Remove the folder recursively if it exists.

If error_on_file==True, raise OSError if there’s a file with the same name.

pylablib.core.utils.files.remove_dir_if_empty(path, error_on_file=True)

Remove the folder only if it’s empty.

If error_on_file==True, raise OSError if there’s a file with the same name.

pylablib.core.utils.files.clean_dir(path, error_on_file=True)

Remove the folder and then recreate it.

If error_on_file==True, raise OSError if there’s a file with the same name.

class pylablib.core.utils.files.FolderList

Bases: pylablib.core.utils.files.FolderList

Describes folder content.

pylablib.core.utils.files.list_dir(folder='', folder_filter=None, file_filter=None, separate_kinds=True, error_on_file=True)

Return folder content filtered by folder_filter and file_filter.

Parameters:

• folder (str) – Path to the folder.
• folder_filter – Folder filter function (more description at string.get_string_filter()).
• file_filter – File filter function (more description at string.get_string_filter()).
• separate_kinds (bool) – if True, return FolderList with files and folder separate; otherwise, return a single list (works much faster).
• error_on_file (bool) – if True, raise OSError if there’s a file with the same name as the target folder.

pylablib.core.utils.files.dir_empty(folder, folder_filter=None, file_filter=None, level='single', error_on_file=True)

Check if the folder is empty (only checks content filtered by folder_filter and file_filter).

Parameters:

• folder (str) – Path to the folder.
• folder_filter – Folder filter function (more description at string.get_string_filter()).
• file_filter – File filter function (more description at string.get_string_filter()).
• level (str) – if 'single', check only immediate folder content; if 'recursive', follow recursively in all folders passing folder_filter.
• error_on_file (bool) – if True, raise OSError if there’s a file with the same name as the target folder.

pylablib.core.utils.files.walk_dir(folder, folder_filter=None, file_filter=None, rel_path=True, topdown=True, visit_folder_filter=None, max_depth=None)

Modification of os.walk() function.

Acts in a similar way, but followlinks is always False and errors of os.listdir() are always passed.

Parameters:

• folder (str) – Path to the folder.
• folder_filter – Folder filter function (more description at string.get_string_filter()).
• file_filter – File filter function (more description at string.get_string_filter()).
• rel_path (bool) – If True, the returned folder path is specified relative to the initial path.
• topdown (bool) – If True, return folder before its subfolders.
• visit_folder_filter – Filter for visiting folders (more description at string.get_string_filter()). If not None, specifies filter for visiting folders which is different from folder_filter (filter for returned folders).
• max_depth (int) – If not None, limits the recursion depth.

Yields:

For each folder (including the original) yields a tuple (folder_path, folders, files),

where folder_path is the containing folder name and folders and files are its content (similar to list_dir()).

pylablib.core.utils.files.list_dir_recursive(folder, folder_filter=None, file_filter=None, topdown=True, visit_folder_filter=None, max_depth=None)

Recursive walk analog of list_dir().

Parameters are the same as walk_dir().

Returns:FolderList

pylablib.core.utils.files.copy_dir(source, dest, folder_filter=None, file_filter=None, overwrite=True, cmp_on_overwrite=True)

Copy files satisfying the filtering conditions.

Parameters:

• source (str) – Source path.
• dest (str) – Destination path.
• folder_filter – Folder filter function (more description at string.get_string_filter()).
• file_filter – File filter function (more description at string.get_string_filter()).
• overwrite (bool) – If True, overwrite existing files.
• cmp_on_overwrite (bool) – If True and the two files are compared to be the same, don’t perform overwrite.

pylablib.core.utils.files.move_dir(source, dest, folder_filter=None, file_filter=None, overwrite=True, cmp_on_overwrite=True, preserve_if_not_move=False)

Move files satisfying the filtering conditions.

Parameters:

• source (str) – Source path.
• dest (str) – Destination path.
• folder_filter – Folder filter function (more description at string.get_string_filter()).
• file_filter – File filter function (more description at string.get_string_filter()).
• overwrite (bool) – If True, overwrite existing files (if the existing file isn’t overwritten, preserve the original).
• cmp_on_overwrite (bool) – If True and the two files are compared to be the same, don’t perform overwrite.
• preserve_if_not_move (bool) – If True and the files are identical, preserve the original.

pylablib.core.utils.files.combine_diff(d1, d2)

pylablib.core.utils.files.cmp_dirs(a, b, folder_filter=None, file_filter=None, shallow=True, return_difference=False)

Compare the folders based on the content filtered by folder_filter and file_filter.

Parameters:

• a (str) – First folder path
• b (str) – Second folder path
• folder_filter – Folder filter function (more description at string.get_string_filter()).
• file_filter – File filter function (more description at string.get_string_filter()).
• shallow – If True, do shallow comparison of the files (see filecmp.cmp()).
• return_difference – If False, simply return bool; otherwise, return difference type ('=', '+', '-' or '*').

pylablib.core.utils.files.retry_copy(source, dest, overwrite=True, cmp_on_overwrite=True, try_times=5, delay=0.3)

Retrying version of copy_file().

If the operation raises error, wait for delay (in seconds) and call it again. Try total of try_times times.

pylablib.core.utils.files.retry_move(source, dest, overwrite=True, cmp_on_overwrite=True, preserve_if_not_move=False, try_times=5, delay=0.3)

Retrying version of move_file() (see retry_copy() for details on retrying).

pylablib.core.utils.files.retry_remove(path, try_times=5, delay=0.3)

Retrying version of os.remove() (see retry_copy() for details on retrying).

pylablib.core.utils.files.retry_ensure_dir(path, error_on_file=True, try_times=5, delay=0.3)

Retrying version of ensure_dir() (see retry_copy() for details on retrying).

pylablib.core.utils.files.retry_copy_dir(source, dest, folder_filter=None, file_filter=None, overwrite=True, cmp_on_overwrite=True, try_times=5, delay=0.3)

Retrying version of copy_dir() (see retry_copy() for details on retrying).

pylablib.core.utils.files.retry_move_dir(source, dest, folder_filter=None, file_filter=None, overwrite=True, cmp_on_overwrite=True, preserve_if_not_move=False, try_times=5, delay=0.3)

Retrying version of move_dir() (see retry_copy() for details on retrying).

pylablib.core.utils.files.retry_remove_dir(path, error_on_file=True, try_times=5, delay=0.3)

Retrying version of remove_dir() (see retry_copy() for details on retrying).

pylablib.core.utils.files.retry_remove_dir_if_empty(path, error_on_file=True, try_times=5, delay=0.3)

Retrying version of remove_dir_if_empty() (see retry_copy() for details on retrying).

pylablib.core.utils.files.retry_clean_dir(path, error_on_file=True, try_times=5, delay=0.3)

Retrying version of clean_dir() (see retry_copy() for details on retrying).

pylablib.core.utils.files.zip_folder(zip_path, source_path, inside_path='', folder_filter=None, file_filter=None, mode='a', compression=8)

Add a folder into a zip archive.

Parameters:

• zip_path (str) – Path to the .zip file.
• source_path (str) – Path to the source folder.
• inside_path (str) – Destination path inside the zip archive.
• folder_filter – Folder filter function (more description at string.get_string_filter()).
• file_filter – File filter function (more description at string.get_string_filter()).
• mode (str) – Zip archive adding mode (see zipfile.ZipFile).
• compression – Zip archive compression (see zipfile.ZipFile).

pylablib.core.utils.files.zip_file(zip_path, source_path, inside_name=None, mode='a', compression=8)

Add a file into a zip archive.

Parameters:

• zip_path (str) – Path to the .zip file.
• source_path (str) – Path to the source file.
• inside_name (str) – Destination file name inside the zip archive (source name on the top level by default).
• mode (str) – Zip archive adding mode (see zipfile.ZipFile).
• compression – Zip archive compression (see zipfile.ZipFile).

pylablib.core.utils.files.zip_multiple_files(zip_path, source_paths, inside_names=None, mode='a', compression=8)

Add a multiple files into a zip archive.

Parameters:

• zip_path (str) – Path to the .zip file.
• source_paths ([str]) – List of path to the source files.
• inside_names ([str] or None) – List of destination file names inside the zip archive (source name on the top level by default).
• mode (str) – Zip archive adding mode (see zipfile.ZipFile).
• compression – Zip archive compression (see zipfile.ZipFile).

pylablib.core.utils.files.unzip_folder(zip_path, dest_path, inside_path='', folder_filter=None, file_filter=None)

Extract a folder from a zip archive (create containing folder if necessary).

Parameters:

• zip_path (str) – Path to the .zip file.
• dest_path (str) – Path to the destination folder.
• inside_path (str) – Source path inside the zip archive; extracted data paths are relative (i.e., they don’t include inside_path).
• folder_filter – Folder filter function (more description at string.get_string_filter()).
• file_filter – File filter function (more description at string.get_string_filter()).

pylablib.core.utils.files.unzip_file(zip_path, dest_path, inside_path)

Extract a file from a zip archive (create containing folder if necessary).

Parameters:

• zip_path (str) – Path to the .zip file.
• dest_path (str) – Destination file path.
• inside_path (str) – Source path inside the zip archive.

pylablib.core.utils.files.openfiledialog(**options)

Open file dialog, wrapper for tkFileDialog.

pylablib.core.utils.files.savefiledialog(**options)

Save file dialog, wrapper for tkFileDialog.

pylablib.core.utils.files.opendirdialog(**options)

Open directory dialog, wrapper for tkFileDialog.

pylablib.core.utils.funcargparse module

Contains routines for checking arguments passed into a function for better flexibility.

pylablib.core.utils.funcargparse.parameter_value_error(par_val, par_name, message=None, error_type=None)

Raise parameter value error (ValueError by default).

pylablib.core.utils.funcargparse.parameter_range_error(par_val, par_name, par_set=None, message=None, error_type=None)

Raise parameter range error (ValueError by default).

pylablib.core.utils.funcargparse.check_parameter_range(par_val, par_name, par_set, message=None, error_type=None)

Raise error if par_val is not in in the par_set (par_name is used in the error message).

pylablib.core.utils.funcargparse.getdefault(value, default_value, unassigned_value=None, conflict_action='ignore', message=None, error_type=None)

Analog of dict’s getdefault.

If value is unassigned_value, return default_value instead. If conflict_action=='error' and value!=default_value, raise value error using message and error_type.

pylablib.core.utils.funcargparse.is_sequence(value, sequence_type='builtin;nostring')

Check if value is a sequence.

sequence_type is semicolon separated list of possible sequence types:

• 'builtin' - list, tuple or str
• 'nostring' - str is not allows
• 'array' - list, tuple or numpy.ndarray
• 'indexable' - anything which can be indexed
• 'haslength' - anything with length property

pylablib.core.utils.funcargparse.make_sequence(element, length=1, sequence_type='list')

Turn element into a sequence of sequence_type ('list' or 'tuple') repeated length times.

pylablib.core.utils.funcargparse.as_sequence(value, multiply_length=1, allowed_type='builtin;nostring', wrapping_type='list', length_conflict_action='ignore', message=None, error_type=None)

Ensure that value is a sequence.

If value is not a sequence of allowed_type (as checked by is_sequence()), turn it into a sequence specified by wrapping_type and multiply_length.

If value is a sequence and length_conflict_action=='error', raise error with error_type and error_message if the length doesn’t match multiply_length. Otherwise, return value unchanged.

pylablib.core.utils.functions module

Utilities for dealing with function, methods and function signatures.

class pylablib.core.utils.functions.FunctionSignature(arg_names=None, defaults=None, varg_name=None, kwarg_name=None, kwonly_arg_names=None, cls=None, obj=None, name=None, doc=None)

Bases: object

Description of a function signature, including name, argument names, default values, names of varg and kwarg arguments, class and object (for methods) and docstring.

Parameters:

• arg_names (list) – Names of the arguments.
• default (dict) – Dictionary {name: value} of default values.
• varg_name (str) – Name of *varg parameter (None means no such parameter).
• kwarg_name (str) – Name of **kwarg parameter (None means no such parameter).
• cls – Caller class, for methods.
• obj – Caller object, for methods.
• name (str) – Function name.
• doc (str) – Function docstring.

get_defaults_list()

Get list of default values for arguments in the order specified in the signature.

signature(pass_order=None)

Get string containing a signature (arguments list) of the function (call or definition), including *vargs and **kwargs.

If pass_order is not None, it specifies the order in which the arguments are passed.

wrap_function(func, pass_order=None)

Wrap a function func into a containing function with this signature.

Sets function name, argument names, default values, object and class (for methods) and docstring. If pass_order is not None, it determines the order in which the positional arguments are passed to the wrapped function.

mandatory_args_num()

Get minimal number of arguments which have to be passed to the function.

The mandatory arguments are the ones which are not bound to caller object (i.e., not self) and don’t have default values.

max_args_num(include_positional=True, include_keywords=True)

Get maximal number of arguments which can be passed to the function.

Parameters:

• include_positional (bool) – If True and function accepts *vargs, return None (unlimited number of arguments).
• include_keywords (bool) – If True and function accepts **kwargs, return None (unlimited number of arguments).

static from_function(func, follow_wrapped=True)

Get signature of the given function or method.

If follow_wrapped==True, follow __wrapped__ attributes until the innermost function (useful for getting signatures of functions wrapped using functools methods).

copy()

Return a copy

as_simple_func()

Turn the signature into a simple function (as opposed to a bound method).

If the signature corresponds to a bound method, get rid of the first argument in the signature (self) and the bound object. Otherwise, return unchanged.

static merge(inner, outer, add_place='front', merge_duplicates=True, overwrite=None, hide_outer_obj=False)

Merge two signatures (used for wrapping functions).

The signature describes the function would take arguments according to the outer signature and pass them accroding to the inner signature.

The arguments are combined:

• if add_place=='front', the outer arguments are placed in the beginning, followed by inner arguments not already listed;
• if add_place=='back', the inner arguments are placed in the beginning, followed by outer arguments not already listed.

The default values are joined, with the outer values superseding the inner values.

overwrite is a set or a list specifying which inner parameters are overwritten by the outer. It includes 'name', 'doc', 'cls', 'obj', 'varg_name' and 'kwarg_name'; the default value is all parameters.

If the inner signature is a bound method and hide_inner_obj==True, treat it as a function (with self argument missing). In this case, the wrapped signature .obj field will be None.

Returns:(signature, pass_order)

pass_order is the order in which the arguments of the combined signature may be passed to the inner signature; it may be different from the signature order if add_place=='front'. If merge_duplicates==True, duplicate entries in pass_order are omitted; otherwise, they’re repeated.

Return type:tuple

pylablib.core.utils.functions.funcsig(func, follow_wrapped=True)

Return a function signature object

pylablib.core.utils.functions.getargsfrom(source, **merge_params)

Decorator factory.

Returns decorator that conforms function signature to the source function. **merge_params are passed to the FunctionSignature.merge() method merging wrapped and source signature.

The default behavior (conforming parameter names, default values args and kwargs names) is useful for wrapping universal functions like g(*args, **kwargs).

Example:

def f(x, y=2):
    return x+y

@getargsfrom(f)
def g(*args): # Now g has the same signature as f, including parameter names and default values.
    return prod(args)

pylablib.core.utils.functions.call_cut_args(func, *args, **kwargs)

Call func with the given arguments, omitting the ones that don’t fit its signature.

pylablib.core.utils.functions.getattr_call(obj, attr_name, *args, **vargs)

Call the getter for the attribute attr_name of obj.

If the attribute is a property, pass *args and **kwargs to the getter (fget); otherwise, ignore them.

pylablib.core.utils.functions.setattr_call(obj, attr_name, *args, **vargs)

Call the setter for the attribute attr_name of obj.

If the attribute is a propert, pass *args and **kwargs to the setter (fset); otherwise, the set value is assumed to be either the first argument, or the keyword argument with the name 'value'.

pylablib.core.utils.functions.delattr_call(obj, attr_name, *args, **vargs)

Call the deleter for the attribute attr_name of obj.

If the attribute is a property, pass *args and **kwargs to the deleter (fdel); otherwise, ignore them.

class pylablib.core.utils.functions.IObjectCall

Bases: object

Universal interface for object method call (makes methods, attributes and properties look like methods).

Should be called with an object as a first argument.

class pylablib.core.utils.functions.MethodObjectCall(method)

Bases: pylablib.core.utils.functions.IObjectCall

Object call created from an object method.

Parameters:method – Either a method object or a method name which is used for the call.

class pylablib.core.utils.functions.AttrObjectCall(name, as_getter)

Bases: pylablib.core.utils.functions.IObjectCall

Object call created from an object attribute (makes attributes and properties look like methods).

Parameters:

• name (str) – Attribute name.
• as_getter (bool) – If True, call the getter when invoked; otherwise, call the setter.

If an attribute is a simple attribute, than getter gets no arguments and setter gets one argument (either the first argument, or the keyword argument named 'value'). If it’s a property, pass all the parameters to the property call.

class pylablib.core.utils.functions.IObjectProperty

Bases: object

Universal interface for an object property (makes methods, attributes and properties look like properties).

Can be used to get, set or remove a property.

get(obj, params=None)

set(obj, value)

rem(obj, params=None)

class pylablib.core.utils.functions.MethodObjectProperty(getter=None, setter=None, remover=None, expand_tuple=True)

Bases: pylablib.core.utils.functions.IObjectProperty

Object property created from object methods (makes methods look like properties).

Parameters:

• getter (callable) – Method invoked on get(). If None, raise RuntimeError when called.
• setter (callable) – Method invoked on set(). If None, raise RuntimeError when called.
• remover (callable) – Method invoked on rem(). If None, raise RuntimeError when called.
• expand_tuple (bool) – If True and if the first argument in the method call is a tuple, expand it as an argument list for the underlying function call.

get(obj, params=None)

set(obj, value)

rem(obj, params=None)

class pylablib.core.utils.functions.AttrObjectProperty(name, use_getter=True, use_setter=True, use_remover=True, expand_tuple=True)

Bases: pylablib.core.utils.functions.IObjectProperty

Object property created from object attribute. Works with attributes or properties.

Parameters:

• name (str) – Attribute name.
• use_getter (bool) – If False, raise RuntimeError when calling get method.
• use_setter (bool) – If False, raise RuntimeError when calling set method.
• use_remover (bool) – If False, raise RuntimeError when calling rem method.
• expand_tuple (bool) – If True and if the first argument in the method call is a tuple, expand it as an argument list for the underlying function call.

get(obj, params=None)

set(obj, value)

rem(obj, params=None)

pylablib.core.utils.functions.empty_object_property(value=None)

Dummy property which does nothing and returns value on get (None by default).

pylablib.core.utils.functions.obj_prop(*args, **kwargs)

Build an object property wrapper.

If no arguments (or a single None argument) are suppled, return a dummy property. If one argument is supplied, return AttrObjectProperty for a property with a given name. Otherwise, return MethodObjectProperty property.

pylablib.core.utils.functions.as_obj_prop(value)

Turn value into an object property using obj_prop() function.

If it’s already IObjectProperty, return unchanged. If value is a tuple, expand as an argument list.

pylablib.core.utils.functions.delaydef(gen)

Wrapper for a delayed definition of a function inside of a module.

Useful if defining a function is computationally costly. The wrapped function should be a generator of the target function rather than the function itself.

On the first call the generator is executed to define the target function, which is then substituted for all subsequent calls.

pylablib.core.utils.general module

Collection of small utilities.

pylablib.core.utils.general.set_props(obj, prop_names, props)

Set multiple attributes of obj.

Names are given by prop_names list and values are given by props list.

pylablib.core.utils.general.get_props(obj, prop_names)

Get multiple attributes of obj.

Names are given by prop_names list.

pylablib.core.utils.general.try_method_wrapper(func, method_name=None, inherit_signature=True)

Decorator that makes the function attempt to call the first argument’s method instead of func.

Before calling the function, try and call a method of the first argument named method_name (func name by default). If the method exists, call it instead of the wrapped function. If inherit_signature==True, completely copy the signature of the wrapped method (name, args list, docstring, etc.).

pylablib.core.utils.general.to_predicate(x)

Turn x into a predicate.

If x is callable, it will be called with a single argument and returned value determines if the argument passes. If x is a container, an argument passes if it’s contained in x.

pylablib.core.utils.general.map_container(value, func)

Map values in the container.

value can be a tuple, a list or a dict (mapping is applied to the values) raises ValueError if it’s something else.

pylablib.core.utils.general.recursive_map(value, func)

Map container recursively.

value can be a tuple, a list or a dict (mapping is applied to the values).

pylablib.core.utils.general.any_item(d)

Return arbitrary tuple (key, value) contained in the dictionary (works both in Python 2 and 3)

pylablib.core.utils.general.merge_dicts(*dicts)

Combine multiple dict objects together.

If multiple dictionaries have the same keys, later arguments have higher priority.

pylablib.core.utils.general.filter_dict(pred, d, exclude=False)

Filter dictionary based on a predicate.

pred can be a callable or a container (in which case the predicate is true if a value is in the container). If exclude==True, the predicate is inverted.

pylablib.core.utils.general.map_dict_keys(func, d)

Map dictionary keys with func

pylablib.core.utils.general.map_dict_values(func, d)

Map dictionary values with func

pylablib.core.utils.general.to_dict(d, default=None)

Convert a dict or a list of pairs or single keys (or mixed) into a dict.

If a list element is single, default value is used.

pylablib.core.utils.general.to_pairs_list(d, default=None)

Convert a dict or a list of pairs or single keys (or mixed) into a list of pairs.

If a list element is single, default value is used. When converting list into list, the order is preserved.

pylablib.core.utils.general.invert_dict(d, kmap=None)

Invert dictionary (switch keys and values).

If kmap is supplied, it’s a function mapping dictionary values into inverted dictionary keys (identity by default).

pylablib.core.utils.general.flatten_list(l)

Flatten nested list/tuple structure into a single list.

pylablib.core.utils.general.partition_list(pred, l)

Split the lis` l into two parts based on the predicate.

pylablib.core.utils.general.split_in_groups(key_func, l, continuous=True, max_group_size=None)

Split the list l into groups according to the key_func.

Go over the list and group the elements with the same key value together. If continuous==False, groups all elements with the same key together regardless of where they are in the list. otherwise, group only continuous sequences of the elements with the same key together (element with different key in the middle will result in two groups). If continuous==True and max_group_size is not None, it determines the maximal size of a group; larger groups are split into separate groups.

pylablib.core.utils.general.sort_set_by_list(s, l, keep_duplicates=True)

Convert the set s into a list ordered by a list l.

Elements in s which are not in l are omitted. If keep_duplicates==True, keep duplicate occurrences in l in the result; otherwise, only keep the first occurrence.

pylablib.core.utils.general.compare_lists(l1, l2, sort_lists=False, keep_duplicates=True)

Return three lists (l1 and l2, l1-l2, l2-l1).

If sort_lists==True, sort the first two lists by l1, and the last one by l2; otherwise, the order is undefined. If sort_lists==True, keep_duplicated determines if duplicate elements show up in the result.

class pylablib.core.utils.general.DummyResource

Bases: object

Object that acts as a resource (has __enter__ and __exit__ methods), but doesn’t do anything.

Analog of:

@contextlib.contextmanager
def dummy_resource():
    yield

class pylablib.core.utils.general.RetryOnException(tries=None, exceptions=None)

Bases: object

Wrapper for repeating the same block of code several time if an exception occurs

Useful for filesystem or communication operations, where retrying a failed operation is a valid option.

Parameters:

• tries (int) – Determines how many time will the chunk of code execute before re-rasing the exception; None (default) means no limit
• exceptions (Exception or list) – A single exception class or a list of exception classes which are going to be silenced.

Example:

for t in RetryOnException(tries,exceptions):
    with t:
        ... do stuff ...

is analogue of:

for i in range(tries):
    try:
        ... do stuff ...
    except exceptions:
        if i==tries-1:
            raise

class ExceptionCatcher(retrier, try_number)

Bases: object

reraise()

pylablib.core.utils.general.retry_wait(func, try_times=1, delay=0.0, exceptions=None)

Try calling function (with no arguments) at most try_times as long as it keeps raising exception.

If exceptions is not None, it specifies which exception types should be silenced. If an exception has been raised, wait delay seconds before retrying.

class pylablib.core.utils.general.SilenceException(exceptions=None, on_exception=None, reraise=False)

Bases: object

Context which silences exceptions raised in a block of code.

Parameters:

• exceptions (Exception or list) – A single exception class or a list of exception classes which are going to be silenced.
• on_exception (callable) – A callback to be invoked if an exception occurs.
• reraise (bool) – Defines if the exception is re-rased after the callback has been invoked.

A simple bit of syntax sugar. The code:

with SilenceException(exceptions,on_exception,reraise):
    ... do stuff ...

is exactly analogous to:

try:
    ... do stuff ...
except exceptions:
    on_exception()
    if reraise:
        raise

pylablib.core.utils.general.full_exit(code=<Signals.SIGTERM: 15>)

Terminate the current process and all of its threads.

Doesn’t perform any cleanup or resource release; should only be used if the process is irrevocably damaged.

pylablib.core.utils.general.topological_order(graph, visit_order=None)

Get a topological order of a graph.

Return a list of nodes where each node is listed after its children. If visit_order is not None, it is a list specifying nodes visiting order (nodes earlier in the list are visited first). Otherwise, the visit order is undefined. graph is a dictionary {node: [children]}. If graph contains loops, raise ValueError.

class pylablib.core.utils.general.UIDGenerator(thread_safe=False)

Bases: object

Generator of unique numeric IDs.

Parameters:thread_safe (bool) – If True, using lock to ensure that simultaneous calls from different threads are handled properly.

class pylablib.core.utils.general.NamedUIDGenerator(name_template='{0}{1:03d}', thread_safe=False)

Bases: object

Generator of unique string IDs based on a name.

Parameters:

• name_template (str) – Format string with two parameters (name and numeric ID) used to generate string IDs.
• thread_safe (bool) – If True, using lock to ensure that simultaneous calls from different threads are handled properly.

pylablib.core.utils.general.call_every(func, times=1, cooldown=0.0, default=None)

Wrap func such that calls to it are forwarded only under certain conditions.

If times>1, then func is called after at least times calls to the wrapped function. If cooldown>0, then func is called after at least cooldown seconds passed since the last call. If both conditions are specified, they should be satisfied simultaneously. default specifies return value if func wasn’t called.

pylablib.core.utils.general.call_limit(func, times=1, cooldown=0.0, limit=None, default=None)

Wrap func such that calls to it are forwarded only under certain conditions.

If times>1, then func is called after at least times calls to the wrapped function. If cooldown>0, then func is called after at least cooldown seconds passed since the last call. if limit is not None, then func is called only first limit times. If several conditions are specified, they should be satisfied simultaneously. default specifies return value if func wasn’t called. Returned function also has an added method reset, which resets the internal call and time counters.

pylablib.core.utils.general.doc_inherit(parent)

Wrapper for inheriting docstrings from parent classes.

Takes parent class as an argument and replaces the docstring of the wrapped function by the docstring of the same-named function from the parent class (if available).

class pylablib.core.utils.general.Countdown(timeout)

Bases: object

Object for convenient handling of timeouts and countdowns with interrupts.

Parameters:timeout (float) – Countdown timeout; if None, assumed to be infinite.

reset()

time_left(bound_below=True)

Return the amount of time left. For infinite timeout, return None.

If bound_below==True, instead of negative time return zero.

time_passed()

Return the amount of time passed since the countdown start/reset.

passed()

Check if the timeout has passed.

class pylablib.core.utils.general.Timer(period, skip_first=False)

Bases: object

Object for keeping time of repeating tasks.

Parameters:period (float) – Timer period.

change_period(period, method='current')

Change the timer period.

method specifies the changing method. Could be "current" (change the period of the ongoing tick), "next" (change the period starting from the next tick), "reset_skip" (reset the timer and skip the first tick) or "reset_noskip" (reset the timer and don’t skip the first tick).

reset(skip_first=False)

Reset the timer.

If skip_first==False, timer ticks immediately; otherwise, it starts ticking only after one period.

time_left(t=None, bound_below=True)

Return the amount of time left before the next tick.

If bound_below==True, instead of negative time return zero.

passed(t=None)

Return the number of ticks passed.

If timer period is zero, always return 1.

acknowledge(n=None, nmin=0)

Acknowledge the timer tick.

n specifies the number of tick to acknowledge (by default, all passed). Return number of actually acknowledged ticks (0 if the timer hasn’t ticked since the last acknowledgement).

class pylablib.core.utils.general.StreamFileLogger(path, stream=None, lock=None, autoflush=False)

Bases: object

Steam logger that replaces standard output stream (usually stdout or stderr) and logs them into a file.

Parameters:

• path – path to the destination logfile. The file is always appended.
• stream – an optional output stream into which the output will be duplicated; usually, the original stream which is being replaced
• lock – a thread lock object, which is used for any file writing operation; necessary if replacing standard streams (such as sys.stdout or sys.stderr) in a multithreading environment.
• autoflush – if True, flush after any write operation into stream

It is also possible to subclass the file and overload write_header() method to write a header before the first file write operation during the execution.

The intended use is to log stdout or stderr streams:

import sys, threading
sys.stderr = StreamFileLogger("error_log.txt", stream=sys.stderr, lock=threading.Lock())

write_header(f)

Write header to file stream f

add_path(path)

Add another logging path to the list

remove_path(path)

Remove logging path to the list

write(s)

flush()

pylablib.core.utils.general.setbp()

pylablib.core.utils.general.wait_for_keypress(message='Waiting...')

pylablib.core.utils.ipc module

Universal interface for inter-process communication.

Focus on higher throughput for large numpy arrays via shared memory.

class pylablib.core.utils.ipc.IIPCChannel

Bases: object

Generic IPC channel interface

send(data)

Send data

recv(timeout=None)

Receive data

send_numpy(data)

Send numpy array

recv_numpy(timeout=None)

Receive numpy array

get_peer_args()

Get arguments required to create a peer connection

classmethod from_args(*args)

Create a peer connection from teh supplied arguments

class pylablib.core.utils.ipc.TPipeMsg(id, data)

Bases: tuple

data

id

class pylablib.core.utils.ipc.PipeIPCChannel(pipe_conn=None)

Bases: pylablib.core.utils.ipc.IIPCChannel

Generic IPC channel interface using pipe.

get_peer_args()

Get arguments required to create a peer connection

send(data)

Send data

recv(timeout=None)

Receive data

class pylablib.core.utils.ipc.SharedMemIPCChannel(pipe_conn=None, arr=None, arr_size=None)

Bases: pylablib.core.utils.ipc.PipeIPCChannel

Generic IPC channel interface using pipe and shared memory for large arrays.

get_peer_args()

Get arguments required to create a peer connection

send_numpy(data, method='auto', timeout=None)

Send numpy array

recv_numpy(timeout=None)

Receive numpy array

class pylablib.core.utils.ipc.TShmemVarDesc(offset, size, kind, fixed_size)

Bases: tuple

fixed_size

kind

offset

size

class pylablib.core.utils.ipc.SharedMemIPCTable(pipe_conn=None, arr=None, arr_size=None, lock=True)

Bases: object

Shared memory table for exchanging shared variables between processes.

Can be used instead of channels for variables which are rarely changed but frequently checked (e.g., status), or when synchronization of sending and receiving might be difficult

add_variable(name, size, kind='pickle')

Add a variable with a given name.

The variable info is also communicated to the other endpoint. size determines maximal variable size in bytes. If the actual size ever exceeds it, an exception will be raised. kind determines the way to convert variable into bytes; can be "pickle" (universal, but large size overhead), "nps_###"` (where ### can be any numpy scalar dtype description, e.g., "float" or "<u2") for numpy scalars, or "npa_###"` (where ### means the same as for nps) for numpy arrays (in this case the array size and shape need to be communicated separately).

set_variable(name, value)

Set a variable with a given name.

If the variable is missing, raise an exception.

get_variable(name, default=None)

Get a variable with a given name.

If the variable is missing, return default.

is_peer_connected()

Check if the peer is connected (i.e., the other side of the pipe is initialized)

close_connection()

Mark the connection as closed

is_peer_closed()

Check if the peer is closed

get_peer_args()

Get arguments required to create a peer connection

classmethod from_args(*args)

Create a peer connection from teh supplied arguments

pylablib.core.utils.iterator module

Class for building common iterators

class pylablib.core.utils.iterator.AccessIterator(obj, access_function=None)

Bases: object

Simple sequential access iterator with customizable access function (by default it’s 1D indexing).

Determines end of iterations by IndexError.

Parameters:

• obj – Container to be iterated over.
• access_function (callable) – A function which takes two parameteres obj and idx and either returns the element or raises IndexError. By default, a simple __getitem__ operation.

next()

pylablib.core.utils.library_parameters module

Storage for global library parameters

class pylablib.core.utils.library_parameters.LibraryParametersStorage(root_name)

Bases: object

Global library parameters storage.

Parameters:root_name (str) – name of the root module

On creation goes through the root module and all of its submodules and checks if any have '_module_parameters' variable. If they do, this variable is interpreted as a dictionary with parameters from that module. All of the found dictionaries are stored together and the contained parameters can be transparently read and changed.

refresh()

Repeat the modules scan (should be called if any of the modules are reloaded)

update(d)

Update parameters with the supplied dictionary

pylablib.core.utils.log module

Logging class that incorporates making file and console logs. The actions taken can be altered for different message types and level ranges.

pylablib.core.utils.log.normalize_level(level)

class pylablib.core.utils.log.ILogAction

Bases: object

The interface for a log action.

report(message='', origin=None, kind=None, level='note', display=None, continued=False)

Report an event.

Parameters:

• message (str) – log message
• origin (str) – origin of the message. Can be anything, generally specifies position in the code/runtime
• kind (str) – kind of the message. Can be anything, predefined kinds are: 'progress' (general execution flow), 'debug' (debugging messages), 'unexpected' (unexpected situation), 'info' (generic information), 'exectime' (execution timing)
• level (int or str) – severity of the event. Can be an integer between 0 and 50, or one of the predefined levels: 'misc' (0), 'note' (10), 'warning' (20), 'error' (30), 'failure' (40)
• display (str or tuple) – message display parameters, which overrides default class parameters. List of symbols specifying which elements show up in the log record: 't' (event time), 'l' (level), 'k' (kind), 'o' (origin) If display is a tuple, it defines two formats: for a full record, and for a continued record (added after a record with continued==True)
• continued (bool) – if True, the event is assumed to be incomplete, and the next report will continue the current record (applicable to console or file logging, but not error raising)

class pylablib.core.utils.log.LogAction_None

Bases: pylablib.core.utils.log.ILogAction

Log action that does nothing.

report(message='', origin=None, kind=None, level='note', display=None, continued=False)

Report an event.

Parameters:

• message (str) – log message
• origin (str) – origin of the message. Can be anything, generally specifies position in the code/runtime
• kind (str) – kind of the message. Can be anything, predefined kinds are: 'progress' (general execution flow), 'debug' (debugging messages), 'unexpected' (unexpected situation), 'info' (generic information), 'exectime' (execution timing)
• level (int or str) – severity of the event. Can be an integer between 0 and 50, or one of the predefined levels: 'misc' (0), 'note' (10), 'warning' (20), 'error' (30), 'failure' (40)
• display (str or tuple) – message display parameters, which overrides default class parameters. List of symbols specifying which elements show up in the log record: 't' (event time), 'l' (level), 'k' (kind), 'o' (origin) If display is a tuple, it defines two formats: for a full record, and for a continued record (added after a record with continued==True)
• continued (bool) – if True, the event is assumed to be incomplete, and the next report will continue the current record (applicable to console or file logging, but not error raising)

class pylablib.core.utils.log.IMessageLogAction(display=('lko', ''), min_display=('', ''))

Bases: pylablib.core.utils.log.ILogAction

A generic log action that generates a text message.

Operation performed on the text is defined in subclass by overloading report_text().

Parameters:

• display (str or tuple) – default display format (see report() for details)
• min_display (str or tuple) – default “minimal” display format (applied to all messages regardless of other display parameters)

format_time(time=None)

Build a time string.

format_level(level)

Build a level string.

format_kind(kind)

Build a kind string.

format_origin(origin)

Build an origin string.

special_format(message)

Build a format string.

format_message(message='', origin=None, kind=None, level='note', display=None)

Build a report string.

Parameters are the same as report(), except for the missing continuing parameter.

report_text(txt)

Report the generated text.

To be overloaded in subclasses.

report(message='', origin=None, kind=None, level='note', display=None, continued=False)

Report an event.

Parameters:

• message (str) – log message
• origin (str) – origin of the message. Can be anything, generally specifies position in the code/runtime
• kind (str) – kind of the message. Can be anything, predefined kinds are: 'progress' (general execution flow), 'debug' (debugging messages), 'unexpected' (unexpected situation), 'info' (generic information), 'exectime' (execution timing)
• level (int or str) – severity of the event. Can be an integer between 0 and 50, or one of the predefined levels: 'misc' (0), 'note' (10), 'warning' (20), 'error' (30), 'failure' (40)
• display (str or tuple) – message display parameters, which overrides default class parameters. List of symbols specifying which elements show up in the log record: 't' (event time), 'l' (level), 'k' (kind), 'o' (origin) If display is a tuple, it defines two formats: for a full record, and for a continued record (added after a record with continued==True)
• continued (bool) – if True, the event is assumed to be incomplete, and the next report will continue the current record (applicable to console or file logging, but not error raising)

class pylablib.core.utils.log.LogAction_Console

Bases: pylablib.core.utils.log.IMessageLogAction

The log action that generates a report text and writes it to the console.

report_text(txt)

Write the message to the console.

class pylablib.core.utils.log.LogAction_File(path=None, display=('tlko', ''), min_display=('tlko', ''))

Bases: pylablib.core.utils.log.IMessageLogAction

The log action that generates a report text and writes it to a log file.

Parameters:

• path (str) – path to the log file. The file is always appended.
• display (str or tuple) – default display format (see IMessageLogAction.report() for details)
• min_display (str or tuple) – default “minimal” display format (applied to all messages regardless of other display parameters)

report_text(txt)

Write the message to the log file.

exception pylablib.core.utils.log.LogError(message)

Bases: Exception

The standard log exception. Raised on LogAction_Exception action.

class pylablib.core.utils.log.LogAction_Exception(display='tlko')

Bases: pylablib.core.utils.log.IMessageLogAction

The log action that generates a report text and raises a LogError exception with it.

report_text(txt)

Report the generated text.

To be overloaded in subclasses.

class pylablib.core.utils.log.Log(enabled=True)

Bases: object

Log object.

Handles dispatching of report to appropriate actions according to the set rules (see add_rule()).

Actions can be added using add_named_action(). The pre-existing actions are:

• 'console' (prints the message to the console, see LogAction_Console)
• 'file' (writes the message to a log file, see LogAction_File)
• 'exception' (raises message as an exception, see LogAction_Exception).

Parameters:enabled (bool) – determines if the logger is enabled by default (a disabled logger never invokes any actions).

enable(enabled=True)

Enable or disable the log.

add_named_action(name, action)

Add an action to the log under the given name

get_named_action(name)

Get the action with the given name

set_logfile(path, name='file')

Set the log file path of the log action with the given name.

add_rule(action, origin='', kind='*', level_range=None, **kwargs)

Add a reporting rule.

Parameters:

• action (str) – name of the action invoked
• origin (str) – origin prefix specifying the rule domain (by default, empty prefix, i.e., any origin)
• kind (str) – a kind name or '*' (all kinds; default) for which the action is invoked
• level_range (tuple) – a range (lower and upper bound) of levels for which the action is invoked
• **kwargs – additional arguments passed to the action report method (override the default action arguments, but are overridden by the report arguments)

If several rules are applicable to the same message, all of them are invoked.

remove_rule(action=None, origin='', kind='*', level_range=None)

Remove a reporting rule.

Parameters:

• action (str) – name of the action to be removed
• origin (str) – origin prefix specifying the rule domain (by default, empty prefix, i.e., any origin)
• kind (str) – a kind name or '*' (all kinds; default) for which the action is removed
• level_range (tuple) – a range (lower and upper bound) of levels for which the action will be removed

report(message='', origin=None, kind=None, level='note', display=None, continued=False, action=None)

Report an event.

Parameters:

• message (str) – log message
• origin (str) – origin of the message. Can be anything, generally specifies position in the code/runtime
• kind (str) – kind of the message. Can be anything, predefined kinds are: 'progress' (general execution flow), 'debug' (debugging messages), 'unexpected' (unexpected situation), 'info' (generic information), 'exectime' (execution timing)
• level (int or str) – severity of the event. Can be an integer between 0 and 50, or one of the predefined levels: 'misc' (0), 'note' (10), 'warning' (20), 'error' (30), 'failure' (40)
• display (str or tuple) – message display parameters, which overrides default class parameters. List of symbols specifying which elements show up in the log record: 't' (event time), 'l' (level), 'k' (kind), 'o' (origin) If display is a tuple, it defines two formats: for a full record, and for a continued record (added after a record with continued==True)
• continued (bool) – if True, the event is assumed to be incomplete, and the next report will continue the current record (applicable to console or file logging, but not error raising)

error(message='', origin=None, kind='unexpected', level='error', display=None, continued=False, action=None)

Generate an error report (same as report(), but with different default arguments).

prog(message='', origin=None, kind='progress', level='note', display=('o', ''), continued=False, action=None)

Generate a progress report (same as report(), but with different default arguments).

progc(message='', origin=None, kind='progress', level='note', display=('o', ''), continued=True, action=None)

Generate a continued progress report (same as report(), but with different default arguments).

info(message='', origin=None, kind='info', level='note', display=('o', ''), continued=False, action=None)

Generate an info report (same as report(), but with different default arguments).

infoc(message='', origin=None, kind='info', level='note', display=('o', ''), continued=True, action=None)

Generate a continued info report (same as report(), but with different default arguments).

debug(message='', origin=None, kind='debug', level='note', display=('o', ''), continued=False, action=None)

Generate a debug report (same as report(), but with different default arguments).

debugc(message='', origin=None, kind='debug', level='note', display=('o', ''), continued=True, action=None)

Generate a continued debug report (same as report(), but with different default arguments).

etime(message='', origin=None, kind='exectime', level='note', display=('to', ''), continued=False, action=None)

Generate an execution time report (same as report(), but with different default arguments).

etimec(message='', origin=None, kind='exectime', level='note', display=('to', ''), continued=True, action=None)

Generate a continued execution time report (same as report(), but with different default arguments).

enable_kind(kind, enable=True, level=0, action='console', **kwargs)

pylablib.core.utils.log.default_log = <pylablib.core.utils.log.Log object>

Default logger.

Raises exception on any reports with level=='error'; prints to console on any reports with kind=='unexpected'.

pylablib.core.utils.log.read_log(path, strict=True, required=None, bad_line_action='append')

Read the log file.

Return a list of lines, where each line is a dictionary {'time':time, 'level':level, 'kind':kind, 'origin':origin, 'message':message'}. If some elements of the line are missing from the message, set them to None. Try to convert the time string into a datetime.datetime object and the level string into an integer.

Parameters:

• path (str) – path to the file.
• strict (bool) – if True and time or level can’t be converted, treat the line as invalid
• required (list) – list of required non-None line items to consider it valid
• bad_line_action (str) – if 'ingore', ignore all bad lines; if 'append', append bad lines to messages of the previous valid lines (i.e., assume that the messages can span several lines)

pylablib.core.utils.module module

Library for dealing with python module properties.

pylablib.core.utils.module.get_package_version(pkg)

Get the version of the package.

If the package version is unavailable, return None.

pylablib.core.utils.module.cmp_versions(ver1, ver2)

Compare two package versions.

Return '<' if the first version is older (smaller), '>' if it’s younger (larger) or '=' if it’s the same.

pylablib.core.utils.module.cmp_package_version(pkg, ver)

Compare current package version to ver.

ver should be a name of the package (rather than the module). Return '<' if current version is older (smaller), '>' if it’s younger (larger) or '=' if it’s the same. If the package version is unavailable, return None.

pylablib.core.utils.module.expand_relative_path(module_name, rel_path)

Turn a relative module path into an absolute one.

module_name is the absolute name of the reference module, rel_path is the path relative to this module.

pylablib.core.utils.module.get_loaded_package_modules(pkg_name)

Get all modules in the package pkg_name.

Returns a dict {name: module}.

pylablib.core.utils.module.get_reload_order(modules)

Find reload order for modules which respects dependencies (a module is loaded before its dependants).

modules is a dict {name: module}.

The module dependencies (i.e., the modules which the current module depends on) are described in the variable _depends_local defined at its toplevel (missing variable means no dependencies).

pylablib.core.utils.module.reload_package_modules(pkg_name, ignore_errors=False)

Reload package pkg_name, while respecting dependencies of its submodules.

If ignore_errors=True, ignore ImportError exceptions during the reloading process.

pylablib.core.utils.module.unload_package_modules(pkg_name, ignore_errors=False)

Reload package pkg_name, while respecting dependencies of its submodules.

If ignore_errors=True, ignore ImportError exceptions during the reloading process.

pylablib.core.utils.module.get_library_path()

Get a filesystem path for the pyLabLib library (the one containing current the module).

pylablib.core.utils.module.get_library_name()

Get the name for the pyLabLib library (the one containing current the module).

pylablib.core.utils.module.pip_install(pkg, upgrade=True)

Call pip install for a given package.

If upgrade==True, call with --upgrade key (upgrade current version if it is already installed).

pylablib.core.utils.module.install_if_older(pkg, min_ver='')

Install pkg from the default PyPI repository if its version is lower that min_ver

If min_ver is None, upgrade to the newest version regardless; if min_ver=="", install only if no version is installed

pylablib.core.utils.net module

A wrapper for built-in TCP/IP routines.

exception pylablib.core.utils.net.SocketError

Bases: OSError

Base socket error class.

exception pylablib.core.utils.net.SocketTimeout

Bases: pylablib.core.utils.net.SocketError

Socket timeout error.

pylablib.core.utils.net.get_local_addr()

Get local IP address.

pylablib.core.utils.net.get_all_local_addr()

Get a list of all local IP address.

pylablib.core.utils.net.get_local_hostname()

Get a local host name.

class pylablib.core.utils.net.ClientSocket(sock=None, timeout=None, wait_callback=None, send_method='decllen', recv_method='decllen', datatype='auto', nodelay=False)

Bases: object

A client socket (used to connect to a server socket).

Parameters:

• sock (socket.socket) – If not None, use already created socket.
• timeout (float) – The timeout used for connecting and sending/receiving (None means no timeout).
• wait_callback (callable) – Called periodically (every 100ms by default) while waiting for connecting or sending/receiving.
• send_method (str) – Default sending method.
• recv_method (str) – Default receiving method.
• datatype (str) – Type of the returned data; can be "bytes" (return bytes object), "str" (return str object), or "auto" (default Python result: str in Python 2 and bytes in Python 3)
• nodelay (bool) – Whether to enable TCP_NODELAY.

Possible sending/receiving methods are:

• 'fixedlen': data is sent as is, and receiving requires to know the length of the message;
• 'decllen': data is prepended by a length, and receiving reads this length and doesn’t need predetermined length info.

sock

Correpsonding Python socket.

Type:socket.socket

decllen_bo

Byteorder of the prependend length for 'decllen' sending method. Can be either '>' (big-endian, default) or '<'.

Type:str

decllen_ll

Length of the prependend length for 'decllen' sending method; default is 4 bytes.

Type:int

set_wait_callback(wait_callback=None)

Set callback function for waiting during connecting or sending/receiving.

set_timeout(timeout=None)

Set timeout for connecting or sending/receiving.

get_timeout()

Get timeout for connecting or sending/receiving.

using_timeout(timeout=None)

Context manager for usage of a different timeout inside a block.

connect(host, port)

Connect to a remote host.

close()

Close the connection.

is_connected()

Check if the connection is opened

get_local_name()

Return IP address and port of this socket.

get_peer_name()

Return IP address and port of the peer socket.

recv_fixedlen(l)

Receive fixed-length message of length l.

recv_delimiter(delim, lmax=None, chunk_l=1024, strict=False)

Receive a single message ending with a delimiter delim (can be several characters, or list several possible delimiter strings).

lmax specifies the maximal received length (None means no limit). chunk_l specifies the size of data chunk to be read in one try. If strict==False, keep receiving as much data as possible until a delimiter is found in the end (only works properly if a single line is expected); otherwise, receive the data byte-by-byte and stop as soon as a delimiter is found (equivalent ot setting chunk_l=1).

recv_decllen()

Receive variable-length message (prepended by its length).

Length format is described by decllen_bo and decllen_ll attributes.

recv(l=None)

Receive a message using the default method.

recv_all(chunk_l=1024)

Receive all of the data currently in the socket.

chunk_l specifies the size of data chunk to be read in one try. For technical reasons, use 1ms timeout (i.e., this operation takes 1ms).

recv_ack(l=None)

Receive a message using the default method and send an acknowledgement (message length).

send_fixedlen(msg)

Send a message as is.

send_decllen(msg)

Send a message as a variable-length (prepending its length in the sent message).

Length format is described by decllen_bo and decllen_ll attributes.

send_delimiter(msg, delimiter)

Send a message with a delimiter delim (can be several characters).

send(msg)

Send a message using the default method.

send_ack(msg)

Send a message using default method and wait for acknowledgement (message length).

If the acknowledgement message length doesn’t agree, raise SocketError.

pylablib.core.utils.net.recv_JSON(socket, chunk_l=1024, strict=True)

Receive a complete JSON token from the socket.

chunk_l specifies the size of data chunk to be read in one try. If strict==False, keep receiving as much data as possible until the received data forms a complete JSON token. otherwise, receive the data byte-by-byte and stop as soon as a token is formed (equivalent ot setting chunk_l=1).

pylablib.core.utils.net.listen(host, port, conn_func, port_func=None, wait_callback=None, timeout=None, backlog=10, wrap_socket=True, connections_number=None, socket_args=None)

Run a server socket at the given host and port.

Parameters:

• host (str) – Server host address. If None, use the local host defined by socket.gethostname().
• port (int) – Server port. If 0, generate an arbitrary free port.
• conn_func (callable) – Called with the client socket as a single argument every time a connection is established.
• port_func (callable) – Called with the port as a single argument when the listening starts (useful with port=0).
• wait_callback (callable) – A callback function which is called periodically (every 100ms by default) while awaiting for connections.
• timeout (float) – Timeout for waiting for the connections (None is no timeout).
• backlog (int) – Backlog length for the socket (see socket.socket.listen()).
• wrap_socket (bool) – If True, wrap the client socket of the connection into ClientSocket class; otherwise, return socket.socket object.
• connections_number (int) – Specifies maximal number of connections before the listening function returns (by default, the number is unlimited).
• socket_args (dict) – parameters passed to ClientSocket constructor.

Checking for connections is paused until conn_func returns. If multiple connections are expected, conn_func should spawn a separate processing thread and return.

pylablib.core.utils.numclass module

Contains utilities for simplifying emulation of numerical classes.

pylablib.core.utils.numclass.iadd(a, b)

pylablib.core.utils.numclass.isub(a, b)

pylablib.core.utils.numclass.imul(a, b)

pylablib.core.utils.numclass.idiv(a, b)

pylablib.core.utils.numclass.ipow(a, b)

pylablib.core.utils.numclass.ifloordiv(a, b)

pylablib.core.utils.numclass.imod(a, b)

pylablib.core.utils.numclass.ilshift(a, b)

pylablib.core.utils.numclass.irshift(a, b)

pylablib.core.utils.numclass.iand(a, b)

pylablib.core.utils.numclass.ior(a, b)

pylablib.core.utils.numclass.ixor(a, b)

class pylablib.core.utils.numclass.NumClass

Bases: object

Simplifies numerical class emulation.

Defines all of the built-in arithmetic functions and generic methods for handling them.

The most generic method is

_perform_numop(x, op_func, op_sym)

Perform a numeric operation.

Abstract method, has to be overloaded in subclasses.

Parameters:

• x – A second argument (None if the operation is unary).
• op_func (callable) – A default function which performs the operation (e.g., lambda (x, y): x + y for addition).
• op_sym (str) – A symbol for the operation.

The other more specialized methods are (by default they all invoke _perform_numop()):

_perform_numop_l()

Binary operations with the object as the first argument (e.g., obj + x).

_perform_numop_r()

Binary operations with the object as the second argument (e.g., x + obj).

_perform_numop_i()

Binary in-place operations (e.g., obj += x).

_perform_numop_u()

Unary operations (e.g., -obj).

_perform_numop_comp()

Binary comparison operation (e.g., obj < x).

A class-level set _numops_impl contains symbols for operations which are implemented. By default, the operations not contained in _numops_impl raise NotImplementedError.

The operation symbols are:

• arithmetic binary operations with the object as the first argument:

  'l+', 'l-', 'l*', 'l/', 'l**', 'l//', 'l%' (contained in constant _numops_arith_l)

• arithmetic binary operations with the object as the second argument:

  'r+', 'r-', 'r*', 'r/', 'r**', 'r//', 'r%' (contained in constant _numops_arith_r)

• arithmetic binary in-place operations:

  'i+', 'i-', 'i*', 'i/', 'i**', 'i//', 'i%' (contained in constant _numops_arith_i)

• arithmetic unary operations:

  'u+', 'u-', 'uabs' (contained in constant _numops_arith_u)

• bitwise binary operations with the object as the first argument:

  'l<<', 'l>>', 'l&', 'l|', 'l^' (contained in constant _numops_bitwise_l)

• bitwise binary operations with the object as the second argument:

  'r<<', 'r>>', 'r&', 'r|', 'r^' (contained in constant _numops_bitwise_r)

• bitwise binary in-place operations:

  'i<<', 'i>>', 'i&', 'i|', 'i^' (contained in constant _numops_bitwise_i)

• bitwise unary operations:

  'u~' (contained in constant _numops_bitwise_u)

• comparison operations:

  '<', '<=', '>', '>=', '==', '!=' (contained in constant _numpos_comp)

pylablib.core.utils.numerical module

Numerical functions that don’t deal with sequences (those are contained in waveforms.general).

pylablib.core.utils.numerical.gcd(*numbers)

Euclid’s algorithm for GCD. Arguments are cast to integer.

pylablib.core.utils.numerical.integer_distance(x)

Get distance to the closes integer

pylablib.core.utils.numerical.gcd_approx(a, b, min_fraction=1e-08, tolerance=1e-05)

Approximate Euclid’s algorithm for possible non-integer values.

Try to find a number d such that a/d and b/d are less than tolerance away from a closest integer. If GCD becomes less than min_fraction * min(a, b), raise ArithmeticError.

pylablib.core.utils.numerical.round_significant(x, n)

Rounds x to n significant digits (not the same as n decimal places!).

pylablib.core.utils.numerical.limit_to_range(x, min_val=None, max_val=None, default=0)

Confine x to the given limit.

Default limit values are None, which means no limit.

class pylablib.core.utils.numerical.infinite_list(start=0, step=1)

Bases: object

Mimics the behavior of the usual list, but is infinite and immutable.

Supports accessing elements, slicing (including slices giving infinite lists) and iterating. Iterating over it naturally leads to an infinite loop, so it should only be used either for finite slices or for loops with break condition.

Parameters:

• start – The first element of the list.
• step – List step.

class counter(lst)

Bases: object

next()

pylablib.core.utils.numerical.unity()

Return a unity function

pylablib.core.utils.numerical.constant(c)

Return a function which returns a constant c.

c can only be either a scalar, or an array-like object with the shape matching the expected argument.

pylablib.core.utils.numerical.polynomial(coeffs)

Return a polynomial function which with coefficients coeffs.

Coefficients are list lowest-order first, so that coeffs[i] is the coefficient in front of x**i.

pylablib.core.utils.observer_pool module

A simple observer pool (notification pool) implementeation.

class pylablib.core.utils.observer_pool.ObserverPool(expand_tuple=True)

Bases: object

An observer pool.

Stores notification functions (callbacks), and calls them whenever notify() is called. The callbacks can have priority (higher priority ones are called first) and filter (observer is only called if the filter function passes the notification tag).

Parameters:expand_tuple (bool) – if True and the notification value is a tuple, treat it as an argument list for the callback functions.

class Observer(filt, callback, priority, attr, cacheable)

Bases: tuple

attr

cacheable

callback

filt

priority

add_observer(callback, name=None, filt=None, priority=0, attr=None, cacheable=False)

Add the observer callback.

Parameters:

• callback (callable) – callback function; takes at least one argument (notification tag), and possible more depending on the notification value.
• name (str) – stored callback name; by default, a unique name is auto-generated
• filt (callable or None) – a filter function for this observer (the observer is called only if the notify() function tag and value pass the filter); by default, all tags are accepted
• priority (int) – callback priority; higher priority callback are invoked first.
• attr – additional observer attributes (can be used by ObserverPool subclasses to change their behavior).
• cacheable (bool) – if True, assumes that the filter function only depends on the tag, so its calls can be cached.

Returns:

callback name (equal to name if supplied; an automatically generated name otherwise).

remove_observer(name)

Remove the observer callback with the given name.

find_observers(tag, value)

notify(tag, value=())

Notify the obserevers by calling their callbacks.

Return a dictionary of the callback results. By default the value is an empty tuple: for expand_tuple==True this means that only one argument (tag) is passed to the callbacks.

pylablib.core.utils.plotting module

Utilities for matplotlib plotting.

class pylablib.core.utils.plotting.IRecurrentPlot(fig=None, auto_clear=True, auto_relim=True, auto_layout=True)

Bases: object

Recurrent plot.

Can be used to plot multiple similar datasets in the same plot.

First ploting call creates figure and axes (calling plot_prepare() method); all consecutive calls only change the data (calling plot_next() method). This way the figure is preserved between the plotting calls, which decreases resource consumption and minimizes matplotlib memory leaks.

Parameters:

• fig (matplotlib.figure.Figure) – If not None, the figure to use for plotting.
• auto_clear (bool) – If True, clear plot (empty data) before each subsequent plotting.
• auto_relim (bool) – If True, rescale plot after each plotting.
• auto_layout (bool) – If True, call tight_layout after each plotting.

plot_prepare(*args, **vargs)

Prepare plot.

Abstract method, has to be overloaded in subclasses.

Called once before the first plotting happens.

plot_clear()

Clear the ploted data.

plot_next(*args, **kwargs)

Plot data.

Abstract method, has to be overloaded in subclasses.

Called every time the data is updated.

setup_figure()

Create a figure if it hasn’t been created already.

setup_prepare(*args, **kwargs)

Prepare the plot if it hasn’t been prepared already.

plot(*args, **kwargs)

Plot the data.

The supplied arguments are redirected to the overloaded methods plot_prepare() and plot_next().

savefig(path, *args, **kwargs)

Save the figure to the location defined by path.

Arguments are passed to matplotlib.figure.Figure.savefig().

close()

Clear and close the figure.

class pylablib.core.utils.plotting.GenericPlotter(axes_num=1, plots_num=1, axes_names=None, log_x=False, log_y=False, xlabel='', ylabel='', legend=None, xscale=1.0, yscale=1.0, fig=None)

Bases: pylablib.core.utils.plotting.IRecurrentPlot

Generic multi-axes plotter.

Parameters:

• axes_num (int) – Number of axes in the figure (all are aligned vertically).
• plots_num – Number of plot lines; can be either an integer (all plots have the same number of lines) or an integer list of length axes_num.
• axes_names ([str]) – Names of axes for referencing in plotting (ordered integers by default).
• log_x/log_y – Use log scale for x/y axes; can be either a single bool (all plots have the same scale) or list of a bool list of length axes_num.
• xlabel/ylabel – Labels for for x/y axes; can be either a single string (all plots have the same axes labels) or a string list with length axes_num.
• legend ([str]) – Plot legends
• xscale/yscale – Scales for x/y axes (supplied data is multiplied by these scales before plotting); can be either a single float (all axes have the same scale) or list of floats with length axes_num (all plots in the same axes have the same scale) or list of lists of floats (specifies scale for each line in each plot separately)
• fig (matplotlib.figure.Figure) – If not None, the figure to use for plotting.

plot_prepare(*args, **kwargs)

Prepare the plot if it hasn’t been prepared already.

plot_next(data, legend=None)

Plot data.

Data is a list of lists [axes_num][plot_num] of 1D or 2D 2-columns array.

pylablib.core.utils.plotting.add_all_subplots(fig, r, c=None, *args, **vargs)

pylablib.core.utils.plotting.iterlabels(obj, include=['axes', 'ticks', 'title'])

Iterate over text labels in obj.

Parameters:

• obj – can be a single Axes or a Figure (in which case iteration goes over all contained axes).
• include ([str]) – determines which kind of labels are iterated over. Can contain "axes" (axes label), "ticks" (axes tick labels) and "title" (plot title).

pylablib.core.utils.plotting.plot_func(func, rng, *args, **kwargs)

Plot a callable function over a given range.

rng is a tuple (start, stop, points_number) passed to numpy.linspace() to generate plot points. The rest of the arguments is the same as in matplotlib.pyplot.plot().

pylablib.core.utils.plotting.plot_columns(data, x_column, y_column, *args, **kwargs)

Plot two data columns vs each other.

pylablib.core.utils.pstorage module

Implements names lifting feature (works both for attributes and for methods), similar to the standard class inheritance. The goal is solely to simplify syntax (shortcut access to parameters deep in the subarguments tree). Multiple inclusion is allowed (e.g., A->B->C and A->D->C), but recursive containment (A->B->C->A) is not.

class pylablib.core.utils.pstorage.ParametersStorage

Bases: object

The basic class that stores local parameters and references to other (imported) storages.

class pylablib.core.utils.pstorage.ParametersStorageInterface

Bases: pylablib.core.utils.pstorage.ParametersStorage

An interface of the ParametersStorage.

Lets access parameters as if they were attributes or container items, and implements __dir__ method for autocomplete. By default, the attribute interface (s.p=0) adds a usual attribute, while the container interface (s["p"]=0) adds a new local parameter.

descriptor_to_storage(desc_name, stored_name=None)

Add a descriptor (usually a property) with the name desc_name to the storage.

If stored_name is not None, use it as the name of the parameter inside the storage.

pylablib.core.utils.pstorage.init(init_func)

__init__ function decorator.

pylablib.core.utils.py3 module

Dealing with Python2 / Python3 compatibility.

pylablib.core.utils.py3.as_str(data)

pylablib.core.utils.py3.as_bytes(data)

pylablib.core.utils.py3.as_builtin_bytes(data)

pylablib.core.utils.py3.as_datatype(data, datatype)

pylablib.core.utils.rpyc module

Routines and classes related to RPyC package

pylablib.core.utils.rpyc.obtain(proxy, serv=None)

Obtain a remote netfref object by value (i.e., copy it to the local Python instance).

Wrapper around rpyc.utils.classic.obtain() with some special cases handling. serv specifies the current remote service. If it is of type SocketTunnelService, use its socket tunnel for faster transfer.

pylablib.core.utils.rpyc.transfer(obj, serv)

Send a local object to the remote PC by value (i.e., copy it to the remote Python instance).

A ‘reversed’ version of obtain().

class pylablib.core.utils.rpyc.SocketTunnelService(server=False)

Bases: sphinx.ext.autodoc.importer._MockObject

Extension of the standard rpyc.core.service.SlaveService with built-in network socket tunnel for faster data transfer.

In order for the tunnel to work, services on both ends need to be subclasses of SocketTunnelService. Because of the initial setup protocol, the two services are asymmetric: one should be ‘server’ (corresponding to the listening server), and one should be ‘client’ (external connection). The roles are decided by the server constructor parameter.

tunnel_send(obj, packer=None)

Send data through the socket tunnel.

If packer is not None, it defines a function to convert obj to a bytes string.

tunnel_recv(unpacker=None)

Receive data sent through the socket tunnel.

If unpacker is not None, it defines a function to convert the received bytes string into an object.

obtain(proxy)

Execute obtain() on the local instance

transfer(obj)

Execute transfer() on the local instance

on_connect(conn)

on_disconnect(conn)

class pylablib.core.utils.rpyc.DeviceService(verbose=False)

Bases: pylablib.core.utils.rpyc.SocketTunnelService

Device RPyC service.

Expands on SocketTunnelService by adding get_device() method, which opens local devices, tracks them, and closes them automatically on disconnect.

on_connect(conn)

on_disconnect(conn)

get_device(module, cls, *args, **kwargs)

Connect to a device.

cls and module are names of the device class and the containing module (for module name the "pylablib.aux_libs.devices" prefix can be omitted)

pylablib.core.utils.rpyc.run_device_service(port=18812, verbose=False)

Start DeviceService at the given port

pylablib.core.utils.rpyc.connect_device_service(addr, port=18812, timeout=3, attempts=2)

Connect to the DeviceService running at the given address and port

timeout and attempts define respectively timeout of a single connection attempt, and the number of attempts (RPyC default is 3 seconds timeout and 6 attempts).

pylablib.core.utils.serializable module

Mixing class for converting object into dict structure. If an attribute of an object is also serializable, it is going to be added to the top level of the dict. Avoid recursive attributes (x.a=y; y.b=x), since they will lead to errors while trying to load an object. Avoid recursive containers (x[0]=y; y[0]=x), since they will lead to an infinite loop while serializing. Choice of the attributes to serialize is the same for all objects of the same class

class pylablib.core.utils.serializable.Serializable(name=None)

Bases: object

A serializable object: can be converted to/from a dictionary structure.

pylablib.core.utils.serializable.init_name(object_name_arg='name')

__init__ function decorator.

pylablib.core.utils.serializable.init(init_func)

pylablib.core.utils.serializable.to_dict(objects, full_dict=None, deep_copy=True)

Serialize the list of objects into the dictionary.

Return full_dict. If deep_copy==True, attempt to copy (call .copy() method) all non-serializable attributes before storing them in the dictionary. Only Serializable objects get serialized.

pylablib.core.utils.serializable.from_dict(full_dict, case_sensitive=True, deep_copy=True)

Deserialize objects from the dictionary.

Return a dictionary {name: object} containing the extracted objects. If deep_copy==True, attempt to copy (call .copy() method) all non-serializable attributes before assigning them as objects attributes. Only Serializable objects get deserialized.

pylablib.core.utils.strdump module

Utils for converting variables into standard python objects (lists, dictionaries, strings, etc.) and back (e.g., for a more predictable LAN transfer). Provides an extension for pickle for more customized classes (numpy arrays, Datatable, Dictionary).

class pylablib.core.utils.strdump.StrDumper

Bases: object

Class for dumping and loading an object.

Stores procedures for dumping and loading, i.e., conversion from complex classes (such as Dictionary) to simple built-in classes (such as dict or str).

add_class(cls, dumpf=None, loadf=None, name=None, allow_subclass=True, recursive=False)

Add a rule for dumping/loading an object of class cls.

Parameters:

• cls –
• dumpf (callable) – Function for dumping an object of the class; None means identity function.
• loadf (callable) – Function for loading an object of the class; None means identity function.
• name (str) – Name of class, which is stored in the packed data (cls.__name__ by default).
• allow_subclass (bool) – If True, this rule is also used for subclasses of this class.
• recursive (bool) – If True, the functions are given a second argument, which is a dumping/loading function for their sub-elements.

dump(obj)

Convert an object into a dumped value.

load(obj)

Convert a dumped value into an object.

loads(s)

Convert a pickled string of a damped object into an object.

dumps(obj)

Dumps an object into a pickled string.

pylablib.core.utils.strdump.dumper = <pylablib.core.utils.strdump.StrDumper object>

Default dumper for converting into standard Python classes and pickling.

Converts numpy.ndarray, Dictionary and DataTable objects (these conversion routines are defined when corresponding modules are imported). The converted values include non-printable characters (conversion uses numpy.ma.loads() and numpy.ma.dumps()), so they can’t be saved into text files. However, they’re suited for pickling.

pylablib.core.utils.strdump.dump(obj)

Convert obj into standard Python classes using default dumper.

pylablib.core.utils.strdump.load(s)

Convert standard Python class representation s into an object using default dumper.

pylablib.core.utils.strdump.dumps(obj)

Convert obj into a pickled string using default dumper.

pylablib.core.utils.strdump.loads(s)

Convert a pickled string into an object. using default dumper

pylablib.core.utils.string module

String search, manipulation and conversion routines.

pylablib.core.utils.string.string_equal(name1, name2, case_sensitive=True, as_prefix=False)

Determine if name1 and name2 are equal with taking special rules (case_sensitive and as_prefix) into account.

If as_prefix==True, strings match even if name1 is just a prefix of name2.

pylablib.core.utils.string.find_list_string(name, str_list, case_sensitive=True, as_prefix=False, first_matched=False)

Find name in the string list.

Comparison parameters are defined in string_equal(). If first_matched==True, stop at the first match; otherwise if multiple occurrences happen, raise ValueError.

Returns:tuple (index, value).

pylablib.core.utils.string.find_dict_string(name, str_dict, case_sensitive=True, as_prefix=False)

Find name in the string list.

Comparison parameters are defined in string_equal(). If multiple occurrences happen, raise ValueError.

Returns:tuple (key, value).

pylablib.core.utils.string.find_first_entry(line, elements, start=0, not_found_value=-1)

Find the index of the earliest position inside the line of any of the strings in elements, starting from start.

If none are found, return not_found_value.

pylablib.core.utils.string.translate_string_filter(filt, syntax, match_case=True, default=False)

Turns filt into a matching function.

The matching function takes single str argument, returns bool value.

filt can be

• None: function always returns default,
• bool: function always returns this value,
• str: pattern, determined by syntax,
• anything else: returned as is (assumed to already be a callable).

syntax can be 're' (re), 'glob' (glob) or 'pred' (simply matching predicate). match_case determines whether the filter cares about the string case when matching.

class pylablib.core.utils.string.StringFilter(include=None, exclude=None, syntax='re', match_case=False)

Bases: object

String filter function.

Matches string if it matches include (matches all strings by default) and doesn’t match exclude (matches nothing by default).

Parameters:

• include – Inclusion filter (translated by translate_string_filter(); regex by default).
• exclude – Exclusion filter (translated by translate_string_filter(); regex by default).
• syntax – Default syntax for pattern filters.
• match_case (bool) – Determines whether filter ignores case when matching.

pylablib.core.utils.string.get_string_filter(include=None, exclude=None, syntax='re', match_case=False)

Generate StringFilter with the given parameters.

If the first argument is already StringFilter, return as is. If it’s a tuple, expand as argument list.

pylablib.core.utils.string.sfglob(include=None, exclude=None)

Return string filter based on glob syntax.

pylablib.core.utils.string.sfregex(include=None, exclude=None, match_case=False)

Return string filter based on re syntax.

pylablib.core.utils.string.filter_string_list(l, filt)

Filter string list based on the filter.

pylablib.core.utils.string.escape_string(value, location='element', quote_type='"')

Escape string.

Escaping can be partially skipped depending on location:

• "parameter": escape only if it contains hard delimiters ("\n\t\v\r") anywhere

  or _border_escaped (", ' or space) on the sides (suited for parameters taking the full string);

• "entry": same as above, plus containing soft delimiters anywhere (suited for entries of a table);
• "element": always escaped

If quote_type is not None, automatically put the string into the specified quotation marks;

if quote_type is None, all quotation marks are escaped; if it’s not None, only quote_type marks are escaped.

class pylablib.core.utils.string.TConversionClass(label, cls, to_str, from_str)

Bases: tuple

cls

from_str

label

to_str

pylablib.core.utils.string.to_string(value, location='element', custom_representations=None, parenthesis_rules='text', use_classes=False)

Convert value to string with an option of modifying format string.

Parameters:

• value –
• location (str) – Used for converting strings (see escape_string()).
• custom_representations (dict) – dictionary {value_type: fmt}, where value type can be 'int', 'float' or 'complex' and fmt is a str.format() string.
• parenthesis_rules (str) – determine how to deal with single-element tuples and complex numbers can be "text" (single-element tuples are represented with simple parentheses, e.g., "(1)"; complex number are represented without parentheses, e.g., "1+2j") or "python" (single-element tuples are represented with a comma in the end, e.g., "(1,)"; complex number are represented with parentheses, e.g., "(1+2j)")
• use_classes (bool) – if True, use additional representation classes for special objects (e.g., numpy arrays will be represented as "array([1, 2, 3])" instead of just "[1, 2, 3]"). This improves conversion fidelity, but makes result harder to parse (e.g., by external string parsers).

pylablib.core.utils.string.is_convertible(value)

Check if the value can be converted to a string using standard to_string() function.

pylablib.core.utils.string.extract_escaped_string(line, start=0)

Extract escaped string in quotation marks from the line, starting from start.

line[start] should be a quotation mark (' or ") or b followed by a quotation mark (for binary strings).

Returns:tuple (end position, un-escaped string).

pylablib.core.utils.string.unescape_string(value)

Un-escape string.

Assume that all quotation marks have been escaped.

pylablib.core.utils.string.to_range(range_tuple)

pylablib.core.utils.string.from_string(value, case_sensitive=True, parenthesis_rules='text', use_classes=True)

Parse a string.

Recognizes integers, floats, complex numbers (with i or j for complex part), strings (in quotation marks), dicts, sets, list and tuples, booleans and None. If item is unrecognizable, assumed to be a string.

case_sensitive is applied when compared to None, True or False.

parenthesis_rules determine how to deal with empty entries (e.g., [1,,3]) and complex number representation ("1+2j" vs. "(1+2j)"):

• 'text': any empty entries are translated into empty_string (i.e., [,] -> [empty_string, empty_string]), except for completely empty structures ([] or ());

  complex numbers are represented without parentheses, so that "(1+2j)" will be interpreted as a single-element tuple (1+2j,).

• 'python': empty entries in the middle are not allowed; empty entries at the end are ignored (i.e., [2,] -> [2])

  (single-element tuple can still be expressed in two ways: (e,) or (e)); complex numbers are by default represented with parentheses, so that "(1+2j)" will be interpreted as a complex number, and only (1+2j,), ((1+2j)) or ((1+2j),) as a single-element tuple.

use_classes: if True, try to find additional representation classes for special objects

(e.g., numpy arrays will be represented as "array([1, 2, 3])" instead of just "[1, 2, 3]").

pylablib.core.utils.string.from_string_partial(value, delimiters=re.compile('\\s*, \\s*|\\s+'), case_sensitive=True, parenthesis_rules='text', use_classes=True, return_string=False)

Convert the first part of the supplied string (bounded by delimiters) into a value.

delimites is a string or a regexp (default is "\s*,\s*|\s+").

If return_string==False, return tuple (end position, converted value); else, return tuple (end position, value string).

The rest of the parameters is the same as in from_string().

pylablib.core.utils.string.from_row_string(value, delimiters=re.compile('\\s*, \\s*|\\s+'), case_sensitive=True, parenthesis_rules='text', use_classes=True, return_string=False)

Convert the row string into a list of values, separated by delimiters.

If return_string==False, return list of converted objects; otherwise, return list of unconverted strings.

The rest of the parameters is the same as in from_string_partial().

pylablib.core.utils.strpack module

Utilities for packing values into bitstrings. Small extension of the struct module.

pylablib.core.utils.strpack.int2bytes(val, l, bo='>')

Convert integer into a list of bytes of length l.

bo determines byte order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.bytes2int(val, bo='>')

Convert a list of bytes into an integer.

bo determines byte order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.int2bits(val, l, bo='>')

Convert integer into a list of bits of length l.

bo determines byte (and bit) order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.bits2int(val, bo='>')

Convert a list of bits into an integer.

bo determines byte (and bit) order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.pack_uint(val, l, bo='>')

Convert an unsigned integer into a bytestring of length l.

Return bytes object in Py3 and builtins.bytes object in Py2. bo determines byte order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.pack_int(val, l, bo='>')

Convert a signed integer into a bytestring of length l.

Return bytes object in Py3 and builtins.bytes object in Py2. bo determines byte order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.unpack_uint(msg, bo='>')

Convert a bytestring into an unsigned integer.

bo determines byte order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.unpack_int(msg, bo='>')

Convert a bytestring into an signed integer.

bo determines byte order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.unpack_numpy_u12bit(buffer, byteorder='<', count=-1)

pylablib.core.utils.thread module

Simple threading routines.

A more extensive threading library is contained in the core.mthread package.

class pylablib.core.utils.thread.PeriodicThread

Bases: object

A thread that runs in an infinite loop (until externally stopped) and executes its task with a given periodicity.

To use, it needs to be inherited, with the subclass redefining execute() or process_message() method.

execute()

Perform a single iteration of the loop.

To be overridden in subclasses.

process_message(msg)

Process a message sent from the parent thread.

To be overridden in subclasses.

loop(period, sync)

Main loop methods. Called automatically in a new thread when start() is invoked.

wait_for_execution()

Synchronize with the thread (wait until the current iteration is executed).

send_message(msg, sync=True)

Send a message to the thread.

The message is processed by the thread in the process_message() method (by default does nothing). If sync==True, wait until the thread received (not necessarily processed) the message.

start(period, sync=True)

Start the thread.

period specifies the job execution period, in seconds. If sync==True, wait until the thread is started.

stop(sync=True)

Stop the thread.

If sync==True, wait until the thread is stopped.

pause(sync=True)

Pause the thread execution.

If sync==True, wait until the thread is paused.

resume(sync=True)

Resume the thread execution.

If sync==True, wait until the thread is resumed.

is_looping()

Check if the thread is actively executing (not paused).

is_running()

Check if the thread is running (possibly paused).

pylablib.core.utils.versioning module

Fetching, storing and recalling different script and library configurations.

pylablib.core.utils.versioning.generate_version_idx()

Generate a unique string id based on the current time and a random number.

class pylablib.core.utils.versioning.VersionConfig(store_path='.', lib_path='lib', libs=None, additional_scripts=None, versions=None, comments=None)

Bases: object

Current version configuration.

Combines information about included library packages, additional scripts, package versions, and comments.

Parameters:

• store_path (str) – root folder of the configuration
• lib_path (str) – library path inside the root folder
• libs (list or None) – included packages inside the library (any package includes all of its sub-packages); by default, includes the 'core' package and all individual 'aux_libs' sub-packages
• additional_scripts (list or None) – additional scripts (outside of the included libraries); by default, none are included
• versions (dict) – list of versions of versions (e.g., of library packages); by default, none are supplied
• comments (str) – config comment.

to_file(f)

Save the config data to a file.

static from_path(path, **additional_args)

Build the config data from the folder.

additional_args include arguments supplied to the VersionConfig constructor overriding the ones extracted from the config file (if it’s present).

update(other, override=True)

Merge this config with the other one.

Combine information about libraries, comments, and additional scripts. If override==True, the priority versions data is taken from the other config; otherwise, it’s taken from this config.

copy()

update_versions(parts, force_update=True)

Update versions to a new unique idx.

includes(other)

Check if this config includes the other one.

get_lib_path(spec_lib='', relative=False)

Generate a library path in this version config.

common_lib_folders()

Return all of the libs folders, including intermediate containing folders.

common_lib_files(path_type='absolute')

Return all of the files in lib folders and intermediate containing folders (i.e., all files potentially used by libs).

template_script_files()

Get all script files in the main folder.

script_files()

Get all included standalone script files (all script files in the main folder and the additional scripts).

exists()

Check if this configuration exists on the hard drive.

class pylablib.core.utils.versioning.RevisionDiff(libs_diff=None, common_lib_files_diff='=', scripts_diff='=')

Bases: object

Describes the difference between two different script folder states.

Parameters:

• lib_diffs (dict) – dictionary with four entries: - "+" - list of libraries which are added in the first state compared to the second one; - "-" - list of libraries which are removed in the first state compared to the second one; - "*" - list of libraries which are changed in the first state compared to the second one; - "=" - list of libraries which are the same in both states;
• common_lib_files_diff (str) – difference in the common library files (files in the folders containing library subfolders). Can be "+", "-", "*", or "="; the meaning is the same as for lib_diffs.
• scripts_diff (str) – difference in the script files (files outside of the root library folder); the meaning is the same as for common_lib_files_diff.

all_libs_diff()

Combine all library difference into one verdict.

all_diff()

Combine all differences (libraries and files) into one verdict.

coverage(replace=True)

Get the coverage required to store changes of the first revision if it is to be replaced with the second one.

Return either 'script' (only store scripts, libraries are the same) or 'full' (store both scripts and libraries). If replace==True, assume that the new revision completely replaces the current one; otherwise, assume that it’s added (i.e., if the current revision is larger than the new one, no need to store it).

pylablib.core.utils.versioning.set_history_size(full=None, script=None)

Set the size of the history shelf.

If not None, full and script define the size of the corresponding coverage shelves. Values <=0 mean no limit.

pylablib.core.utils.versioning.get_history_size()

Get the sizes of the history shelves.

Returns tuple (full_history_size, script_history_size). Values <=0 mean no limit.

pylablib.core.utils.versioning.store(shelf, label=None, coverage='full', source_path='.', cleanup_shelf=False, **config_params)

Store the code to the shelf with a given label.

Parameters:

• shelf (str) – Storage shelf (folder).
• label (str) – Snapshot label to be appended to the default filename (date and time of creation).
• coverage (str) – Either 'script' (include only application scripts) or 'full' (include libraries as well).
• source_path (str) – The code root folder.
• cleanup_shelf (bool) – If True and the shelf size is above the limit, remove the oldest entries.

**config_params are passed to the version config parameters.

pylablib.core.utils.versioning.recall_from_folder(source_path, dest_path='.', replace=False, overwrite=True, source_config_params=None, recall_config_params=None)

Recall code from the source folder to the dest folder.

If the configuration is changed, store the current state on the history shelf before changing it.

Parameters:

• source_path (str) – Path to the source folder.
• dest_path (str) – Path to the destination folder.
• replace (bool) – If True, remove all parts of the dest revision which are different from the source; otherwise, source is added to dest.
• overwrite (bool) – If True, overwrite the old files.
• source_config_params (dict) – Overrides source version config parameters.
• recall_config_params (dict) – Overrides merged version config parameters.

pylablib.core.utils.versioning.recall_from_file(file_name, dest_path='.', replace=False, overwrite=True, source_config_params=None, recall_config_params=None)

Recall code from the zip file.

If the configuration is changed, store the current state on the history shelf before changing it.

Arguments are the same as in recall_from_folder(), only instead of folder path it’s zip file path.

pylablib.core.utils.versioning.recall_from_shelf(shelf, name, coverage, dest_path='.', replace=False, overwrite=True, source_config_params=None, recall_config_params=None)

Recall code from the shelf.

If the configuration is changed, store the current state on the history shelf before changing it.

Arguments are the same as in recall_from_file(), only instead of a path the file is specified by shelf, name and coverage.

pylablib.core.utils.versioning.recall(shelf, name, coverage=None, replace=False, overwrite=True, **recall_config_params)

Alias for recall_from_shelf().

pylablib.core.utils.versioning.fetch(source_path, dest_path='.', overwrite=True, preserve_libs=True, **recall_config_params)

Fetch code library.

If the configuration is changed, store the current state on the history shelf before changing it.

Parameters:

• source_path (str) – Path to the library root.
• dest_path (str) – Path to the destination folder.
• overwrite (bool) – If True, overwrite the old files.
• preserve_libs (bool) – If True, preserve sub-libraries even if they are not specified in the recall_config_params.
• recall_config_params (dict) – Overrides merged version config parameters (see VersionConfig).

pylablib.core.utils.versioning.diff_from_latest(store_path='.', coverage=None)

Compare current revision with the latest item on the history shelf.

Parameters:

• store_path (str) – Path to the revision to check.
• coverage (str) – Defines coverage levels to check. None means both 'script' and 'full'.

pylablib.core.utils.versioning.store_script_if_changed(store_path='.', coverage=None)

Store script files (not library) if they are different from the latest revision.

When storing changes, the shelves sizes are limited, and if the limit is exceeded, the oldest entries are deleted. For more details, see get_history_size() and set_history_size().

Parameters:

• store_path (str) – Path to the revision to check.
• coverage (str) – Defines coverage levels to check. None means both 'script' and 'full'.

pylablib.core.utils.versioning.fetch_and_store_script(source_path=None, dest_path='.', overwrite=True, once_per_run=True, preserve_libs=True, **recall_config_params)

Fetch code library from the source_path (current library path by default); store any recent changes of script or library on the history shelf.

When storing changes, the shelves sizes are limited, and if the limit is exceeded, the oldest entries are deleted. For more details, see get_history_size() and set_history_size().

Parameters:

• source_path (str) – Path to the library root. None means the current library path (library containing this module).
• dest_path (str) – Path to the destination folder.
• overwrite (bool) – If True, overwrite the old files.
• once_per_run (bool) – If True, only perform operation once per run of the script. All subsequent call will return immediately.
• preserve_libs (bool) – If True, preserve sub-libraries even if they are not specified in the recall_config_params.
• recall_config_params (dict) – Overrides merged version config parameters (see VersionConfig).

Module contents

pylablib.core.utils.setbp()