pylablib.core.datatable package

Submodules

pylablib.core.datatable.column module

Single column classes. Used in ColumnDataTableStorage.

class pylablib.core.datatable.column.IDataColumn

Bases: pylablib.core.utils.numclass.NumClass

expand(length)

Expand column. Usually fill with zeros, unless the column values can be auto-predicted.

subcolumn(idx, force_copy=False)

Return value(s) at idx as a column object.

insert(idx, val)

Add a value val to the column at a position given by idx.

append(val)

Append a value to the column

as_array(force_copy=False)

Turn the column into a numpy array

copy()

shape

nrows()

ndim

classmethod add_array_function(func, alias=None, wrap_into_column=False, as_property=False, doc=None)

Turns a function into a method, which is automatically applied to the array representation.

Parameters:

• func (callable) – a function which takes the column converted into a numpy array as a first argument, and then the rest if the supplied arguments
• alias (str) – the method name; by default, it’s func.__name__
• wrap_into_column (bool) – if True, the returned result is wrapped into ArrayDataColumn
• as_property (bool) – if True, the function is added as a property getter instead
• doc (str) – the method docstring; by default, it’s func.__doc__

argmax(*args, **vargs)

Same as numpy.argmax().

argmin(*args, **vargs)

Same as numpy.argmin().

argsort(*args, **vargs)

Same as numpy.argsort().

conjugate(*args, **vargs)

Same as numpy.conj.

cut_out_regions(regions, x_column=None, ordered=False, multi_pass=True)

Cut the regions out of the wf based on x_column.

x_column is used to determine which colmn’s values to use to check if the point is in range (see waveforms.get_x_column()). If ordered_x==True, then the function assumes that x_column in ascending order. If multi_pass==False, combine all indices before deleting the data in a single operation (works faster, but only for non-intersecting regions).

cut_to_range(xs_range, x_column=None, ordered=False)

Cut the waveform to the given range based on x_column.

The range is defined as xs_range[0]:xs_range[1], or infinite if xs_range=None. x_column is used to determine which colmn’s values to use to check if the point is in range (see waveforms.get_x_column()). If ordered_x==True, then the function assumes that x_column in ascending order.

decimate(n=1, dec_mode='skip', axis=0, mode='drop')

Decimate the data.

Parameters:

• wf – Data.
• n (int) – Decimation factor.
• dec_mode (str) – Decimation mode. Can be - 'skip' - just leave every n’th point while completely omitting everything else; - 'bin' or 'mean' - do a binning average; - 'sum' - sum points; - 'min' - leave min point; - 'max' - leave max point; - 'median' - leave median point (works as a median filter).
• axis (int) – Axis along which to perform the decimation.
• mode (str) – Determines what to do with the last bin if it’s incomplete. Can be either 'drop' (omit the last bin) or 'leave' (keep it).

find_closest_arg(*args, **vargs)

Find the index of a value in xs that is closest to x.

approach can take values 'top', 'bottom' or 'both' and denotes from which side should array elements approach x (meaning that the found array element should be >x, <x or just the closest one). If there are no elements lying on the desired side of x (e.g. approach=='top' and all elements of xs are less than x), the function returns None. if ordered==True, then xs is assumed to be in ascending or descending order, and binary search is implemented (works only for 1D arrays). if there are recurring elements, return any of them.

get_range_indices(xs_range, ordered=False)

Find waveform indices correspoding to the given range.

The range is defined as xs_range[0]:xs_range[1], or infinite if xs_range=None (so the data is returned unchanged in that case). If ordered_x==True, then the function assumes that x_column in ascending order.

imag

Same as numpy.imag().

is_ascending(*args, **vargs)

Check the if waveform is ascending.

If it has more than 1 dimension, check all lines along 0’th axis.

is_descending(*args, **vargs)

Check if the waveform is descending.

If it has more than 1 dimension, check all lines along 0’th axis.

is_linear(*args, **vargs)

Check if the waveform is linear (values go with a constant step).

If it has more than 1 dimension, check all lines along 0’th axis (with the same step for all).

is_ordered()

Check if the waveform is ordered (ascending or descending).

If it has more than 1 dimension, check all lines along 0’th axis.

max(*args, **vargs)

Same as numpy.amax().

mean(*args, **vargs)

Same as numpy.mean().

min(*args, **vargs)

Same as numpy.amin().

nonzero(*args, **vargs)

Same as numpy.nonzero().

real

Same as numpy.real().

sliding_filter(n=1, dec_mode='bin', mode='reflect', cval=0.0)

Perform sliding filtering on the data.

Parameters:

• wf – 1D array-like object.
• n (int) – bin width.
• dec_mode (str) –

  Decimation mode. Can be

  • 'bin' or 'mean' - do a binning average;
  • 'sum' - sum points;
  • 'min' - leave min point;
  • 'max' - leave max point;
  • 'median' - leave median point (works as a median filter).

• mode (str) – Expansion mode. Can be 'constant' (added values are determined by cval), 'nearest' (added values are endvalues of the waveform), 'reflect' (reflect waveform wrt its endpoint) or 'wrap' (wrap the values from the other size).
• cval (float) – If mode=='constant', determines the expanded values.

std(*args, **vargs)

Same as numpy.std().

sum(*args, **vargs)

Same as numpy.sum().

unique(*args, **vargs)

Same as numpy.unique().

class pylablib.core.datatable.column.WrapperDataColumn(column)

Bases: pylablib.core.datatable.column.IDataColumn

Wraps potentially mutable column types and proxies all the requests to them.

Used when the underlying column object can change in the runtime (e.g., to accommodate a new element type).

insert(idx, val)

Only supports adding at a single specific location.

expand(length)

Expand column. Usually fill with zeros, unless the column values can be auto-predicted.

subcolumn(idx, force_copy=False, wrap=True)

Return value(s) at idx as a column object.

as_array(force_copy=False)

Turn the column into a numpy array

copy()

shape

real

Same as numpy.real().

imag

Same as numpy.imag().

conjugate()

Return the complex-conjugate of the column

class pylablib.core.datatable.column.IStoredDataColumn

Bases: pylablib.core.datatable.column.IDataColumn

Abstract class to distinguish data columns with stored data, as opposed to the ones with generated data.

class pylablib.core.datatable.column.ArrayDataColumn(column)

Bases: pylablib.core.datatable.column.IStoredDataColumn

Column which stores its data in a numpy array.

Automatically expands the stored data type (int -> float -> complex) if needed.

copy()

shape

class pylablib.core.datatable.column.ListDataColumn(column)

Bases: pylablib.core.datatable.column.IStoredDataColumn

Column which stores its data in a list.

subcolumn(idx, force_copy=False)

Return value(s) at idx as a column object.

as_array(force_copy=False)

Turn the column into a numpy array

copy()

shape

real

Same as numpy.real().

imag

Same as numpy.imag().

conjugate()

Return the complex-conjugate of the column

class pylablib.core.datatable.column.LinearDataColumn(length, start=0, step=1)

Bases: pylablib.core.datatable.column.IDataColumn

A linear data column.

Represents a linear data (essentially, a range object). Doesn’t store the full data, just the length, start and step. Automatically increments upon expansion.

expand(length)

Expand column. Usually fill with zeros, unless the column values can be auto-predicted.

subcolumn(idx, force_copy=False)

Return value(s) at idx as a column object.

as_array(force_copy=False)

Turn the column into a numpy array

copy()

shape

real

Same as numpy.real().

imag

Same as numpy.imag().

conjugate()

Return the complex-conjugate of the column

decimate(n=1, dec_mode='skip', axis=0, mode='drop')

Decimate a linear column data.

For parameters, see filters.decimate().

find_closest_arg(x, approach='both', ordered=None)

get_range_indices(xs_range, ordered=None)

is_ascending()

is_descending()

is_linear()

pylablib.core.datatable.column.as_linear_column(column)

Try and turn a column into a linear column.

If it is not linear, raise ValueError.

pylablib.core.datatable.column.as_column(column, force_numpy=True, try_linear=False, force_copy=False)

Turn an object into a column.

If column is a list and force_numpy==True, turn it into a numpy array and return ArrayDataColumn (by default lists are wrapped into ListDataColumn). If try_linear==True, try to represent it as a linear data first. If force_copy==True, create the copy of the data.

pylablib.core.datatable.column.crange(*args)

[start,] stop[, step]

Analogue of range which creates a linear data column.

pylablib.core.datatable.column.zeros(length)

Create a column of the given length filled with zeros.

pylablib.core.datatable.datatable_utils module

pylablib.core.datatable.datatable_utils.as_array(data, force_copy=False, try_object=True)

Turn data into a numpy array.

If force_copy==True, copy the data if it’s already a numpy array. If try_object==False, only try to convert to numerical numpy arrays; otherwise, generic numpy arrays (with dtype=="object") are acceptable.

pylablib.core.datatable.datatable_utils.get_shape(data, strict=False)

Get the data shape.

If the data is a nested list and strict==True, raise an error unless all sublists have the same length (i.e., the data is rectangular).

pylablib.core.datatable.indexing module

Processing and normalization of different indexing styles.

pylablib.core.datatable.indexing.string_list_idx(names_to_find, names_list, only_exact=False)

Index through a list of strings in names_list.

Return corresponding numerical indices. Case sensitive; first look for exact matching, then for prefix matching (unless only_exact=True).

pylablib.core.datatable.indexing.is_slice(idx)

Check if idx is slice.

pylablib.core.datatable.indexing.is_range(idx)

Check if idx is iterable (list, numpy array, or builtins.range).

pylablib.core.datatable.indexing.is_bool_array(idx)

Check if idx is a boolean array.

pylablib.core.datatable.indexing.to_range(idx, length)

Turn list, array, builtins.range, slice into an iterable.

pylablib.core.datatable.indexing.covers_all(idx, length, strict=False, ordered=True)

Check if idx covers all of the elements (indices from 0 to length).

If strict==True, strictly checks the condition; otherwise may return False even if idx actually covers everything, but takes less time (i.e., can be used for optimization). If ordered==True, only returns True when indices follow in order.

class pylablib.core.datatable.indexing.IIndex

Bases: object

A generic index object.

Used to transform a variety of indexes into a subset applicable for specific objects (numpy arrays or lists).

Allowed input index types:

• scalar: integer, string
• vector: integer lists or numpy arrays, bool lists or numpy arrays, string lists or numpy arrays, builtin.ranges, slices and string slices

tup()

Represent index as a tuple for easy unpacking.

class pylablib.core.datatable.indexing.NumpyIndex(idx, ndim=None)

Bases: pylablib.core.datatable.indexing.IIndex

NumPy compatible index: allows for integers, slices, numpy integer or boolean arrays, integer lists or builtin.ranges

pylablib.core.datatable.indexing.to_numpy_idx(idx)

class pylablib.core.datatable.indexing.ListIndex(idx, names=None, ndim=None)

Bases: pylablib.core.datatable.indexing.IIndex

List compatible index: allows for integers, slices, numpy integer arrays, integer lists or builtin.ranges;

pylablib.core.datatable.indexing.to_list_idx(idx, names=None)

class pylablib.core.datatable.indexing.ListIndexNoSlice(idx, names=None, length=None, ndim=None)

Bases: pylablib.core.datatable.indexing.ListIndex

List compatible index with slice unwrapped into bultin.range: allows for integers, numpy integer arrays, integer lists or builtin.ranges;

pylablib.core.datatable.indexing.to_list_idx_noslice(idx, names=None, length=None)

pylablib.core.datatable.indexing.to_double_index(idx, names)

pylablib.core.datatable.table module

class pylablib.core.datatable.table.DataTable(data=None, column_names=None, transposed='auto', force_copy=True, storage_type=None)

Bases: object

A data table which is designed to store data in several named heterogeneous columns.

Differences from the regular numpy array:

• The columns have names which can be used for indexing; this proves to be more convenient for large (10+ columns) datatables
• Different columns can have different types, so, e.g., non-numeric datatypes (such as str) still allow to preserve the rest of the data as numeric
• Automatic typecasting for assignment (e.g., if the table starts as int, part of it can be assigned float without re-creating the whole table)
• Various useful methods: insert, append, modified.

Parameters:

• data – table data; can be a numpy array, a list of columns, a 2D list, or a dict of data columns (in which case column_names determines which columns to get from the dict)
• column_names (list) – list of column names; by default, the column names are autogenerated: "col00", "col01", etc.
• transposed – if True, the columns arguments is assumed to be column-wise (list of columns) if False, the columns arguments is assumed to be row-wise (list of rows) if "auto", assumed to be False for numpy arrays and True otherwise
• force_copy (bool) – if True, make sure that the supplied data is copied
• storage_type (str) – determines the type of underlying DataTable storage: 'columns' (default) stores each column separately in an IDataColumn object; 'array' stores all the data in a 2D numpy array (limited functionality, but faster execution)

shape

ndim

nrows()

Get number of rows.

ncols()

Get number of columns.

idx()

Create an index column (analogous to numpy.arange(len(table))).

as_array(force_copy=False)

Turn the table into a numpy array.

If force_copy==True, ensure that the result is a copy of the data.

as_pandas(force_copy=False)

Turn the table into a pandas DataFrame.

If force_copy==True, ensure that the result is a copy of the data.

class RowAccessor(storage, return_numpy=True)

Bases: object

A row accessor: creates a simple interface to treat the table row-wise (append/insert/delete/iterate over rows).

Generated automatically for each table on creation, doesn’t need to be created explicitly.

insert(idx, val)

Insert a row or a list of rows to index idx (1D).

append(val)

Append a row or a list of rows to the end of the table.

idx()

Create an index column (analogous to numpy.arange(len(table))).

class ColumnAccessor(storage, return_numpy=True)

Bases: object

A column accessor: creates a simple interface to treat the table column-wise (append/insert/delete/iterate over columns).

Generated automatically for each table on creation, doesn’t need to be created explicitly.

keys()

Get keys (column names)

iterkeys()

Iterate over keys (column names)

itervalues()

Iterate over values (IDataColumn objects)

iteritems()

Iterate over items (tuples of column names and corresponding IDataColumn objects)

insert(idx, val, names=None, transposed='auto')

Add new columns at index idx (1D).

Columns data is given by val and their names are given by names (a string for a single column, or a list of strings for multiple columns). If transposed==True, val is assumed to be arranged column-wise (list of columns). If transposed==False, val is assumed to be arranged row-wise (list of rows). If transposed=="auto", it is assumed to be True if val is a 2D numpy array, and False otherwise.

append(val, names=None, transposed='auto')

Append a column or a list of column to the end of the table.

See IDataTableStorage.add_columns() for description.

move(source, dest)

Move a column with the source index to the dest position (should be integer).

The column names are adjusted accordingly.

idx()

Create an index row (analogous to numpy.arange(table.shape[1])).

class TableAccessor(storage, return_numpy=True)

Bases: object

A table accessor: acessing the table data through this interface returns DataTable objects (acessing DataTable object directly returns numpy objects for better compatibility).

Generated automatically for each table on creation, doesn’t need to be created explicitly.

idx()

expand(length)

Expand the table by length. Usually fill with zeros, unless the column values can be auto-predicted.

resize(length)

Resize the table to length.

If the current length is larger, the table is truncated. Otherwise, it’s expanded (using expand()).

get_column_names(idx=None)

Return the list of column names at index idx (by default, all of the names).

set_column_names(new_names)

Return the list of column names.

change_column_names(idx, val)

Change names of columns at index idx.

swap_columns(idx1, idx2)

Swap two columns at indices idx1 and idx2.

Names are adjusted accordingly.

rearrange_columns(idx)

Rearrange columns according to the new index (analogous to replacing the table by its subtable given by index idx).

modified(idx, val, names=None, force_copy=True)

Return a copy of this table with the columns at index idx modified.

val can be either a single number or column (if idx is a single index), or a list of numbers or columns (if idx is a list of indices). names specifices new names of modified columns (by default the old names are preserved). If force_copy==True, all of the table data is deep-copied.

set_x_column_name(idx='#')

Set the dedicated x column.

Can be either a valid column name (or numeric index), or "#" which means the index column (returned by idx()).

get_x_column_name()

Get the name of the dedicated.

get_x_column(x_column=None, idx_default=False)

Get the dedicated x column.

copy(copy_columns=True)

Copy the table.

If y_columns==True, deep-copy the columns as well.

concatenate(table, diff_columns_action='exclude', fill_value=nan)

classmethod add_array_function(func, alias=None, doc=None)

Turns a function into a method, which is automatically applied to the array representation.

Parameters:

• func (callable) – a function which takes the column converted into a numpy array as a first argument, and then the rest if the supplied arguments
• alias (str) – the method name; by default, it’s func.__name__
• doc (str) – the method docstring; by default, it’s func.__doc__

classmethod add_columnwise_function(func, alias=None, collection_type='list', column_arg_name=None, doc=None)

Add function to the class definition, which is automatically applied to each column.

Parameters:

• func (callable) – a function which takes the column converted into a numpy array as a first argument, and then the rest if the supplied arguments
• alias (str) – the method name; by default, it’s func.__name__
• collection_type (str) – determines the type of the result; can be "list", "array", or "table"
• column_arg_name (str) – name of the column argument supplied to the function, in which case it is only applied to this specified column (by default, the function has no such argument)
• doc (str) – the method docstring; by default, it’s func.__doc__

classmethod add_column_function(func, alias=None, column_arg_name='column', column_arg_default=None, doc=None)

Add function to the class definition, which is automatically applied to a single column. Column number should be given as a first argument of the function.

Parameters:

• func (callable) – a function which takes the column converted into a numpy array as a first argument, and then the rest if the supplied arguments
• alias (str) – the method name; by default, it’s func.__name__
• column_arg_name (str) – name of the column argument supplied to the function, in which case it is only applied to this specified column
• column_arg_default – default name of the column, if no column argument os supplied to the function
• doc (str) – the method docstring; by default, it’s func.__doc__

argmax(*args, **vargs)

Same as numpy.argmax().

argmin(*args, **vargs)

Same as numpy.argmin().

argsort(*args, **vargs)

Same as numpy.argsort().

cut_out_regions(regions, x_column=None, ordered=False, multi_pass=True)

Cut the regions out of the wf based on x_column.

x_column is used to determine which colmn’s values to use to check if the point is in range (see waveforms.get_x_column()). If ordered_x==True, then the function assumes that x_column in ascending order. If multi_pass==False, combine all indices before deleting the data in a single operation (works faster, but only for non-intersecting regions).

cut_to_range(xs_range, x_column=None, ordered=False)

Cut the waveform to the given range based on x_column.

The range is defined as xs_range[0]:xs_range[1], or infinite if xs_range=None. x_column is used to determine which colmn’s values to use to check if the point is in range (see waveforms.get_x_column()). If ordered_x==True, then the function assumes that x_column in ascending order.

decimate(*args, **vargs)

Decimate the data.

Parameters:

• wf – Data.
• n (int) – Decimation factor.
• dec_mode (str) – Decimation mode. Can be - 'skip' - just leave every n’th point while completely omitting everything else; - 'bin' or 'mean' - do a binning average; - 'sum' - sum points; - 'min' - leave min point; - 'max' - leave max point; - 'median' - leave median point (works as a median filter).
• axis (int) – Axis along which to perform the decimation.
• mode (str) – Determines what to do with the last bin if it’s incomplete. Can be either 'drop' (omit the last bin) or 'leave' (keep it).

filter_by(columns=None, pred=None, exclude=False)

Filter 1D or 2D array using a predicate.

If the data is 2D, columns contains indices of columns to be passed to the pred function. If exclude==False, drop all of the rows satisfying pred rather than keep them.

max(*args, **vargs)

Same as numpy.amax().

mean(*args, **vargs)

Same as numpy.mean().

min(*args, **vargs)

Same as numpy.amin().

nonzero(*args, **vargs)

Same as numpy.nonzero().

sliding_filter(*args, **vargs)

Perform sliding filtering on the data.

Parameters:

• wf – 1D array-like object.
• n (int) – bin width.
• dec_mode (str) –

  Decimation mode. Can be

  • 'bin' or 'mean' - do a binning average;
  • 'sum' - sum points;
  • 'min' - leave min point;
  • 'max' - leave max point;
  • 'median' - leave median point (works as a median filter).

• mode (str) – Expansion mode. Can be 'constant' (added values are determined by cval), 'nearest' (added values are endvalues of the waveform), 'reflect' (reflect waveform wrt its endpoint) or 'wrap' (wrap the values from the other size).
• cval (float) – If mode=='constant', determines the expanded values.

sort_by(x_column=None, reverse=False, stable=False)

Sort 2D array using selected column as a key and preserving rows.

If reverse==True, sort in descending order. x_column values are described in waveforms.get_x_column(). If stable==True, use stable sort (could be slower and uses more memory)

std(*args, **vargs)

Same as numpy.std().

sum(*args, **vargs)

Same as numpy.sum().

unique(*args, **vargs)

Same as numpy.unique().

unique_slices(u_column)

Split a table into subtables with different values in a given column.

Return a list of wf subtables, each of which has a different (and equal among all rows in the subtable) value in u_column.

pylablib.core.datatable.table_storage module

Data table storage. Does not implement any indexing or iterator interface (it’s delegated to IDataTable). Should not be accessed directly by users of DataTable.

class pylablib.core.datatable.table_storage.IDataTableStorage

Bases: object

shape

ndim

as_array(force_copy=False)

Turn the storage into a numpy array.

If force_copy==True, ensure that the result is a copy of the data.

as_pandas(force_copy=False)

Turn the storage into a pandas DataFrame.

If force_copy==True, ensure that the result is a copy of the data.

get_item(idx)

Return the data at the index idx (1D or 2D) as a numpy array.

set_item(idx, val)

Return the data at the index idx (1D or 2D) to val.

get_columns(idx)

Return a column or a list of columns at the index idx (1D).

get_single_column(idx)

Return a single column at the index idx (1D).

Same as get_columns(), but only accepts single column index.

set_columns(idx, val)

Set a column or a list of columns at the index idx (1D) to val.

add_columns(idx, val, names, transposed, force_copy=False)

Add new columns at index idx (1D).

Columns data is given by val and their names are given by names (a string for a single column, or a list of strings for multiple columns). If transposed==True, val is assumed to be arranged column-wise (list of columns). If transposed==False, val is assumed to be arranged row-wise (list of rows). If transposed=="auto", it is assumed to be True if val is a 2D numpy array, and False otherwise. If force_copy==True, make sure that val data is copied.

del_columns(idx)

Delete a column or a list of columns at the index idx (1D)

get_rows(idx)

Return a row or a list of rows at the index idx (1D).

Each row is represented as a tuple.

get_single_row(idx)

Return a single row at the index idx (1D) as a tuple.

Same as get_rows(), but only accepts single column index.

get_single_row_item(idx)

Return a single row at the index idx (1D) as a numpy array.

Same as get_item(), but only accepts single column index.

set_rows(idx, val)

Set a row or a list of rows at the index idx (1D) to val.

add_rows(idx, val)

Add new rows at index idx (1D).

del_rows(idx)

Delete a row or a list of rows at the index idx (1D)

get_subtable(idx)

Return the data at the index idx (1D or 2D) as an IDataTableStorage object of the same type.

expand(length)

Expand the table by length. Usually fill with zeros, unless the column values can be auto-predicted.

get_column_names(idx=None)

Return the list of column names at index idx (by default, all of the names).

get_column_indices(idx=None)

Return the list of column numerical indices corresponding to the index idx.

set_column_names(new_names)

Set new column names.

swap_columns(idx1, idx2)

Swap two columns at indices idx1 and idx2.

copy()

class pylablib.core.datatable.table_storage.ColumnDataTableStorage(columns=None, names=None, transposed='auto', force_copy=True)

Bases: pylablib.core.datatable.table_storage.IDataTableStorage

Table storage which stores the data as a list of columns (defined in datatable.column).

More flexible compared to the ArrayDataTableStorage, but potentially slower.

Parameters:

• columns – table data; can be a numpy array, a list of columns, or a 2D list
• names (list) – list of column names; by default, the column names are autogenerated: "col00", "col01", etc.
• transposed – if True, the columns arguments is assumed to be column-wise (list of columns) if False, the columns arguments is assumed to be row-wise (list of rows) if "auto", assumed to be False for numpy arrays and True otherwise
• force_copy (bool) – if True, make sure that the supplied data is copied

shape

as_pandas(force_copy=False)

Turn the storage into a pandas DataFrame.

If force_copy==True, ensure that the result is a copy of the data.

get_column_names(idx=None)

Return the list of column names.

get_column_indices(idx=None)

Return the list of column numerical indices corresponding to the index idx.

set_column_names(new_names)

Set new column names.

swap_columns(idx1, idx2)

Swap two columns at indices idx1 and idx2.

copy()

get_item(idx)

Return the data at the index idx (1D or 2D) as a numpy array.

set_item(idx, val)

Return the data at the index idx (1D or 2D) to val.

get_columns(idx)

Return a column or a list of columns at the index idx (1D).

get_single_column(idx)

Return a single column at the index idx (1D).

Same as get_columns(), but only accepts single column index.

set_columns(idx, val, force_copy=False)

Set a column or a list of columns at the index idx (1D) to val.

add_columns(idx, val, names, transposed='auto', force_copy=False)

Add new columns at index idx (1D).

Columns data is given by val and their names are given by names (a string for a single column, or a list of strings for multiple columns). If transposed==True, val is assumed to be arranged column-wise (list of columns). If transposed==False, val is assumed to be arranged row-wise (list of rows). If transposed=="auto", it is assumed to be False if val is a 2D numpy array, and True otherwise. If force_copy==True, make sure that val data is copied.

del_columns(idx)

Delete a column or a list of columns at the index idx (1D)

get_rows(idx)

Return a row or a list of rows at the index idx (1D).

Each row is represented as a tuple.

get_single_row(idx)

Return a single row at the index idx (1D) as a tuple.

Same as get_rows(), but only accepts single column index.

get_single_row_item(idx)

Return a single row at the index idx (1D) as a numpy array.

Same as get_item(), but only accepts single column index.

add_rows(idx, val)

Add new rows at index idx (1D).

del_rows(idx)

Delete a row or a list of rows at the index idx (1D)

get_subtable(idx, force_copy=False)

Return the data at the index idx (1D or 2D) as an IDataTableStorage object of the same type.

expand(length)

Expand the table. Usually fill with zeros, unless the column values can be auto-predicted.

class pylablib.core.datatable.table_storage.ArrayDataTableStorage(columns=None, names=None, transposed='auto', force_copy=True)

Bases: pylablib.core.datatable.table_storage.IDataTableStorage

Table storage which stores the data as a 2D numpy array.

Faster, but less flexible than ColumnDataTableStorage. Indexing is only numpy-style or column-wise. All columns have the same datatype and are stored in the same array. All columns and rows are returned as numpy arrays.

Parameters:

• columns – table data; can be a numpy array, a list of columns, or a 2D list
• names (list) – list of column names; by default, the column names are autogenerated: "col00", "col01", etc.
• transposed – if True, the columns arguments is assumed to be column-wise (list of columns) if False, the columns arguments is assumed to be row-wise (list of rows) if "auto", assumed to be False for numpy arrays and True otherwise
• force_copy (bool) – if True, make sure that the supplied data is copied

shape

get_column_names(idx=None)

Return the list of column names.

get_column_indices(idx=None)

Return the list of column numerical indices corresponding to the index idx.

set_column_names(new_names)

Set new column names.

swap_columns(idx1, idx2)

Swap two columns at indices idx1 and idx2.

copy()

get_item(idx)

Return the data at the index idx (1D or 2D) as a numpy array.

set_item(idx, val)

Return the data at the index idx (1D or 2D) to val.

get_columns(idx)

Return a column or a list of columns at the index idx (1D).

get_single_column(idx)

Return a single column at the index idx (1D).

Same as get_columns(), but only accepts single column index.

set_columns(idx, val, force_copy=False)

Set a column or a list of columns at the index idx (1D) to val.

add_columns(idx, val, names, transposed='auto', force_copy=False)

Add new columns at index idx (1D).

Columns data is given by val and their names are given by names (a string for a single column, or a list of strings for multiple columns). If transposed==True, val is assumed to be arranged column-wise (list of columns). If transposed==False, val is assumed to be arranged row-wise (list of rows). If transposed=="auto", it is assumed to be True if val is a 2D numpy array, and False otherwise. If force_copy==True, make sure that val data is copied.

del_columns(idx)

Delete a column or a list of columns at the index idx (1D)

get_rows(idx)

Return the data at the index idx (1D or 2D) as a numpy array.

get_single_row(idx)

Return a single row at the index idx (1D) as a tuple.

Same as get_rows(), but only accepts single column index.

get_single_row_item(idx)

Return a single row at the index idx (1D) as a numpy array.

Same as get_item(), but only accepts single column index.

add_rows(idx, val)

Add new rows at index idx (1D).

del_rows(idx)

Delete a row or a list of rows at the index idx (1D)

get_subtable(idx, force_copy=True)

Return the data at the index idx (1D or 2D) as an IDataTableStorage object of the same type.

expand(length)

Expand the table. Usually fill with zeros, unless the column values can be auto-predicted.

pylablib.core.datatable.wrapping module

Utilities for uniform treatment of DataTable and numpy array objects for functions which can deal with them both.

class pylablib.core.datatable.wrapping.IGenWrapper(container)

Bases: object

The interface for a wrapper that gives a uniform access to basic methods of wrapped objects’.

get_type()

Get a string representing the wrapped object type

copy(wrapped=True)

Copy the object. If wrapped==True, return a new wrapper contating the object copy; otherwise, just return the copy.

ndim()

shape()

class pylablib.core.datatable.wrapping.I1DWrapper(container)

Bases: pylablib.core.datatable.wrapping.IGenWrapper

A wrapper containing a 1D object (a 1D numpy array or a IDataColumn object).

Provides a uniform access to basic methods of a wrapped object.

class Accessor(wrapper)

Bases: object

An accessor: creates a simple uniform interface to treat the wrapped object element-wise (get/set/iterate over elements).

Generated automatically for each table on creation, doesn’t need to be created explicitly.

subcolumn(idx, wrapped=True)

Return a subcolumn at index idx.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

static from_array(array, force_copy=False, force_numpy=True, try_optimizing=False, wrapped=True)

Build a new object of the type corresponding to the wrapper from the supplied array (a 1D numpy array or a list).

If force_copy==True, make a copy of supplied array. For a Column1DWrapper, if column is a list and force_numpy==True, turn it into a numpy array and return ArrayDataColumn (by default lists are wrapped into ListDataColumn). For a Column1DWrapper, if try_optimizing==True, check if the column can be turned into a LinearDataColumn. If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

classmethod from_columns(columns, column_names=None, wrapped=True)

Build a new object of the type corresponding to the wrapper from the supplied columns (a list of columns; only length-1 lists is supported).

column_names parameter is ignored. If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

array_replaced(array, force_copy=False, force_numpy=True, try_optimizing=False, wrapped=True)

Return a copy of the column with the data replaced by array.

All of the parameters are the same as in from_array().

get_type()

Get a string representing the wrapped object type

copy(wrapped=True)

Copy the object. If wrapped==True, return a new wrapper contating the object copy; otherwise, just return the copy.

ndim()

class pylablib.core.datatable.wrapping.Array1DWrapper(container)

Bases: pylablib.core.datatable.wrapping.I1DWrapper

A wrapper for a 1D numpy array.

Provides a uniform access to basic methods of a wrapped object.

get_deleted(idx, wrapped=True)

Return a copy of the column with the data at index idx deleted.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

get_inserted(idx, val, wrapped=True)

Return a copy of the column with the data val added at index idx.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

insert(idx, val)

Add data val to index idx.

get_appended(val, wrapped=True)

Return a copy of the column with the data val appended at the end.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

append(val)

Append data val to the end.

subcolumn(idx, wrapped=True)

Return a subcolumn at index idx.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

static from_array(array, force_copy=False, force_numpy=True, try_optimizing=False, wrapped=True)

Build a new object of the type corresponding to the wrapper from the supplied array (a 1D numpy array or a list).

If force_copy==True, make a copy of supplied array. If wrapped==True, return a new wrapper contating the column; otherwise, just return the column. force_numpy and try_optimizing parameters are ignored.

get_type()

Get a string representing the wrapped object type

copy(wrapped=True)

Copy the object. If wrapped==True, return a new wrapper contating the object copy; otherwise, just return the copy.

class pylablib.core.datatable.wrapping.Column1DWrapper(container)

Bases: pylablib.core.datatable.wrapping.I1DWrapper

A wrapper for an IDataColumn object.

Provides a uniform access to basic methods of a wrapped object.

get_deleted(idx, wrapped=True)

Return a copy of the column with the data at index idx deleted.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

get_inserted(idx, val, wrapped=True)

Return a copy of the column with the data val added at index idx.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

insert(idx, val)

Add data val to index idx.

get_appended(val, wrapped=True)

Return a copy of the column with the data val appended at the end.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

append(val)

Append data val to the end.

subcolumn(idx, wrapped=True)

Return a subcolumn at index idx.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

static from_array(array, force_copy=False, force_numpy=True, try_optimizing=False, wrapped=True)

Build a new object of the type corresponding to the wrapper from the supplied array (a 1D numpy array or a list).

If force_copy==True, make a copy of supplied array. If column is a list and force_numpy==True, turn it into a numpy array and return ArrayDataColumn (by default lists are wrapped into ListDataColumn). If try_optimizing==True, check if the column can be turned into a LinearDataColumn. If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

get_type()

Get a string representing the wrapped object type

copy(wrapped=True)

Copy the object. If wrapped==True, return a new wrapper contating the object copy; otherwise, just return the copy.

class pylablib.core.datatable.wrapping.Series1DWrapper(container)

Bases: pylablib.core.datatable.wrapping.I1DWrapper

A wrapper for a pandas Series object.

Provides a uniform access to basic methods of a wrapped object.

get_deleted(idx, wrapped=True)

Return a copy of the column with the data at index idx deleted.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

get_inserted(idx, val, wrapped=True)

Return a copy of the column with the data val added at index idx.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

get_appended(val, wrapped=True)

Return a copy of the column with the data val appended at the end.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

subcolumn(idx, wrapped=True)

Return a subcolumn at index idx.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

static from_array(array, force_copy=False, force_numpy=True, try_optimizing=False, wrapped=True)

Build a new object of the type corresponding to the wrapper from the supplied array (a 1D numpy array or a list).

If force_copy==True, make a copy of supplied array. If wrapped==True, return a new wrapper contating the column; otherwise, just return the column. force_numpy and try_optimizing parameters are ignored.

get_type()

Get a string representing the wrapped object type

copy(wrapped=True)

Copy the object. If wrapped==True, return a new wrapper contating the object copy; otherwise, just return the copy.

class pylablib.core.datatable.wrapping.I2DWrapper(container, r=None, c=None, t=None)

Bases: pylablib.core.datatable.wrapping.IGenWrapper

A wrapper containing a 2D object (a 2D numpy array or a DataTable object).

Provides a uniform access to basic methods of a wrapped object.

static from_columns(columns, column_names=None, wrapped=True)

Build a new object of the type corresponding to the wrapper from the supplied columns (a list of columns).

column_names supplies names of the columns (only relevant for Table2DWrapper). If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

columns_replaced(columns, wrapped=True)

Return copy of the object with the data replaced by columns.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

static from_array(array, column_names=None, force_copy=False, wrapped=True)

Build a new object of the type corresponding to the wrapper from the supplied array (a list of rows or a 2D numpy array).

column_names supplies names of the columns (only relevant for Table2DWrapper). If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

array_replaced(array, force_copy=False, wrapped=True)

Return a copy of the column with the data replaced by array.

All of the parameters are the same as in from_array().

get_type()

Get a string representing the wrapped object type

copy(wrapped=True)

Copy the object. If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

column(idx, wrapped=True)

Get a column at index idx.

Return a 1D numpy array for a 2D numpy array object, and an IDataColumn object for a DataTable. If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

subtable(idx, wrapped=True)

Return a subtable at index idx.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

ndim()

class pylablib.core.datatable.wrapping.Array2DWrapper(container)

Bases: pylablib.core.datatable.wrapping.I2DWrapper

A wrapper for a 2D numpy array.

Provides a uniform access to basic methods of a wrapped object.

set_container(cont)

class RowAccessor(wrapper, storage)

Bases: object

A row accessor: creates a simple uniform interface to treat the wrapped object row-wise (append/insert/delete/iterate over rows).

Generated automatically for each table on creation, doesn’t need to be created explicitly.

get_deleted(idx, wrapped=True)

Return a new table with the rows at idx deleted.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

get_inserted(idx, val, wrapped=True)

Return a new table with new rows given by val inserted at idx.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

insert(idx, val)

Insert new rows given by val at index idx.

get_appended(val, wrapped=True)

Return a new table with new rows given by val appended to the end of the table.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

append(val)

Insert new rows given by val to the end of the table.

class ColumnAccessor(wrapper, storage)

Bases: object

A column accessor: creates a simple uniform interface to treat the wrapped object column-wise (append/insert/delete/iterate over columns).

Generated automatically for each table on creation, doesn’t need to be created explicitly.

get_deleted(idx, wrapped=True)

Return a new table with the columns at idx deleted.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

get_inserted(idx, val, wrapped=True)

Return a new table with new columns given by val inserted at idx.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

insert(idx, val)

Insert new columns given by val at index idx.

get_appended(val, wrapped=True)

Return a new table with new columns given by val appended to the end of the table.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

append(val)

Insert new columns given by val to the end of the table.

set_names(names)

Set column names (does nothing).

get_names()

Get column names (all names are None).

get_index(idx)

Get number index for a given column index

class TableAccessor(storage)

Bases: object

A table accessor: acessing the table data through this interface returns an object of the appropriate type (numpy array for numpy wrapped object, and DataTable for DataTable wrapped object).

Generated automatically for each table on creation, doesn’t need to be created explicitly.

subtable(idx, wrapped=True)

Return a subtable at index idx of the appropriate type (2D numpy array).

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

column(idx, wrapped=True)

Get a column at index idx as a 1D numpy array.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

static from_columns(columns, column_names=None, wrapped=True)

Build a new object of the type corresponding to the wrapper from the supplied columns (a list of columns).

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table. column_names parameter is ignored.

static from_array(array, column_names=None, force_copy=False, wrapped=True)

Build a new object of the type corresponding to the wrapper from the supplied array (a list of rows or a 2D numpy array).

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table. column_names parameter is ignored.

get_type()

Get a string representing the wrapped object type

copy(wrapped=True)

Copy the object. If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

class pylablib.core.datatable.wrapping.Table2DWrapper(container)

Bases: pylablib.core.datatable.wrapping.I2DWrapper

A wrapper for a DataTable object.

Provides a uniform access to basic methods of a wrapped object.

class RowAccessor(wrapper, storage)

Bases: object

A row accessor: creates a simple uniform interface to treat the wrapped object row-wise (append/insert/delete/iterate over rows).

Generated automatically for each table on creation, doesn’t need to be created explicitly.

get_deleted(idx, wrapped=True)

Return a new table with the rows at idx deleted.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

get_inserted(idx, val, wrapped=True)

Return a new table with new rows given by val inserted at idx.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

insert(idx, val)

Insert new rows given by val at index idx.

get_appended(val, wrapped=True)

Return a new table with new rows given by val appended to the end of the table.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

append(val)

Insert new rows given by val to the end of the table.

class ColumnAccessor(wrapper, storage)

Bases: object

A column accessor: creates a simple uniform interface to treat the wrapped object column-wise (append/insert/delete/iterate over columns).

Generated automatically for each table on creation, doesn’t need to be created explicitly.

get_deleted(idx, wrapped=True)

Return a new table with the columns at idx deleted.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

get_inserted(idx, val, wrapped=True)

Return a new table with new columns given by val inserted at idx.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

insert(idx, val)

Insert new columns given by val at index idx.

get_appended(val, wrapped=True)

Return a new table with new columns given by val appended to the end of the table.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

append(val)

Insert new columns given by val to the end of the table.

set_names(names)

Set column names.

get_names()

Get column names.

get_index(idx)

Get number index for a given column index

class TableAccessor(storage)

Bases: object

A table accessor: acessing the table data through this interface returns an object of the appropriate type (numpy array for numpy wrapped object, and DataTable for DataTable wrapped object).

Generated automatically for each table on creation, doesn’t need to be created explicitly.

subtable(idx, wrapped=True)

Return a subtable at index idx of the appropriate type (DataTable).

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

column(idx, wrapped=True)

Get a column at index idx as an IDataColumn object.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

static from_columns(columns, column_names=None, wrapped=True)

Build a new object of the type corresponding to the wrapper from the supplied columns (a list of columns).

column_names supplies names of the columns (only relevant for Table2DWrapper). If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

static from_array(array, column_names=None, force_copy=False, wrapped=True)

Build a new object of the type corresponding to the wrapper from the supplied array (a list of rows or a 2D numpy array).

column_names supplies names of the columns (only relevant for Table2DWrapper). If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

get_type()

Get a string representing the wrapped object type

copy(wrapped=True)

Copy the object. If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

class pylablib.core.datatable.wrapping.DataFrame2DWrapper(container)

Bases: pylablib.core.datatable.wrapping.I2DWrapper

A wrapper for a pandas DataFrame object.

Provides a uniform access to basic methods of a wrapped object.

class RowAccessor(wrapper, storage)

Bases: object

A row accessor: creates a simple uniform interface to treat the wrapped object row-wise (append/insert/delete/iterate over rows).

Generated automatically for each table on creation, doesn’t need to be created explicitly.

get_deleted(idx, wrapped=True)

Return a copy of the column with the data at index idx deleted.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

get_inserted(idx, val, wrapped=True)

Return a new table with new rows given by val inserted at idx.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

insert(idx, val)

Insert new rows given by val at index idx.

get_appended(val, wrapped=True)

Return a new table with new rows given by val appended to the end of the table.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

append(val)

Insert new rows given by val to the end of the table.

class ColumnAccessor(wrapper, storage)

Bases: object

A column accessor: creates a simple uniform interface to treat the wrapped object column-wise (append/insert/delete/iterate over columns).

Generated automatically for each table on creation, doesn’t need to be created explicitly.

get_deleted(idx, wrapped=True)

Return a new table with the columns at idx deleted.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

get_inserted(idx, val, column_name=None, wrapped=True)

Return a new table with new columns given by val inserted at idx.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

insert(idx, val, column_name=None)

Insert new columns given by val at index idx.

get_appended(val, column_name=None, wrapped=True)

Return a new table with new columns given by val appended to the end of the table.

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

append(val, column_name=None)

Insert new columns given by val to the end of the table.

set_names(names)

Set column names.

get_names()

Get column names.

get_index(idx)

Get number index for a given column index

class TableAccessor(storage)

Bases: object

A table accessor: acessing the table data through this interface returns an object of the appropriate type (numpy array for numpy wrapped object, and DataTable for DataTable wrapped object).

Generated automatically for each table on creation, doesn’t need to be created explicitly.

subtable(idx, wrapped=True)

Return a subtable at index idx of the appropriate type (DataTable).

If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

column(idx, wrapped=True)

Get a column at index idx as an IDataColumn object.

If wrapped==True, return a new wrapper contating the column; otherwise, just return the column.

static from_columns(columns, column_names=None, wrapped=True)

Build a new object of the type corresponding to the wrapper from the supplied columns (a list of columns).

column_names supplies names of the columns (only relevant for DataFrame2DWrapper). If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

static from_array(array, column_names=None, force_copy=False, wrapped=True)

Build a new object of the type corresponding to the wrapper from the supplied array (a list of rows or a 2D numpy array).

column_names supplies names of the columns (only relevant for DataFrame2DWrapper). If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

get_type()

Get a string representing the wrapped object type

copy(wrapped=True)

Copy the object. If wrapped==True, return a new wrapper contating the table; otherwise, just return the table.

pylablib.core.datatable.wrapping.wrap1d(container)

Wrap a 1D container (a 1D numpy array or an IDataColumn) into an appropriate wrapper.

pylablib.core.datatable.wrapping.wrap2d(container)

Wrap a 2D container (a 2D numpy array or a DataTable) into an appropriate wrapper.

pylablib.core.datatable.wrapping.wrap(container)

Wrap container (a numpy array, an IDataColumn or a DataTable) into an appropriate wrapper.

Module contents