helpers.data_handling

Helper functions for handling data.

In order to make the pipeline from lta.helpers.pipeline more testable, most of its constructor and processing functions have been moved here. This makes the code more atomic - and, thus, testable - by removing it from the harder to test context of the object.

lta.helpers.data_handling.construct_df(file: pathlib.Path, n_rows: int, metadata: List[str], index_names: Optional[List[str]] = None, **kwargs: Any) → pandas.core.frame.DataFrame

Construct a dataframe from the given path.

A light wrapper to handing the reading of complex metadata. It reads the whole dataframe, then sices of n_rows to treat as metadata. Any undesired rows are dropped, then metadata is added as a multiindex onto the raw data, less those rows that were metadata.

Any row metadata can be retained by specifying the index_col kwarg to pd.read_csv.

Parameters

  • file (Path) – The path to the data file

  • n_rows (int) – The number of rows in the column metadata

  • metadata (List[str]) – The metadata rows to include

  • index_names (Optional[List[str]]) – Names for the data frame multi-index

  • **kwargs (Any) – Further argument passed to pd.read_csv

Returns

The created dataframe

Return type

pd.DataFrame

lta.helpers.data_handling.enfc(df: pandas.core.frame.DataFrame, axis: Literal['index', 'columns'], level: str, order: Optional[Tuple[str, str]] = None) → pandas.core.frame.DataFrame

Calculate the error normalised fold change for a dataframe.

ENFC is defined as the log foldchange of lipid levels divide by the propagated error. The mean and standard deviation are calculated on the groups in level, before applying the above calculation.

Notes

By definition, fold change requires an experimental and control group, othewise the notion of up- or down-regulated makes no sense. This function assumes these groups are named “experimental” and “control”, respectively, though alternatives can be passed to the order argument.

Additionally, the notion of fold change requires both samples to be non-0. When either x/0, 0/x, or 0/0 occurs during the FC calculation, these values are returned as NaN.

Parameters

  • df (pd.DataFrame) – The dataframe containing lipid expression values

  • axis (Literal[‘index’, ‘columns’]) – Which multiindex to consider

  • level (str) – The level of the multiindex containing experimental conditions

  • order (Optional[Tuple[str, str]]) – default=(‘experimental’, ‘control’) The names of the conditions to compare. Fold change will be order[0] / order[1].

Returns

The processed data.

Return type

pd.DataFrame

lta.helpers.data_handling.not_zero(df: pandas.core.frame.DataFrame, axis: Literal['index', 'columns'], level: str, compartment: str, thresh: float) → pandas.core.frame.DataFrame

Mark any group that has more than thresh fraction 0 as 0.

Given a counts matrix, mark all counts that are zero. Then, groupby the level of the specified multiindex, and mark a whole group as 0 if there are more than thresh * len(group) 0s. Finally, any lipid that is all 0 is dropped on the specified axis

Grouping occurs across compartment and level in that order.

Parameters

  • df (pd.DataFrame) – The lipid data to convert to boolean

  • axis (Literal[‘index’, ‘columns’]) – Which multiindex to consider

  • level (str) – The level of the multiindex to groupby

  • compartment (str) – The level of the multiindex containing compartment sample

  • thresh (float) – The fraction of samples above which a group is said to be 0

Returns

The processed data.

Return type

pd.DataFrame