blue_cwl.staging

Staging utils.

Functions

get_distribution_path_entry(entry_id, ...)

get_entry_id(entry)

Get a NEXUS resource id from a dictionary entry.

get_entry_property(entry_id, entry_data, ...)

materialize_cell_composition_summary(obj, *)

Materialize a cell composition summary distribution.

materialize_cell_composition_volume(obj, *)

Materializa a cell composition volume distribution.

materialize_density_distribution(forge, ...)

Bacwards compatible function using forge.

materialize_macro_connectome_config(obj[, ...])

Materialize the macro connectome config.

materialize_micro_connectome_config(obj[, ...])

Materialize micro connectome config.

materialize_ph_catalog(obj, *[, output_dir, ...])

Materialize placement hints catalog resources.

materialize_synapse_config(obj, output_dir, *)

Materialize a synapse editor config.

stage_atlas(id_or_entity, output_dir, *[, ...])

Stage an atlas to the given output_dir.

stage_detailed_circuit(obj, *, output_file)

Stage DetailedCircuit a resource to its circuit config file.

stage_distribution_file(id_or_entity, ...[, ...])

Stage a distribution file from a NEXUS resource.

stage_file(source, target[, symbolic])

Stage a source file to the target location.

stage_variant(obj, *, output_file[, base, ...])

Stage variant nexus resource to a cwl local definition.

transform_cached(func)

Cache for transform functions.

transform_nested_dataset(dataset, ...[, ...])

Transform a nested dataset using the level transforms.

Classes

AtlasInfo(ontology_path, annotation_path, ...)

Atlas paths information.
class blue_cwl.staging.AtlasInfo(ontology_path: str, annotation_path: str, hemisphere_path: str | None, ph_catalog: dict | None, cell_orientation_field_path: str | None, directory: Path)

Atlas paths information.

classmethod from_file(filepath: str | PathLike[str])

Instantiate from a file.

to_file(filepath: str | PathLike[str])

Dump atlas info to a json file.

blue_cwl.staging.get_entry_id(entry: dict) → str

Get a NEXUS resource id from a dictionary entry.

This function makes sure that both the id and the revision are fetched and combined to create the full resource’s id with revision.

Parameters:

entry – The entry dictionary to extract the id with revision from.

Returns:

Full resource id with revision.

blue_cwl.staging.materialize_cell_composition_summary(obj: dict | str | Entity, *, output_file: str | PathLike[str] | None = None, base: str | None = None, org: str | None = None, proj: str | None = None, token: str | None = None) → DataFrame

Materialize a cell composition summary distribution.

Parameters:

  • obj – One of the following: - Resource nexus id of entity to get distribution from1 - Dictiorary of the data - Entity to get distribution from

  • output_file – Optional output file to write the dataframe.

  • base – The nexus base endpoint. If None entity-management’s runtime base is used.

  • org – The nexus organization. If None entity-management’s runtime org is used.

  • proj – The nexus project. If None entity-management’s runtime proj is used.

  • token – Optional OAuth token. If None entity-management’s runtime token is used.

Returns:

  • region

  • region_url

  • region_label

  • mtype

  • mtype_url

  • etype

  • etype_url

  • density

Return type:

DataFrame with the following columns

blue_cwl.staging.materialize_cell_composition_volume(obj: dict | str | Entity, *, output_file: str | PathLike[str] | None = None, base: str | None = None, org: str | None = None, proj: str | None = None, token: str | None = None) → DataFrame

Materializa a cell composition volume distribution.

Parameters:

  • obj – One of the following: - Resource nexus id of entity to get distribution from1 - Dictiorary of the data - Entity to get distribution from

  • output_file – Optional output file to write the dataframe.

  • base – The nexus base endpoint. If None entity-management’s runtime base is used.

  • org – The nexus organization. If None entity-management’s runtime org is used.

  • proj – The nexus project. If None entity-management’s runtime proj is used.

  • token – Optional OAuth token. If None entity-management’s runtime token is used.

Returns:

  • mtype

  • mtype_url

  • etype

  • etype_url

  • path

Return type:

DataFrame with the following columns

blue_cwl.staging.materialize_density_distribution(forge, dataset, output_file: str | PathLike[str])

Bacwards compatible function using forge.

blue_cwl.staging.materialize_macro_connectome_config(obj: dict | str | Entity, output_file: str | PathLike[str] | None = None, *, base: str | None = None, org: str | None = None, proj: str | None = None, token: str | None = None) → dict

Materialize the macro connectome config.

Parameters:

  • obj –

    One of the following:

    • Resource nexus id of entity to get distribution from1

    • Dictiorary of the data

    • Entity to get distribution from

  • output_file – Optional path to write the result to a file. Default is None.

  • base – The nexus base endpoint. If None entity-management’s runtime base is used.

  • org – The nexus organization. If None entity-management’s runtime org is used.

  • proj – The nexus project. If None entity-management’s runtime proj is used.

  • token – Optional OAuth token. If None entity-management’s runtime token is used.

Returns:

{
    "initial": {"connection_strength": path / to / initial / arrow / file},
    "overrides": {"connection_strength": path / to / overrides / arrow / file},
}

Return type:

The materialized dictionary of the macro connectome configuration. Example

Note: overrides key is mandatory but can be empty.

blue_cwl.staging.materialize_micro_connectome_config(obj, output_file: DataFrame | None = None, *, base: str | None = None, org: str | None = None, proj: str | None = None, token: str | None = None) → dict

Materialize micro connectome config.

Parameters:

  • obj – One of the following: - Resource nexus id of entity to get distribution from1 - Dictiorary of the data - Entity to get distribution from

  • output_file – Optional filepath to write dictionary as json.

  • base – The nexus base endpoint. If None entity-management’s runtime base is used.

  • org – The nexus organization. If None entity-management’s runtime org is used.

  • proj – The nexus project. If None entity-management’s runtime proj is used.

  • token – Optional OAuth token. If None entity-management’s runtime token is used.

Returns:

{
    "variants": {"var1": {"params": {"weight": {"default": 1.0}}}},
    "initial": {
        "variants": path / to / variants / arrow / file,
        "var1": path / to / var1 - parameters / arrow / file,
    },
    "overrides": {
        "variants": path / to / variants - overrides / arrow / file,
        "var1": path / to / var1 - parameters - overrides / arrow / file,
    },
}

Return type:

Materialized dictionary of the micro connectome configuration. Example

Note: overrides key is mandatory but can be empty or include subset of keys in ‘initial’.

blue_cwl.staging.materialize_ph_catalog(obj, *, output_dir=None, output_filenames: dict | None = None, symbolic=True, base: str | None = None, org: str | None = None, proj: str | None = None, token: str | None = None)

Materialize placement hints catalog resources.

Parameters:

  • obj – One of the following: - Resource nexus id of entity to get distribution from1 - Dictiorary of the data - Entity to get distribution from

  • output_dir – Output directory to stage the files.

  • output_filenames – Filenames to use for the output datasets.

  • symbolic – Make symlinks where possible.

  • base – The nexus base endpoint. If None entity-management’s runtime base is used.

  • org – The nexus organization. If None entity-management’s runtime org is used.

  • proj – The nexus project. If None entity-management’s runtime proj is used.

  • token – Optional OAuth token. If None entity-management’s runtime token is used.

Returns:

Materialized dataset dictionary.

blue_cwl.staging.materialize_synapse_config(obj, output_dir, *, output_file: str | PathLike[str] | None = None, base: str | None = None, org: str | None = None, proj: str | None = None, token: str | None = None) → dict

Materialize a synapse editor config.

Parameters:

  • obj – One of the following: - Resource nexus id of entity to get distribution from1 - Dictiorary of the data - Entity to get distribution from

  • output_dir – Output directory to stage data.

  • output_file – Optional output file to write the json config to.

  • base – The nexus base endpoint. If None entity-management’s runtime base is used.

  • org – The nexus organization. If None entity-management’s runtime org is used.

  • proj – The nexus project. If None entity-management’s runtime proj is used.

  • token – Optional OAuth token. If None entity-management’s runtime token is used.

Returns:

Materialized dataset dictionary.

blue_cwl.staging.stage_atlas(id_or_entity: str, output_dir: Path, *, parcellation_ontology_basename: str = 'hierarchy.json', parcellation_volume_basename: str = 'brain_regions.nrrd', parcellation_hemisphere_basename: str = 'hemisphere.nrrd', cell_orientation_field_basename: str = 'orientation.nrrd', symbolic=True, output_file: str | PathLike[str] | None = None, base: str | None = None, org: str | None = None, proj: str | None = None, token: str | None = None) → AtlasInfo

Stage an atlas to the given output_dir.

Parameters:

  • id_or_entity – The resource id or the entity.

  • output_dir – The output directory to put the files in.

  • parcellation_ontology_basename – The filename of the retrieved hierarchy.

  • parcellation_volume_basename – The filename of the retrieved brain annotations

  • parcellation_hemisphere_basename – The filename of the retrieved hemisphere volume.

  • cell_orientation_field_basename – The filename of the cell orientation field file.

  • symbolic – If True symbolic links will be attempted if the datasets exist on gpfs otherwise the files will be downloaded.

  • output_file – Optional output file to write the atlas info file.

  • base – The nexus base endpoint. If None entity-management’s runtime base is used.

  • org – The nexus organization. If None entity-management’s runtime org is used.

  • proj – The nexus project. If None entity-management’s runtime proj is used.

  • token – Optional OAuth token. If None entity-management’s runtime token is used.

Returns:

An AtlasInfo instance with the staged file paths.

blue_cwl.staging.stage_detailed_circuit(obj, *, output_file: str | PathLike[str], base: str | None = None, org: str | None = None, proj: str | None = None, token: str | None = None) → None

Stage DetailedCircuit a resource to its circuit config file.

blue_cwl.staging.stage_distribution_file(id_or_entity: str | Entity, output_dir: str | PathLike[str], *, encoding_format: str | None = None, filename: str | None = None, symbolic: bool = True, base: str | None = None, org: str | None = None, proj: str | None = None, token: str | None = None) → Path

Stage a distribution file from a NEXUS resource.

Parameters:

  • id_or_entity – Nexus id of the entity or the entity itself.

  • output_dir – The output directory to copy the file or store the link.

  • encoding_format – The encoding format of the distribution. Example: application/json

  • filename – Filename to use. If None the source’s filename will be used.

  • symbolic – Whether to make a symbolic link or copy the file.

  • base – The nexus base endpoint. If None entity-management’s runtime base is used.

  • org – The nexus organization. If None entity-management’s runtime org is used.

  • proj – The nexus project. If None entity-management’s runtime proj is used.

  • token – Optional OAuth token. If None entity-management’s runtime token is used.

Returns:

Path to the staged file.

Note

A resource may have many distributions with a different encoding format. If that’s the case the encoding format argument is mandatory to select the respective distribution.

blue_cwl.staging.stage_file(source: Path, target: Path, symbolic: bool = True) → None

Stage a source file to the target location.

Parameters:

  • source – File path to stage.

  • target – Location to stage to.

  • symbolic – If True a soft link will be created at target, pointing to source. Otherwise, a copy will be performed.

Note: The directory structure of the target will be created if it doesn’t exist.

blue_cwl.staging.stage_variant(obj, *, output_file: str | PathLike[str], base: str | None = None, org: str | None = None, proj: str | None = None, token: str | None = None)

Stage variant nexus resource to a cwl local definition.

blue_cwl.staging.transform_cached(func)

Cache for transform functions.

blue_cwl.staging.transform_nested_dataset(dataset: dict, level_transforms: Sequence[Callable[[str, dict], tuple[str, dict]]], branch_token: str = 'hasPart') → dict

Transform a nested dataset using the level transforms.

Parameters:

  • dataset – The nested dataset.

  • level_transforms – A list of callables that transform each nested level.

  • branch_token – Token to access the children of each level. Default is ‘hasPart’.

Returns:

Transformed dataset.