mmap_ninja package

Submodules

mmap_ninja.base module

class mmap_ninja.base.BytesSlices(buffer: bytes, starts: List[int], ends: List[int])

Bases: object

buffer: bytes
ends: List[int]
starts: List[int]
class mmap_ninja.base.Wrapped(data, wrapper_fn, copy_before_wrapper_fn=True)

Bases: object

mmap_ninja.base.from_generator_base(out_dir, sample_generator, batch_size, batch_ctor, extend_fn=None, **kwargs)

Creates an output from a generator, flushing every batch to disk.

Parameters:

  • out_dir – The output directory.

  • sample_generator – The generator of samples.

  • batch_size – The batch size, which controls how often the output should be written to disk.

  • batch_ctor – The constructor used to initialize the output.

  • extend_fn – Functon to call when doing .extend. By default, this will call memmap.extend(samples)

  • kwargs – Additional keyword arguments to be passed when initializing the output.

Returns:

mmap_ninja.numpy module

class mmap_ninja.numpy.NumpyBytesSlices(buffer: numpy.ndarray, starts: List[int], ends: List[int], flattened_shapes: List[int], shapes: List[Tuple[int]])

Bases: object

buffer: ndarray
ends: List[int]
flattened_shapes: List[int]
shapes: List[Tuple[int]]
starts: List[int]
mmap_ninja.numpy.append(np_mmap: memmap, arr: ndarray) → None

Append a single sample to an already existing numpy array

Parameters:

  • np_mmap –

  • arr –

mmap_ninja.numpy.extend(np_mmap: memmap, arr: ndarray) → None

Extend a numpy memory map with new samples

Parameters:

  • np_mmap – The numpy memory map object

  • arr – The new samples

Returns:

mmap_ninja.numpy.extend_dir(out_dir: str | Path, arr: ndarray) → None

Extend an already existing memory map by adding new samples

Parameters:

  • out_dir – The directory, in which the memory-mapped array is stored.

  • arr – The numpy array of new samples

Returns:

mmap_ninja.numpy.from_generator(out_dir: str | Path, sample_generator, batch_size: int, verbose=False) → memmap

Create a numpy memory-map from a sample generator.

Parameters:

  • sample_generator – A generator of the samples

  • out_dir – The output directory

  • batch_size – How often to flush to disk

  • verbose – Whether to show the progress bar.

Returns:

mmap_ninja.numpy.from_ndarray(out_dir: str | Path, arr: ndarray) → memmap

Initializes a memory map, in which all samples should be of the same shape

Parameters:

  • out_dir – The directory in which the memory map will be persisted

  • arr – The numpy array which should be memory mapped.

Returns:

The memory mapped file

mmap_ninja.numpy.open_existing(out_dir: str | Path, mode='r') → memmap

Open an already existing numpy array.

Parameters:

  • out_dir – The output directory.

  • mode – The mode with which to open the memory-mapped file.

Returns:

The np.memmap object.

mmap_ninja.ragged module

class mmap_ninja.ragged.RaggedMmap(out_dir: str | Path, wrapper_fn=None, mode='r', starts_key='starts', ends_key='ends', shapes_key='shapes', flattened_shapes_key='flattened_shapes', copy_before_wrapper_fn=True)

Bases: object

append(array: ndarray)
extend(arrays: Sequence[ndarray])
classmethod from_generator(out_dir: str | Path, sample_generator, batch_size: int, verbose=False, **kwargs)
classmethod from_lists(out_dir: str | Path, lists: Sequence[ndarray], dtype=None, mode='r+', wrapper_fn=None, starts_key='starts', ends_key='ends', shapes_key='shapes', flattened_shapes_key='flattened_shapes')
get_multiple(item)
get_single(item)
set_multiple(item, value)
set_single(idx, value)

mmap_ninja.string module

class mmap_ninja.string.StringsMmap(out_dir: str | Path, mode='r+b', starts_key='starts', ends_key='ends')

Bases: object

append(string: str)
close()
extend(list_of_strings: Sequence[str], verbose=False)
classmethod from_generator(out_dir: str | Path, sample_generator, batch_size: int, verbose=False, **kwargs)
classmethod from_strings(out_dir: str | Path, strings: Sequence[str], mode='r+b', starts_key='starts', ends_key='ends', verbose=False)
get_multiple(item)
get_single(item)
set_multiple(key, value)
set_single(idx, new_value)

Module contents