5. Transforms¶

A transform is a callable that takes the dataset's underlying DataBackend and returns (transformed_backend, metadata). They are applied in order. Built-ins:

• Filter — include / exclude rows by column value
• Deduplicate — drop duplicate rows
• Subsample / BalancedSample / LongTailUpsample — resampling
• LabelFromFeature / MultiLabelFromFeatures — categorical → integer labels (returns a label_map in metadata)
• SelectColumns — keep a subset of columns

There are three ways to apply them: directly, via DatasetConfig, or via YAML.

(a) Direct: apply_transformations¶

In [ ]:

from alp_data import Beans
from alp_data.transforms import (
    DeduplicateConfig,
    LabelFromFeatureConfig,
)

ds = Beans(split="dogs_test", sample_rate=16000)
print(f"before: {len(ds)} rows")

metadata = ds.apply_transformations([
    DeduplicateConfig(type="deduplicate", subset=["file_name", "label"], keep_first=True),
    LabelFromFeatureConfig(type="label_from_feature", feature="label", output_feature="label", override=True),
])

print(f"after:  {len(ds)} rows")
print("metadata keys:", list(metadata.keys()))
print("label map:", metadata["label_from_feature"]["label_map"])
print("num classes:", metadata["label_from_feature"]["num_classes"])

from alp_data import Beans from alp_data.transforms import ( DeduplicateConfig, LabelFromFeatureConfig, ) ds = Beans(split="dogs_test", sample_rate=16000) print(f"before: {len(ds)} rows") metadata = ds.apply_transformations([ DeduplicateConfig(type="deduplicate", subset=["file_name", "label"], keep_first=True), LabelFromFeatureConfig(type="label_from_feature", feature="label", output_feature="label", override=True), ]) print(f"after: {len(ds)} rows") print("metadata keys:", list(metadata.keys())) print("label map:", metadata["label_from_feature"]["label_map"]) print("num classes:", metadata["label_from_feature"]["num_classes"])

(b) Python DatasetConfig¶

Same effect, but declared as a config object up front.

In [ ]:

from alp_data import DatasetConfig

config = DatasetConfig(
    dataset_name="beans",
    split="dogs_test",
    sample_rate=16000,
    output_take_and_give={"audio": "raw_wav", "label": "label"},
    transformations=[
        DeduplicateConfig(type="deduplicate", subset=["file_name", "label"], keep_first=True),
        LabelFromFeatureConfig(type="label_from_feature", feature="label", output_feature="label", override=True),
    ],
)

ds, meta = Beans.from_config(config)
print(f"len: {len(ds)}")
print("metadata keys:", list(meta.keys()))
print("sample keys:", list(ds[0].keys()))

from alp_data import DatasetConfig config = DatasetConfig( dataset_name="beans", split="dogs_test", sample_rate=16000, output_take_and_give={"audio": "raw_wav", "label": "label"}, transformations=[ DeduplicateConfig(type="deduplicate", subset=["file_name", "label"], keep_first=True), LabelFromFeatureConfig(type="label_from_feature", feature="label", output_feature="label", override=True), ], ) ds, meta = Beans.from_config(config) print(f"len: {len(ds)}") print("metadata keys:", list(meta.keys())) print("sample keys:", list(ds[0].keys()))

(c) YAML¶

The same pipeline declared in YAML — easy to commit alongside training code.

In [ ]:

from alp_data.io import read_text

print(read_text("configs/beans_with_transforms.yaml"))

from alp_data.io import read_text print(read_text("configs/beans_with_transforms.yaml"))

In [ ]:

from alp_data import dataset_from_config

ds, meta = dataset_from_config("configs/beans_with_transforms.yaml")
print(f"len: {len(ds)}")
print("metadata keys:", list(meta.keys()))

from alp_data import dataset_from_config ds, meta = dataset_from_config("configs/beans_with_transforms.yaml") print(f"len: {len(ds)}") print("metadata keys:", list(meta.keys()))