6. Concatenating and chaining datasets¶

Two ways to combine datasets:

	ConcatenatedDataset	ChainedDataset
Merges DataBackends	yes	no
Joint transforms	yes	no
Streaming	no	yes
Memory	heavier	lightweight

ConcatenatedDataset (Python)¶

In [1]:

from alp_data import Beans, ConcatenatedDataset
from alp_data.transforms import LabelFromFeatureConfig

ds_dogs = Beans(split="dogs_test", sample_rate=16000)
ds_esc  = Beans(split="esc50_validation", sample_rate=16000)
print(f"dogs:  {len(ds_dogs)}")
print(f"esc50: {len(ds_esc)}")

combined = ConcatenatedDataset([ds_dogs, ds_esc], merge_level="soft")
print(f"combined: {len(combined)}")
print("info name:", combined.info.name)

from alp_data import Beans, ConcatenatedDataset from alp_data.transforms import LabelFromFeatureConfig ds_dogs = Beans(split="dogs_test", sample_rate=16000) ds_esc = Beans(split="esc50_validation", sample_rate=16000) print(f"dogs: {len(ds_dogs)}") print(f"esc50: {len(ds_esc)}") combined = ConcatenatedDataset([ds_dogs, ds_esc], merge_level="soft") print(f"combined: {len(combined)}") print("info name:", combined.info.name)

dogs:  139
esc50: 400
combined: 539
info name: beans+beans

Apply a joint transform after concatenation:

In [2]:

meta = combined.apply_transformations([
    LabelFromFeatureConfig(type="label_from_feature", feature="label", output_feature="label", override=True),
])
print("num classes after merge:", meta["label_from_feature"]["num_classes"])

meta = combined.apply_transformations([ LabelFromFeatureConfig(type="label_from_feature", feature="label", output_feature="label", override=True), ]) print("num classes after merge:", meta["label_from_feature"]["num_classes"])

num classes after merge: 60

ConcatenatedDataset (YAML)¶

In [6]:

from alp_data.io import anypath, read_text
from alp_data import dataset_from_config

print(read_text("configs/beans_concat.yaml"))
ds, meta = dataset_from_config("configs/beans_concat.yaml")
print(f"len: {len(ds)}, meta keys: {list(meta.keys())}")

from alp_data.io import anypath, read_text from alp_data import dataset_from_config print(read_text("configs/beans_concat.yaml")) ds, meta = dataset_from_config("configs/beans_concat.yaml") print(f"len: {len(ds)}, meta keys: {list(meta.keys())}")

concat:
  datasets:
    - dataset_name: beans
      split: dogs_test
    - dataset_name: beans
      split: esc50_validation
  merge_level: soft
  transformations:
    - type: label_from_feature
      feature: label
      output_feature: label
      override: true

len: 539, meta keys: ['label_from_feature']

ChainedDataset¶

Lightweight iteration across datasets — no DataBackend merge.

In [7]:

from alp_data import ChainedDataset

ds_a = Beans(split="dogs_test", sample_rate=16000)
ds_b = Beans(split="cbi_test", sample_rate=16000)

chained = ChainedDataset([ds_a, ds_b])
print(f"chained len: {len(chained)} (= {len(ds_a)} + {len(ds_b)})")

for i, sample in enumerate(chained):
    print(f"{i}: keys={list(sample.keys())[:4]}")
    if i >= 2:
        break

from alp_data import ChainedDataset ds_a = Beans(split="dogs_test", sample_rate=16000) ds_b = Beans(split="cbi_test", sample_rate=16000) chained = ChainedDataset([ds_a, ds_b]) print(f"chained len: {len(chained)} (= {len(ds_a)} + {len(ds_b)})") for i, sample in enumerate(chained): print(f"{i}: keys={list(sample.keys())[:4]}") if i >= 2: break

chained len: 3759 (= 139 + 3620)
0: keys=['label', 'file_name', 'local_path', 'labels_as_list']
1: keys=['label', 'file_name', 'local_path', 'labels_as_list']
2: keys=['label', 'file_name', 'local_path', 'labels_as_list']

ChainedDataset also loads from YAML via the chain: top-level key:

In [8]:

print(read_text("configs/beans_chain.yaml"))
ds, _ = dataset_from_config("configs/beans_chain.yaml")
print(f"len: {len(ds)}")

print(read_text("configs/beans_chain.yaml")) ds, _ = dataset_from_config("configs/beans_chain.yaml") print(f"len: {len(ds)}")

chain:
  datasets:
    - dataset_name: beans
      split: dogs_test
    - dataset_name: beans
      split: cbi_test

len: 3759

In [ ]: