1. Basics¶

List registered datasets, instantiate one, inspect its info, and iterate.

List all registered datasets¶

In [1]:

from alp_data import list_registered_datasets, print_registered_datasets

names = list_registered_datasets()
print(f"{len(names)} registered datasets:")
for name in names:
    print(f"  - {name}")

from alp_data import list_registered_datasets, print_registered_datasets names = list_registered_datasets() print(f"{len(names)} registered datasets:") for name in names: print(f" - {name}")

37 registered datasets:
  - animal-sound-archive
  - animalspeak
  - anuraset_strong
  - arctic_bird_sounds
  - audioset
  - audioset_strong
  - beans
  - beans_zero
  - Bengalese Finch Calls
  - birdeep
  - birdset
  - chiffchaff_id
  - corvid_wascher
  - dclde2026
  - dinardo_dolphin_whistles
  - esp_raincoast
  - geladas
  - giant_otters
  - gibbon_solos
  - hawaiian_birds
  - inaturalist
  - InfantMarmosetsVox
  - insectset_459
  - littleowl_id
  - macaques_coo_calls
  - nocturnal_bird_migration
  - pipit_id
  - powdermill
  - subsegmentation
  - superb_starling
  - voxaboxen
  - voxaboxen_events
  - wabad
  - watkins
  - xeno-canto
  - xeno_canto_annotated_jeantet_23
  - zebra_finch_julie_elie

print_registered_datasets() dumps the full DatasetInfo for every dataset (commented to keep output short):

In [ ]:

# print_registered_datasets()

# print_registered_datasets()

Instantiate a dataset¶

Lets look at the BEANS benchmark [https://arxiv.org/abs/2210.12300]

In [1]:

from alp_data import Beans

from alp_data import Beans

We'll use the dogs subset of BEANS (small, quick to load).

In [2]:

beans = Beans(split="dogs_test", sample_rate=16000)
print(f"Length: {len(beans)}")

beans = Beans(split="dogs_test", sample_rate=16000) print(f"Length: {len(beans)}")

Length: 139

All available splits are under:

In [22]:

print(beans.available_splits)

print(beans.available_splits)

['train', 'validation', 'test', 'cbi_test', 'cbi_validation', 'cbi_train', 'watkins_test', 'watkins_validation', 'watkins_train', 'dogs_test', 'dogs_validation', 'dogs_train', 'egyptian_fruit_bats_test', 'egyptian_fruit_bats_validation', 'egyptian_fruit_bats_train', 'hiceas_test', 'hiceas_validation', 'hiceas_train', 'dcase_test', 'dcase_validation', 'dcase_train', 'enabirds_test', 'enabirds_validation', 'enabirds_train', 'esc50_test', 'esc50_validation', 'esc50_train', 'speech_commands_test', 'speech_commands_validation', 'speech_commands_train', 'humbugdb_test', 'humbugdb_validation', 'humbugdb_train', 'rfcx_test', 'rfcx_validation', 'rfcx_train', 'hainan_gibbons_test', 'hainan_gibbons_validation', 'hainan_gibbons_train']

Lets have a quick look at the columns.

In [4]:

beans.columns

beans.columns

Out[4]:

['label', 'file_name', 'local_path', 'labels_as_list']

Inspect the info attribute¶

Every dataset carries a DatasetInfo describing name, owner, version, sources, license, splits.

In [5]:

info = beans.info
print(f"name:        {info.name}")
print(f"owner:       {info.owner}")
print(f"version:     {info.version}")
print(f"license:     {info.license}")
print(f"sources:     {info.sources}")
print(f"description: {info.description}")
print(f"splits:      {list(info.split_paths)[:5]} ...")

info = beans.info print(f"name: {info.name}") print(f"owner: {info.owner}") print(f"version: {info.version}") print(f"license: {info.license}") print(f"sources: {info.sources}") print(f"description: {info.description}") print(f"splits: {list(info.split_paths)[:5]} ...")

name:        beans
owner:       gagan
version:     0.1.0
license:     CC-BY-4.0, CC0
sources:     ['cbi', 'watkins', 'dogs', 'egyptian_fruit_bats', 'hiceas', 'dcase', 'enabirds', 'esc50', 'speech_commands', 'humbugdb', 'rfcx', 'hainan_gibbons']
description: BEANS benchmark dataset
splits:      ['train', 'validation', 'test', 'cbi_test', 'cbi_validation'] ...

Random access¶

In [3]:

sample = beans[0]
print("keys:", list(sample.keys()))
print("audio shape:", sample["audio"].shape)

sample = beans[0] print("keys:", list(sample.keys())) print("audio shape:", sample["audio"].shape)

keys: ['label', 'file_name', 'local_path', 'labels_as_list', 'audio', 'sample_rate']
audio shape: (133235,)

Iterate¶

In [4]:

for i, sample in enumerate(beans):
    print(f"{i}: audio={sample['audio'].shape}, label={sample.get('label')}")
    if i >= 4:
        break

for i, sample in enumerate(beans): print(f"{i}: audio={sample['audio'].shape}, label={sample.get('label')}") if i >= 4: break

0: audio=(133235,), label=Mac
1: audio=(395434,), label=Mac
2: audio=(212904,), label=Mac
3: audio=(718379,), label=Mac
4: audio=(342935,), label=Mac

Streaming mode¶

One of the advantages we offer is streaming data instead of having to download data first.

In [5]:

from alp_data import INaturalist

from alp_data import INaturalist

In [6]:

inat = INaturalist(split="val", streaming=True, backend="pandas")  # takes 0 sec to instantiate

inat = INaturalist(split="val", streaming=True, backend="pandas") # takes 0 sec to instantiate

📝 Streaming mode doesn't allow __getitem__ or __len__, so running:

print(len(inat))

raises: NotImplementedError: Length is not available in streaming mode. Iterate over the dataset instead.

In [7]:

for i, sample in enumerate(inat):
    print(f"{i}: audio={sample['audio'].shape}, label={sample.get('species_scientific')}")
    if i >= 4:
        break

for i, sample in enumerate(inat): print(f"{i}: audio={sample['audio'].shape}, label={sample.get('species_scientific')}") if i >= 4: break

0: audio=(1107968,), label=Cracticus torquatus
1: audio=(1059610,), label=Centropus sinensis
2: audio=(536576,), label=Zhangixalus arvalis
3: audio=(523264,), label=Corvus corax
4: audio=(991232,), label=Haemorhous mexicanus

In [ ]: