2. The alp_data.io module¶

Every dataset in alp-data reads its files through alp_data.io. The point of this module is protocol transparency: the same call works whether a file lives on your local disk, in Google Cloud Storage (gs://), or in Cloudflare R2 (r2://). You dont have to worry about the backend!

It has three layers:

• paths — anypath builds the right path object from a string; cloud paths get a pathlib-like API without any IO.
• filesystems — filesystem / filesystem_from_path hand you a cached fsspec filesystem for a protocol or a path.
• read/utility helpers — read_text, read_yaml, read_json, read_audio, get_audio_info, audio_stereo_to_mono, plus exists / rm.

anypath — one factory for local and cloud paths¶

anypath inspects the prefix and returns the matching path type:

input prefix	returned type
(none) / local	pathlib.Path
gs://	PureGSPath
r2:// or s3://	PureR2Path

Note: r2:// URIs are normalized to the s3:// protocol internally (R2 speaks the S3 API).

In [ ]:

from alp_data.io import anypath

local = anypath("../../tests/samples/noise.wav")
gs = anypath("gs://esp-data-274503/clips/audio.flac")
r2 = anypath("r2://my-bucket/clips/call.wav")

for p in (local, gs, r2):
    print(f"{type(p).__name__:12} {p}")

from alp_data.io import anypath local = anypath("../../tests/samples/noise.wav") gs = anypath("gs://esp-data-274503/clips/audio.flac") r2 = anypath("r2://my-bucket/clips/call.wav") for p in (local, gs, r2): print(f"{type(p).__name__:12} {p}")

PosixPath    ../../tests/samples/noise.wav
PureGSPath   gs://esp-data-274503/clips/audio.flac
PureR2Path   s3://my-bucket/clips/call.wav

Cloud paths expose a pathlib-like API (name, stem, suffix, parent, parts, with_suffix, / joining, ...) plus a cloud-specific bucket property — all pure string manipulation, no network calls.

In [ ]:

print("name:       ", gs.name)
print("stem:       ", gs.stem)
print("suffix:     ", gs.suffix)
print("parent:     ", gs.parent)
print("bucket:     ", gs.bucket)
print("parts:      ", gs.parts)
print("with_suffix:", gs.with_suffix(".wav"))
print("joinpath:   ", gs / "nested" / "file.wav")

print("name: ", gs.name) print("stem: ", gs.stem) print("suffix: ", gs.suffix) print("parent: ", gs.parent) print("bucket: ", gs.bucket) print("parts: ", gs.parts) print("with_suffix:", gs.with_suffix(".wav")) print("joinpath: ", gs / "nested" / "file.wav")

name:        audio.flac
stem:        audio
suffix:      .flac
parent:      gs://esp-data-274503/clips
bucket:      esp-data-274503
parts:       ('gs://esp-data-274503/', 'clips', 'audio.flac')
with_suffix: gs://esp-data-274503/clips/audio.wav
joinpath:    gs://esp-data-274503/clips/audio.flac/nested/file.wav

DATA_HOME¶

DATA_HOME is the default root bucket where official datasets live. It reads the ALP_DATA_HOME environment variable, and defaults to the public GCS bucket.

In [1]:

from alp_data.io import DATA_HOME

print(DATA_HOME)

from alp_data.io import DATA_HOME print(DATA_HOME)

gs://esp-data-274503

Filesystems: filesystem and filesystem_from_path¶

filesystem(protocol) returns a cached fsspec filesystem for "local", "gcs"/"gs", or "r2" (R2 credentials are pulled from GCP Secret Manager automatically). filesystem_from_path(path) is the convenience wrapper the read helpers use internally — it picks the protocol straight from the path. Because the results are cached, repeated calls return the same instance.

In [ ]:

from alp_data.io import filesystem, filesystem_from_path

local_fs = filesystem("local")
print(type(local_fs).__name__)
print("cached (same instance):", filesystem("local") is filesystem("local"))

# filesystem_from_path infers the backend from the path's protocol
print("from local path:", type(filesystem_from_path("../../tests/samples/noise.wav")).__name__)
# filesystem_from_path("gs://esp-data-274503/x")  -> GCSFileSystem
# filesystem_from_path("r2://my-bucket/x")        -> S3FileSystem

from alp_data.io import filesystem, filesystem_from_path local_fs = filesystem("local") print(type(local_fs).__name__) print("cached (same instance):", filesystem("local") is filesystem("local")) # filesystem_from_path infers the backend from the path's protocol print("from local path:", type(filesystem_from_path("../../tests/samples/noise.wav")).__name__) # filesystem_from_path("gs://esp-data-274503/x") -> GCSFileSystem # filesystem_from_path("r2://my-bucket/x") -> S3FileSystem

LocalFileSystem
cached (same instance): True
from local path: LocalFileSystem

exists and rm¶

Both dispatch through filesystem_from_path, so they work on local and cloud paths identically.

In [ ]:

from alp_data.io import exists, rm

print(exists("../../tests/samples/noise.wav"))   # True
print(exists("../../tests/samples/nope.wav"))    # False

# create then delete a scratch file through the same fsspec filesystem
local_fs.makedirs("_demo_data", exist_ok=True)
local_fs.pipe_file("_demo_data/io_scratch.txt", b"hello")
print("after write:", exists("_demo_data/io_scratch.txt"))
rm("_demo_data/io_scratch.txt")
print("after rm:   ", exists("_demo_data/io_scratch.txt"))

from alp_data.io import exists, rm print(exists("../../tests/samples/noise.wav")) # True print(exists("../../tests/samples/nope.wav")) # False # create then delete a scratch file through the same fsspec filesystem local_fs.makedirs("_demo_data", exist_ok=True) local_fs.pipe_file("_demo_data/io_scratch.txt", b"hello") print("after write:", exists("_demo_data/io_scratch.txt")) rm("_demo_data/io_scratch.txt") print("after rm: ", exists("_demo_data/io_scratch.txt"))

True
False
after write: True
after rm:    False

Reading structured files: read_text, read_yaml, read_json¶

These open the file on the right filesystem and parse it for you.

In [ ]:

from alp_data.io import read_text, read_yaml

print(read_text("configs/beans_basic.yaml"))

cfg = read_yaml("configs/beans_basic.yaml")
print("parsed:", cfg)

from alp_data.io import read_text, read_yaml print(read_text("configs/beans_basic.yaml")) cfg = read_yaml("configs/beans_basic.yaml") print("parsed:", cfg)

dataset:
  dataset_name: beans
  split: dogs_test
  sample_rate: 16000

parsed: {'dataset': {'dataset_name': 'beans', 'split': 'dogs_test', 'sample_rate': 16000}}

In [ ]:

import json

from alp_data.io import read_json

local_fs.pipe_file("_demo_data/meta.json", json.dumps({"sr": 16000, "n": 3}).encode())
print(read_json("_demo_data/meta.json"))
rm("_demo_data/meta.json")

import json from alp_data.io import read_json local_fs.pipe_file("_demo_data/meta.json", json.dumps({"sr": 16000, "n": 3}).encode()) print(read_json("_demo_data/meta.json")) rm("_demo_data/meta.json")

{'sr': 16000, 'n': 3}

Reading audio: read_audio, get_audio_info, audio_stereo_to_mono¶

get_audio_info reads the header only (no decode). read_audio decodes to a NumPy array and supports frame- or time-based slicing. Supported formats: .wav, .flac, .ogg, .mp3.

In [ ]:

from alp_data.io import get_audio_info, read_audio

print(get_audio_info("../../tests/samples/noise.wav"))

audio, sr = read_audio("../../tests/samples/noise.wav")
print("full clip:", audio.shape, "@", sr, "Hz")

from alp_data.io import get_audio_info, read_audio print(get_audio_info("../../tests/samples/noise.wav")) audio, sr = read_audio("../../tests/samples/noise.wav") print("full clip:", audio.shape, "@", sr, "Hz")

{'sr': 16000, 'duration': 32.768, 'num_frames': 524288, 'num_channels': 1, 'format': 'WAV', 'subtype': 'PCM_16'}
full clip: (524288,) @ 16000 Hz

Read just a 1-second window using start_time / end_time (seconds). Frame-based slicing via frames / start is also available, but time and frame slicing are mutually exclusive.

In [ ]:

clip, sr = read_audio("../../tests/samples/noise.wav", start_time=1.0, end_time=2.0)
print("1s window:", clip.shape, "@", sr, "Hz")

clip, sr = read_audio("../../tests/samples/noise.wav", start_time=1.0, end_time=2.0) print("1s window:", clip.shape, "@", sr, "Hz")

1s window: (16000,) @ 16000 Hz

audio_stereo_to_mono collapses a multi-channel array to mono ("average" or "keep_first"), auto-detecting which axis is the channel axis.

In [ ]:

from alp_data.io import audio_stereo_to_mono

stereo, sr = read_audio("../../tests/samples/stereo.wav")
print("stereo:", stereo.shape)

mono = audio_stereo_to_mono(stereo, mono_method="average")
print("mono:  ", mono.shape)

from alp_data.io import audio_stereo_to_mono stereo, sr = read_audio("../../tests/samples/stereo.wav") print("stereo:", stereo.shape) mono = audio_stereo_to_mono(stereo, mono_method="average") print("mono: ", mono.shape)

stereo: (10000, 2)
mono:   (10000,)

Why this matters¶

Every helper above takes the path as a string or path object and figures out the protocol itself. Swap ../../tests/samples/noise.wav for gs://esp-data-274503/.../noise.wav and the exact same call streams the file straight from the cloud — no download step, no backend branching. The dataset classes in the other notebooks rely on this layer under the hood.