Dataset & DataLoader Batching Pipelines

1Concept

`torch.utils.data.Dataset` abstracts custom data access implementing `__len__()` and `__getitem__()`. `DataLoader` provides parallel batching, data shuffling, and multi-process worker loading (`num_workers=4`).

2Architecture Diagram

Raw Data ---> [ Custom Dataset ] ---> [ DataLoader (Batches of 32 + Shuffle) ] ---> GPU Training Loop

3Code Example

dataloader_code = '''
from torch.utils.data import Dataset, DataLoader
import torch

class SyntheticDataset(Dataset):
    def __init__(self, size: int):
        self.x = torch.randn(size, 8)
        self.y = torch.randint(0, 2, (size,))

    def __len__(self):
        return len(self.x)

    def __getitem__(self, idx):
        return self.x[idx], self.y[idx]

dataset = SyntheticDataset(100)
loader = DataLoader(dataset, batch_size=16, shuffle=True, num_workers=2)
print(f"Total Batches per Epoch: {len(loader)}")
'''
print("=== DataLoader Pipeline Architecture ===")
print(dataloader_code.strip())

4Expected Output

=== DataLoader Pipeline Architecture ===
from torch.utils.data import Dataset, DataLoader
import torch

class SyntheticDataset(Dataset):
    def __init__(self, size: int):
        self.x = torch.randn(size, 8)
        self.y = torch.randint(0, 2, (size,))

    def __len__(self):
        return len(self.x)

    def __getitem__(self, idx):
        return self.x[idx], self.y[idx]

dataset = SyntheticDataset(100)
loader = DataLoader(dataset, batch_size=16, shuffle=True, num_workers=2)
print(f"Total Batches per Epoch: {len(loader)}")

5Key Takeaways

• ✓Set `pin_memory=True` in DataLoader for faster page-locked memory transfers to GPU.
• ✓`num_workers > 0` utilizes multi-processing to load batches concurrently.
• ✓Custom collate functions (`collate_fn`) handle dynamic variable-length sequences.