Data Cleaning: Handling Missing Values & Types

1Concept

Data cleaning handles missing values via `isna()`, `fillna()` (imputation with mean/median or forward-fill), and `dropna()`. Casting to optimized nullable types (e.g. `Int64`, `category`, `boolean`) drastically reduces memory consumption.

2Architecture Diagram

Raw Data with NaN ---> fillna(df['salary'].median()) ---> Imputed Clean Dataset!

3Code Example

# Data cleaning pattern
dataset = [100.0, None, 150.0, 200.0, None]
clean_values = [x for x in dataset if x is not None]
median_val = sorted(clean_values)[len(clean_values) // 2]
imputed = [x if x is not None else median_val for x in dataset]

print(f"Original dataset: {dataset}")
print(f"Imputed dataset:  {imputed} (Imputed with median {median_val})")

4Expected Output

Original dataset: [100.0, None, 150.0, 200.0, None]
Imputed dataset:  [100.0, 150.0, 150.0, 200.0, 150.0] (Imputed with median 150.0)

5Key Takeaways

• ✓Standard `float64` was historically required for NaNs; Pandas now has nullable `Int64` and `boolean`.
• ✓Categorical columns (`astype('category')`) can reduce string memory usage by 80%+.
• ✓Use `df.dropna(subset=['critical_column'])` to filter invalid rows.