Step-by-Step Guide: End-to-End Exploratory Data Analysis & Feature Engineering in pandas

Hands-On Data Science Tutorial: A practical walkthrough of data cleaning, outlier detection, categorical encoding, and feature scaling using pandas, NumPy, and scikit-learn.

Step 1: Automated Missing Data & Outlier Inspection

import pandas as pd
import numpy as np

df = pd.read_csv("dataset.csv")

# Missing data ratio
missing_pct = df.isnull().mean() * 100
print("Missing Value Ratios (%):
", missing_pct[missing_pct > 0])

# IQR Outlier Detection
Q1 = df['amount'].quantile(0.25)
Q3 = df['amount'].quantile(0.75)
IQR = Q3 - Q1
df_clean = df[(df['amount'] >= Q1 - 1.5 * IQR) & (df['amount'] <= Q3 + 1.5 * IQR)]

Step 2: Feature Transformation & Standard Scaling Pipeline

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

preprocessor = ColumnTransformer(
    transformers=[
        ('num', StandardScaler(), ['age', 'income', 'credit_score']),
        ('cat', OneHotEncoder(drop='first'), ['gender', 'region'])
    ]
)

X_processed = preprocessor.fit_transform(df_clean)