Pandas Tutorial for Beginners (2025): Learn Data Analysis with Python

Pandas is Python's go-to library for data analysis — used by data scientists, analysts, and engineers worldwide to load, clean, transform, and analyze tabular data. This tutorial takes you from zero to performing real data analysis, with no prior data science experience required.

What you'll learn

Pandas version used: Pandas 2.x (latest stable as of 2025)

Part 1 — Why Pandas?

Before Pandas, analyzing data in Python meant writing lots of manual loops. Pandas provides two powerful data structures — Series and DataFrame — that make data work feel like SQL plus Excel, powered by Python.

Without Pandas:
  rows = []
  with open("sales.csv") as f:
      for line in f:
          rows.append(line.split(","))
  # then manually filter, sort, group...

With Pandas:
  df = pd.read_csv("sales.csv")
  df[df["revenue"] > 1000].groupby("region")["revenue"].sum()

Part 2 — Setup

Install Pandas

pip install pandas

For data science work, install the full stack:

pip install pandas numpy matplotlib openpyxl

Verify installation

import pandas as pd
import numpy as np

print(pd.__version__)  # 2.x.x

Run Pandas code

Options:

• Jupyter Notebook: pip install jupyter → jupyter notebook (recommended for exploration)
• VS Code with Python extension
• Google Colab: free browser-based Jupyter (no install needed)
• Python script: python analysis.py

Part 3 — Core Data Structures

Pandas has two main data structures:

Series

import pandas as pd

# Create from list
s = pd.Series([10, 20, 30, 40])
print(s)
# 0    10
# 1    20
# 2    30
# 3    40
# dtype: int64

# Create with custom index
s = pd.Series([10, 20, 30], index=["a", "b", "c"])
print(s["b"])  # 20

# Series from dict
prices = pd.Series({"apple": 1.20, "banana": 0.50, "cherry": 3.00})
print(prices["cherry"])  # 3.0

DataFrame

# Create from dict
data = {
    "name": ["Alice", "Bob", "Carol", "Dave"],
    "age": [25, 30, 35, 28],
    "city": ["New York", "London", "Paris", "Tokyo"],
    "salary": [70000, 85000, 90000, 75000],
}
df = pd.DataFrame(data)
print(df)
#     name  age      city  salary
# 0  Alice   25  New York   70000
# 1    Bob   30    London   85000
# 2  Carol   35     Paris   90000
# 3   Dave   28     Tokyo   75000

# DataFrame properties
print(df.shape)       # (4, 4) — rows × columns
print(df.columns)     # Index(['name', 'age', 'city', 'salary'])
print(df.dtypes)      # column data types
print(df.index)       # RangeIndex(start=0, stop=4, step=1)

Part 4 — Reading Files

Pandas can read almost any tabular format:

Read CSV

# Basic read
df = pd.read_csv("sales.csv")

# Common options
df = pd.read_csv(
    "sales.csv",
    sep=",",              # delimiter (default comma)
    header=0,             # row to use as column names (default 0)
    index_col="id",       # column to use as row index
    parse_dates=["date"], # parse these columns as datetime
    nrows=1000,           # read only first 1000 rows
    skiprows=[1, 2],      # skip these row numbers
    encoding="utf-8",     # file encoding
    na_values=["N/A", "missing"],  # treat as NaN
)

Read Excel

# Requires: pip install openpyxl
df = pd.read_excel("report.xlsx", sheet_name="Sales")

# Read all sheets
sheets = pd.read_excel("report.xlsx", sheet_name=None)
# sheets is a dict: {"Sales": df1, "Costs": df2}

Read JSON

df = pd.read_json("data.json")

# JSON lines format (one JSON object per line)
df = pd.read_json("data.jsonl", lines=True)

Create sample data (for practice)

import pandas as pd
import numpy as np

np.random.seed(42)
n = 200

df = pd.DataFrame({
    "date": pd.date_range("2024-01-01", periods=n, freq="D"),
    "product": np.random.choice(["Widget", "Gadget", "Donut"], n),
    "region": np.random.choice(["North", "South", "East", "West"], n),
    "units": np.random.randint(1, 50, n),
    "price": np.random.uniform(10, 200, n).round(2),
    "returned": np.random.choice([True, False], n, p=[0.05, 0.95]),
})
df["revenue"] = (df["units"] * df["price"]).round(2)

Part 5 — Exploring Data

Always start by profiling your dataset before any analysis.

# Shape
print(df.shape)        # (200, 7) — 200 rows, 7 columns

# First/last rows
print(df.head())       # first 5 rows
print(df.head(10))     # first 10 rows
print(df.tail())       # last 5 rows

# Column names and types
print(df.dtypes)
# date        datetime64[ns]
# product             object
# region              object
# units                int64
# price              float64
# returned              bool
# revenue            float64

# Summary statistics
print(df.describe())
#          units       price    revenue
# count  200.000  200.000000  200.000000
# mean    25.030   103.95...   2655...
# std     14.225    54.88...   1784...
# min      1.000    10.04...     11...
# 25%     13.000    57.11...   1022...
# 50%     25.000   103.26...   2420...
# 75%     37.000   150.86...   3962...
# max     49.000   199.71...   8537...

# Include non-numeric columns
print(df.describe(include="all"))

# Concise summary with nulls
print(df.info())
# RangeIndex: 200 entries, 0 to 199
# Data columns (total 7 columns):
# ...non-null values per column...

# Unique values in a column
print(df["product"].unique())    # ['Widget' 'Gadget' 'Donut']
print(df["product"].nunique())   # 3
print(df["product"].value_counts())
# Widget    72
# Gadget    67
# Donut     61

# Check for missing values
print(df.isnull().sum())
# date        0
# product     0
# ...all zeros = no nulls

Part 6 — Selecting Data

Select columns

# Single column → returns Series
col = df["product"]

# Multiple columns → returns DataFrame
subset = df[["product", "region", "revenue"]]

# All numeric columns
numeric_df = df.select_dtypes(include="number")

Select rows with loc and iloc

# loc: label-based (index labels and column names)
# iloc: integer-based (row/column positions)

# Single row by index label
row = df.loc[0]

# Slice rows by label
rows = df.loc[0:4]         # rows 0, 1, 2, 3, 4 (inclusive)

# Row + specific columns
subset = df.loc[0:4, ["product", "revenue"]]

# iloc: by integer position
subset = df.iloc[0:5]       # rows 0–4 (exclusive end)
subset = df.iloc[0:5, 0:3]  # rows 0–4, columns 0–2

# Last row
last = df.iloc[-1]

# Every other row
every_other = df.iloc[::2]

at and iat (single cell access)

# Fast single-value access
val = df.at[0, "product"]   # label-based
val = df.iat[0, 2]          # integer-based (row 0, column 2)

Part 7 — Filtering Data

Boolean indexing

# Single condition
high_revenue = df[df["revenue"] > 5000]

# Multiple conditions (use & | ~, NOT and/or/not)
widget_north = df[(df["product"] == "Widget") & (df["region"] == "North")]

# OR condition
north_or_south = df[(df["region"] == "North") | (df["region"] == "South")]

# NOT condition
not_returned = df[~df["returned"]]

isin() — filter by list

two_regions = df[df["region"].isin(["North", "East"])]

between() — filter numeric range

mid_revenue = df[df["revenue"].between(1000, 3000)]

query() — SQL-like string syntax

result = df.query("revenue > 5000 and region == 'North'")
result = df.query("product in ['Widget', 'Donut']")
result = df.query("units > 30 and price < 100")

# Use @ to reference external variables
min_rev = 2000
result = df.query("revenue > @min_rev")

str accessor — filter on string columns

# Contains
gadgets = df[df["product"].str.contains("Gadget")]

# Starts/ends with
starts_with_w = df[df["product"].str.startswith("W")]

# Case-insensitive
matches = df[df["product"].str.lower().str.contains("widget")]

Part 8 — Handling Missing Data

Detect missing values

# Add some missing values for demo
df_with_nulls = df.copy()
df_with_nulls.loc[5:10, "price"] = np.nan
df_with_nulls.loc[15:20, "product"] = np.nan

# Count nulls per column
print(df_with_nulls.isnull().sum())
# price      6
# product    6

# Percentage of nulls
print(df_with_nulls.isnull().mean() * 100)

# Any null in each row
rows_with_null = df_with_nulls[df_with_nulls.isnull().any(axis=1)]

Fill missing values

# Fill with a constant
df_filled = df_with_nulls.fillna(0)

# Fill per column
df_filled = df_with_nulls.fillna({
    "price": df_with_nulls["price"].mean(),
    "product": "Unknown",
})

# Forward fill (use previous value)
df_ffill = df_with_nulls.fillna(method="ffill")

# Backward fill
df_bfill = df_with_nulls.fillna(method="bfill")

Drop missing values

# Drop rows with any null
df_clean = df_with_nulls.dropna()

# Drop rows where all values are null
df_clean = df_with_nulls.dropna(how="all")

# Drop rows with nulls in specific columns
df_clean = df_with_nulls.dropna(subset=["price", "product"])

# Drop columns with more than 50% nulls
threshold = len(df_with_nulls) * 0.5
df_clean = df_with_nulls.dropna(thresh=threshold, axis=1)

Part 9 — Data Manipulation

Add and modify columns

# New column from calculation
df["revenue_per_unit"] = df["revenue"] / df["units"]

# Conditional column
df["size"] = np.where(df["units"] > 30, "large", "small")

# Multiple conditions with np.select
conditions = [
    df["revenue"] < 1000,
    df["revenue"].between(1000, 3000),
    df["revenue"] > 3000,
]
choices = ["low", "medium", "high"]
df["revenue_tier"] = np.select(conditions, choices)

# Apply a function to a column
df["price_rounded"] = df["price"].apply(lambda x: round(x / 10) * 10)

Rename columns

# Rename specific columns
df = df.rename(columns={"units": "quantity", "price": "unit_price"})

# Rename all columns at once
df.columns = ["date", "product", "region", "qty", "price", "returned", "revenue"]

# Clean column names (lowercase, replace spaces)
df.columns = df.columns.str.lower().str.replace(" ", "_")

Drop columns and rows

# Drop columns
df_slim = df.drop(columns=["revenue_per_unit", "size"])

# Drop rows by index
df_trimmed = df.drop(index=[0, 1, 2])

# Drop rows matching condition
df_no_returns = df[~df["returned"]]  # filter instead of drop

Change data types

# Convert column type
df["units"] = df["units"].astype(float)
df["product"] = df["product"].astype("category")  # saves memory

# Parse dates
df["date"] = pd.to_datetime(df["date"])

# Convert to numeric (coerce errors to NaN)
df["price"] = pd.to_numeric(df["price"], errors="coerce")

Remove duplicates

# Check for duplicates
print(df.duplicated().sum())

# Remove duplicate rows
df_unique = df.drop_duplicates()

# Remove duplicates based on specific columns
df_unique = df.drop_duplicates(subset=["product", "region"])

# Keep last occurrence instead of first
df_unique = df.drop_duplicates(subset=["product"], keep="last")

Reset index

df_filtered = df[df["revenue"] > 2000]
df_filtered = df_filtered.reset_index(drop=True)  # 0, 1, 2... new index

Part 10 — Sorting

# Sort by single column (ascending)
df_sorted = df.sort_values("revenue")

# Sort descending
df_sorted = df.sort_values("revenue", ascending=False)

# Sort by multiple columns
df_sorted = df.sort_values(["region", "revenue"], ascending=[True, False])

# Sort index
df_sorted = df.sort_index()

Part 11 — GroupBy and Aggregation

GroupBy is one of Pandas' most powerful features — like SQL's GROUP BY.

# Total revenue by product
revenue_by_product = df.groupby("product")["revenue"].sum()
# product
# Donut     158430.12
# Gadget    178543.90
# Widget    192011.44

# Multiple aggregations
summary = df.groupby("product")["revenue"].agg(["sum", "mean", "count", "min", "max"])

# Custom aggregation with named columns
summary = df.groupby("product").agg(
    total_revenue=("revenue", "sum"),
    avg_revenue=("revenue", "mean"),
    order_count=("revenue", "count"),
    avg_units=("units", "mean"),
)

# Multiple group keys
region_product = df.groupby(["region", "product"])["revenue"].sum()

# GroupBy with filter
df.groupby("product").filter(lambda g: g["revenue"].mean() > 2500)

Aggregation functions

Pivot tables

# Pivot: product × region → sum of revenue
pivot = df.pivot_table(
    values="revenue",
    index="product",
    columns="region",
    aggfunc="sum",
    fill_value=0,
)

# Multi-value pivot
pivot = df.pivot_table(
    values=["revenue", "units"],
    index="product",
    columns="region",
    aggfunc={"revenue": "sum", "units": "mean"},
)

Part 12 — Merging DataFrames

concat — stack DataFrames

# Stack vertically (add rows)
df_all = pd.concat([df_q1, df_q2, df_q3], ignore_index=True)

# Stack horizontally (add columns)
df_combined = pd.concat([df_left, df_right], axis=1)

merge — SQL-style joins

# Sample DataFrames
orders = pd.DataFrame({
    "order_id": [1, 2, 3, 4, 5],
    "customer_id": [101, 102, 103, 101, 104],
    "amount": [250, 150, 400, 300, 200],
})

customers = pd.DataFrame({
    "customer_id": [101, 102, 103, 105],
    "name": ["Alice", "Bob", "Carol", "Eve"],
    "city": ["NY", "LA", "NY", "SF"],
})

# Inner join (only matching rows)
result = pd.merge(orders, customers, on="customer_id", how="inner")

# Left join (all orders, matched customers or NaN)
result = pd.merge(orders, customers, on="customer_id", how="left")

# Right join
result = pd.merge(orders, customers, on="customer_id", how="right")

# Outer join (all rows from both)
result = pd.merge(orders, customers, on="customer_id", how="outer")

# Different column names in each table
result = pd.merge(
    orders, customers,
    left_on="customer_id",
    right_on="customer_id",
)

join — merge on index

df1 = df1.set_index("customer_id")
df2 = df2.set_index("customer_id")
result = df1.join(df2, how="left")

Part 13 — String Operations

Access string methods via .str accessor:

df["product"] = df["product"].str.lower()        # lowercase
df["product"] = df["product"].str.upper()        # uppercase
df["product"] = df["product"].str.strip()        # strip whitespace
df["product"] = df["product"].str.replace("-", " ")  # replace
df["product"] = df["product"].str.title()        # Title Case

# Extract
df["first_char"] = df["product"].str[0]          # first character
df["length"] = df["product"].str.len()           # string length

# Split
df[["first", "last"]] = df["name"].str.split(" ", expand=True)

# Contains / starts / ends
mask = df["product"].str.contains("widget", case=False)
mask = df["product"].str.startswith("W")

# Extract with regex
df["digits"] = df["code"].str.extract(r"(\d+)")

Part 14 — DateTime Operations

# Parse dates
df["date"] = pd.to_datetime(df["date"])

# Extract components
df["year"] = df["date"].dt.year
df["month"] = df["date"].dt.month
df["day"] = df["date"].dt.day
df["weekday"] = df["date"].dt.day_name()    # 'Monday', 'Tuesday'...
df["week"] = df["date"].dt.isocalendar().week
df["quarter"] = df["date"].dt.quarter

# Date arithmetic
df["days_ago"] = (pd.Timestamp.now() - df["date"]).dt.days

# Filter by date
recent = df[df["date"] >= "2024-06-01"]
q1 = df[(df["date"] >= "2024-01-01") & (df["date"] < "2024-04-01")]

# Resample time series (requires date as index)
df_ts = df.set_index("date")
monthly = df_ts["revenue"].resample("ME").sum()   # monthly totals
weekly = df_ts["revenue"].resample("W").mean()    # weekly averages

Part 15 — Apply and Map

apply — row or column operations

# Apply function to a column
df["price_log"] = df["price"].apply(np.log)

# Apply with custom function
def revenue_tier(rev):
    if rev < 1000:
        return "low"
    elif rev < 3000:
        return "medium"
    return "high"

df["tier"] = df["revenue"].apply(revenue_tier)

# Apply to whole row (axis=1)
df["label"] = df.apply(
    lambda row: f"{row['product']}-{row['region']}", axis=1
)

map — element-wise on Series

# Map with dict
region_code = {"North": "N", "South": "S", "East": "E", "West": "W"}
df["region_code"] = df["region"].map(region_code)

# Map with function
df["price_cent"] = df["price"].map(lambda x: int(x * 100))

transform — GroupBy with same-size output

# Add group mean as column (keeps all rows)
df["group_mean_rev"] = df.groupby("product")["revenue"].transform("mean")
df["normalized"] = df["revenue"] / df["group_mean_rev"]

Part 16 — Exporting Data

# Write CSV
df.to_csv("output.csv", index=False)    # index=False: don't write row numbers

# Write Excel
df.to_excel("output.xlsx", sheet_name="Sales", index=False)

# Write multiple sheets
with pd.ExcelWriter("report.xlsx") as writer:
    df_q1.to_excel(writer, sheet_name="Q1", index=False)
    df_q2.to_excel(writer, sheet_name="Q2", index=False)
    summary.to_excel(writer, sheet_name="Summary")

# Write JSON
df.to_json("output.json", orient="records", indent=2)

# Write to SQL
from sqlalchemy import create_engine
engine = create_engine("sqlite:///data.db")
df.to_sql("sales", engine, if_exists="replace", index=False)

Part 17 — 3 Real Projects

Project 1: Sales Dashboard Analysis

import pandas as pd
import numpy as np

# Generate sample sales data
np.random.seed(42)
n = 500

df = pd.DataFrame({
    "date": pd.date_range("2024-01-01", periods=n, freq="D")[:n],
    "product": np.random.choice(["Widget", "Gadget", "Donut", "Sprocket"], n),
    "region": np.random.choice(["North", "South", "East", "West"], n),
    "rep": np.random.choice(["Alice", "Bob", "Carol", "Dave", "Eve"], n),
    "units": np.random.randint(1, 50, n),
    "unit_price": np.random.uniform(10, 200, n).round(2),
})
df["revenue"] = (df["units"] * df["unit_price"]).round(2)
df["date"] = pd.to_datetime(df["date"])

print("=== SALES DASHBOARD ===\n")

# 1. Overall summary
print("Overall Summary:")
print(f"  Total orders:  {len(df):,}")
print(f"  Total revenue: ${df['revenue'].sum():,.2f}")
print(f"  Avg order:     ${df['revenue'].mean():.2f}")
print(f"  Date range:    {df['date'].min().date()} to {df['date'].max().date()}")

# 2. Revenue by product
print("\nRevenue by Product:")
by_product = df.groupby("product").agg(
    orders=("revenue", "count"),
    total_revenue=("revenue", "sum"),
    avg_order=("revenue", "mean"),
).sort_values("total_revenue", ascending=False)
print(by_product.round(2).to_string())

# 3. Top sales reps
print("\nTop 3 Sales Reps by Revenue:")
top_reps = (
    df.groupby("rep")["revenue"]
    .sum()
    .sort_values(ascending=False)
    .head(3)
)
for rep, rev in top_reps.items():
    print(f"  {rep}: ${rev:,.2f}")

# 4. Monthly trend
print("\nMonthly Revenue Trend:")
df["month"] = df["date"].dt.to_period("M")
monthly = df.groupby("month")["revenue"].sum()
for month, rev in monthly.head(6).items():
    print(f"  {month}: ${rev:,.2f}")

# 5. Best region per product
print("\nBest Region per Product:")
pivot = df.pivot_table(
    values="revenue",
    index="product",
    columns="region",
    aggfunc="sum",
    fill_value=0,
)
for product in pivot.index:
    best = pivot.loc[product].idxmax()
    val = pivot.loc[product].max()
    print(f"  {product}: {best} (${val:,.2f})")

Project 2: Customer Churn Analysis

import pandas as pd
import numpy as np

np.random.seed(0)
n = 1000

# Simulate customer data
df = pd.DataFrame({
    "customer_id": range(1001, 1001 + n),
    "signup_date": pd.date_range("2021-01-01", periods=n, freq="3D"),
    "plan": np.random.choice(["basic", "pro", "enterprise"], n, p=[0.5, 0.35, 0.15]),
    "monthly_spend": np.random.exponential(100, n).round(2),
    "support_tickets": np.random.poisson(1.5, n),
    "last_login_days_ago": np.random.exponential(20, n).astype(int),
    "churned": np.random.choice([0, 1], n, p=[0.75, 0.25]),
})

# Clean: cap outliers
df["monthly_spend"] = df["monthly_spend"].clip(upper=1000)
df["last_login_days_ago"] = df["last_login_days_ago"].clip(upper=365)

print("=== CUSTOMER CHURN ANALYSIS ===\n")

# Overall churn rate
churn_rate = df["churned"].mean() * 100
print(f"Overall Churn Rate: {churn_rate:.1f}%")

# Churn by plan
print("\nChurn Rate by Plan:")
churn_by_plan = df.groupby("plan").agg(
    customers=("churned", "count"),
    churned=("churned", "sum"),
    churn_rate=("churned", "mean"),
).round(3)
churn_by_plan["churn_rate"] = (churn_by_plan["churn_rate"] * 100).round(1)
churn_by_plan["churn_rate"] = churn_by_plan["churn_rate"].astype(str) + "%"
print(churn_by_plan.to_string())

# Churn vs spend
print("\nAvg Monthly Spend (churned vs retained):")
spend_comp = df.groupby("churned")["monthly_spend"].agg(["mean", "median"]).round(2)
spend_comp.index = ["Retained", "Churned"]
print(spend_comp.to_string())

# High-risk customers (not churned yet but likely)
at_risk = df[
    (df["churned"] == 0) &
    (df["last_login_days_ago"] > 30) &
    (df["support_tickets"] >= 3)
]
print(f"\nAt-risk customers (inactive 30d+ with 3+ tickets): {len(at_risk)}")
print(at_risk[["customer_id", "plan", "monthly_spend", "last_login_days_ago"]].head())

# Revenue at risk
revenue_at_risk = at_risk["monthly_spend"].sum()
print(f"Monthly revenue at risk: ${revenue_at_risk:,.2f}")

Project 3: CSV Data Cleaner Pipeline

import pandas as pd
import numpy as np
import io

# Simulate dirty CSV data
dirty_csv = """id,name,email,age,salary,join_date,department
1,Alice Smith,alice@company.com,28,75000,2022-03-15,Engineering
2,Bob Jones,bob@company.com,,80000,15-04-2021,Marketing
3,  carol white  ,CAROL@COMPANY.COM,35,90000,2021-07-20,Engineering
4,Dave Brown,,31,85000,2022/01/10,HR
5,Eve Davis,eve@company.com,27,70000,2023-02-28,Marketing
6,Frank Green,invalid-email,150,60000,2023-05-01,Engineering
7,Grace Lee,grace@company.com,30,78000,2023-08-15,
8,Henry Wu,henry@company.com,32,82000,2022-11-30,HR
9,Ivy Chen,ivy@company.com,29,71000,2021-09-01,Marketing
10,Alice Smith,alice@company.com,28,75000,2022-03-15,Engineering
"""

def clean_pipeline(csv_string: str) -> pd.DataFrame:
    """Full data cleaning pipeline."""
    # Load
    df = pd.read_csv(io.StringIO(csv_string))
    print(f"Loaded: {df.shape[0]} rows × {df.shape[1]} columns")
    print(f"Nulls: {df.isnull().sum().to_dict()}\n")

    # 1. Strip whitespace from string columns
    str_cols = df.select_dtypes("object").columns
    df[str_cols] = df[str_cols].apply(lambda c: c.str.strip())

    # 2. Normalize text case
    df["name"] = df["name"].str.title()
    df["email"] = df["email"].str.lower()

    # 3. Remove duplicates
    before = len(df)
    df = df.drop_duplicates(subset=["email"], keep="first")
    print(f"Removed {before - len(df)} duplicate emails")

    # 4. Parse dates (handle multiple formats)
    df["join_date"] = pd.to_datetime(df["join_date"], dayfirst=False, errors="coerce")

    # 5. Validate email (simple check)
    valid_email = df["email"].str.match(r"^[\w.]+@[\w.]+\.\w+$", na=False)
    bad_emails = df[~valid_email & df["email"].notna()]
    if not bad_emails.empty:
        print(f"Invalid emails found: {bad_emails['email'].tolist()}")
        df.loc[~valid_email, "email"] = np.nan

    # 6. Validate age (0–120)
    df.loc[(df["age"] < 0) | (df["age"] > 120), "age"] = np.nan

    # 7. Fill missing values
    df["age"] = df["age"].fillna(df["age"].median())
    df["department"] = df["department"].fillna("Unknown")

    # 8. Fix salary: ensure > 0
    df = df[df["salary"] > 0]

    # 9. Report final state
    print(f"\nAfter cleaning: {df.shape[0]} rows")
    print(f"Remaining nulls: {df.isnull().sum().to_dict()}")
    print(f"Date range: {df['join_date'].min().date()} to {df['join_date'].max().date()}")

    return df

clean_df = clean_pipeline(dirty_csv)
print("\nClean data:")
print(clean_df.to_string(index=False))

# Export
clean_df.to_csv("employees_clean.csv", index=False)
print("\nSaved to employees_clean.csv")

Part 18 — Performance Tips

Process large CSV in chunks

# Process a large file without loading it all
chunk_size = 10_000
results = []

for chunk in pd.read_csv("big_file.csv", chunksize=chunk_size):
    # Process each chunk
    result = chunk[chunk["revenue"] > 1000].groupby("region")["revenue"].sum()
    results.append(result)

final = pd.concat(results).groupby(level=0).sum()

Pandas vs Related Tools

Common Mistakes

Quick Reference Cheat Sheet

# Load
df = pd.read_csv("file.csv")
df = pd.read_excel("file.xlsx", sheet_name="Sheet1")

# Explore
df.shape          # (rows, cols)
df.head()         # first 5 rows
df.info()         # types + nulls
df.describe()     # stats
df["col"].value_counts()

# Select
df["col"]                    # Series
df[["col1", "col2"]]        # DataFrame
df.loc[0:5, ["col1"]]       # label-based
df.iloc[0:5, 0:2]           # position-based

# Filter
df[df["col"] > 100]
df.query("col > 100 and other == 'x'")

# Missing
df.isnull().sum()
df.fillna(0)
df.dropna()

# Manipulate
df["new"] = df["a"] + df["b"]
df = df.rename(columns={"old": "new"})
df = df.drop(columns=["col"])
df = df.drop_duplicates()
df["col"] = df["col"].astype("category")

# Sort
df.sort_values("col", ascending=False)

# GroupBy
df.groupby("col")["val"].sum()
df.groupby("col").agg(total=("val", "sum"), avg=("val", "mean"))

# Merge
pd.merge(df1, df2, on="key", how="left")
pd.concat([df1, df2], ignore_index=True)

# DateTime
df["date"] = pd.to_datetime(df["date"])
df["year"] = df["date"].dt.year
df.set_index("date").resample("ME").sum()

# Export
df.to_csv("out.csv", index=False)
df.to_excel("out.xlsx", index=False)

Learning Path

Free resources

Pandas vs Related Terms

FAQ

Do I need to know NumPy before Pandas? No, but knowing NumPy basics (arrays, np.where, np.nan) helps. You can learn them side by side — use Pandas for tables, NumPy when you need raw array operations.

What's the difference between loc and iloc? loc is label-based: df.loc[0:5] includes row 5. iloc is integer position-based: df.iloc[0:5] excludes position 5. Use loc for named indexes and column names; use iloc for numeric positions.

Should I use Pandas or Polars in 2025? Start with Pandas — it has a larger ecosystem, more tutorials, and integrates with everything. Switch to Polars when you need 5–10× speed on large datasets (1M+ rows) or want lazy evaluation. Polars syntax is similar once you know Pandas.

How do I avoid SettingWithCopyWarning? This warning appears when you modify a slice of a DataFrame. Fix: either filter and call .copy() before modifying (df2 = df[df["col"] > 0].copy()), or use .loc to assign directly (df.loc[df["col"] > 0, "new"] = value).

How do I handle large files that don't fit in RAM? Use chunksize in read_csv to process in chunks, use Parquet format (much faster I/O), switch to Dask (parallel Pandas), or use DuckDB (SQL on Parquet files with minimal RAM).

Can Pandas connect to databases? Yes. Use pd.read_sql("SELECT ...", connection) with any SQLAlchemy-compatible database (PostgreSQL, MySQL, SQLite, etc.). Use df.to_sql("table_name", engine) to write back.