File Handling & Data IO

Introduction

Every DataFrame so far was built directly in code. Real data usually lives in a file first — this topic covers moving a DataFrame to and from a CSV file, the most common exchange format.

Concept

• df.to_csv("filename.csv", index=False) — writes a DataFrame to a CSV file. index=False leaves the row index out of the file, which is usually what you want, since it isn't real data.
• pd.read_csv("filename.csv") — reads a CSV file directly into a new DataFrame.

Worked Example

import pandas as pd

data = {"Name": ["Aisha", "Rahul", "Meera"], "Marks": [91, 78, 85]}
students = pd.DataFrame(data)

students.to_csv("students.csv", index=False)

loaded = pd.read_csv("students.csv")
print(loaded)

students.to_csv("students.csv", index=False) writes the DataFrame's two columns to a plain CSV file, one row per line, without the default 0, 1, 2 index cluttering it. pd.read_csv("students.csv") reads that same file straight back into a new DataFrame, loaded — structurally identical to students, since nothing about the actual data changed in the round trip.

Note: Without index=False, to_csv() would also write an extra unnamed column holding the row index — usually not what's wanted, since that index isn't part of the actual data.

Explanation

This mirrors exactly the CSV file handling from Class 12 CS's Python course, just at the DataFrame level instead of row-by-row: writing and reading back the same data is the standard way to confirm a save actually worked correctly, before trusting a much larger real file.

Summary

df.to_csv("file.csv", index=False) writes a DataFrame to CSV, leaving out the row index. pd.read_csv("file.csv") reads a CSV file back into a new DataFrame — the standard way real data moves in and out of Pandas.

import sys

# SMART CHECK: Only install if missing
if "pandas" not in sys.modules:
    print("Toolkit: Initializing Python Environment...")
    import pyodide_js
    await pyodide_js.loadPackage("micropip")
    import micropip
    await micropip.install(['pandas', 'numpy'])
    print("Toolkit: Pandas & NumPy installed. You are ready to code!")
else:
    print("Toolkit: System is already ready. Proceed to the lesson.")

import pandas as pd

# 1. Create our Report DataFrame
data = {
    'Student': ['Amit', 'Sumit', 'Ravi'],
    'Score': [95, 85, 92]
}
df = pd.DataFrame(data)

print("--- Data to Save ---")
print(df)

# 2. Save to CSV
# We use index=False to prevent saving the 0,1,2 row numbers into the file
df.to_csv('report_card.csv', index=False)

print("\nSuccess: File 'report_card.csv' created.")

# 3. Read the file back into a NEW variable
loaded_df = pd.read_csv('report_card.csv')

print("--- Data Loaded from File ---")
print(loaded_df)

Introduction

Real data is rarely perfectly complete — a value might genuinely be unrecorded. This topic covers how Pandas represents that, and what to do about it.

Concept

Pandas represents a missing value as NaN ("Not a Number"), Python's None typically becoming NaN inside a numeric column.

• isnull() — returns True/False for each value, marking which are missing.
• dropna() — returns a new DataFrame with any row containing a NaN removed.
• fillna(value) — returns a new DataFrame with every NaN replaced by value.

Worked Example

import pandas as pd
import numpy as np

data = {"Name": ["Aisha", "Rahul", "Meera"], "Marks": [91, np.nan, 85]}
students = pd.DataFrame(data)

print(students.isnull())

print(students.dropna())
print(students.fillna(0))

students.isnull() prints True next to Rahul's Marks and False everywhere else — the one genuinely missing value. students.dropna() returns a new DataFrame with Rahul's entire row removed, since it contains a NaN. students.fillna(0) instead returns a new DataFrame with Rahul's missing mark replaced by 0, keeping his row.

Note: dropna() and fillna() solve missing data two genuinely different ways — losing the row entirely versus keeping it with a substituted value — which one is appropriate depends entirely on whether that row's other data is still useful without the missing piece.

Explanation

NaN deliberately isn't the same as 0 or an empty string — it specifically means "this value is unknown," which is exactly why isnull() exists as a dedicated check, rather than comparing a column against 0 and hoping every genuine zero and every missing value never need telling apart.

Summary

Pandas represents missing data as NaN. isnull() marks which values are missing; dropna() removes rows containing any NaN; fillna(value) replaces every NaN with a given value — both return a new DataFrame, leaving the original untouched.

import sys

# SMART CHECK: Only install if missing
if "pandas" not in sys.modules:
    print("Toolkit: Initializing Python Environment...")
    import pyodide_js
    await pyodide_js.loadPackage("micropip")
    import micropip
    await micropip.install(['pandas', 'numpy'])
    print("Toolkit: Pandas & NumPy installed. You are ready to code!")
else:
    print("Toolkit: System is already ready. Proceed to the lesson.")

import pandas as pd
import numpy as np # Importing numpy to create NaN values

# Data with missing values (np.nan)
data = {
    'Student': ['Amit', 'Sumit', 'Ravi', 'Pooja'],
    'Score': [95, np.nan, 80, np.nan] # Two missing scores
}
df = pd.DataFrame(data)

print("--- Raw Data ---")
print(df)

print("\n--- Who has missing data? (True/False) ---")
print(df.isnull())

print("\n--- Total Missing Values ---")
print(df.isnull().sum())

# Method 1: Drop (Remove rows with ANY missing value)
df_dropped = df.dropna()

print("--- After DROPNA (Rows Removed) ---")
print(df_dropped)

# Method 2: Fill (Replace NaN with 0)
df_filled = df.fillna(0)

print("\n--- After FILLNA (NaN becomes 0.0) ---")
print(df_filled)

Introduction

describe(), from earlier in this unit, summarised a whole DataFrame at once. This topic covers calling those same underlying statistics individually, on exactly the column you need.

Concept

Called directly on a numeric column (a Series):

• sum() — total of all values.
• mean() — average.
• max() / min() — largest / smallest value.
• count() — how many non-null values there are.

Worked Example

import pandas as pd

data = {"Name": ["Aisha", "Rahul", "Meera"], "Marks": [91, 78, 85]}
students = pd.DataFrame(data)

print(students["Marks"].sum())
print(students["Marks"].mean())
print(students["Marks"].max())

students["Marks"] selects the Marks column as a Series; .sum() on it gives 254 (91 + 78 + 85); .mean() gives the average, 84.666...; .max() gives 91, the highest mark in the column.

Note: These same methods work directly on a plain Series too — a DataFrame column, once selected with df["column"], genuinely is a Series, exactly as covered at the very start of this unit.

Explanation

Calling .sum()/.mean() directly on one column, rather than reading them off describe()'s full table, is the right choice once you know exactly which single statistic you need — describe() is for exploring an unfamiliar dataset broadly; these are for a specific, targeted calculation.

Summary

sum(), mean(), max(), min(), and count() — called directly on a numeric Series (including a single DataFrame column) — compute exactly the statistic named, the targeted alternative to describe()'s full summary table.

import sys

# SMART CHECK: Only install if missing
if "pandas" not in sys.modules:
    print("Toolkit: Initializing Python Environment...")
    import pyodide_js
    await pyodide_js.loadPackage("micropip")
    import micropip
    await micropip.install(['pandas', 'numpy'])
    print("Toolkit: Pandas & NumPy installed. You are ready to code!")
else:
    print("Toolkit: System is already ready. Proceed to the lesson.")

import pandas as pd

data = {
    'Month': ['Jan', 'Feb', 'Mar'],
    'Sales': [5000, 7000, 4500],
    'Expenses': [2000, 3000, 2500]
}
df = pd.DataFrame(data)
# Set Month as index so it doesn't get calculated
df = df.set_index('Month')

print("--- Sales Data ---")
print(df)

print("\n1. Total Sales (Sum):", df['Sales'].sum())
print("2. Average Expense (Mean):", df['Expenses'].mean())
print("3. Best Month (Max Sales):", df['Sales'].max())

import pandas as pd

data = {
    'Math': [90, 80],
    'Eng': [85, 95]
}
df = pd.DataFrame(data, index=['Amit', 'Sumit'])

print("--- The Scores ---")
print(df)

print("\n--- Axis 0 (Vertical: Subject Averages) ---")
print(df.mean(axis=0))

print("\n--- Axis 1 (Horizontal: Student Averages) ---")
print(df.mean(axis=1))

Introduction

The math operations from the last topic worked on one whole column at once. This topic covers computing them separately for each category within a column — grouping rows before summarising them.

Concept

df.groupby("column")

Splits a DataFrame into groups, one per unique value in column. A statistic method (like .mean() or .sum()) called on the result computes that statistic separately within each group.

Worked Example

import pandas as pd

data = {
    "Name": ["Aisha", "Rahul", "Meera", "Karan"],
    "Stream": ["Science", "Commerce", "Science", "Commerce"],
    "Marks": [91, 78, 85, 82]
}
students = pd.DataFrame(data)

print(students.groupby("Stream")["Marks"].mean())

students.groupby("Stream") splits the four rows into two groups — one for "Science" (Aisha, Meera) and one for "Commerce" (Rahul, Karan). ["Marks"].mean() then computes the average Marks within each group separately: Science averages 88.0 (91 and 85), Commerce averages 80.0 (78 and 82) — two results, one per Stream, not one overall average.

Note: groupby() on its own doesn't print anything useful by itself — it produces a grouped object waiting for a statistic method (.mean(), .sum(), .count(), and so on) to actually compute something from it.

Explanation

groupby() answers a genuinely different question than the last topic's plain .mean(): not "what's the average across everyone," but "what's the average within each Stream" — exactly the kind of per-category breakdown a spreadsheet's own grouping/pivot feature is built for, done here in one line.

Summary

df.groupby("column")["other_column"].statistic() splits a DataFrame into groups by a column's unique values, then computes the named statistic separately within each group — a per-category breakdown, not one single overall result.

import sys

# SMART CHECK: Only install if missing
if "pandas" not in sys.modules:
    print("Toolkit: Initializing Python Environment...")
    import pyodide_js
    await pyodide_js.loadPackage("micropip")
    import micropip
    await micropip.install(['pandas', 'numpy'])
    print("Toolkit: Pandas & NumPy installed. You are ready to code!")
else:
    print("Toolkit: System is already ready. Proceed to the lesson.")

import pandas as pd

# Data: Sales in different cities
data = {
    'City': ['Delhi', 'Mumbai', 'Delhi', 'Mumbai', 'Delhi'],
    'Sales': [100, 200, 150, 250, 120]
}
df = pd.DataFrame(data)

print("--- Original Data ---")
print(df)

# Group by 'City' and calculate SUM
# This combines all 'Delhi' rows and all 'Mumbai' rows
city_sales = df.groupby('City').sum()

print("\n--- Total Sales per City ---")
print(city_sales)

# Group by 'City' but get multiple stats: Mean, Max, and Count
stats = df.groupby('City')['Sales'].agg(['mean', 'max', 'count'])

print("--- Detailed Stats per City ---")
print(stats)

Introduction

This closing topic of the chapter covers combining two related DataFrames into one, matched on a shared column — the DataFrame equivalent of a SQL join from your database chapter.

Concept

pd.merge(df1, df2, on="shared_column")

Combines two DataFrames into one, matching rows where the value in shared_column is the same in both.

Worked Example

import pandas as pd

students = pd.DataFrame({"RollNo": [1, 2, 3], "Name": ["Aisha", "Rahul", "Meera"]})
marks = pd.DataFrame({"RollNo": [1, 2, 3], "Marks": [91, 78, 85]})

combined = pd.merge(students, marks, on="RollNo")
print(combined)

students holds names, marks holds scores — both keyed by the same RollNo. pd.merge(students, marks, on="RollNo") matches each row in students to the row in marks sharing the same RollNo, producing one combined DataFrame with RollNo, Name, and Marks together, one row per student.

Note: on="RollNo" requires that exact column name to exist, spelled identically, in both DataFrames — merging fails to find any matches at all if the shared column is named differently in each.

Explanation

This is precisely the same idea as an Equi-Join from your SQL chapter — combining two tables based on a shared key column — just performed on two DataFrames in Pandas instead of two tables in a database.

Summary

pd.merge(df1, df2, on="shared_column") combines two DataFrames into one, matching rows on a shared key column — the Pandas equivalent of a SQL join, closing out this chapter's coverage of reading, cleaning, summarising, and combining data.

import sys

# SMART CHECK: Only install if missing
if "pandas" not in sys.modules:
    print("Toolkit: Initializing Python Environment...")
    import pyodide_js
    await pyodide_js.loadPackage("micropip")
    import micropip
    await micropip.install(['pandas', 'numpy'])
    print("Toolkit: Pandas & NumPy installed. You are ready to code!")
else:
    print("Toolkit: System is already ready. Proceed to the lesson.")

import pandas as pd

# 1. Create DataFrames
df_students = pd.DataFrame({
    'ID': [1, 2, 3],
    'Name': ['Amit', 'Sumit', 'Ravi']
})

df_marks = pd.DataFrame({
    'ID': [1, 2, 4],
    'Score': [95, 85, 90]
})

print("--- Students ---")
print(df_students)
print("\n--- Marks ---")
print(df_marks)

# 2. Inner Merge (Only Common IDs: 1 and 2)
# Ravi (3) and Pooja (4) are dropped because they don't exist in BOTH tables.
merged_inner = pd.merge(df_students, df_marks, on='ID', how='inner')

print("\n--- Inner Join (Common Only) ---")
print(merged_inner)

# Outer Merge (Keep Everyone)
# Ravi gets NaN for Score. Pooja gets NaN for Name.
merged_outer = pd.merge(df_students, df_marks, on='ID', how='outer')

print("--- Outer Join (Full Data) ---")
print(merged_outer)