Pandas: Python Library for Data Manipulation and Analysis

Real-world data is almost never ready to use straight out of the box. It arrives as messy CSV files, JSON exports, or database dumps — incomplete, inconsistently formatted, and riddled with missing values. Pandas is the Python library that bridges the gap between raw data and a clean, analysis-ready dataset. It gives you two powerful data structures, a rich API for filtering and transformation, and seamless integration with NumPy, Matplotlib, and Scikit-learn. Before you write a single line of machine learning code, you’ll spend a significant amount of time here.

Installation and Import

pip install pandas

import pandas as pd

Pandas Data Structures

Series (1D)

A Series is a one-dimensional labeled array. Unlike a Python list, it comes with built-in statistical methods.

data_list = [1, 2, 3, 4, 5]
series = pd.Series(data_list)

# Series with custom index labels
series_labeled = pd.Series(data_list, index=['a', 'b', 'c', 'd', 'e'])
print(series_labeled)
# a    1
# b    2
# ...

DataFrame (2D)

A DataFrame is a two-dimensional table with labeled rows and columns — think of it as a spreadsheet or SQL table in Python. Unlike a NumPy 2D array, a DataFrame can hold different data types in different columns.

data = [[1, 2, 3]]
df = pd.DataFrame(data, columns=['c0', 'c1', 'c2'], index=['r0'])

Creating DataFrames from Files

# CSV (most common format)
df = pd.read_csv('obesity_prediction.csv')

# Other formats
df_excel   = pd.read_excel('data.xlsx')
df_json    = pd.read_json('data.json')
df_parquet = pd.read_parquet('data.parquet')

# Exporting
df.to_csv('output.csv', index=False)
df.to_parquet('output.parquet')

For large datasets, prefer Parquet or Feather formats over CSV. A 1 GB CSV file can compress to ~100 MB in Parquet — roughly 10× smaller — while also being faster to read and write.

Exploring a Dataset

df.head(3)               # First 3 rows
df.tail(15)              # Last 15 rows
df.sample(n=10, random_state=29)  # 10 reproducible random rows

print(df.shape)          # (rows, columns) — e.g. (2111, 17)
print(df.size)           # Total elements (rows × columns)
print(df.columns.tolist())  # List of column names
print(df.index.tolist())    # List of row index labels

df.info()                # Column names, non-null counts, dtypes, memory
df.describe()            # Mean, std, min, max, quartiles for numeric columns

Accessing Data

`loc[]` — Label-Based Indexing

loc uses row labels and column names. Slicing with loc is inclusive of both endpoints.

# Single cell
age = df.loc[0, 'age']

# Row and column slice (inclusive)
subset = df.loc[0:5, 'age':'weight']

# Specific rows and columns
list_subset = df.loc[[0, 7, 10], ['age', 'height', 'weight']]

# All rows for a column range
all_rows = df.loc[:, 'age':'weight']

`iloc[]` — Integer Position-Based Indexing

iloc uses integer positions. Slicing is exclusive of the upper bound (Python-style).

subset = df.iloc[0:10, 0:5]       # Rows 0–9, columns 0–4
coords = df.iloc[[10, 20, 30], [0, 1, 2]]

`at[]` and `iat[]` — Fast Scalar Access

val = df.at[0, 'age']    # Label-based, faster than loc for single values
val = df.iat[5, 0]       # Integer-based, faster than iloc for single values

Column Access

df['age']                     # Single column as Series
df[['age', 'height']]         # Multiple columns as DataFrame
df.age                        # Dot notation — only works for simple names

Prefer bracket notation df['column'] over dot notation df.column. Dot notation fails silently when a column name contains spaces or matches a built-in DataFrame attribute.

Filtering Data

# Single condition
tall = df[df['height'] > 1.75]

# AND — both conditions must be true
filtered = df[(df['weight'] < 50) & (df['category'] == 'normal weight')]

# OR — at least one condition must be true
filtered = df[(df['weight'] < 50) | (df['category'] == 'normal weight')]

# String pattern matching
contains_normal = df[df['category'].str.contains('normal', regex=True)]
starts_with     = df[df['category'].str.startswith('normal')]

Always wrap each condition in its own parentheses when combining with & or |. Without the parentheses, Python’s operator precedence will produce unexpected results or raise a TypeError.

Updating and Transforming Data

Updating with `loc`

df.loc[0, 'age'] = 22                   # Single cell
df.loc[:, 'smoke'] = 'yes'              # Entire column
df.loc[0:2, 'smoke'] = 'yes'            # Range of rows
df.loc[[2, 3], 'height'] = 1.6          # Specific rows

Transforming with `apply()`

# Named function
def add_five(x):
    return x + 5

df['age'] = df['age'].apply(add_five)

# Lambda (anonymous function)
df['age'] = df['age'].apply(lambda x: x - 5)

# Conditional logic with lambda
df['age_category'] = df['age'].apply(
    lambda x: 'very young' if x < 25 else 'mature'
)

Faster Conditionals with `np.where()`

For simple conditions, np.where() is significantly faster than apply() because it is fully vectorized.

import numpy as np

df['age_category'] = np.where(df['age'] < 25, 'very young', 'not so young')

When to use each:

Scenario	Use
Simple Pass/Fail or Yes/No condition	`np.where()`
Multiple conditions or complex logic	`apply()`
Maximum performance on large datasets	`np.where()`
Custom multi-step calculations	`apply()`

Column Operations

# Add a column at a specific position
df.insert(2, 'BMI', df['weight'] / (df['height'] ** 2))

# Drop columns
df.drop(columns=['BMI'], inplace=True)
df = df.drop(columns=['col1', 'col2'])

# Delete in-place
del df['age_category']

# Rename columns
df.rename(columns={'obesity': 'category', 'smoke': 'smoker_status'}, inplace=True)

Combining DataFrames

Merging (SQL-style Joins)

df1 = pd.DataFrame({'ID': [1, 2, 3, 4], 'name': ['Alice', 'Bob', 'Charlie', 'David']})
df2 = pd.DataFrame({'ID': [3, 4, 5, 6], 'course': ['Math', 'Science', 'History', 'Art']})

inner = pd.merge(df1, df2, how='inner', on='ID')   # Intersection — IDs 3 & 4
outer = pd.merge(df1, df2, how='outer', on='ID')   # Union — all IDs with NaN gaps
left  = pd.merge(df1, df2, how='left',  on='ID')   # All df1 rows
right = pd.merge(df1, df2, how='right', on='ID')   # All df2 rows

# Different key column names
merged = pd.merge(df1, df2, how='inner', left_on='ID_1', right_on='ID_2')

Concatenating (Stacking)

# Vertical stack (rows on top of each other)
vert = pd.concat([df1, df2], axis=0, ignore_index=True)

# Horizontal stack (columns side-by-side)
horiz = pd.concat([df1, df2], axis=1)

Handling Missing Values

# Detect missing values
print(df.isna().sum())    # Count NaN per column
print(df.notna().sum())   # Count non-null per column

# Fill missing values
df.fillna(0, inplace=True)                            # All NaN → 0
df['age'].fillna(df['age'].mean(), inplace=True)      # Fill with column mean
df.fillna({'smoke': 'no', 'family_history': 'no'}, inplace=True)
df['age'].interpolate(inplace=True)                   # Interpolate from neighbors

# Drop rows/columns with any missing value
df.dropna(axis=0, inplace=True)  # Drop rows
df.dropna(axis=1, inplace=True)  # Drop columns

Data Aggregation

# Group by gender, compute mean height
print(df.groupby('gender')['height'].mean())

# Group by gender, compute multiple aggregations at once
stats = df.groupby('gender')['height'].agg(['sum', 'mean', 'std'])

Working with Strings

df['employee_name'] = df['employee_name'].str.strip()          # Remove whitespace
df['employee_name'] = df['employee_name'].str.lower()          # Lowercase
df['employee_name'] = df['employee_name'].str.upper()          # Uppercase
df['employee_name'] = df['employee_name'].str.title()          # Title Case
df['role']          = df['role'].str.replace('-', ' ')         # Replace substring
df['role_split']    = df['role'].str.split(' ')                # Split into list
df['name_short']    = df['employee_name'].str[0:4]             # Slice characters

# Check if column contains a substring
df['is_dev'] = df['role'].str.contains('Developer')

Working with Dates

# Convert string column to datetime
df['event_date'] = pd.to_datetime(df['date_str'], format='mixed')

# Extract date components using the .dt accessor
df['year']        = df['event_date'].dt.year
df['month']       = df['event_date'].dt.month
df['day']         = df['event_date'].dt.day
df['day_of_week'] = df['event_date'].dt.day_name()

# Calculate date differences
start = pd.to_datetime('2026-07-01')
end   = pd.to_datetime('2026-07-10')
duration = end - start
print(duration.days)   # 9

SQL & Databases

Data Analysis

FastAPI

Frontend

Pandas: Python Library for Data Manipulation and Analysis

Installation and Import

Pandas Data Structures

Series (1D)

DataFrame (2D)

Creating DataFrames from Files

Exploring a Dataset

Accessing Data

`loc[]` — Label-Based Indexing

`iloc[]` — Integer Position-Based Indexing

`at[]` and `iat[]` — Fast Scalar Access

Column Access

Filtering Data

Updating and Transforming Data

Updating with `loc`

Transforming with `apply()`

Faster Conditionals with `np.where()`

Column Operations

Combining DataFrames

Merging (SQL-style Joins)

Concatenating (Stacking)

Handling Missing Values

Data Aggregation

Working with Strings

Working with Dates

​Installation and Import

​Pandas Data Structures

​Series (1D)

​DataFrame (2D)

​Creating DataFrames from Files

​Exploring a Dataset

​Accessing Data

​loc[] — Label-Based Indexing

​iloc[] — Integer Position-Based Indexing

​at[] and iat[] — Fast Scalar Access

​Column Access

​Filtering Data

​Updating and Transforming Data

​Updating with loc

​Transforming with apply()

​Faster Conditionals with np.where()

​Column Operations

​Combining DataFrames

​Merging (SQL-style Joins)

​Concatenating (Stacking)

​Handling Missing Values

​Data Aggregation

​Working with Strings

​Working with Dates

Installation and Import

Pandas Data Structures

Series (1D)

DataFrame (2D)

Creating DataFrames from Files

Exploring a Dataset

Accessing Data

`loc[]` — Label-Based Indexing

`iloc[]` — Integer Position-Based Indexing

`at[]` and `iat[]` — Fast Scalar Access

Column Access

Filtering Data

Updating and Transforming Data

Updating with `loc`

Transforming with `apply()`

Faster Conditionals with `np.where()`

Column Operations

Combining DataFrames

Merging (SQL-style Joins)

Concatenating (Stacking)

Handling Missing Values

Data Aggregation

Working with Strings

Working with Dates