> ## Documentation Index
> Fetch the complete documentation index at: https://fastapi2day.codewithsiva.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Pandas: Python Library for Data Manipulation and Analysis

> Learn Pandas DataFrames and Series for loading, exploring, filtering, transforming, merging, and cleaning real-world datasets in Python.

Real-world data is almost never ready to use straight out of the box. It arrives as messy CSV files, JSON exports, or database dumps — incomplete, inconsistently formatted, and riddled with missing values. **Pandas** is the Python library that bridges the gap between raw data and a clean, analysis-ready dataset. It gives you two powerful data structures, a rich API for filtering and transformation, and seamless integration with NumPy, Matplotlib, and Scikit-learn. Before you write a single line of machine learning code, you'll spend a significant amount of time here.

## Installation and Import

```bash theme={null}
pip install pandas
```

```python theme={null}
import pandas as pd
```

## Pandas Data Structures

### Series (1D)

A `Series` is a one-dimensional labeled array. Unlike a Python list, it comes with built-in statistical methods.

```python theme={null}
data_list = [1, 2, 3, 4, 5]
series = pd.Series(data_list)

# Series with custom index labels
series_labeled = pd.Series(data_list, index=['a', 'b', 'c', 'd', 'e'])
print(series_labeled)
# a    1
# b    2
# ...
```

### DataFrame (2D)

A `DataFrame` is a two-dimensional table with labeled rows and columns — think of it as a spreadsheet or SQL table in Python. Unlike a NumPy 2D array, a DataFrame can hold **different data types in different columns**.

```python theme={null}
data = [[1, 2, 3]]
df = pd.DataFrame(data, columns=['c0', 'c1', 'c2'], index=['r0'])
```

## Creating DataFrames from Files

```python theme={null}
# CSV (most common format)
df = pd.read_csv('obesity_prediction.csv')

# Other formats
df_excel   = pd.read_excel('data.xlsx')
df_json    = pd.read_json('data.json')
df_parquet = pd.read_parquet('data.parquet')

# Exporting
df.to_csv('output.csv', index=False)
df.to_parquet('output.parquet')
```

<Tip>
  For large datasets, prefer **Parquet** or **Feather** formats over CSV. A 1 GB CSV file can compress to \~100 MB in Parquet — roughly 10× smaller — while also being faster to read and write.
</Tip>

## Exploring a Dataset

```python theme={null}
df.head(3)               # First 3 rows
df.tail(15)              # Last 15 rows
df.sample(n=10, random_state=29)  # 10 reproducible random rows

print(df.shape)          # (rows, columns) — e.g. (2111, 17)
print(df.size)           # Total elements (rows × columns)
print(df.columns.tolist())  # List of column names
print(df.index.tolist())    # List of row index labels

df.info()                # Column names, non-null counts, dtypes, memory
df.describe()            # Mean, std, min, max, quartiles for numeric columns
```

## Accessing Data

### `loc[]` — Label-Based Indexing

`loc` uses **row labels** and **column names**. Slicing with `loc` is **inclusive** of both endpoints.

```python theme={null}
# Single cell
age = df.loc[0, 'age']

# Row and column slice (inclusive)
subset = df.loc[0:5, 'age':'weight']

# Specific rows and columns
list_subset = df.loc[[0, 7, 10], ['age', 'height', 'weight']]

# All rows for a column range
all_rows = df.loc[:, 'age':'weight']
```

### `iloc[]` — Integer Position-Based Indexing

`iloc` uses **integer positions**. Slicing is **exclusive** of the upper bound (Python-style).

```python theme={null}
subset = df.iloc[0:10, 0:5]       # Rows 0–9, columns 0–4
coords = df.iloc[[10, 20, 30], [0, 1, 2]]
```

### `at[]` and `iat[]` — Fast Scalar Access

```python theme={null}
val = df.at[0, 'age']    # Label-based, faster than loc for single values
val = df.iat[5, 0]       # Integer-based, faster than iloc for single values
```

### Column Access

```python theme={null}
df['age']                     # Single column as Series
df[['age', 'height']]         # Multiple columns as DataFrame
df.age                        # Dot notation — only works for simple names
```

<Note>
  Prefer bracket notation `df['column']` over dot notation `df.column`. Dot notation fails silently when a column name contains spaces or matches a built-in DataFrame attribute.
</Note>

## Filtering Data

```python theme={null}
# Single condition
tall = df[df['height'] > 1.75]

# AND — both conditions must be true
filtered = df[(df['weight'] < 50) & (df['category'] == 'normal weight')]

# OR — at least one condition must be true
filtered = df[(df['weight'] < 50) | (df['category'] == 'normal weight')]

# String pattern matching
contains_normal = df[df['category'].str.contains('normal', regex=True)]
starts_with     = df[df['category'].str.startswith('normal')]
```

<Warning>
  Always wrap each condition in its own parentheses when combining with `&` or `|`. Without the parentheses, Python's operator precedence will produce unexpected results or raise a TypeError.
</Warning>

## Updating and Transforming Data

### Updating with `loc`

```python theme={null}
df.loc[0, 'age'] = 22                   # Single cell
df.loc[:, 'smoke'] = 'yes'              # Entire column
df.loc[0:2, 'smoke'] = 'yes'            # Range of rows
df.loc[[2, 3], 'height'] = 1.6          # Specific rows
```

### Transforming with `apply()`

```python theme={null}
# Named function
def add_five(x):
    return x + 5

df['age'] = df['age'].apply(add_five)

# Lambda (anonymous function)
df['age'] = df['age'].apply(lambda x: x - 5)

# Conditional logic with lambda
df['age_category'] = df['age'].apply(
    lambda x: 'very young' if x < 25 else 'mature'
)
```

### Faster Conditionals with `np.where()`

For simple conditions, `np.where()` is significantly faster than `apply()` because it is fully vectorized.

```python theme={null}
import numpy as np

df['age_category'] = np.where(df['age'] < 25, 'very young', 'not so young')
```

**When to use each:**

| Scenario                              | Use          |
| ------------------------------------- | ------------ |
| Simple Pass/Fail or Yes/No condition  | `np.where()` |
| Multiple conditions or complex logic  | `apply()`    |
| Maximum performance on large datasets | `np.where()` |
| Custom multi-step calculations        | `apply()`    |

## Column Operations

```python theme={null}
# Add a column at a specific position
df.insert(2, 'BMI', df['weight'] / (df['height'] ** 2))

# Drop columns
df.drop(columns=['BMI'], inplace=True)
df = df.drop(columns=['col1', 'col2'])

# Delete in-place
del df['age_category']

# Rename columns
df.rename(columns={'obesity': 'category', 'smoke': 'smoker_status'}, inplace=True)
```

## Combining DataFrames

### Merging (SQL-style Joins)

```python theme={null}
df1 = pd.DataFrame({'ID': [1, 2, 3, 4], 'name': ['Alice', 'Bob', 'Charlie', 'David']})
df2 = pd.DataFrame({'ID': [3, 4, 5, 6], 'course': ['Math', 'Science', 'History', 'Art']})

inner = pd.merge(df1, df2, how='inner', on='ID')   # Intersection — IDs 3 & 4
outer = pd.merge(df1, df2, how='outer', on='ID')   # Union — all IDs with NaN gaps
left  = pd.merge(df1, df2, how='left',  on='ID')   # All df1 rows
right = pd.merge(df1, df2, how='right', on='ID')   # All df2 rows

# Different key column names
merged = pd.merge(df1, df2, how='inner', left_on='ID_1', right_on='ID_2')
```

### Concatenating (Stacking)

```python theme={null}
# Vertical stack (rows on top of each other)
vert = pd.concat([df1, df2], axis=0, ignore_index=True)

# Horizontal stack (columns side-by-side)
horiz = pd.concat([df1, df2], axis=1)
```

## Handling Missing Values

```python theme={null}
# Detect missing values
print(df.isna().sum())    # Count NaN per column
print(df.notna().sum())   # Count non-null per column

# Fill missing values
df.fillna(0, inplace=True)                            # All NaN → 0
df['age'].fillna(df['age'].mean(), inplace=True)      # Fill with column mean
df.fillna({'smoke': 'no', 'family_history': 'no'}, inplace=True)
df['age'].interpolate(inplace=True)                   # Interpolate from neighbors

# Drop rows/columns with any missing value
df.dropna(axis=0, inplace=True)  # Drop rows
df.dropna(axis=1, inplace=True)  # Drop columns
```

## Data Aggregation

```python theme={null}
# Group by gender, compute mean height
print(df.groupby('gender')['height'].mean())

# Group by gender, compute multiple aggregations at once
stats = df.groupby('gender')['height'].agg(['sum', 'mean', 'std'])
```

## Working with Strings

```python theme={null}
df['employee_name'] = df['employee_name'].str.strip()          # Remove whitespace
df['employee_name'] = df['employee_name'].str.lower()          # Lowercase
df['employee_name'] = df['employee_name'].str.upper()          # Uppercase
df['employee_name'] = df['employee_name'].str.title()          # Title Case
df['role']          = df['role'].str.replace('-', ' ')         # Replace substring
df['role_split']    = df['role'].str.split(' ')                # Split into list
df['name_short']    = df['employee_name'].str[0:4]             # Slice characters

# Check if column contains a substring
df['is_dev'] = df['role'].str.contains('Developer')
```

## Working with Dates

```python theme={null}
# Convert string column to datetime
df['event_date'] = pd.to_datetime(df['date_str'], format='mixed')

# Extract date components using the .dt accessor
df['year']        = df['event_date'].dt.year
df['month']       = df['event_date'].dt.month
df['day']         = df['event_date'].dt.day
df['day_of_week'] = df['event_date'].dt.day_name()

# Calculate date differences
start = pd.to_datetime('2026-07-01')
end   = pd.to_datetime('2026-07-10')
duration = end - start
print(duration.days)   # 9
```