# PipeCraft: Python & Pandas for Data Engineering Cheat Sheet

This cheatsheet provides highly detailed explanations and production code snippets for Python and Pandas virtual environments, data cleaning, and processing.

---

## 📂 1. Virtual Environment & Packages

### `python3 -m venv venv`
* **What it does:** Creates a lightweight, isolated virtual environment folder named `venv` in your project directory.
* **Why it's used:** Prevents Python library version conflicts between different software projects on the same host machine.

### `source venv/bin/activate`
* **What it does:** Activates the virtual environment, switching the host's active python and pip executables to the local `venv` folder.
* **Note:** On Windows, run `venv\\Scripts\\activate.bat` instead.

### `pip install -r requirements.txt`
* **What it does:** Installs all the Python library versions specified in the `requirements.txt` list.

---

## 🐼 2. Pandas Core Data Transformations

### `pd.read_csv('raw.csv', sep=',', encoding='utf-8')`
* **What it does:** Reads a flat CSV text file into a tabular DataFrame object.
* **Parameters:** `sep` defines the delimiter (comma), and `encoding` ensures special characters are decoded correctly.

### `df.dropna(subset=['user_id'])`
* **What it does:** Removes row rows where the specified columns (`user_id`) have null or empty values.
* **Why it's used:** Maintains data referential integrity prior to writing transactions to the data warehouse.

### `df['email'] = df['email'].str.strip().str.lower()`
* **What it does:** Trims leading/trailing whitespace spaces and converts the string characters to lowercase.
* **Why it's used:** Standardizes user identifiers so that database JOINs succeed without case mismatch issues.

### `df['age'] = df['age'].fillna(df['age'].mean()).astype(int)`
* **What it does:** Fills empty `NaN` values in the 'age' column with the column's mean average, then casts the data type to integer.

### `df.groupby('category').agg({'amount': ['sum', 'count']})`
* **What it does:** Groups the dataset by a categorical column and runs mathematical aggregations (finding sum total and transaction count).

### `df.to_parquet('output.parquet', compression='snappy')`
* **What it does:** Exports the DataFrame object to a columnar, binary Parquet format compressed using the Snappy algorithm.
* **Why it's used:** Reduces storage costs in S3/MinIO by up to 80% compared to flat CSV files and increases scan speeds.