NCA-ADS: Data Manipulation
& Preparation

1. cuDF Basics — GPU DataFrames

Creating DataFrames:

• cudf.DataFrame({'col': [1,2,3]}) — from Python dict
• cudf.read_csv('file.csv') — GPU-accelerated CSV ingestion via cuIO
• cudf.read_parquet('file.parquet') — columnar format, fastest read

Selecting and Filtering:

• df['col'] — select single column (returns Series)
• df[['col1','col2']] — select multiple columns
• df.iloc[0:5] — integer-location based row selection
• df[df['col'] > 5] — boolean mask filtering

Basic Operations:

• .head() / .tail() — preview first / last rows
• .shape — tuple of (rows, columns)
• .dtypes — data type of each column
• .describe() — summary statistics (count, mean, std, min, max)

Null Handling:

• .isnull() — boolean mask of null positions
• .fillna(value) — replace nulls with a value
• .dropna() — remove rows containing nulls
• Key exam fact: cuDF nulls use Apache Arrow bitmask — represented as NA, not NaN. This is a fundamental difference from pandas float-NaN nulls.

2. Data Integration — Joins and Merges

Core syntax: cudf.merge(left, right, on='key', how='inner')

• Inner join: only rows with matching keys in both DataFrames
• Left join: all rows from left DataFrame; NaN where right has no match
• Right join: all rows from right DataFrame; NaN where left has no match
• Outer join: all rows from both DataFrames; NaN where no match on either side

Join Decision Table:

• Concatenation: cudf.concat([df1, df2]) — stacks DataFrames vertically (same schema)
• Deduplication after merge: .drop_duplicates(subset=['col']) — removes duplicate rows that can appear when join keys are not unique

3. Data Cleaning and Quality

Identifying missing data:

• df.isnull().sum() — count of nulls per column
• df.notnull() — boolean mask of non-null positions

Imputation strategies:

• df['col'].fillna(df['col'].mean()) — mean imputation for numerical columns
• df['col'].fillna(df['col'].mode()[0]) — mode imputation for categorical columns
• df['col'].fillna(method='ffill') — forward fill for time-series data

Outlier detection:

• Z-score method: flag values beyond ±3 standard deviations from the mean
• IQR method: flag values below Q1 − 1.5×IQR or above Q3 + 1.5×IQR

Other cleaning operations:

• Data type conversion: .astype('float32'), .astype('category')
• String cleaning: .str.strip(), .str.lower(), .str.replace()
• Removing duplicates: .drop_duplicates()
• Data governance: identify and handle PII columns before modeling — mask, anonymize, or drop sensitive data (names, SSNs, email addresses) to comply with regulations

4. Feature Engineering — Numerical Variables

Scaling methods:

• Min-Max normalization: scales to range 0–1. Formula: (x − min) / (max − min)
• Standard scaling (Z-score): transforms to mean=0, std=1. Formula: (x − mean) / std

When to scale:

• Required for: distance-based algorithms (KNN, SVM), linear models, neural networks — these algorithms use numeric distances between points
• Not required for: tree-based models (RandomForest, XGBoost) — trees split on thresholds, not distances

Other numerical transformations:

• Binning: cudf.cut() (equal-width bins) or cudf.qcut() (equal-frequency bins) — converts continuous to ordinal categorical
• Polynomial features: x², x×y — capture non-linear relationships between features
• Log transform: cupy.log(df['col']) — normalizes right-skewed distributions (common for income, prices)
• Interaction features: multiply or add two columns to capture joint effects between predictors

5. Feature Engineering — Categorical Variables

One-hot encoding: creates one binary column per unique category value

• Use for: nominal (unordered) categories; linear models; neural networks
• Code: cudf.get_dummies(df, columns=['category_col'])
• Caution: high-cardinality columns (hundreds+ unique values) create too many columns — memory and sparsity issues

Label encoding: assigns an integer to each unique category

• Use for: ordinal (ordered) categories (Small < Medium < Large); tree-based models
• Code: df['col'] = df['col'].astype('category').cat.codes
• Warning: implies ordering — incorrect for nominal categories with linear models (implies false ordinality)

cuDF category dtype:

• Stores unique string values once in a dictionary + integer codes per row
• Benefits: lower memory usage, faster groupby and sort (integer vs string comparison)
• Create: df['col'].astype('category')

High-cardinality columns (thousands of unique values): use target encoding (encode by target mean) or hashing instead of one-hot to avoid dimensionality explosion.

6. Class Imbalance — Handling Skewed Target Variables

The problem: when the target variable is highly imbalanced (e.g., 99% non-fraud, 1% fraud), a naive model achieves high accuracy by always predicting the majority class — while completely failing its real purpose.

Solutions:

• Oversampling minority (SMOTE): Synthetic Minority Oversampling Technique — generates synthetic samples by interpolating between existing minority class examples. Better than random duplication.
• Undersampling majority: randomly remove samples from the majority class to balance the dataset
• Class weights: pass class_weight='balanced' to cuML models — the model is penalized more for misclassifying the minority class. Available in cuML LogisticRegression and SVC.

Evaluation metrics for imbalanced data:

• Accuracy is misleading — a model predicting majority class 100% of the time achieves high accuracy
• Use: Precision, Recall, F1-score, AUC-ROC to measure performance on the minority class

7. Dimensionality Reduction and Data Sampling

Why reduce dimensions:

• Curse of dimensionality: too many features degrades model performance and increases training time
• Remove redundant or correlated features
• Speed up downstream training

PCA (Principal Component Analysis) — linear:

• Finds directions of maximum variance in the data
• Code: cuml.PCA(n_components=50).fit_transform(X)
• Choose n_components by explained variance ratio — aim for ~95% cumulative explained variance
• Best for: feature compression before modeling, removing correlated features

UMAP (Uniform Manifold Approximation and Projection) — non-linear:

• Preserves local neighborhood structure and cluster patterns better than PCA
• Code: cuml.UMAP(n_components=2).fit_transform(X)
• cuML UMAP is dramatically faster than CPU umap-learn on large datasets
• Best for: 2D/3D visualization of clusters and high-dimensional data exploration

Sampling:

• Random sampling: df.sample(frac=0.1) — take 10% random sample for quick EDA
• Stratified sampling: preserve class distribution in sample — critical for imbalanced datasets to avoid sampling bias

8. Efficient Storage — Parquet and Modern Formats

Parquet — columnar storage format (vs row-based CSV):

• Column pruning: read only needed columns — cudf.read_parquet('file.parquet', columns=['col1','col2'])
• Predicate pushdown: filter rows at read time without loading the full file
• Compression: Snappy or ZSTD compression reduces file size dramatically
• GPU-native: cudf.read_parquet() reads directly into GPU VRAM

Best practice: convert CSV sources to Parquet for repeated processing — dramatically faster reads and much lower I/O cost.

9. GPU-Accelerated ETL Workflows

ETL = Extract, Transform, Load

• Extract: cudf.read_parquet() or cudf.read_csv() — loads data into GPU VRAM
• Transform: all cuDF operations (filter, merge, fillna, groupby, etc.) execute on GPU
• Load: df.to_parquet() writes results back to storage

Keeping the pipeline on GPU: avoid .to_pandas() mid-pipeline — it triggers a slow PCIe transfer from GPU VRAM to CPU RAM, then back again for the next GPU operation.

Scaling beyond a single GPU:

• Dask: distributes cuDF operations across multiple GPUs or nodes — dask_cudf.read_parquet() returns a lazy Dask DataFrame that executes across a cluster
• GPUDirect Storage: reads NVMe data directly into GPU VRAM, bypassing CPU RAM entirely — the fastest possible data ingestion path for GPU workloads