Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Foundations of Big Data & Spark

• Overview of the Big Data ecosystem and Spark’s role in contemporary data platforms
• Exploring Spark architecture: drivers, executors, cluster managers, lazy evaluation, DAGs, and execution planning
• Distinguishing between RDD and DataFrame APIs and determining the appropriate use case for each
• Creating and configuring SparkSession, along with understanding the fundamentals of application configuration

Module 2: PySpark DataFrames

• Ingesting and exporting data from enterprise sources and formats such as CSV, JSON, Parquet, and Delta
• Manipulating PySpark DataFrames through transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing advanced operations including window functions, timestamp handling, and processing nested data structures
• Applying data quality checks and writing clean, reusable, and maintainable PySpark code

Module 3: Kubernetes & Docker Essentials for Spark Users

• Conceptual understanding of how Docker images and Kubernetes pods map to Spark driver and executor processes, focused on cluster consumers rather than administrators
• Visualizing the lifecycle of a Spark job on Kubernetes, including pod scheduling, resource requests/limits, and session setting mappings
• Distinguishing between parameters controlled via Spark session settings and those managed by platform teams through cluster provisioning, and why this distinction is critical for tuning
• Hands-on exercise: Examine a running Spark-on-Kubernetes job within a sandbox cluster to identify driver and executor pods

Module 4: Deep Dive into Spark Session Configuration on Kubernetes

• Reasoning about the trade-offs between executor count and executor size, specifically regarding memory versus cores
• Allocating memory for drivers and executors, managing memory overhead, and translating these requirements into pod resources
• Exploring dynamic allocation on Kubernetes, identifying scenarios where it conserves resources and where it may not
• Hands-on exercise: Execute the same job with varying executor and core configurations to compare runtime and resource consumption

Module 5: Scaling Behavior & Cost Optimization

• Analyzing how adding or removing nodes impacts job behavior, completion times, and resource usage
• Evaluating the performance and cost trade-offs between using many small executors versus fewer larger ones
• Managing shuffle behavior and tuning shuffle partitions, including estimating the cost implications of configuration choices
• Hands-on exercise: Expand the sandbox cluster from five to ten nodes to observe the resulting changes in job completion time and resource consumption

Module 6: Efficient Data Ingestion & Partitioning

• Addressing the small-files problem: understanding why sources split into numerous 1–5 MB Parquet files degrade performance and distort partitioning
• Implementing repartitioning and coalescing strategies
• Controlling partition sizes during both read and write operations
• Writing Parquet efficiently to prevent the recurrence of small-file issues downstream
• Hands-on exercise: Load a dataset consisting of many small Parquet partitions, apply various repartitioning strategies, and compare performance metrics before and after optimization

Module 7: Pandas Memory Management & Failure Diagnosis

• Identifying root causes of out-of-memory errors in Pandas and recognizing associated symptoms
• Applying memory-efficient conversion patterns between Spark and Pandas
• Preventing memory spikes when exporting large datasets to CSV
• Utilizing chunked processing and data type (dtype) optimization in resource-constrained environments
• Hands-on exercise: Reproduce a common Pandas out-of-memory scenario and resolve it using chunking and dtype optimization techniques

Module 8: Polars as a Complementary Tool

• Positioning Polars relative to Pandas, focusing on performance characteristics, lazy evaluation, and memory behavior
• Understanding where Polars fits alongside PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS environments
• Hands-on exercise: Refactor a Pandas-heavy transformation using Polars to compare memory usage and processing speed

Module 9: Applying Optimization to ETL & ML Workloads

• Applying configuration, partitioning, and memory management principles across a realistic ETL pipeline
• Understanding optimization considerations specific to machine learning workloads running on the same cluster
• Implementing a practical tuning workflow to systematically diagnose cost and performance issues
• Hands-on exercise: Complete an end-to-end mini-project that combines data loading, transformation, and a simple model training step, with participants independently tuning the Spark configuration