PySpark – Apache Spark Programming in Python for beginners

Course Curriculum

Chapter 1: Understanding Big Data and Data Lake

Lecture 1: Section Overview

Lecture 2: What is Big Data and How it Started

Lecture 3: Hadoop Architecture, History, and Evolution

Lecture 4: What is Data Lake and How it works

Lecture 5: Introducing Apache Spark and Databricks Cloud

Chapter 2: Installing and Using Apache Spark

Lecture 1: Section Overview

Lecture 2: Spark Development Environments

Lecture 3: Setup your Databricks Community Cloud Environment

Lecture 4: Introduction to Databricks Workspace

Lecture 5: Create your First Spark Application in Databricks Cloud

Lecture 6: Setup your Local Development IDE

Lecture 7: Mac Users – Setup your Local Development IDE

Lecture 8: Create your First Spark Application using IDE

Lecture 9: Source Code and Other Resources

Chapter 3: Getting Started with Apache Spark

Lecture 1: Micro Project – Problem Statement

Lecture 2: Introduction to Spark Data Frames

Lecture 3: Creating Spark Dataframe

Lecture 4: Creating Spark Tables

Lecture 5: Common problem with Databricks Community

Lecture 6: Working with Spark SQL

Lecture 7: Dataframe Transformations and Actions

Lecture 8: Applying Transformations

Lecture 9: Querying Spark Dataframe

Lecture 10: More Dataframe Transformations

Chapter 4: Spark Execution Model and Architecture

Lecture 1: Execution Methods – How to Run Spark Programs?

Lecture 2: Spark Distributed Processing Model – How your program runs?

Lecture 3: Spark Execution Modes and Cluster Managers

Lecture 4: Summarizing Spark Execution Models – When to use What?

Lecture 5: Working with PySpark Shell – Demo

Lecture 6: Installing Multi-Node Spark Cluster – Demo

Lecture 7: Working with Notebooks in Cluster – Demo

Lecture 8: Working with Spark Submit – Demo

Lecture 9: Section Summary

Chapter 5: Spark Programming Model and Developer Experience

Lecture 1: Creating Spark Project Build Configuration

Lecture 2: Configuring Spark Project Application Logs

Lecture 3: Creating Spark Session

Lecture 4: Configuring Spark Session

Lecture 5: Data Frame Introduction

Lecture 6: Data Frame Partitions and Executors

Lecture 7: Spark Transformations and Actions

Lecture 8: Spark Jobs Stages and Task

Lecture 9: Understanding your Execution Plan

Lecture 10: Unit Testing Spark Application

Lecture 11: Rounding off Summary

Chapter 6: Spark Structured API Foundation

Lecture 1: Introduction to Spark APIs

Lecture 2: Introduction to Spark RDD API

Lecture 3: Working with Spark SQL

Lecture 4: Spark SQL Engine and Catalyst Optimizer

Lecture 5: Section Summary

Chapter 7: Spark Data Sources and Sinks

Lecture 1: Spark Data Sources and Sinks

Lecture 2: Spark DataFrameReader API

Lecture 3: Reading CSV, JSON and Parquet files

Lecture 4: Creating Spark DataFrame Schema

Lecture 5: Spark DataFrameWriter API

Lecture 6: Writing Your Data and Managing Layout

Lecture 7: Spark Databases and Tables

Lecture 8: Working with Spark SQL Tables

Chapter 8: Spark Dataframe and Dataset Transformations

Lecture 1: Introduction to Data Transformation

Lecture 2: Working with Dataframe Rows

Lecture 3: DataFrame Rows and Unit Testing

Lecture 4: Dataframe Rows and Unstructured data

Lecture 5: Working with Dataframe Columns

Lecture 6: Creating and Using UDF

Lecture 7: Misc Transformations

Chapter 9: Aggregations in Apache Spark

Lecture 1: Aggregating Dataframes

Lecture 2: Grouping Aggregations

Lecture 3: Windowing Aggregations

Chapter 10: Spark Dataframe Joins

Lecture 1: Dataframe Joins and column name ambiguity

Lecture 2: Outer Joins in Dataframe

Lecture 3: Internals of Spark Join and shuffle

Lecture 4: Optimizing your joins

Lecture 5: Implementing Bucket Joins

Chapter 11: Capstone Project

Lecture 1: Project Scope and Background

Lecture 2: Data Transformation Requirement

Lecture 3: Setup your starter project

Lecture 4: Test your starter project

Lecture 5: Setup your source control and process

Lecture 6: Creating your Project CI CD Pipeline

Lecture 7: Develop Code

Lecture 8: Write Test Cases

Lecture 9: Working with Kafka integration

Lecture 10: Estimating resources for your application

Chapter 12: Keep Learning

Lecture 1: Final Word