Spark 3 on Google Cloud Platform-Beginner to Advanced Level

Course Curriculum

Chapter 1: Introduction

Lecture 1: Course Introduction and Overview

Lecture 2: GitHub repository for the course

Lecture 3: Setup a Trial GCP Account

Lecture 4: Install and Setup the Gcloud SDK

Chapter 2: Getting Started with Spark Fundamentals

Lecture 1: Introduction to Dataproc on GCP

Lecture 2: Overview of Sparks Architecture

Lecture 3: Datalake vs Datawarehouse

Lecture 4: Role of Spark in Big Data Ecosystem

Lecture 5: Overview of Spark APIs

Lecture 6: Whats new in Spark3 ?

Lecture 7: Should i be learning Spark in 2023?

Chapter 3: Getting started with Spark DataFrame API

Lecture 1: Section Introduction

Lecture 2: Lab – Create a Dataproc Cluster

Lecture 3: Lab – Walkthrough of Jupyter Notebook and different components

Lecture 4: Lab- Basic Dataframe Operations in PySpark

Lecture 5: Lab – Typecasting & timestamp column extraction

Lecture 6: Labs – Dataframe Aggregations

Lecture 7: Transformations and Actions in Spark

Lecture 8: Lab – Advanced transformations using Window Functions

Lecture 9: Lab – Rolling Window Operations

Lecture 10: Lab – Write transformed data back to a sink : GCS Bucket and BigQuery

Lecture 11: Lab – Use Spark-Submit to submit jobs to dataproc clusters

Chapter 4: Getting started with SparkSql in Spark3

Lecture 1: Introduction to SparkSql

Lecture 2: Different Types of Tables in Spark

Lecture 3: Lab – Create Tables for SparkSql

Lecture 4: Lab – Analytical Window Functions and creating permanent tables

Lecture 5: Lab – Perform Joins on Dataframes

Lecture 6: What are Partitions in Spark Dataframes?

Lecture 7: Lab – Perform repartitioning of dataframes

Lecture 8: Data Shuffling in Joins

Lecture 9: Lab – User defined functions in Spark

Chapter 5: Spark Concepts – Autoscaling , Optimization and Alerting

Lecture 1: What is a catalyst optimizer in spark ?

Lecture 2: Cache and Persist in Spark

Lecture 3: What is Autoscaling in spark and dataproc?

Lecture 4: Lab – Apply Autoscaling Policies to Dataproc Clusters

Lecture 5: Introduction to Dataproc Workflows

Lecture 6: Lab – Execute GCP Workflows

Lecture 7: Lab – Cloud Scheduler to automate Workflow Execution

Lecture 8: What is Checkpointing in Spark?

Lecture 9: What are Broadcast Joins?

Lecture 10: Lab – Setup Alerting Policies for Spark Jobs

Chapter 6: Project – End to End Batch processing pipeline using Spark

Lecture 1: Project Introduction

Lecture 2: Lab – Setup MySql Instance and Database on GCP

Lecture 3: Lab – Ingest Data into MySql

Lecture 4: Lab – Setup Dataproc with initialization actions

Lecture 5: Assignment Lab – Setup Connectivity from PySpark to MySql Db

Lecture 6: Assignment Lab – Perform transformations using PySpark

Lecture 7: Lab – Setup Workflows to execute end-to-end pipeline

Chapter 7: Real Time Analytics With Spark Structured Streaming

Lecture 1: Section Introduction

Lecture 2: Overview of PusSub Lite

Lecture 3: What are Tumbling Windows ?

Lecture 4: What is Watermarking?

Lecture 5: What are Sliding Windows?

Lecture 6: Lab – Create PubSub Lite Reservation

Lecture 7: Lab – Publish Data to PubSub and Testing using PySpark

Lecture 8: Lab – Implement Tumbling Windows

Lecture 9: Lab -Implement Tumbling Window with Watermarking

Lecture 10: Lab- Implement Sliding Windows

Chapter 8: Joins on Streaming Data

Lecture 1: Overview of Joining Streaming Dataframe

Lecture 2: Lab -Join Streaming Dataframe with Static Dataframe

Lecture 3: Lab – Join 2 Streaming Dataframes

Lecture 4: Lab – Use Watermarking in Streaming Joins

Chapter 9: Real Time Collaborative Filtering Project

Lecture 1: Overview of the Use Case

Lecture 2: Lab – Model Training using ML Library and Code Walkthrough

Lecture 3: Lab – Code Walkthrough and Publish Data

Lecture 4: Lab – Real Time Product Recommendation Model in Action

Chapter 10: Prep Up for the Interview Questions on Spark

Lecture 1: Introduction and Tips

Lecture 2: Batch Data Processing Interview Questions – Part 1

Lecture 3: Batch Data Processing Interview Questions – Part 2

Lecture 4: Batch Processing Interview Questions – Part 3

Lecture 5: Real Time Data Processing Interview Questions – Part 1

Lecture 6: Real Time Data Processing Interview Questions – Part 2