Apache Spark 3 – Spark Programming in Scala for Beginners

Course Curriculum

Chapter 1: Understanding Big Data and Data Lake

Lecture 1: Section Overview

Lecture 2: What is Big Data and How it Started

Lecture 3: Hadoop Architecture, History, and Evolution

Lecture 4: What is Data Lake and How it works

Lecture 5: Introducing Apache Spark and Databricks Cloud

Lecture 6: Source Code and Other Resources

Chapter 2: Installing and Using Apache Spark

Lecture 1: Spark Development Environments

Lecture 2: Apache Spark in Local Mode Command Line REPL

Lecture 3: Apache Spark in the IDE – IntelliJ IDEA

Lecture 4: Apache Spark in Cloud – Databricks Community and Notebooks

Lecture 5: Apache Spark in Hadoop Ecosystem – Zeppelin Notebooks

Chapter 3: Spark Execution Model and Architecture

Lecture 1: Execution Methods – How to Run Spark Programs?

Lecture 2: Spark Distributed Processing Model – How your program runs?

Lecture 3: Spark Execution Modes and Cluster Managers

Lecture 4: Summarizing Spark Execution Models – When to use What?

Lecture 5: Working with Spark Shell – Demo

Lecture 6: Installing Multi-Node Spark Cluster – Demo

Lecture 7: Working with Notebooks in Cluster – Demo

Lecture 8: Working with Spark Submit – Demo

Lecture 9: Section Summary

Chapter 4: Spark Programming Model and Developer Experience

Lecture 1: Creating Spark Project Build Configuration

Lecture 2: Configuring Spark Project Application Logs

Lecture 3: Creating Spark Session

Lecture 4: Configuring Spark Session

Lecture 5: Data Frame Introduction

Lecture 6: Data Frame Partitions and Executors

Lecture 7: Spark Transformations and Actions

Lecture 8: Spark Jobs Stages and Task

Lecture 9: Understanding your Execution Plan

Lecture 10: Unit Testing Spark Application

Lecture 11: Debugging Spark Driver and Executor

Lecture 12: Spark Application Logs in a Cluster

Lecture 13: Rounding off Summary

Chapter 5: Spark Structured API Foundation

Lecture 1: Introduction to Spark APIs

Lecture 2: Introduction to Spark RDD API

Lecture 3: Dataset Vs Dataframe

Lecture 4: Working with Spark Dataset

Lecture 5: Working with Spark SQL

Lecture 6: Spark SQL Engine and Catalyst Optimizer

Lecture 7: Section Summary

Chapter 6: Spark Data Sources and Sinks

Lecture 1: Introduction to Spark Sources and Sinks

Lecture 2: Spark DataFrameReader API

Lecture 3: Reading CSV, JSON and Parquet files

Lecture 4: Creating Spark DataFrame Schema

Lecture 5: Spark DataFrameWriter API

Lecture 6: Writing Your Data and Managing Layout

Lecture 7: Spark Databases and Tables

Lecture 8: Working with Spark SQL Tables

Chapter 7: Spark Dataframe and Dataset Transformations

Lecture 1: Introduction to Data Transformation

Lecture 2: Working with Dataframe Rows

Lecture 3: Dataframe Rows and Unit Testing

Lecture 4: Dataframe Rows and Unstructured data

Lecture 5: Working with Dataframe Columns

Lecture 6: Creating and Using UDF

Lecture 7: Miscellaneous Transformations

Chapter 8: Aggregations in Apache Spark

Lecture 1: Aggregating Dataframes

Lecture 2: Grouping Aggregations

Lecture 3: Windowing Aggregations

Chapter 9: Spark Dataframe Joins

Lecture 1: Dataframe Joins and column name ambiguity

Lecture 2: Outer Joins in Dataframe

Lecture 3: Internals of Spark Join and shuffle

Lecture 4: Optimizing your joins

Lecture 5: Implementing Bucket Joins

Chapter 10: Keep Learning

Lecture 1: Final Word

Lecture 2: Bonus Lecture : Get Extra

Chapter 11: Archived- Apache Spark Introduction

Lecture 1: BigData History and Primer

Lecture 2: Understanding the Data Lake Landscape

Lecture 3: What is Apache Spark – An Introduction and Overview