Apache Spark with Scala – Hands On with Big Data!

Course Curriculum

Chapter 1: Getting Started

Lecture 1: Udemy 101: Getting the Most From This Course

Lecture 2: Alternate download link for the ml-100k dataset

Lecture 3: WARNING: DO NOT INSTALL JAVA 21+ IN THE NEXT LECTURE

Lecture 4: Introduction, and installing the course materials, IntelliJ, and Scala

Lecture 5: Introduction to Apache Spark

Lecture 6: Important note

Chapter 2: Scala Crash Course [Optional]

Lecture 1: [Activity] Scala Basics

Lecture 2: [Exercise] Flow Control in Scala

Lecture 3: [Exercise] Functions in Scala

Lecture 4: [Exercise] Data Structures in Scala

Chapter 3: Using Resilient Distributed Datasets (RDDs)

Lecture 1: The Resilient Distributed Dataset

Lecture 2: Ratings Histogram Example

Lecture 3: Spark Internals

Lecture 4: Key / Value RDD's, and the Average Friends by Age example

Lecture 5: [Activity] Running the Average Friends by Age Example

Lecture 6: Filtering RDD's, and the Minimum Temperature by Location Example

Lecture 7: [Activity] Running the Minimum Temperature Example, and Modifying it for Maximum

Lecture 8: [Activity] Counting Word Occurrences using Flatmap()

Lecture 9: [Activity] Improving the Word Count Script with Regular Expressions

Lecture 10: [Activity] Sorting the Word Count Results

Lecture 11: [Exercise] Find the Total Amount Spent by Customer

Lecture 12: [Exercise] Check your Results, and Sort Them by Total Amount Spent

Lecture 13: Check Your Results and Implementation Against Mine

Chapter 4: SparkSQL, DataFrames, and DataSets

Lecture 1: Introduction to SparkSQL

Lecture 2: [Activity] Using SparkSQL

Lecture 3: [Activity] Using DataSets

Lecture 4: [Exercise] Implement the "Friends by Age" example using DataSets

Lecture 5: Exercise Solution: Friends by Age, with Datasets.

Lecture 6: [Activity] Word Count example, using Datasets

Lecture 7: [Activity] Revisiting the Minimum Temperature example, with Datasets

Lecture 8: [Exercise] Implement the "Total Spent by Customer" problem with Datasets

Lecture 9: Exercise Solution: Total Spent by Customer with Datasets

Chapter 5: Advanced Examples of Spark Programs

Lecture 1: [Activity] Find the Most Popular Movie

Lecture 2: [Activity] Use Broadcast Variables to Display Movie Names

Lecture 3: [Activity] Find the Most Popular Superhero in a Social Graph

Lecture 4: [Exercise] Find the Most Obscure Superheroes

Lecture 5: Exercise Solution: Find the Most Obscure Superheroes

Lecture 6: Superhero Degrees of Separation: Introducing Breadth-First Search

Lecture 7: Superhero Degrees of Separation: Accumulators, and Implementing BFS in Spark

Lecture 8: [Activity] Superhero Degrees of Separation: Review the code, and run it!

Lecture 9: Item-Based Collaborative Filtering in Spark, cache(), and persist()

Lecture 10: [Activity] Running the Similar Movies Script using Spark's Cluster Manager

Lecture 11: [Exercise] Improve the Quality of Similar Movies

Chapter 6: Running Spark on a Cluster

Lecture 1: [Activity] Using spark-submit to run Spark driver scripts

Lecture 2: [Activity] Packaging driver scripts with SBT

Lecture 3: [Exercise] Package a Script with SBT and Run it Locally with spark-submit

Lecture 4: Exercise solution: Using SBT and spark-submit

Lecture 5: Introducing Amazon Elastic MapReduce

Lecture 6: Creating Similar Movies from One Million Ratings on EMR

Lecture 7: Partitioning

Lecture 8: Best Practices for Running on a Cluster

Lecture 9: Troubleshooting, and Managing Dependencies

Chapter 7: Machine Learning with Spark ML

Lecture 1: Introducing MLLib

Lecture 2: [Activity] Using MLLib to Produce Movie Recommendations

Lecture 3: Linear Regression with MLLib

Lecture 4: [Activity] Running a Linear Regression with Spark

Lecture 5: [Exercise] Predict Real Estate Values with Decision Trees in Spark

Lecture 6: Exercise Solution: Predicting Real Estate with Decision Trees in Spark

Chapter 8: Intro to Spark Streaming

Lecture 1: The DStream API for Spark Streaming

Lecture 2: [Activity] Real-time Monitoring of the Most Popular Hashtags on Twitter

Lecture 3: Structured Streaming

Lecture 4: [Activity] Using Structured Streaming for real-time log analysis

Lecture 5: [Exercise] Windowed Operations with Structured Streaming

Lecture 6: Exercise Solution: Top URL's in a 30-second Window

Chapter 9: Intro to GraphX

Lecture 1: GraphX, Pregel, and Breadth-First-Search with Pregel.

Lecture 2: Using the Pregel API with Spark GraphX

Lecture 3: [Activity] Superhero Degrees of Separation using GraphX

Chapter 10: You Made It! Where to Go from Here.

Lecture 1: Learning More, and Career Tips

Lecture 2: Bonus Lecture: More courses to explore!