Scalable programming with Scala and Spark

Course Curriculum

Chapter 1: You, This Course and Us

Lecture 1: You, This Course and Us

Lecture 2: Course Materials

Lecture 3: Installing Scala and Hello World

Chapter 2: Introduction to Spark

Lecture 1: What does Donald Rumsfeld have to do with data analysis?

Lecture 2: Why is Spark so cool?

Lecture 3: An introduction to RDDs – Resilient Distributed Datasets

Lecture 4: Built-in libraries for Spark

Lecture 5: Installing Spark

Lecture 6: The Spark Shell

Lecture 7: See it in Action : Munging Airlines Data with Spark

Lecture 8: Transformations and Actions

Chapter 3: Resilient Distributed Datasets

Lecture 1: RDD Characteristics: Partitions and Immutability

Lecture 2: RDD Characteristics: Lineage, RDDs know where they came from

Lecture 3: What can you do with RDDs?

Lecture 4: Create your first RDD from a file

Lecture 5: Average distance travelled by a flight using map() and reduce() operations

Lecture 6: Get delayed flights using filter(), cache data using persist()

Lecture 7: Average flight delay in one-step using aggregate()

Lecture 8: Frequency histogram of delays using countByValue()

Chapter 4: Advanced RDDs: Pair Resilient Distributed Datasets

Lecture 1: Special Transformations and Actions

Lecture 2: Average delay per airport, use reduceByKey(), mapValues() and join()

Lecture 3: Average delay per airport in one step using combineByKey()

Lecture 4: Get the top airports by delay using sortBy()

Lecture 5: Lookup airport descriptions using lookup(), collectAsMap(), broadcast()

Chapter 5: Advanced Spark: Accumulators, Spark Submit, MapReduce , Behind The Scenes

Lecture 1: Get information from individual processing nodes using accumulators

Lecture 2: Long running programs using spark-submit

Lecture 3: Spark-Submit with Scala – A demo

Lecture 4: Behind the scenes: What happens when a Spark script runs?

Lecture 5: Running MapReduce operations

Chapter 6: PageRank: Ranking Search Results

Lecture 1: What is PageRank?

Lecture 2: The PageRank algorithm

Lecture 3: Implement PageRank in Spark

Lecture 4: Join optimization in PageRank using Custom Partitioning

Chapter 7: Spark SQL

Lecture 1: Dataframes: RDDs + Tables

Chapter 8: MLlib in Spark: Build a recommendations engine

Lecture 1: Collaborative filtering algorithms

Lecture 2: Latent Factor Analysis with the Alternating Least Squares method

Lecture 3: Music recommendations using the Audioscrobbler dataset

Lecture 4: Implement code in Spark using MLlib

Chapter 9: Spark Streaming

Lecture 1: Introduction to streaming

Lecture 2: Implement stream processing in Spark using Dstreams

Lecture 3: Stateful transformations using sliding windows

Chapter 10: Graph Libraries

Lecture 1: The Marvel social network using Graphs

Chapter 11: Scala Language Primer

Lecture 1: Scala – A "better Java"?

Lecture 2: How do Classes work in Scala?

Lecture 3: Classes in Scala – continued

Lecture 4: Functions are different from Methods

Lecture 5: Collections in Scala

Lecture 6: Map, Flatmap – The Functional way of looping

Lecture 7: First Class Functions revisited

Lecture 8: Partially Applied Functions

Lecture 9: Closures

Lecture 10: Currying

Chapter 12: Supplementary Installs

Lecture 1: Installing Intellij

Lecture 2: Installing Anaconda

Lecture 3: [For Linux/Mac OS Shell Newbies] Path and other Environment Variables