Tuning Apache Spark: Powerful Big Data Processing Recipes

Course Curriculum

Chapter 1: Data Stream Development with Apache Spark, Kafka, and Spring Boot

Lecture 1: The Course Overview

Lecture 2: Discovering the Data Streaming Pipeline Blueprint Architecture

Lecture 3: Analyzing Meetup RSVPs in Real-Time

Lecture 4: Running the Collection Tier (Part I – Collecting Data)

Lecture 5: Collecting Data Via the Stream Pattern and Spring WebSocketClient API

Lecture 6: Explaining the Message Queuing Tier Role

Lecture 7: Introducing Our Message Queuing Tier –Apache Kafka

Lecture 8: Running The Collection Tier (Part II – Sending Data)

Lecture 9: Dissecting the Data Access Tier

Lecture 10: Introducing Our Data Access Tier – MongoDB

Lecture 11: Exploring Spring Reactive

Lecture 12: Exposing the Data Access Tier in Browser

Lecture 13: Diving into the Analysis Tier

Lecture 14: Streaming Algorithms For Data Analysis

Lecture 15: Introducing Our Analysis Tier – Apache Spark

Lecture 16: Plug-in Spark Analysis Tier to Our Pipeline

Lecture 17: Brief Overview of Spark RDDs

Lecture 18: Spark Streaming

Lecture 19: DataFrames, Datasets and Spark SQL

Lecture 20: Spark Structured Streaming

Lecture 21: Machine Learning in 7 Steps

Lecture 22: MLlib (Spark ML)

Lecture 23: Spark ML and Structured Streaming

Lecture 24: Spark GraphX

Lecture 25: Fault Tolerance (HML)

Lecture 26: Kafka Connect

Lecture 27: Securing Communication between Tiers

Chapter 2: Apache Spark: Tips, Tricks, & Techniques

Lecture 1: The Course Overview

Lecture 2: Using Spark Transformations to Defer Computations to a Later Time

Lecture 3: Avoiding Transformations

Lecture 4: Using reduce and reduceByKey to Calculate Results

Lecture 5: Performing Actions That Trigger Computations

Lecture 6: Reusing the Same RDD for Different Actions

Lecture 7: Delve into Spark RDDs Parent/Child Chain

Lecture 8: Using RDD in an Immutable Way

Lecture 9: Using DataFrame Operations to Transform It

Lecture 10: Immutability in the Highly Concurrent Environment

Lecture 11: Using Dataset API in an Immutable Way

Lecture 12: Detecting a Shuffle in a Processing

Lecture 13: Testing Operations That Cause Shuffle in Apache Spark

Lecture 14: Changing Design of Jobs with Wide Dependencies

Lecture 15: Using keyBy() Operations to Reduce Shuffle

Lecture 16: Using Custom Partitioner to Reduce Shuffle

Lecture 17: Saving Data in Plain Text

Lecture 18: Leveraging JSON as a Data Format

Lecture 19: Tabular Formats – CSV

Lecture 20: Using Avro with Spark

Lecture 21: Columnar Formats – Parquet

Lecture 22: Available Transformations on Key/Value Pairs

Lecture 23: Using aggregateByKey Instead of groupBy()

Lecture 24: Actions on Key/Value Pairs

Lecture 25: Available Partitioners on Key/Value Data

Lecture 26: Implementing Custom Partitioner

Lecture 27: Separating Logic from Spark Engine – Unit Testing

Lecture 28: Integration Testing Using SparkSession

Lecture 29: Mocking Data Sources Using Partial Functions

Lecture 30: Using ScalaCheck for Property-Based Testing

Lecture 31: Testing in Different Versions of Spark

Lecture 32: Creating Graph from Datasource

Lecture 33: Using Vertex API

Lecture 34: Using Edge API

Lecture 35: Calculate Degree of Vertex

Lecture 36: Calculate Page Rank

Chapter 3: Troubleshooting Apache Spark

Lecture 1: The Course Overview

Lecture 2: Eager Computations: Lazy Evaluation

Lecture 3: Caching Values: In-Memory Persistence

Lecture 4: Unexpected API Behavior: Picking the Proper RDD API

Lecture 5: Wide Dependencies: Using Narrow Dependencies

Lecture 6: Making Computations Parallel: Using Partitions

Lecture 7: Defining Robust Custom Functions: Understanding User-Defined Functions

Lecture 8: Logical Plans Hiding the Truth: Examining the Physical Plans

Lecture 9: Slow Interpreted Lambdas: Code Generation Spark Optimization

Lecture 10: Avoid Wrong Join Strategies: Using a Join Type Based on Data Volume

Lecture 11: Slow Joins: Choosing an Execution Plan for Join

Lecture 12: Distributed Joins Problem: DataFrame API

Lecture 13: TypeSafe Joins Problem: The Newest DataSet API

Lecture 14: Minimizing Object Creation: Reusing Existing Objects

Lecture 15: Iterating Transformations – The mapPartitions() Method

Lecture 16: Slow Spark Application Start: Reducing Setup Overhead

Lecture 17: Performing Unnecessary Recomputation: Reusing RDDs

Lecture 18: Repeating the Same Code in Stream Pipeline: Using Sources and Sinks

Lecture 19: Long Latency of Jobs: Understanding Batch Internals

Lecture 20: Fault Tolerance: Using Data Checkpointing

Lecture 21: Maintaining Batch and Streaming: Using Structured Streaming Pros