Spark Scala coding framework, testing, Structured streaming

Course Curriculum

Chapter 1: Introduction

Lecture 1: Introduction

Lecture 2: What is Big Data Spark?

Lecture 3: Big Data Hadoop concepts and hands-on labs for beginners

Chapter 2: Environment Setup & Spark Scala basics

Lecture 1: Installing JDK 11 on a Windows Machine

Lecture 2: Installing IntelliJ and Winutils for Spark Scala Hive programming on Windows

Lecture 3: Scala Basics

Lecture 4: For Mac users – Installing JDK and IntelliJ and Spark Scala Hive Hello World

Lecture 5: Installing PostgreSQL

Lecture 6: psql command line interface for PostgreSQL

Lecture 7: Fetching PostgresSQL data to a Spark DataFrame

Lecture 8: Importing a project into IntelliJ

Chapter 3: Coding Best Practices

Lecture 1: Organizing code with Objects and Methods

Lecture 2: Implementing Log4j SLf4j Logging

Lecture 3: Exception Handling with try, catch, Option, Some and None

Chapter 4: A Data Pipeline with Hive, Spark and Postgres

Lecture 1: Reading from Hive and Writing to Postgres

Lecture 2: Reading Configuration from JSON using Typesafe

Lecture 3: Reading command-line arguments and debugging in InjtelliJ

Lecture 4: Writing data to a Hive Table

Lecture 5: Managing input parameters using a Scala Case Class

Lecture 6: Intellij Maven troubleshooting tips

Chapter 5: Spark Scala Unit Testing using ScalaTest

Lecture 1: Scala Unit Testing using JUnit & ScalaTest

Lecture 2: Spark Transformation unit testing using ScalaTest

Lecture 3: Unit testing to catch an Exception

Lecture 4: Catching Exception using assertThrows

Lecture 5: Throwing Custom Error and Intercepting Error Message

Lecture 6: Testing with assertResult

Lecture 7: Testing with Matchers

Lecture 8: Failing tests intentionally

Lecture 9: Sharing fixtures

Chapter 6: Running the application on Cloudera QuickStart VM on GCP

Lecture 1: Exporting the project to an uber jar

Lecture 2: Signing up for GCP free trial

Lecture 3: Cloudera QuickStart VM Installation on GCP

Lecture 4: Running Spark 2 with Hive on Cloudera QuickStart VM

Lecture 5: Uber Jar spark-submit on Cloudera QuickStart VM

Lecture 6: Doing spark submit locally

Chapter 7: Spark Scala – Structured Streaming

Lecture 1: Structured Streaming concepts

Lecture 2: Streaming data from files

Lecture 3: Batch Vs Streaming code

Lecture 4: Writing streaming data to a Hive table

Lecture 5: Streaming Aggregation

Lecture 6: Filtering Stream

Lecture 7: Adding timestamp to streaming data

Lecture 8: Aggregation in a time window

Lecture 9: Tumbling window and Sliding window

Lecture 10: PySpark coding framework course preview

Lecture 11: Congratulations & Thank You

Chapter 8: Appendix – Big Data Hadoop Hive for beginners

Lecture 1: Big Data concepts

Lecture 2: Hadoop concepts

Lecture 3: Hadoop Distributed File System (HDFS)

Lecture 4: Understanding Google Cloud (GCP) Dataproc

Lecture 5: Signing up for a Google Cloud free trial

Lecture 6: Storing a file in HDFS

Lecture 7: MapReduce and YARN

Lecture 8: Hive

Lecture 9: Querying HDFS data using Hive

Lecture 10: Deleting the Cluster

Lecture 11: Analyzing a billion records with Hive