Learn By Example: Hadoop, MapReduce for Big Data problems

Course Curriculum

Chapter 1: Introduction

Lecture 1: You, this course and Us

Chapter 2: Why is Big Data a Big Deal

Lecture 1: The Big Data Paradigm

Lecture 2: Serial vs Distributed Computing

Lecture 3: What is Hadoop?

Lecture 4: HDFS or the Hadoop Distributed File System

Lecture 5: MapReduce Introduced

Lecture 6: YARN or Yet Another Resource Negotiator

Chapter 3: Installing Hadoop in a Local Environment

Lecture 1: Hadoop Install Modes

Lecture 2: Hadoop Standalone mode Install

Lecture 3: Hadoop Pseudo-Distributed mode Install

Chapter 4: The MapReduce "Hello World"

Lecture 1: The basic philosophy underlying MapReduce

Lecture 2: MapReduce – Visualized And Explained

Lecture 3: MapReduce – Digging a little deeper at every step

Lecture 4: "Hello World" in MapReduce

Lecture 5: The Mapper

Lecture 6: The Reducer

Lecture 7: The Job

Chapter 5: Run a MapReduce Job

Lecture 1: Get comfortable with HDFS

Lecture 2: Run your first MapReduce Job

Chapter 6: Juicing your MapReduce – Combiners, Shuffle and Sort and The Streaming API

Lecture 1: Parallelize the reduce phase – use the Combiner

Lecture 2: Not all Reducers are Combiners

Lecture 3: How many mappers and reducers does your MapReduce have?

Lecture 4: Parallelizing reduce using Shuffle And Sort

Lecture 5: MapReduce is not limited to the Java language – Introducing the Streaming API

Lecture 6: Python for MapReduce

Chapter 7: HDFS and Yarn

Lecture 1: HDFS – Protecting against data loss using replication

Lecture 2: HDFS – Name nodes and why they're critical

Lecture 3: HDFS – Checkpointing to backup name node information

Lecture 4: Yarn – Basic components

Lecture 5: Yarn – Submitting a job to Yarn

Lecture 6: Yarn – Plug in scheduling policies

Lecture 7: Yarn – Configure the scheduler

Chapter 8: MapReduce Customizations For Finer Grained Control

Lecture 1: Setting up your MapReduce to accept command line arguments

Lecture 2: The Tool, ToolRunner and GenericOptionsParser

Lecture 3: Configuring properties of the Job object

Lecture 4: Customizing the Partitioner, Sort Comparator, and Group Comparator

Chapter 9: The Inverted Index, Custom Data Types for Keys, Bigram Counts and Unit Tests!

Lecture 1: The heart of search engines – The Inverted Index

Lecture 2: Generating the inverted index using MapReduce

Lecture 3: Custom data types for keys – The Writable Interface

Lecture 4: Represent a Bigram using a WritableComparable

Lecture 5: MapReduce to count the Bigrams in input text

Lecture 6: Setting up your Hadoop project

Lecture 7: Test your MapReduce job using MRUnit

Chapter 10: Input and Output Formats and Customized Partitioning

Lecture 1: Introducing the File Input Format

Lecture 2: Text And Sequence File Formats

Lecture 3: Data partitioning using a custom partitioner

Lecture 4: Make the custom partitioner real in code

Lecture 5: Total Order Partitioning

Lecture 6: Input Sampling, Distribution, Partitioning and configuring these

Lecture 7: Secondary Sort

Chapter 11: Recommendation Systems using Collaborative Filtering

Lecture 1: Introduction to Collaborative Filtering

Lecture 2: Friend recommendations using chained MR jobs

Lecture 3: Get common friends for every pair of users – the first MapReduce

Lecture 4: Top 10 friend recommendation for every user – the second MapReduce

Chapter 12: Hadoop as a Database

Lecture 1: Structured data in Hadoop

Lecture 2: Running an SQL Select with MapReduce

Lecture 3: Running an SQL Group By with MapReduce

Lecture 4: A MapReduce Join – The Map Side

Lecture 5: A MapReduce Join – The Reduce Side

Lecture 6: A MapReduce Join – Sorting and Partitioning

Lecture 7: A MapReduce Join – Putting it all together

Chapter 13: K-Means Clustering

Lecture 1: What is K-Means Clustering?

Lecture 2: A MapReduce job for K-Means Clustering

Lecture 3: K-Means Clustering – Measuring the distance between points

Lecture 4: K-Means Clustering – Custom Writables for Input/Output

Lecture 5: K-Means Clustering – Configuring the Job

Lecture 6: K-Means Clustering – The Mapper and Reducer

Lecture 7: K-Means Clustering : The Iterative MapReduce Job

Chapter 14: Setting up a Hadoop Cluster

Lecture 1: Manually configuring a Hadoop cluster (Linux VMs)

Lecture 2: Getting started with Amazon Web Servicies

Lecture 3: Start a Hadoop Cluster with Cloudera Manager on AWS

Chapter 15: Appendix

Lecture 1: Setup a Virtual Linux Instance (For Windows users)

Lecture 2: [For Linux/Mac OS Shell Newbies] Path and other Environment Variables