System Design for Big Data Pipelines

Course Curriculum

Chapter 1: Introduction & Expectations

Lecture 1: Need for Quality Pipeline Design

Lecture 2: Course Coverage and Pre-requisites

Lecture 3: Cloud Serverless Technologies

Chapter 2: Building Blocks for Big Data Pipelines

Lecture 1: The Big Data Pipeline Network

Lecture 2: Data Acquisition Blocks

Lecture 3: Data Transport Blocks

Lecture 4: Data Processing Blocks

Lecture 5: Data Storage Blocks

Lecture 6: Data Serving Blocks

Lecture 7: Data Pipeline Infrastructure

Lecture 8: Data Pipeline Operations

Chapter 3: System Design Process

Lecture 1: System Design Process Overview

Lecture 2: Analyze Functional Requirements

Lecture 3: Analyze Pipeline Input

Lecture 4: Analyze Non-functional Requirements

Lecture 5: Draw a Pipeline Flowchart

Lecture 6: Create a Skeleton Design

Lecture 7: Analyze Scaling

Lecture 8: Select Technologies

Lecture 9: Design Infrastructure and Operations

Lecture 10: Develop a Test Strategy

Chapter 4: Scalable Pipelines – Design Principles

Lecture 1: Batch vs Realtime Pipelines

Lecture 2: Distributed Architectures

Lecture 3: Microservices based Architectures

Lecture 4: Batch Pipelines – Best Practices

Lecture 5: Realtime Pipelines – Best Practices

Lecture 6: Performance Benchmarking for Big Data Pipelines

Chapter 5: Data Acquisition Design

Lecture 1: File Transfer Pattern

Lecture 2: Extraction Client Pattern

Lecture 3: Ingestion API Pattern

Lecture 4: Pub Sub Acquisition Pattern

Lecture 5: Data Acquisition Design Practices

Chapter 6: Data Transport Design

Lecture 1: Extract Load Pattern

Lecture 2: Request Response Pattern

Lecture 3: Event Streaming Pattern

Lecture 4: Data Transport Design Practices

Chapter 7: Data Processing & Transformation Design

Lecture 1: Data Processing Patterns

Lecture 2: Distributed Processing with Big Data

Lecture 3: Batch Processing Design Practices – Part 1

Lecture 4: Batch Processing Design Practices – Part 2

Lecture 5: Stream Processing Design Practices

Lecture 6: Batch vs Realtime Processing

Lecture 7: Input and Output Considerations for Processing

Lecture 8: Processing Engine Technologies

Chapter 8: Storage Design

Lecture 1: Distributed File System Pattern

Lecture 2: Relational Database Pattern

Lecture 3: Document Database Pattern

Lecture 4: Columnar Database Pattern

Lecture 5: Graph Database Pattern

Lecture 6: Distributed Cache Pattern

Lecture 7: Data Storage Design Practices – 1

Lecture 8: Data Storage Design Practices – 2

Chapter 9: Serving Design

Lecture 1: Query Interface Pattern

Lecture 2: Serving API Pattern

Lecture 3: Push Client Pattern

Lecture 4: Publish Subscribe Pattern

Lecture 5: Data Serving Design Practices

Chapter 10: Infrastructure and Deployments

Lecture 1: Infrastructure Technologies

Lecture 2: Microservices Deployments

Lecture 3: Processing Jobs Deployments

Lecture 4: Databases and Queues Deployments

Lecture 5: Geographical Distribution

Chapter 11: Security

Lecture 1: Pipeline Security by Design

Lecture 2: Secure External Interfaces

Lecture 3: Secure Data Storage

Lecture 4: Privacy Considerations

Lecture 5: Multi-Tenancy Considerations

Chapter 12: Serviceability

Lecture 1: Elements of Serviceability

Lecture 2: Monitoring Pipelines

Lecture 3: Data to Monitor

Lecture 4: Pipeline Troubleshooting

Chapter 13: Use Case I : Customer Journey Analytics (CJA)

Lecture 1: Problem Definition for CJA

Lecture 2: Study CJA Functional Requirements

Lecture 3: Analyze CJA Input Data

Lecture 4: Study CJA Non-Functional Requirements

Lecture 5: Study CJA Pipeline Flowchart

Lecture 6: Create CJA Skeleton Design

Lecture 7: Analyze CJA Scaling

Lecture 8: Select Technologies for CJA

Lecture 9: Design Infrastructure and Operations for CJA

Chapter 14: Use Case II : Suspicious Login Alerting (SLA)

Lecture 1: Problem Definition for SLA

Lecture 2: Study SLA Functional Requirements

Lecture 3: Analyze SLA Input Data

Lecture 4: Study SLA Non-Functional Requirements

Lecture 5: Draw SLA Pipeline Flowchart

Lecture 6: Create SLA Skeleton Design