HOT

Real Time Spark Project for Beginners: Hadoop, Spark, Docker

Development

Rating3.9
Enroll Now

Real Time Spark Project for Beginners: Hadoop, Spark, Docker, available at $49.99, has an average rating of 3.85, with 24 lectures, based on 91 reviews, and has 17214 subscribers.

You will learn about Complete Development of Real Time Streaming Data Pipeline using Hadoop and Spark Cluster on Docker Setting up Single Node Hadoop and Spark Cluster on Docker Features of Spark Structured Streaming using Spark with Scala Features of Spark Structured Streaming using Spark with Python(PySpark) How to use PostgreSQL with Spark Structured Streaming Basic understanding of Apache Kafka How to build Data Visualisation using Django Web Framework and Flexmonster Fundamentals of Docker and Containerization This course is ideal for individuals who are Beginners who want to learn Apache Spark/Big Data Project Development Process and Architecture or Beginners who want to learn Real Time Streaming Data Pipeline Development Process and Architecture or Entry/Intermediate level Data Engineers and Data Scientist or Data Engineering and Data Science Aspirants or Data Enthusiast who want to learn, how to develop and run Spark Application on Docker or Anyone who is really willingness to become Big Data/Spark Developer It is particularly useful for Beginners who want to learn Apache Spark/Big Data Project Development Process and Architecture or Beginners who want to learn Real Time Streaming Data Pipeline Development Process and Architecture or Entry/Intermediate level Data Engineers and Data Scientist or Data Engineering and Data Science Aspirants or Data Enthusiast who want to learn, how to develop and run Spark Application on Docker or Anyone who is really willingness to become Big Data/Spark Developer.

Enroll now: Real Time Spark Project for Beginners: Hadoop, Spark, Docker

Summary

Title: Real Time Spark Project for Beginners: Hadoop, Spark, Docker

Price: $49.99

Average Rating: 3.85

Number of Lectures: 24

Number of Published Lectures: 24

Number of Curriculum Items: 24

Number of Published Curriculum Objects: 24

Original Price: $19.99

Quality Status: approved

Status: Live

What You Will Learn

  • Complete Development of Real Time Streaming Data Pipeline using Hadoop and Spark Cluster on Docker
  • Setting up Single Node Hadoop and Spark Cluster on Docker
  • Features of Spark Structured Streaming using Spark with Scala
  • Features of Spark Structured Streaming using Spark with Python(PySpark)
  • How to use PostgreSQL with Spark Structured Streaming
  • Basic understanding of Apache Kafka
  • How to build Data Visualisation using Django Web Framework and Flexmonster
  • Fundamentals of Docker and Containerization

Who Should Attend

  • Beginners who want to learn Apache Spark/Big Data Project Development Process and Architecture
  • Beginners who want to learn Real Time Streaming Data Pipeline Development Process and Architecture
  • Entry/Intermediate level Data Engineers and Data Scientist
  • Data Engineering and Data Science Aspirants
  • Data Enthusiast who want to learn, how to develop and run Spark Application on Docker
  • Anyone who is really willingness to become Big Data/Spark Developer

Target Audiences

  • Beginners who want to learn Apache Spark/Big Data Project Development Process and Architecture
  • Beginners who want to learn Real Time Streaming Data Pipeline Development Process and Architecture
  • Entry/Intermediate level Data Engineers and Data Scientist
  • Data Engineering and Data Science Aspirants
  • Data Enthusiast who want to learn, how to develop and run Spark Application on Docker
  • Anyone who is really willingness to become Big Data/Spark Developer

  • In many data centers, different type of servers generate large amount of data(events, Event in this case is status of the server in the data center) in real-time.

  • There is always a need to process these data in real-time and generate insights which will be used by the server/data center monitoring people and they have to track these server’s status regularly and find the resolution in case of issues occurring, for better server stability.

  • Since the data is huge and coming in real-time, we need to choose the right architecture with scalable storage and computation frameworks/technologies.

  • Hence we want to build the Real Time Data Pipeline Using Apache Kafka, Apache Spark, Hadoop, PostgreSQL, Django and Flexmonster on Docker to generate insights out of this data.

  • The Spark Project/Data Pipeline is built using Apache Spark with Scala and PySpark on Apache Hadoop Cluster which is on top of Docker.

  • Data Visualization is built using Django Web Framework and Flexmonster.

  • Apache Spark is an open-source unified analytics engine for large-scale data processing. Spark provides an interface for programming clusters with implicit data parallelism and fault tolerance.

    Apache Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation written in Java and Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

    Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for distributed storage and processing of big data using the MapReduce programming model.

    A NoSQL (originally referring to “non-SQL” or “non-relational”) database provides a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases.

Course Curriculum

Chapter 1: Introduction

Lecture 1: Introduction to Apache Spark

Lecture 2: Real Time Spark Project Overview | Building End to End Streaming Data Pipeline

Chapter 2: Environment Setup

Lecture 1: Setting up Docker Environment

Lecture 2: Create Single Node Kafka Cluster on Docker

Lecture 3: Create Single Node Apache Hadoop and Spark Cluster on Docker

Lecture 4: Setting up IntelliJ IDEA Community Edition(IDE)

Lecture 5: Setting up PyCharm Community Edition(IDE)

Lecture 6: Setting up Django Web Framework

Chapter 3: Development | Project Code Walk-through

Lecture 1: Event Simulator using Python(Server Status Detail)

Lecture 2: Building Streaming Data Pipeline using Scala | Spark Structured Streaming

Lecture 3: Building Streaming Data Pipeline using PySpark | Spark Structured Streaming

Lecture 4: Setting up PostgreSQL Database(Events Database)

Lecture 5: Building Dashboard using Django Web Framework and Flexmonster | Visualization

Chapter 4: Complete Project Demo

Lecture 1: Real Time Spark Project Demo

Lecture 2: Running Real Time Streaming Data Pipeline using Spark Cluster On Docker

Chapter 5: Docker Beginners Guide

Lecture 1: Introduction to Docker

Lecture 2: Install Docker on Ubuntu 18.04

Lecture 3: Docker Commands | Commonly Used

Lecture 4: Create First Docker Image and Container

Lecture 5: Create MySQL Docker Container

Lecture 6: Cassandra on Docker Container

Lecture 7: MongoDB on Docker Container

Lecture 8: Setting up Docker Compose

Lecture 9: How to create Docker Volume

Instructors

Rating Distribution

  • 1 stars: 6 votes
  • 2 stars: 3 votes
  • 3 stars: 18 votes
  • 4 stars: 32 votes
  • 5 stars: 32 votes

Frequently Asked Questions


How long do I have access to the course materials?


You can view and review the lecture materials indefinitely, like an on-demand channel.


Can I take my courses with me wherever I go?


Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don’t have an internet connection, some instructors also let their students download course lectures. That’s up to the instructor though, so make sure you get on their good side!

You may also like

Podcast: The top 8 ways to monetize by Podcasting

The Art of Storytelling: How to Connect with Audiences

Unbounce for Beginners: Build a Landing Page with Unbounce