Transform Data into Insights with Dagster and Deepnote

Transform Data into Insights with Dagster and Deepnote, available at $59.99, has an average rating of 4.1, with 38 lectures, based on 14 reviews, and has 207 subscribers.

You will learn about Turn messy, real-world data into actionable insights. Gain familiarity with tools such as Deepnote, Dagster, and Metabase. Use Deepnote as a data engineering development environment. Generate realistic development data for analysis and visualization. Learn data exploration and preprocessing techniques using Python and SQL. Clean and normalize data from various sources, such as relational databases, JSON, .xls files and more. Set up Dagster to orchestrate your data pipeline. Integrate the processing logic into a scalable ETL pipeline with Dagster. Deploy your pipeline to Dagster Cloud (serverless) Optimize processing through techniques such as parallelization or streamed processing. Create powerful data visualizations using Metabase. This course is ideal for individuals who are Developers seeking to build scalable and efficient ETL pipelines. or Entrepreneurs looking to leverage data for business growth. or Data analysts and scientists who want to streamline their data processing workflow. or Business professionals looking to improve their data-driven decision-making abilities. or Students and recent graduates interested in a career in data engineering. or Data managers tasked with organizing and making data accessible for analysis. or Project managers looking to implement data-driven solutions for clients or company. or Individuals interested in learning cutting-edge tools and techniques in data engineering. It is particularly useful for Developers seeking to build scalable and efficient ETL pipelines. or Entrepreneurs looking to leverage data for business growth. or Data analysts and scientists who want to streamline their data processing workflow. or Business professionals looking to improve their data-driven decision-making abilities. or Students and recent graduates interested in a career in data engineering. or Data managers tasked with organizing and making data accessible for analysis. or Project managers looking to implement data-driven solutions for clients or company. or Individuals interested in learning cutting-edge tools and techniques in data engineering.

Enroll now: Transform Data into Insights with Dagster and Deepnote

Summary

Title: Transform Data into Insights with Dagster and Deepnote

Price: $59.99

Average Rating: 4.1

Number of Lectures: 38

Number of Published Lectures: 38

Number of Curriculum Items: 38

Number of Published Curriculum Objects: 38

Original Price: $59.99

Quality Status: approved

Status: Live

What You Will Learn

Turn messy, real-world data into actionable insights.
Gain familiarity with tools such as Deepnote, Dagster, and Metabase.
Use Deepnote as a data engineering development environment.
Generate realistic development data for analysis and visualization.
Learn data exploration and preprocessing techniques using Python and SQL.
Clean and normalize data from various sources, such as relational databases, JSON, .xls files and more.
Set up Dagster to orchestrate your data pipeline.
Integrate the processing logic into a scalable ETL pipeline with Dagster.
Deploy your pipeline to Dagster Cloud (serverless)
Optimize processing through techniques such as parallelization or streamed processing.
Create powerful data visualizations using Metabase.

Who Should Attend

Developers seeking to build scalable and efficient ETL pipelines.
Entrepreneurs looking to leverage data for business growth.
Data analysts and scientists who want to streamline their data processing workflow.
Business professionals looking to improve their data-driven decision-making abilities.
Students and recent graduates interested in a career in data engineering.
Data managers tasked with organizing and making data accessible for analysis.
Project managers looking to implement data-driven solutions for clients or company.
Individuals interested in learning cutting-edge tools and techniques in data engineering.

Target Audiences

Developers seeking to build scalable and efficient ETL pipelines.
Entrepreneurs looking to leverage data for business growth.
Data analysts and scientists who want to streamline their data processing workflow.
Business professionals looking to improve their data-driven decision-making abilities.
Students and recent graduates interested in a career in data engineering.
Data managers tasked with organizing and making data accessible for analysis.
Project managers looking to implement data-driven solutions for clients or company.
Individuals interested in learning cutting-edge tools and techniques in data engineering.

Do you struggle with making data-driven decisions for your business due to scattered, inconsistent, and inaccessible data? This course is the solution! Learn to build a streamlined and efficient ETL pipeline that will allow you to turn data into actionable insights.

This course teaches you how to build a system that collects data from multiple sources, normalizes it, and stores it in a consistent and accessible format. You will learn how to extract data, explore and preprocess it, and ultimately visualize it to support better decision-making and optimize business processes.

Forget about big data and cluster management headaches, this course is designed to get you up and running quickly with a real-time ETL pipeline. With infrastructure costs under $50 a month, you can start seeing immediate results and return on investment for your clients or company.

In the first part of the course, I will walk you through the architecture and introduce you to the tools we will be using:

Deepnote, as a setup-free development environment
Dagster, as the pipeline orchestrator
Metabase, as a low-code data visualization platform

While the course will introduce you to the relevant features of Deepnote and Metabase, it is mostly focused on Dagster.

In the next part, we will get started by generating dummy sales data of a hypothetical company using Deepnote. The code will be provided for this. Once we have the data, the course will dive into data exploration and preprocessing techniques using Python and SQL in Deepnote, including cleaning and normalizing data from various sources such as relational and JSON data, Excel sheets, and more. We will implement the processing logic in Deepnote, then commit it to a Git repository that will be shared with Dagster.

In the following section, we will wrap the business logic with Dagster operations and jobs, then deploy them to Dagster Cloud (self-hosted option also available), which will allow you to manage everything from a single, unified view. In this section, you will also learn a few tricks to speed up and optimize processing, such as parallelization or streamed processing.

In the final section of this course, you’ll bring your preprocessed data to life with Metabase. With a few simple clicks, even non-technical individuals will be able to create stunning, powerful visualizations that unlock the full potential of your data.

By the end of this course, you’ll have a comprehensive understanding of the tools used and how they work together, empowering you to provide tangible benefits to your clients or company from day one, measured in thousands or tens of thousands of dollars.

The choice is yours – will you seize this opportunity to deliver massive benefits to your company or clients, and claim your fair share of the rewards?

Course Curriculum

Chapter 1: Introduction

Lecture 1: Welcome to the World of Data Engineering

Lecture 2: The Power of Clean, Organized Data

Lecture 3: The Skills and Tools Needed to be a Successful Data Engineer

Lecture 4: An ETL pipeline for Small and Medium-Sized Businesses

Chapter 2: Exploring the Tools of the ETL pipeline: Deepnote, Dagster, and Metabase

Lecture 1: DeepNote

Lecture 2: Dagster

Lecture 3: Metabase

Lecture 4: Other tools

Chapter 3: Designing the ETL Pipeline: From Data Sources to Dashboards

Lecture 1: Building the Solution Architecture

Chapter 4: Setting Up Your Development Environment and Generating Dummy Data

Lecture 1: Creating a PostgreSQL Database on Google Cloud

Lecture 2: Generating Synthetic Data of a Hypothetical Client

Lecture 3: Explanation of the data generation process (optional)

Lecture 4: Verifying the Generated Data

Chapter 5: Getting Started with Deepnote: An Introduction to Python and SQL for Data Explor

Lecture 1: Extracting and Viewing Data in Deepnote

Lecture 2: Digging Deeper: Identifying Data Issues

Lecture 3: Digging Deeper: Coming Up with a Strategy

Lecture 4: Creating a Database Table for Storing Normalized Data

Chapter 6: Data Preprocessing in Deepnote: Cleaning and Normalizing Data

Lecture 1: Preprocessing Relational Data: POS Transactions

Lecture 2: Preprocessing Relation Data: Crypto Transactions

Lecture 3: Preprocessing JSON Data

Lecture 4: Preprocessing Excel Sheets: Loading Files from Google Drive

Lecture 5: Preprocessing Excel Sheets: Market Transactions

Lecture 6: Refactoring Business Logic: Challenge

Lecture 7: Refactoring Business Logic: Solution

Lecture 8: Unit Testing

Chapter 7: Setting up the ETL pipeline with Dagster

Lecture 1: Overview of Dagster Concepts

Lecture 2: Set up Local Dagster Development

Lecture 3: Extracting Data

Lecture 4: Transforming and Loading Data

Lecture 5: Partitioned Processing

Lecture 6: Job Configuration

Lecture 7: Streamed Data Processing

Lecture 8: Processing Files

Lecture 9: Creating Dagster Schedules

Lecture 10: Creating Dagster Sensors

Lecture 11: Deploying to Dagster Cloud

Chapter 8: Visualizing Data in Metabase

Lecture 1: Creating Visualizations from the Processed Data

Chapter 9: Bonus Content

Lecture 1: Bonus Lecture

Instructors

Simon Szalai
Senior Data Engineer

Rating Distribution

1 stars: 1 votes
2 stars: 1 votes
3 stars: 2 votes
4 stars: 4 votes
5 stars: 6 votes

Frequently Asked Questions

How long do I have access to the course materials?

You can view and review the lecture materials indefinitely, like an on-demand channel.

Can I take my courses with me wherever I go?

Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don’t have an internet connection, some instructors also let their students download course lectures. That’s up to the instructor though, so make sure you get on their good side!

Menu