Hyperparameter Optimization for Machine Learning

Course Curriculum

Chapter 1: Introduction

Lecture 1: Introduction

Lecture 2: Course curriculum

Lecture 3: Course aim and knowledge requirements

Lecture 4: Course material

Lecture 5: Jupyter notebooks

Lecture 6: Presentations

Lecture 7: Datasets

Lecture 8: Set up your computer – required packages

Lecture 9: Resources to learn machine learning skills

Chapter 2: Hyperparameter Tuning – Overview

Lecture 1: Parameters and Hyperparameters

Lecture 2: Hyperparameter Optimization

Chapter 3: Performance metrics

Lecture 1: Performance Metrics – Introduction

Lecture 2: Classification Metrics (Optional)

Lecture 3: Regression Metrics (Optional)

Lecture 4: Creating your own metrics

Lecture 5: Using Scikit-learn metrics

Chapter 4: Cross-Validation

Lecture 1: Cross-Validation

Lecture 2: Bias vs Variance (Optional)

Lecture 3: Cross-Validation schemes

Lecture 4: Estimating the model generalization error with CV – Demo

Lecture 5: Cross-Validation for Hyperparameter Tuning – Demo

Lecture 6: Special Cross-Validation schemes

Lecture 7: Group Cross-Validation – Demo

Chapter 5: Basic Search Algorithms

Lecture 1: Basic Search Algorithms – Introduction

Lecture 2: Manual Search

Lecture 3: Grid Search

Lecture 4: Grid Search – Demo

Lecture 5: Grid Search with different hyperparameter spaces

Lecture 6: Random Search

Lecture 7: Random Search with Scikit-learn

Lecture 8: Random Search with Scikit-Optimize

Lecture 9: Random Search with Hyperopt

Lecture 10: More examples

Chapter 6: Bayesian Optimization

Lecture 1: Sequential Search

Lecture 2: Bayesian Optimization

Lecture 3: Bayesian Inference – Introduction

Lecture 4: Joint and Conditional Probabilities

Lecture 5: Bayes Rule

Lecture 6: Sequential Model-Based Optimization

Lecture 7: Gaussian Distribution

Lecture 8: Multivariate Gaussian Distribution

Lecture 9: Gaussian Process

Lecture 10: Kernels

Lecture 11: Acquisition Functions

Lecture 12: Additional Reading Resources

Lecture 13: Scikit-Optimize – 1-Dimension

Lecture 14: Scikit-Optimize – Manual Search

Lecture 15: Scikit-Optimize – Automatic Search

Lecture 16: Scikit-Optimize – Alternative Kernel

Lecture 17: Scikit-Optimize – Neuronal Networks

Lecture 18: Scikit-Optimize – CNN – Search Analysis

Chapter 7: Other SMBO Algorithms

Lecture 1: Other SMBO Algorithms

Lecture 2: SMAC

Lecture 3: SMAC Demo

Lecture 4: Tree-structured Parzen Estimators – TPE

Lecture 5: TPE Procedure

Lecture 6: TPE hyperparameters

Lecture 7: TPE – why tree-structured?

Lecture 8: TPE with Hyperopt

Lecture 9: Discussion: Bayesian Optimization and Basic Search

Chapter 8: Scikit-Optimize

Lecture 1: Scikit-Optimize

Lecture 2: Section content

Lecture 3: Hyperparameter Distributions

Lecture 4: Defining the hyperparameter space

Lecture 5: Defining the objective function

Lecture 6: Random search

Lecture 7: Bayesian search with Gaussian processes

Lecture 8: Bayesian search with Random Forests

Lecture 9: Bayesian search with GBMs

Lecture 10: Parallelizing a Bayesian search

Lecture 11: Bayesian search with Scikit-learn wrapper

Lecture 12: Changing the kernel of a Gaussian Process

Lecture 13: Optimizing xgboost

Lecture 14: Optimizing Hyperparameters of a CNN

Lecture 15: Analyzing the CNN search

Chapter 9: Hyperopt

Lecture 1: Hyperopt

Lecture 2: Section content

Lecture 3: Search space configuration and distributions

Lecture 4: Sampling from nested spaces

Lecture 5: Search algorithms

Lecture 6: Evaluating the search

Lecture 7: Optimizing multiple ML models simultaneously

Lecture 8: Optimizing Hyperparameters of a CNN

Lecture 9: References

Chapter 10: Optuna

Lecture 1: Optuna

Lecture 2: Optuna main functions

Lecture 3: Section content

Lecture 4: Search algorithms

Lecture 5: Optimizing multiple ML models with simultaneously

Lecture 6: Optimizing hyperparameters of a CNN