Introduction to Testing AI Models, LLMs and Chatbots

Course Curriculum

Chapter 1: Introduction

Lecture 1: Introduction to Material

Lecture 2: Introduction to Generative Artificial Intelligence

Lecture 3: Demo On AI Capabilities

Lecture 4: History of AI from 1950 to 2024

Chapter 2: Setting up Environment

Lecture 1: Install VS Code

Lecture 2: Installing python

Lecture 3: Install Python Dependencies – PIP

Lecture 4: Install NodeJS and NPM

Lecture 5: Get an OPENAI API Key

Chapter 3: 7 Used LLM Testing Benchmarks

Lecture 1: Introduction to Benchmarking for LLMs

Lecture 2: 7 Benchmarking Models

Lecture 3: TruthfulQA – Truthfulness

Lecture 4: Python – Demo – Benchmarking Open AI CHAT GPT – TruthfulQA

Lecture 5: Python – Demo – Benchmarking Open AI CHAT GPT – MMLU

Lecture 6: Python – Demo – Benchmarking Open AI CHAT GPT – HumanEval

Chapter 4: Introduction to Artificial Intelligence

Lecture 1: What makes up AI

Lecture 2: Where do Large Language Models(LLM) fit into AI

Lecture 3: Natural Language Processing

Lecture 4: Types of Machine Learning

Lecture 5: Importance of Training Data

Lecture 6: Machine Learning – Supervised ML

Lecture 7: Machine Learning – Unsupervised ML

Lecture 8: Machine Learning – Reinforced ML

Lecture 9: Neural Networks and Deep Learning

Lecture 10: Importance of Training Data

Lecture 11: What is a Large Language Model – LLM

Lecture 12: Generative AI – What it is

Chapter 5: LLM Functional Testing – Traditional Software Perspective

Lecture 1: Types of Testing in Software

Lecture 2: Testing Types for LLMs | Foundation Models

Lecture 3: Overall Testing Approach to LLMs

Lecture 4: Basic Content Generation for LLMs

Lecture 5: Temperature Testing of LLMs

Lecture 6: Functional Completeness

Lecture 7: Functional Correctness

Lecture 8: Accuracy Testing

Lecture 9: Repeatability Testing

Lecture 10: Multimodal Testing

Lecture 11: Efficiency and Simplicity Validations

Lecture 12: Learning Ability- Chat GPT and Google Vertex AI

Lecture 13: Statistical Correctness

Lecture 14: LLM AI Testing – Long-Term Drift Testing

Lecture 15: [Demo ] – Vertex AI Learning Ability – Model Training – Manual

Lecture 16: [Demo ] – Vertex AI Learning Ability – Model Training – Json

Chapter 6: LLM AI Reasoning | Intelligence Testing

Lecture 1: Creative Logical Abilities – A bit of fun

Lecture 2: Reasoning – Causal Reasoning

Lecture 3: Reasoning Ability | Deductive | Abductive | Inductive Logic

Lecture 4: Reverse Reasoning

Lecture 5: Counterfactual Reasoning

Lecture 6: Abstract Reasoning

Lecture 7: Generative Reasoning

Chapter 7: NLP – Style and Emotions

Lecture 1: Text Capability – Named Entity Recognition (NER)

Lecture 2: Text Capability – Style Transfer

Lecture 3: Text Capability – Sarcasm and Humor Detection

Lecture 4: Text Capability – Discourse Coherence

Chapter 8: Chatbot Testing

Lecture 1: Understand Task Based vs AI Based Chatbots

Lecture 2: Understanding Chatbots based on LLM

Lecture 3: Functional Testing for chatbots

Lecture 4: Context and Memory Testing

Lecture 5: Blabber Testing | Context forcing

Chapter 9: Testing of Ethical AI – LLM Non functional Testing

Lecture 1: EU Regulation of AI Systems

Lecture 2: AI and Biases

Lecture 3: GEN AI and Privacy

Lecture 4: GEN AI and Intellectual Property

Lecture 5: Hallucinations

Lecture 6: Gen AI and Misinformation or Disinformation

Lecture 7: Google Moderation Service

Lecture 8: OPENAI-CHAT GPT Moderation Service

Lecture 9: GEN AI and Deep Fake