Syllabus

Course Meeting Times

Lectures: 3 sessions / week, 1 hour / session

Prerequisites

Permission of instructor is required. Helpful courses (ideal but not required): Theory of Probability (18.175) and either Statistical Learning Theory and Applications (9.520) or Machine Learning (6.867)

Description

The main goal of this course is to study the generalization ability of a number of popular machine learning algorithms such as boosting, support vector machines and neural networks. We will develop a number of technical tools that will allow us to give qualitative explanations of why these learning algorithms work so well in many classification problems.

Topics of the course include Vapnik-Chervonenkis theory, concentration inequalities in product spaces, and other elements of empirical process theory.

Grading

The grade is based upon two problem sets and class attendance.

Course Outline

Introduction

• Classification problem set-up
• Examples of learning algorithms: Voting algorithms (boosting), support vector machines, neural networks
• Analyzing generalization ability

Technical Tools: Elements of Empirical Process Theory

One-dimensional Concentration Inequalities

• Chebyshev (Markov), Rademacher, Hoeffding, Bernstein, Bennett
• Toward uniform bounds: Union bound, clustering

Vapnik-Chervonenkis Theory and More

• VC classes of sets and functions
• Shattering numbers, growth function, covering numbers
• Examples of VC classes, properties
• Uniform deviation bounds
• Symmetrization
• Kolmogorov's chaining technique
• Dudley's entropy integral
• Contraction principles

Concentration Inequalities

• Talagrand's concentration inequality on the cube
• Symmetrization
• Talagrand's concentration inequality for empirical processes
• Vapnik-Chervonenkis type inequalities
• Martingale-difference inequalities

Applications

• Generalization ability of voting classifiers, neural networks, support vector machines