18.465 | Spring 2007 | Graduate

Topics in Statistics: Statistical Learning Theory

Lecture Notes

The lecture notes for this course were prepared by Alexander Rakhlin and Wen Dong, students in the class.

Complete set of lecture notes in one file (PDF - 4.8 MB)

LEC # TOPICS
1 Introduction
2 Voting classifiers, training error of boosting (PDF)
3 Support vector machines (SVM) (PDF)
4 Generalization error of SVM (PDF)
5 One dimensional concentration inequalities. Bennett’s inequality (PDF)
6 Bernstein’s inequality (PDF)
7 Hoeffding, Hoeffding-Chernoff, and Khinchine inequality (PDF)
8 Vapnik-Chervonenkis classes of sets (PDF)
9 Properties of VC classes of sets (PDF)
10 Symmetrization. Pessimistic VC inequality (PDF)
11 Optimistic VC inequality (PDF)
12 VC subgraph classes of functions. Packing and covering numbers (PDF)
13 Covering numbers of the VC subgraph classes (PDF)
14 Kolmogorov’s chaining method. Dudley’s entropy integral (PDF)
15 More symmetrization. Generalized VC inequality (PDF)
16 Consequences of the generalized VC inequality (PDF)
17 Covering numbers of the convex hull (PDF)
18 Uniform entropy condition of VC-hull classes (PDF)
19 Generalization error bound for VC-hull classes (PDF)
20 Bounds on the generalization error of voting classifiers (PDF)
21 Bounds on the generalization error of voting classifiers (cont.) (PDF)
22 Bounds on the generalization error of voting classifiers (cont.) (PDF)
23 Bounds in terms of sparsity (PDF)
24 Bounds in terms of sparsity (cont.) (example) (PDF)
25 Martingale-difference inequalities (PDF)
26 Comparison inequality for Rademacher processes (PDF)
27 Application of martingale inequalities. Generalized martingale inequalities (PDF)
28 Generalization bounds for neural networks (PDF)
29 Generalization bounds for neural networks (cont.) (PDF)
30 Generalization bounds for kernel methods (PDF)
31 Optimistic VC inequality for random classes of sets (PDF)
32 Applications of random VC inequality to voting algorithms and SVM (PDF)
33 Talagrand’s convex-hull distance inequality (PDF)
34 Consequences of Talagrand’s convex-hull distance inequality (PDF)
35 Talagrand’s concentration inequality for empirical processes (PDF)
36 Talagrand’s two-point inequality (PDF)
37 Talagrand’s concentration inequality for empirical processes (PDF)
38 Applications of Talagrand’s concentration inequality (PDF)
39 Applications of talagrand’s convex-hull distance inequality. Bin packing (PDF)
40 Entropy tensorization inequality. Tensorization of Laplace transform (PDF)
41 Application of the entropy tensorization technique (PDF)
42 Stein’s method for concentration inequalities (PDF)

Course Info

Departments
As Taught In
Spring 2007
Level