Information Theory

Lecture Notes

LEC #	TOPICS	LECTURE NOTES
1	Introduction, entropy	(PDF)
2	Jensen’s inequality, data processing theorem, Fanos’s inequality	(PDF)
3	Different types of convergence, asymptotic equipartition property (AEP), typical set, joint typicality	(PDF)
4	Entropies of stochastic processes	(PDF)
5	Data compression, Kraft inequality, optimal codes	(PDF)
6	Huffman codes	(PDF)
7	Shannon-Fano-Elias codes, Slepian-Wolf	(PDF 1) (PDF 2)
8	Channel capacity, binary symmetric and erasure channels	(PDF)
9	Maximizing capacity, Blahut-Arimoto	(PDF)
10	The channel coding theorem	(PDF)
11	Strong coding theorem, types of errors	(PDF)
12	Strong coding theorem, error exponents	(PDF)
13	Fano’s inequality and the converse to the coding theorem	(PDF)
14	Feedback capacity	(PDF)
15	Joint source channel coding	(PDF)
16	Differential entropy, maximizing entropy	(PDF)
17	Additive Gaussian noise channel	(PDF)
18	Gaussian channels: parallel, colored noise, inter-symbol interference	(PDF)
19	Gaussian channels with feedback	(PDF)
20	Multiple access channels	(PDF)
21	Broadcast channels	(PDF)
22	Finite state Markov channels	(PDF)
23	Channel side information, wide-band channels	(PDF)