Information Theory

Calendar

LEC #	TOPICS	KEY DATES
1	Introduction, entropy
2	Jensen’s inequality, data processing theorem, Fanos’s inequality
3	Different types of convergence, asymptotic equipartition property (AEP), typical set, joint typicality
4	Entropies of stochastic processes	Problem set 1 due
5	Data compression, Kraft inequality, optimal codes	Problem set 2 due
6	Huffman codes	Problem set 3 due
7	Shannon-Fano-Elias codes, Slepian-Wolf
8	Channel capacity, binary symmetric and erasure channels
9	Maximizing capacity, Blahut-Arimoto	Problem set 4 due
10	The channel coding theorem
11	Strong coding theorem, types of errors	Problem set 5 due
12	Strong coding theorem, error exponents
	In-class midterm
13	Fano’s inequality and the converse to the coding theorem	Problem set 6 due
14	Feedback capacity
15	Joint source channel coding	Problem set 7 due
16	Differential entropy, maximizing entropy
17	Additive Gaussian noise channel	Problem set 8 due
18	Gaussian channels: parallel, colored noise, inter-symbol interference
19	Gaussian channels with feedback	Problem set 9 due
20	Multiple access channels
21	Broadcast channels	Problem set 10 due
	In-class presentations (2 sessions)
22	Finite state Markov channels
23	Channel side information, wide-band channels