6.441 | Spring 2010 | Graduate

# Information Theory

Reading assignments were taken from the course textbook:

Cover, Thomas, and Joy Thomas. Elements of Information Theory. 2nd ed. New York, NY: Wiley-Interscience, 2006. ISBN: 9780471241959.

1 Introduction, entropy Chapter 1, sections 2.1 - 2.5
2 Jensen’s inequality, data processing theorem, Fanos’s inequality Sections 2.6 - 2.8, 2.11
3 Different types of convergence, asymptotic equipartition property (AEP), typical set, joint typicality Sections 3.1 - 3.3, 8.6
4 Entropies of stochastic processes Chapter 4
5 Data compression, Kraft inequality, optimal codes Sections 5.1 - 5.4
6 Huffman codes Sections 5.5 - 5.7
7 Shannon-Fano-Elias codes, Slepian-Wolf Sections 5.8 - 5.9, section 14 through the end of 14.4.4.1
8 Channel capacity, binary symmetric and erasure channels Sections 8.1 - 8.3
9 Maximizing capacity, Blahut-Arimoto
10 The channel coding theorem Sections 8.4, 8.7
11 Strong coding theorem, types of errors
12 Strong coding theorem, error exponents
In-class midterm
13 Fano’s inequality and the converse to the coding theorem Section 8.9
14 Feedback capacity Section 8.12
15 Joint source channel coding Section 8.13
16 Differential entropy, maximizing entropy Chapter 9, sections 11.1 - 11.6
17 Additive Gaussian noise channel Sections 10.1 - 10.3
18 Gaussian channels: parallel, colored noise, inter-symbol interference Sections 10.4 - 10.5
19 Gaussian channels with feedback Section 10.6
20 Multiple access channels Sections 14.1 - 14.3
In-class presentations (2 sessions)
22 Finite state Markov channels
23 Channel side information, wide-band channels

The supplementary readings are optional, except for the paper that you select as part of your project. The supplementary readings are coded for difficulty. One star: Accessible. Two stars: Requires significant mathematical maturity. Three stars: Expert level.

[* Lectures 4-5] Elias, Peter. “Predictive Coding: Part I.” IRE Transactions: Information Theory 1, no. 16 (1955): 16-24.

[* Lecture 6] Huffman, David. “A Method for the Construction of Minimum-Redundancy Codes.” Proceedings of the IRE 40, no. 9 (1952): 1098-1101.

[** Lecture 7] Slepian, David, and Jack Wolf. “Noiseless Coding of Correlated Information Sources.” IEEE Transactions on Information Theory 19, no. 4 (1973): 471-480.

[** Lecture 7] Elias, Peter. “Predictive Coding: Part II.” IRE Transactions: Information Theory 1, no. 16 (1955): 24-33.

[** Lecture 7] Gallagher, Robert. “Variations on a Theme by Huffman.” IEEE Transactions on Information Theory 24, no. 6 (1978): 668-674.

[** Lectures 10-12] Gallagher, Robert. “A Simple Derivation of the Coding Theorem and Some Applications.” IEEE Transactions on Information Theory 11, no. 1 (1965): 3-18.

[* Lectures 14, 16-20] Cover, Thomas. “The Role of Feedback in Communication.” In Performance Limits in Communication Theory and Practice. Edited by J. K. Skwirzynski. New York, NY: Springer, 1988, pp. 225-235. ISBN: 9789024736959.

[** Lecture 20] El Gamal, Abbas, and Thomas Cover. “Multiple User Information Theory.” Proceedings of the IEEE 68, no. 12 (1980): 1466-1483.

[** Lecture 20] Cover, Thomas, Abbas El Gamal, and Masoud Salehi. “Multiple Access Channels with Arbitrarily Correlated Sources.” IEEE Transactions on Information Theory 26, no. 6 (1980): 648-657.

[** Lecture 20] Gallagher, Robert. “A Perspective on Multiaccess Channels.” IEEE Transactions on Information Theory 31, no. 2 (1985): 124-142.

[** Lecture 21] Cover, Thomas. “Comments on Broadcast Channels.” IEEE Transactions on Information Theory 44, no. 6 (1998): 2524-2530.

[** Lecture 21] Bergmans, Patrick, and Thomas Cover. “Cooperative Broadcasting.” IEEE Transactions on Information Theory 20, no. 3 (1974): 317-324.

[*** Lecture 21] Weingarten, Hanan, Yosef Steinberg, and Shlomo Shamai (Shitz). “The Capacity Region of the Gaussian Multiple-Input Multiple-Output Broadcast Channel.” IEEE Transactions on Information Theory 52, no. 9 (2006): 3936-3964.

[** Lectures 21-22] Jindal, Nihar, Sriram Vishwanath, and Andrea Goldsmith. “On the Duality of Gaussian Multiple-Access and Broadcast Channels.” IEEE Transactions on Information Theory 50, no. 5 (2004): 768-783.

[*** Lectures 22-23] Goldsmith, Andrea, and Pravin Varaiya. “Capacity, Mutual Information, and Coding for Finite-State Markov Channels.” IEEE Transactions on Information Theory 42, no. 3 (1996): 868-886.

[*** Lecture 23] Médard, Muriel, and Robert Gallagher. “Bandwidth Scaling for Fading Multipath Channels.” IEEE Transactions on Information Theory 48, no. 4 (2002): 840-852.

Spring 2010
Problem Sets
Lecture Notes