### Reading Assignments

Reading assignments were taken from the course textbook:

Cover, Thomas, and Joy Thomas. *Elements of Information Theory*. 2nd ed. New York, NY: Wiley-Interscience, 2006. ISBN: 9780471241959.

LEC # | TOPICS | READINGS |
---|---|---|

1 | Introduction, entropy | Chapter 1, sections 2.1 - 2.5 |

2 | Jensen’s inequality, data processing theorem, Fanos’s inequality | Sections 2.6 - 2.8, 2.11 |

3 | Different types of convergence, asymptotic equipartition property (AEP), typical set, joint typicality | Sections 3.1 - 3.3, 8.6 |

4 | Entropies of stochastic processes | Chapter 4 |

5 | Data compression, Kraft inequality, optimal codes | Sections 5.1 - 5.4 |

6 | Huffman codes | Sections 5.5 - 5.7 |

7 | Shannon-Fano-Elias codes, Slepian-Wolf | Sections 5.8 - 5.9, section 14 through the end of 14.4.4.1 |

8 | Channel capacity, binary symmetric and erasure channels | Sections 8.1 - 8.3 |

9 | Maximizing capacity, Blahut-Arimoto | |

10 | The channel coding theorem | Sections 8.4, 8.7 |

11 | Strong coding theorem, types of errors | |

12 | Strong coding theorem, error exponents | |

In-class midterm | ||

13 | Fano’s inequality and the converse to the coding theorem | Section 8.9 |

14 | Feedback capacity | Section 8.12 |

15 | Joint source channel coding | Section 8.13 |

16 | Differential entropy, maximizing entropy | Chapter 9, sections 11.1 - 11.6 |

17 | Additive Gaussian noise channel | Sections 10.1 - 10.3 |

18 | Gaussian channels: parallel, colored noise, inter-symbol interference | Sections 10.4 - 10.5 |

19 | Gaussian channels with feedback | Section 10.6 |

20 | Multiple access channels | Sections 14.1 - 14.3 |

21 | Broadcast channels | Section 14.6 |

In-class presentations (2 sessions) | ||

22 | Finite state Markov channels | |

23 | Channel side information, wide-band channels |

### Supplementary Readings

The supplementary readings are optional, except for the paper that you select as part of your project. The supplementary readings are coded for difficulty. One star: Accessible. Two stars: Requires significant mathematical maturity. Three stars: Expert level.

[* Lectures 4-5] Elias, Peter. “Predictive Coding: Part I.” *IRE Transactions: Information Theory* 1, no. 16 (1955): 16-24.

[* Lecture 6] Huffman, David. “A Method for the Construction of Minimum-Redundancy Codes.” *Proceedings of the IRE* 40, no. 9 (1952): 1098-1101.

[** Lecture 7] Slepian, David, and Jack Wolf. “Noiseless Coding of Correlated Information Sources.” *IEEE Transactions on Information Theory* 19, no. 4 (1973): 471-480.

[** Lecture 7] Elias, Peter. “Predictive Coding: Part II.” *IRE Transactions: Information Theory* 1, no. 16 (1955): 24-33.

[** Lecture 7] Gallagher, Robert. “Variations on a Theme by Huffman.” *IEEE Transactions on Information Theory* 24, no. 6 (1978): 668-674.

[** Lectures 10-12] Gallagher, Robert. “A Simple Derivation of the Coding Theorem and Some Applications.” *IEEE Transactions on Information Theory* 11, no. 1 (1965): 3-18.

[* Lectures 14, 16-20] Cover, Thomas. “The Role of Feedback in Communication.” In *Performance Limits in Communication Theory and Practice*. Edited by J. K. Skwirzynski. New York, NY: Springer, 1988, pp. 225-235. ISBN: 9789024736959.

[** Lecture 20] El Gamal, Abbas, and Thomas Cover. “Multiple User Information Theory.” *Proceedings of the IEEE* 68, no. 12 (1980): 1466-1483.

[** Lecture 20] Cover, Thomas, Abbas El Gamal, and Masoud Salehi. “Multiple Access Channels with Arbitrarily Correlated Sources.” *IEEE Transactions on Information Theory* 26, no. 6 (1980): 648-657.

[** Lecture 20] Gallagher, Robert. “A Perspective on Multiaccess Channels.” *IEEE Transactions on Information Theory* 31, no. 2 (1985): 124-142.

[** Lecture 21] Cover, Thomas. “Comments on Broadcast Channels.” *IEEE Transactions on Information Theory* 44, no. 6 (1998): 2524-2530.

[** Lecture 21] Bergmans, Patrick, and Thomas Cover. “Cooperative Broadcasting.” *IEEE Transactions on Information Theory* 20, no. 3 (1974): 317-324.

[*** Lecture 21] Weingarten, Hanan, Yosef Steinberg, and Shlomo Shamai (Shitz). “The Capacity Region of the Gaussian Multiple-Input Multiple-Output Broadcast Channel.” *IEEE Transactions on Information Theory* 52, no. 9 (2006): 3936-3964.

[** Lectures 21-22] Jindal, Nihar, Sriram Vishwanath, and Andrea Goldsmith. “On the Duality of Gaussian Multiple-Access and Broadcast Channels.” *IEEE Transactions on Information Theory* 50, no. 5 (2004): 768-783.

[*** Lectures 22-23] Goldsmith, Andrea, and Pravin Varaiya. “Capacity, Mutual Information, and Coding for Finite-State Markov Channels.” *IEEE Transactions on Information Theory* 42, no. 3 (1996): 868-886.

[*** Lecture 23] Médard, Muriel, and Robert Gallagher. “Bandwidth Scaling for Fading Multipath Channels.” *IEEE Transactions on Information Theory* 48, no. 4 (2002): 840-852.