Homepage of Abdellatif Zaidi


Abdellatif Zaidi

Assistant Professor
Universite Paris-Est Marne la Vallée



Lecture Notes (Fall and Spring 2011)

  1. EE-TDI: Information Theory: Lecure Notes, Part I
  2. EE-TDI: Information Theory: Lecure Notes, Part II

Lectures: Part I

  1. Introduction
  2. Entropy, Relative Entropy and Mutual Information
    • Entropy, joint entropy and conditional entropy, relative entropy and mutual information, divergence
    • Chain rules for entropy, relative entropy and mutual information
    • Jensen's inequality and its consequences
    • Data processing inequality
    • Fano's inequality
  3. Data Processing and Markov Chains
    • Data processing and inverse processing
    • Data processing and Markov chains
    • Staionary data, memoryless data, Makovian and Gauss-Markovian models
    • Examples of data processing
  4. Properties of Entropy, Mutual Information and Divergence
    • Convexity
    • Case of random vectors
    • Link between entropy and differential entropy, quantization
    • Entropy maximization under constraints

Lectures: Part II

  1. Fundamental limits of Source Coding
    • The source coding problem
    • Rate-Distortion function, properties
    • Shannon source coding theorem
    • Examples
  2. Fundamental limits of Channel Coding
    • The channel coding problem
    • Capacity-Cost function, properties
    • Shannon channel coding theorem
    • Examples
  3. Data Compression
    • Examples of codes
    • Kraft inequality, optimal codes, bounds on the optimal codelength
    • Kraft inequality for uniquely decodable codes
    • Huffman codes, comments on Huffman codes
    • Arithmetic coding
  4. Quantization and Applications
    • Scalar and vector quantization
    • Algorithms for quantization
    • Examples, JPEG, JPEG 2000


  1. Final Exam, Information Theory, Spring 2011


Abdellatif Zaidi
Last modified: 08/18/10
Valid HTML 4.01!