Information Theory and Source Coding

Higher education teachers: Levstek Andrej
Collaborators: Pirc Matija

Subject description

Prerequisits:

• Enrolment in semester

Content (Syllabus outline):

Probability, random variables (trials, events, definition of probability, probability density function, mean values, central limit theorem). Stochastic processes (sample function, time averages, ergodicity, power density spectrum). Information (metrics, information sources, entropy, redundancy). Coding and data compaction (source coding theorem, entropy coding, Lempel-Ziv coding). Mutual information and channel capacity (mutual information, information channel, joint entropy of discrete sources, differential entropy, information capacity theorem). Analogue signal coding – basic formatting (ideal and flat toped sampling, reconstruction of continuous time signals, band-pass signal sampling; quantization, granular and overload noise, dynamic range). Audio signals (sound and hearing, properties of audio signal, perceptual properties of human hearing, frequency masking, redundancy and irrelevance, properties of speech, vocal tract modeling, speech redundancy). Speech coding (non-linear quantization, A- law compression, predictive coding; scalar quantization (DPCM, ADPCM), vector quantization (CELP). Audio signal coding (standard coding formats: CD, DVD-audio, DSD; lossy compression, MP2, MP3, AAC).

Objectives and competences:

Basic principles of information transmission and related backgrounds. Entropy as the basic measure of information. Source coding and basic data compaction algorithms. Fundamental limits of reliable communication over noisy channel. Properties of analogue signals that are important for coding schemes. Distinction between redundancy and irrelevance. Redundancy removal in advanced speech coding. Basic principles of perceptual coding of audio signals.

Intended learning outcomes:

Knowledge and understanding of information theory basics, source coding and data compression. Insight in properties of most common analogue signals and appropriate coding schemes.

Learning and teaching methods:

• lectures,
• tutorial,
• homeworks.

Study materials

1. J.R. Deller, J.G. Proakis, J.H. Hansen, Discrete-time processing of speech signals, MacMillan, New York, 1993
2. N. Moreau, Tools for signal compression, ISTE Ltd. and John Wiley & Sons, Inc., 2009