02_Overview on Suprema and Limits

Reliable

Reliable = Arbitarily small error probability

• Fix a noisy channel.
  • What’s the maximun transmission efficiency(channel code rate)
  • 什么是最大传输效率（用最少的bit位表达传递全部的信息）
• What’s the arbitarily small error probability
  • error 可以任意小，更critical的指标用更高的系统复杂度置换
  • 任意小，不严格为0

Channel Capacity

• Definition of Channel Capacity
  • Channel Capacity is the maximum reliable transmission code rate for a noisy channel.
  • 超过 C(Channel Capacity) 误差率

Mutual information

Mutual information: 共同信息，共识。（Shannon定义的信息是 uncertainty。

• Mutual Information 其实是指系统设计时（信号传输前，这样我们就不知道，系统会传输哪种信号）的内容，是系统本身的性质。

下面的维恩图，反应了左边 8bit 为传输端的内容， 2bit 为公共，6bit 为需要传输的内容。

• 好的系统要，尽可能多地使用公共部分传输信息。

• the design of a good transmission code should relate to the “common uncertainty“(or more formally, the mutual information) between channel inputs and channel outputs.

It is then natual to wonder whether or not this “relation” can be expressed mathematically.

• it was established by Shannon that the bound on the reliable transmission rate(information bits per channel usage) is the maximum channel mutual information(i.e. “common uncertainty” prior to the transmission begins) attainable.

Summary

information theory’s 蹲马步

A.1 Supremum and maximum

Supremum: 一个集合的最小上界 （Least upper bound or supermum)

• Throughout, we work on subsets of. the set of real numbers.

Completeness Axiom:(Least upper bound property) Letbe a non-empty subset ofthat is bounded above. Thenhas a least upper bound(in)

只要集合有上界就一定有supremum

1. 对集合的拓展
2. 对无穷上下界的引入
   

Maximum： If sup, then supis also called the maximum of, and is denoted by max.else we say that the maximum ofdoes not exist.

Properties of the maximum

1. (), if maxexists in{}
2. max

从以上性质证明是对最大值：

1. 反向：中的任意元素都小于等于
2. 正向：在集合中可以取到。

A.2 Infimum and minimum

Infimum跟minimum类似上面的两个数学概念

Minimum: 最小值，在集合内。

A.3 Boundedness and suprema operations

集合乘积拓展不能用以上规律套用，除非集合的值都是正的。

A.4 Sequences and their limist

1. 信息论中，常常将正负无穷引入到研究的集合中，认为正负无穷也是一种收敛（而不是diverage）
2. 对于单调函数（单调序列），没有上界的时候，可以认为是收敛于正无穷

limsup and liminf

• 如果 limsup 和 liminf 两个值重合的话，极限存在。

直观理解：群集点（clustering point

limsup 跟 liminf 是一个无穷远处的概念，序列位号很大以后的上下界。

Limit

Sufficiently large & Infinitely often

A.5 Equivalence

• We close this appendix by providing some equivalent statements that are often used to simplify proofs.
• instead of directly showing that quantityis less than or equal to quantity, one can take an arbitrary constantand prove that.
• Sinceis a larger quantity than, in some cases it might be easier to showthan proving