Information Theory
<p>Jan 15, 2011 06:45 PM
 
Quantification of information:  Claude Shannon
fundamental limits on signal processing such as data compression and on reliably storing data
and communicating data.
 
Broadened to include  statistical infeerence, natural language processing, cryptography, and neurobiology,
evolution of molecular coes, ecology,  etc.
 
ENTROPY.
 
applications include lossless data compression (eg zip files),
lossy data compression (eg mp3s) and channel coding (eg DSL linges)
 
The field is an intersection of math, stats, cs, physics, neurobio, and ee.
 
<b>Overview:  Consider Language</b>:
 
Most common words  (eg  a and the)  should be shorter than less common words 
(eg benefit, generation, and mediocre)
so that sentences will not be too long.
 
data compression;  source coding;  channel coding
 
Note:  has NOTHING TO DO WITH IMPORTANCE of the INFO:
that is, the difference btwn a platitude like  "Thank you, come again"  and
"Call an ambulance"   are NOT CONSIDERED!  only the reliability/ readability of the signal.
 </p>