B03
Entropy and information
What is being counted in each definition of entropy? Follow the change from messages to physical states and identify the assumptions behind each expression.

I.06 / GNOSIS
What did a communication channel have to preserve for a message to survive noise? Follow that engineering question into Shannon’s definition of information.
Shannon's communication theory makes uncertainty in a source and the capacity of a channel mathematically tractable. Its objects are symbol probabilities, codes, and transmission conditions. This gives the site a precise way to compare predictable and unpredictable sequences. The practical question is how much uncertainty remains before a symbol arrives, and how a coding system turns that structure into reliable communication.
A fair binary source has entropy of one bit per symbol. If one outcome occurs with probability 0.9 and the other with 0.1, the binary entropy is about 0.469 bits. Generate short illustrative sequences with these settings, compare their symbol counts, and examine why a short sample may differ noticeably from the generating probabilities.
Use the entropy workbench to vary one probability, then record how code length and prediction error respond under the same source assumptions.
These models isolate a mechanism you can vary. Their assumptions define the part of this topic they can illuminate.
Change one parameter, compare the numerical output, and export the result. The experiment loads when you open it.
Compare symbol frequencies with the ordering that a code can exploit.
The first sweep changes the probability of A while drawing each symbol independently. The other three symbols share the remaining probability.
As one symbol dominates, the observed symbol entropy falls. Huffman codes use short words for frequent symbols; the comparison excludes headers and the codebook.
The second sweep gives all four new draws equal probability and increases repetition. Long runs reward run-length coding, even when a symbol histogram cannot describe the dependence.
Method. The histogram gives an order-zero Shannon entropy. Huffman coding repeatedly joins the two least frequent branches to build a prefix code. The run-length format stores a two-bit symbol and an eight-bit count; counts run from 1 to 255, and a longer run uses another packet.
Nokia Bell Labs · Communication entropy, coding, and noise; does not turn statistical information into a theory of semantic truth.
B03
What is being counted in each definition of entropy? Follow the change from messages to physical states and identify the assumptions behind each expression.
B09
When does shorter code length predict better classification on new examples?
B25
Which distance between probability distributions does a model use, and how does that choice affect inference?