3Blue1Brown explains arithmetic coding for 1-bit text compression

3blue1brown.substack.com1 min read

The author discusses arithmetic codes as a concrete method for turning an arbitrary predictor, such as a large language model, into a compressor. This approach clarifies the perspective behind pretraining and cross-entropy loss, and sets up a discussion on the phrase compression is intelligence. Open article →