My primary interest is in brain function. Because the principal job of
the brain is to process information, it is necessary to define exactly
what information is. For that, there is no substitute for Claude
Shannon’s theory of information. This theory is not only quite
remarkable in its own right, but it is essential for telecoms,
computers, machine learning (and understanding brain function).
I have written ten "tutorial introduction" books, on topics which vary
from quantum mechanics to AI.
In a parallel universe, I am still an Associate Professor at the
University of Sheffield, England.
Pierce was a contemporary of Claude Shannon (inventor of information theory), so he learned information theory shortly after it was published in 1949. Pierce writes in an informal style, but does not flinch from presenting the fundamental theorems of information theory. Some would say his style is too wordy, and the ratio of words/equations is certainly very high. Nevertheless, this book provides a solid introduction to information theory. It was originally published in 1961, so it is a little dated in terms of topics covered. However, because it was re-published by Dover in 1981, it is also fairly cheap. Overall, this is a sensible first book to read on information theory.
"Uncommonly good...the most satisfying discussion to be found." — Scientific American. Behind the familiar surfaces of the telephone, radio, and television lies a sophisticated and intriguing body of knowledge known as information theory. This is the theory that has permitted the rapid development of all sorts of communication, from color television to the clear transmission of photographs from the vicinity of Jupiter. Even more revolutionary progress is expected in the future. To give a solid introduction to this burgeoning field, J. R. Pierce has revised his well-received 1961 study of information theory for a second edition. Beginning with the origins…
This is a more comprehensive and mathematically rigorous book than Pierce’s book. For the novice, it should be read-only after first reading Pierce’s more informal text. Due to its vintage, the layout is fairly cramped, but the content is impeccable. At almost 500 pages, it covers a huge amount of material. This was my main reference book on information theory for many years, but it now sits alongside more recent texts, like MacKay’s book (see below). It is also published by Dover, so it is reasonably priced.
Written for an engineering audience, this book has a threefold purpose: (1) to present elements of modern probability theory — discrete, continuous, and stochastic; (2) to present elements of information theory with emphasis on its basic roots in probability theory; and (3) to present elements of coding theory. The emphasis throughout the book is on such basic concepts as sets, the probability measure associated with sets, sample space, random variables, information measure, and capacity. These concepts proceed from set theory to probability theory and then to information and coding theories. No formal prerequisites are required other than the usual undergraduate…
This is the modern standard text on information theory. It is both comprehensive and highly technical. The layout is spacey, and the authors make good use of the occasional diagram to explain geometric aspects of information theory. One feature I really like is the set of historical notes and a summary of equations at the end of each chapter.
The latest edition of this classic is updated with new problem sets and material The Second Edition of this fundamental textbook maintains the book's tradition of clear, thought-provoking instruction. Readers are provided once again with an instructive mix of mathematics, physics, statistics, and information theory. All the essential topics in information theory are covered in detail, including entropy, data compression, channel capacity, rate distortion, network information theory, and hypothesis testing. The authors provide readers with a solid understanding of the underlying theory and applications. Problem sets and a telegraphic summary at the end of each chapter further assist readers. The…
This is considered to be a modern classic on information theory. It is a fairly readable text that roams far and wide over many topics. MacKay was extraordinarily clever, curious, and generous (which is why he made the book freely available). His wide range of interests included coding theory and artificial neural networks (machine learning), which occupy a few chapters of the book. In describing the difference between Bayesian and frequentist statistical methods, MacKay pulls no punches, and he can be heard laughing through gritted teeth as he tries to make sense of the nonsensical. This is one of those books in which the author’s many throw-away asides are as enlightening as the main text. The book’s website (below) also has a link to an excellent series of video lectures by MacKay. It is also available for free, just use the direct link below.
Information theory and inference, taught together in this exciting textbook, lie at the heart of many important areas of modern technology - communication, signal processing, data mining, machine learning, pattern recognition, computational neuroscience, bioinformatics and cryptography. The book introduces theory in tandem with applications. Information theory is taught alongside practical communication systems such as arithmetic coding for data compression and sparse-graph codes for error-correction. Inference techniques, including message-passing algorithms, Monte Carlo methods and variational approximations, are developed alongside applications to clustering, convolutional codes, independent component analysis, and neural networks. Uniquely, the book covers state-of-the-art error-correcting codes, including low-density-parity-check codes, turbo…
This is really two books, strapped together. The first book by Weaver is an informal introduction to the ideas implicit in the second book by Shannon. Shannon’s book is naturally quite dated in its use of language (e.g. uncertainty is pronounced equivocation), and in its references to ‘current’ technology (e.g. PCM). Despite these caveats, it is still a surprisingly accessible book. Finally, because information theory was developed almost exclusively by Claude Shannon, reading the theory explained in his own words gives some insight into how on Earth he managed to come up with such a radical set of ideas.
Scientific knowledge grows at a phenomenal pace--but few books have had as lasting an impact or played as important a role in our modern world as The Mathematical Theory of Communication, published originally as a paper on communication theory more than fifty years ago. Republished in book form shortly thereafter, it has since gone through four hardcover and sixteen paperback printings. It is a revolutionary work, astounding in its foresight and contemporaneity. The University of Illinois Press is pleased and honored to issue this commemorative reprinting of a classic.
Originally developed by Claude Shannon in the 1940s, information theory laid the foundations for the digital revolution, and is now an essential tool in telecommunications, genetics, linguistics, brain sciences, and deep space communication. In this richly illustrated book, accessible examples are used to introduce information theory in terms of everyday games like ‘20 questions’ before more advanced topics are explored. Online MatLab and Python computer programs provide hands-on experience of information theory in action, and PowerPoint slides give support for teaching.
Written in an informal style, with a comprehensive glossary and tutorial appendices, this text is an ideal primer for novices who wish to learn the essential principles and applications of information theory.
I’ve always been fascinated by group dynamics, large and small. Why things functioned well, why they didn’t. It’s possible my ability to empathize and use humor as a consensus-builder is the reason I was elected president of a homeowners association, a music production association, and even an agricultural group. Books were not particularly involved in this fascination! But in recent years, experiencing the breakdown of civility and trust in our political and cultural discourse, I’ve taken a more analytical view of the dynamics. These books, in their very different ways, have taught me lessons about life, understanding those with different beliefs, and finding ways to connect and move forward.
We’ve all experienced the overwhelming level of political and social divisiveness in our country. This invisible “virus” of negativity is, in part, the result of the name-calling and heated rhetoric that has become commonplace among commentators and elected leaders alike.
My book provides a clear perspective on the historical and modern-day causes of our nation's divisive state. It then proposes easy-to-understand solutions—an action plan for our elected leaders and citizens as well. Rather than a scholarly treatment of a complex topic, the book challenges us to take the obvious steps required of those living in a free democracy. And it…
Uniting the States of America: A Self-Care Plan for a Wounded Nation
Lyle Greenfield's "Uniting the States of America―A Self-Care Plan for a Wounded Nation" is a work of nonfiction and opinion. Incorporating the lessons of history and the ideas and wisdom of many, it is intended as both an educational resource and a call-to-action for citizens concerned about the politically and culturally divided state of our Union. A situation that has raised alarm for the very future of our democracy.
First, the book clearly identifies the causes of what has become a national crisis of belief in and love for our country. How the divisiveness and hostility rampant in our political…
