100 books like The Elements of Statistical Learning

While studying computer networks, Claude Shannon did something pretty impressive. He reformulated the majority of classical statistics from scratch using the language and concepts of computer science. 

Statistical noise? There’s a new word for that; it’s called entropy. Also, it turns out it is a good thing, not a bad thing because entropy is equal to the information content or a data set. Tired of minimizing the squared error of everything? That’s fine, minimize the log of its likelihood instead. It does the same thing. This book challenges the assumptions of classical statistics in a way that fits neatly in the mind of a computer scientist. As a quant trader, this book will help you understand and measure the information content of data, which is critical to your success.

This is a deep and beautifully elegant overview of the ideas underlying statistical inference. It is the finest concise outline I know of the foundations, dealing with the key concepts and ideas in an accessible way. Written by one of the leading creators of modern statistics, without unnecessary mathematics or superfluous detail it includes a balanced description of the fundamentals of distinct schools of thought, such as Bayesian and frequentist schools. The book did not exist when I started learning statistics, but I am certain I would have understood the discipline’s subtleties much sooner if it had.

The text covers classic statistical inference, early computer-age methods, and twenty-century topics. This puts a unique perspective on current analytic technologies labeled machine learning, artificial intelligence, and statical learning. The examples used provide a powerful description of the methods covered and the compare and contrast sections highlight the evolution of analytics. This book by Efron and Hastie is a natural follow-up source for readers interested in more details.

This is my go-to book for when I need to find proofs or examples of the theory or applications of probability. It’s an old book now, but it remains unsurpassed as an outline of the foundations of classical probability theory. The preface to the second edition says “in addition to an unexpected number of users, the book seems to have found friends who read it merely for fun; it is most heartening that they range from pure mathematicians to pure amateurs”. And that must surely be exactly right: I find myself re-reading it because of the insights and perspectives it sheds. 

This is a wonderful book because it says it all. Of course, that’s an exaggeration because no book could possibly encompass the vast breadth of modern statistics, but anyone who read through this multi-volume work would have an enviable knowledge of the discipline. It’s an unsurpassed general source of information about the foundational concepts and tools of statistics, and a reference source I regularly turn to when I need to remind myself of the theory underlying a concept or method.

One of my favorite professors, Gretchen Martinet, used this to teach a course called “Mathematical Statistics” when I was at the University of Virginia. It is an extremely profound course full of dense but fundamental mathematical proofs in classical statistics. 

You will learn why the formula for the normal distribution is the way it is, why the sum of squares appears everywhere in statistics, and how to fit a linear regression by hand. In the same way calculus elevates our understanding of rates of changes, the book elevates your understanding of samples, averages, and distributions. Quant trading requires an intuitive sense of how data, models, and aggregates work, making this content essential for your success.

Everyone knows what probability is, and we all understand how a coin flip works, but not everyone can explain the optimal betting strategies for a roulette table. We don’t study probability to understand the likelihood of events. We study probability to understand the expected outcomes of business processes that depend on those events.

In other words, this book won’t just teach you about probabilities, it will teach you about business strategies associated with those probabilities. It will help you answer a question like: How do I maximize the profit on this life insurance policy, given this set of survival probabilities? It isn’t just a likelihood question, it is a business question. I highly recommend that anyone studying probability does so through an actuarial lens.

This is one of my favorite underappreciated statistics books of all time. Non-parametric statistics can be otherwise described as statistics without assumptions. The entire goal of this field of study is to prove X is greater than Y without making any assumptions about the underlying distributions of X or Y. The methods are different, and they require more data than other methods, but the learning journey is invaluable.

I personally believe that modern machine learning is simply the modeling section of the school of non-parametric statistics. Working through this book will give you a much deeper understanding of why tools like decision trees are so valuable. It will also to teach you to design rigorous numerical experiments on data sets that are beyond the help of classical statistics.

The book is a well-curated collection of the essential mathematical concepts that form ML. You may experience a cultural shock jumping to this book from the previous one, because the writing in this book is a bit formal. However, it is the missing but necessary piece for building solid foundations for practical ML. You will find it more valuable combining the intuition behind ML that you gained previously. And the explanations in the book are succinct and from the ML perspectives. For instance, partial derivatives are explained in terms of neural network weight optimization. I wish the concepts in Linear Algebra, Vector Calculus, and Probability courses back in college were introduced this way so I understand better how they are applied.  

This fine book is about learning the relationships between what is seen in an image, and what is known about the world. It’s a counterpart to our book on feature extraction and it shows you what can be achieved with the features. It’s not for those who shy from maths, as is the case for all of the books here. So that you can build the techniques, Simon’s book also includes a wide variety of algorithms to help you on your way.