Why am I passionate about this?

I had an epiphany at Amazon, when we ran A/B tests at scale and observed the low success rate: we learned to adjust our intuitions. I recall the denial at Microsoft when I proposed to evaluate features with A/B tests “because over 50% of them failed to improve key metrics at Amazon.”  The typical response? We have better program managers. When we started to evaluate ideas at Microsoft, over 2/3 of them failed to improve key metrics, and at Bing, the rate was about 80%. By 2019, most large products at Microsoft were making data-driven decisions with over 100 A/B test treatments launched every workday. I currently teach an A/B Testing class.


I wrote

Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

By Ronny Kohavi, Diane Tang, Ya Xu

Book cover of Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

What is my book about?

Controlled experiments are the gold standard in science for establishing causality, and their use online, under the common name A/B…

Shepherd is reader supported. When you buy books, we may earn an affiliate commission.

The books I picked & why

Book cover of Calling Bullshit: The Art of Skepticism in a Data-Driven World

Ronny Kohavi Why did I love this book?

Most people and organizations aspire to be data-driven, yet some of the most fascinating “discoveries” and stories are incorrect. 

The authors use the term Bullshit; I prefer Twyman’s law: any figure that looks interesting or different is usually wrong—look at very surprising results with skepticism. For instance, the adage that correlation doesn’t imply causation is well known, but correlations don’t help sell new articles, so many correlations are told as causal stories.

The book elucidates pitfalls from uncontrolled experiments, common causes, p-hacking, and selection bias, and amazing “classical” stories are debunked.

Example: If you claim your toothpaste reduces plaque “by up to” 50 percent, the only way that would be false is if the toothpaste worked too well.

By Carl T. Bergstrom, Jevin D. West,

Why should I read it?

3 authors picked Calling Bullshit as one of their favorite books, and they share why you should read it.

What is this book about?

Bullshit isn’t what it used to be. Now, two science professors give us the tools to dismantle misinformation and think clearly in a world of fake news and bad data.
 
“A modern classic . . . a straight-talking survival guide to the mean streets of a dying democracy and a global pandemic.”—Wired

Misinformation, disinformation, and fake news abound and it’s increasingly difficult to know what’s true. Our media environment has become hyperpartisan. Science is conducted by press release. Startup culture elevates bullshit to high art. We are fairly well equipped to spot the sort of old-school bullshit that is based…


Book cover of Hard Facts, Dangerous Half-Truths, and Total Nonsense: Profiting from Evidence-based Management

Ronny Kohavi Why did I love this book?

This insightful book on evidence-based management is rich with great references. It is filled with provoking examples of things we heard or learned, which were debunked, making you think twice about them (Twyman’s law).

Do financial incentives work? They can, but in limited scenarios where individuals could be measured and minimum quality levels kept high; they may, however, backfire and undermine teamwork and attract the wrong people. When hiring, can the best talent be identified? Even in sports, some of the best players aren’t identified early in their careers. 

Rembrandt is considered a great painter; Mozart, one of the greatest composers, but they were largely unrecognized and unrewarded during their lifetimes.  Does supervisor feedback help? In some scenarios, only the supervisors think so.

By Jeffrey Pfeffer, Robert I. Sutton,

Why should I read it?

1 author picked Hard Facts, Dangerous Half-Truths, and Total Nonsense as one of their favorite books, and they share why you should read it.

What is this book about?

The best organizations have the best talent...Financial incentives drive company performance...Firms must change or die. Popular axioms like these drive business decisions every day. Yet too much common management "wisdom" isn't wise at all--but, instead, flawed knowledge based on "best practices" that are actually poor, incomplete, or outright obsolete. Worse, legions of managers use this dubious knowledge to make decisions that are hazardous to organizational health. Jeffrey Pfeffer and Robert I. Sutton show how companies can bolster performance and trump the competition through evidence-based management, an approach to decision-making and action that is driven by hard facts rather than half-truths…


Book cover of Mistakes Were Made (But Not by Me): Why We Justify Foolish Beliefs, Bad Decisions, and Hurtful Acts

Ronny Kohavi Why did I love this book?

A theme in the book is that “Most people, when directly confronted by evidence that they are wrong, do not change their point of view or course of action but justify it even more tenaciously.”  

In software, we have conclusive data from tens of thousands of online controlled experiments (A/B tests) that most (~80%) ideas and features proposed fail to improve the metrics they were designed to improve

The book walks us through great examples on the cognitive dissonance that occurs when our perceived expertise clashes with the sobering reality of failure—humbling. The key, of course, is to think of these experiments as learning opportunities: you win some, you learn some.

By Carol Tavris, Elliot Aronson,

Why should I read it?

4 authors picked Mistakes Were Made (But Not by Me) as one of their favorite books, and they share why you should read it.

What is this book about?

Renowned social psychologists Carol Tavris and Elliot Aronson take a compelling look into how the brain is wired for self-justification. This updated edition concludes with an extended discussion of how we can live with dissonance, learn from it, and perhaps, eventually, forgive ourselves.

Why is it so hard to say “I made a mistake”—and really believe it?

When we make mistakes, cling to outdated attitudes, or mistreat other people, we must calm the cognitive dissonance that jars our feelings of self-worth. And so, unconsciously, we create fictions that absolve us of responsibility, restoring our belief that we are smart, moral,…


Book cover of The Halo Effect... and the Eight Other Business Delusions That Deceive Managers

Ronny Kohavi Why did I love this book?

Nassim Taleb calls this book “one of the most important management books of all time, and an antidote to these bestselling books by gurus presenting false patterns and naïve arguments.” 

Rosenzweig shows that although best-selling management books are inspirational, they are written based on correlational data and lack predictive power. For instance: “Does employee satisfaction lead to high performance? The evidence suggests it’s mainly the other way around—company success has a stronger impact on employee satisfaction.” 

The problem with only studying the “best,” as management classics like In Search of Excellence have done, is that there is no experimental control group. If rich people fly private jets, starting to fly similarly is more likely to lead to bankruptcy than wealth. 

By Phil Rosenzweig,

Why should I read it?

5 authors picked The Halo Effect... and the Eight Other Business Delusions That Deceive Managers as one of their favorite books, and they share why you should read it.

What is this book about?

Why do some companies prosper while others fail? Despite great amounts of research, many of the studies that claim to pin down the secret of success are based in pseudoscience. The Halo Effect is the outcome of that pseudoscience, a myth that Philip Rosenzweig masterfully debunks in THE HALO EFFECT. The Halo Effect describes the tendency of experts to point to the high financial performance of a successful company and then spread its golden glow to all of the company's attributes - clear strategy, strong values, and brilliant leadership. But in fact, as Rosenzweig clearly illustrates, the experts are not…


Book cover of High Output Management

Ronny Kohavi Why did I love this book?

This great management book was written by an engineer who clearly explains the rationale for his recommendations.

Andy Grove, former chairman and CEO of Intel, is credited with driving the growth phase of Silicon Valley, was named Time’s Man of the Year, and is credited as the “Father of OKRs.” One example that I love: a new hire does poor work. His manager says: “He has to make his own mistakes…that’s how he learns!”

Grove writes: “absolutely wrong…the tuition is paid by his customers…[instead of by the manager].” Another gem: “Review rough drafts [you delegated]; don't wait until your subordinates have spent time polishing them into final form before you find out that you have a basic problem with the contents.”

By Andrew S Grove,

Why should I read it?

2 authors picked High Output Management as one of their favorite books, and they share why you should read it.

What is this book about?

The president of Silicon Valley's Intel Corporation sets forth the three basic ideas of his management philosophy and details numerous specific techniques to increase productivity in the manager's work and that of his colleagues and subordinates


Explore my book 😀

Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

By Ronny Kohavi, Diane Tang, Ya Xu

Book cover of Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

What is my book about?

Controlled experiments are the gold standard in science for establishing causality, and their use online, under the common name A/B tests, has flourished. Companies are now able to evaluate changes to software using experimentation platforms that brought the marginal cost of running highly reliable and trustworthy experiments close to zero.

The authors who worked at Microsoft, Google, LinkedIn, and Amazon, ran experiments at high scale: over 20,000 experiments annually. The book has many real examples, focuses on trust, and shares practical advice.  It sold over 22,000 copies and was translated to Chinese, Japanese, Korean, and Russian. It is rated 4.7 on Amazon, where it is commonly in the top 10 best sellers in database and/or data mining.

You might also like...

Uniting the States of America: A Self-Care Plan for a Wounded Nation

By Lyle Greenfield,

Book cover of Uniting the States of America: A Self-Care Plan for a Wounded Nation

Lyle Greenfield Author Of Uniting the States of America: A Self-Care Plan for a Wounded Nation

New book alert!

Why am I passionate about this?

I’ve always been fascinated by group dynamics, large and small. Why things functioned well, why they didn’t. It’s possible my ability to empathize and use humor as a consensus-builder is the reason I was elected president of a homeowners association, a music production association, and even an agricultural group. Books were not particularly involved in this fascination! But in recent years, experiencing the breakdown of civility and trust in our political and cultural discourse, I’ve taken a more analytical view of the dynamics. These books, in their very different ways, have taught me lessons about life, understanding those with different beliefs, and finding ways to connect and move forward. 

Lyle's book list on restoring your belief in human possibility

What is my book about?

We’ve all experienced the overwhelming level of political and social divisiveness in our country. This invisible “virus” of negativity is, in part, the result of the name-calling and heated rhetoric that has become commonplace among commentators and elected leaders alike. 

My book provides a clear perspective on the historical and modern-day causes of our nation's divisive state. It then proposes easy-to-understand solutions—an action plan for our elected leaders and citizens as well. Rather than a scholarly treatment of a complex topic, the book challenges us to take the obvious steps required of those living in a free democracy. And it…

Uniting the States of America: A Self-Care Plan for a Wounded Nation

By Lyle Greenfield,

What is this book about?

Lyle Greenfield's "Uniting the States of America―A Self-Care Plan for a Wounded Nation" is a work of nonfiction and opinion. Incorporating the lessons of history and the ideas and wisdom of many, it is intended as both an educational resource and a call-to-action for citizens concerned about the politically and culturally divided state of our Union. A situation that has raised alarm for the very future of our democracy.

First, the book clearly identifies the causes of what has become a national crisis of belief in and love for our country. How the divisiveness and hostility rampant in our political…


5 book lists we think you will like!

Interested in statistics, decision making, and success in life?

Statistics 30 books
Decision Making 88 books
Success In Life 262 books