How Not to Be Wrong

Author:: Jordan Ellenberg Full Title:: How Not to Be Wrong Tags:#media/book

• themes::
  • theme 1
• Summary::
  • summary 1

* highlights from 2021-02-08

* A mathematician is always asking, “What assumptions are you making? And are they justified?” ([Location 250](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=250))
* Mathematics is the study of things that come out a certain way because there is no other way they could possibly be. ([Location 313](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=313))
* Without the rigorous structure that math provides, common sense can lead you astray. That’s what happened to the officers who wanted to armor the parts of the planes that were already strong enough. But formal mathematics without common sense—without the constant interplay between abstract reasoning and our intuitions about quantity, time, space, motion, behavior, and uncertainty—would just be a sterile exercise in rule-following and bookkeeping. ([Location 329](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=329))
* false linearity—he’s assuming, without coming right out and saying so, that the course of prosperity is described by the line segment in the first picture, ([Location 434](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=434))
* Nonlinear thinking means which way you should go depends on where you already are. ([Location 441](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=441))
* The slogan to keep in mind: straight locally, curved globally. ([Location 618](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=618))
* Zeno’s paradox is much like another conundrum: is the repeating decimal 0.99999. . . . . . equal to 1? ([Location 671](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=671))
* “How shall we define 1 − 1 + 1 − 1 + . . .” but “What is 1 − 1 + 1 − 1 + . . . ?” ([Location 757](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=757))
* This is not just loosey-goosey mathematical relativism. Just because we can assign whatever meaning we like to a string of mathematical symbols doesn’t mean we should. In math, as in life, there are good choices and there are bad ones. In the mathematical context, the good choices are the ones that settle unnecessary perplexities without creating new ones. ([Location 760](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=760))
* a mathematics that values precise answers but also intelligent approximation, that demands the ability to deploy existing algorithms fluently but also the horse sense to work things out on the fly, that mixes rigidity with a sense of play. If we don’t, we’re not really teaching mathematics at all. ([Location 913](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=913))
* In 2048, the line crosses 100%. And that’s why Wang writes that all Americans will be overweight in 2048, if current trends continue. ([Location 926](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=926))
* Nice work, black men! Not until 2095 will all of you be overweight. In 2048, only 80% of you will be. See the problem? If all Americans are supposed to be overweight in 2048, where are those one in five future black men without a weight problem supposed to be? Offshore? ([Location 939](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=939))
    * **Note**: Linear regression must be isolated
* how much is that in dead Americans? ([Location 958](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=958))
* An important rule of mathematical hygiene: when you’re field-testing a mathematical method, try computing the same thing several different ways. If you get several different answers, something’s wrong with your method. ([Location 969](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=969))
* The United States has almost seven times the population of Spain. So if you think of 200 people as 0.0004% of the Spanish population, you find that an equivalent attack would kill 1,300 people in the United States. On the other hand, 200 people is 0.006% of the population of Madrid; scaling up to New York City, which is two and a half times as large, gives you 463 victims. Or should we compare the province of Madrid with the state of New York? That gives you something closer to 600. This multiplicity of conclusions should be a red flag. Something is fishy with the method of proportions. ([Location 972](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=972))
* So it’s better to study rates: deaths as a proportion of total population. ([Location 985](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=985))
    * **Note**: Rates are useful as proportion not hard numbers
* The answer: South Dakota isn’t necessarily causing brain cancer, and North Dakota isn’t necessarily preventing it. The five states at the top have something in common, and the five states at the bottom do, too. And it’s the same thing: hardly anyone lives there. ([Location 992](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=992))
* Measuring the absolute number of brain cancer deaths is biased toward the big states; but measuring the highest rates—or the lowest ones!—puts the smallest states in the lead. That’s how South Dakota can have one of the highest rates of brain cancer death while North Dakota claims one of the lowest. ([Location 1043](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1043))
* That’s how the Law of Large Numbers works: not by balancing out what’s already happened, but by diluting what’s already happened with new data, until the past is so proportionally negligible that it can safely be forgotten. ([Location 1136](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1136))
* Here’s a rule of thumb that makes sense to me: if the magnitude of a disaster is so great that it feels right to talk about “survivors,” then it makes sense to measure the death toll as a proportion of total population. ([Location 1145](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1145))
* It’s because the question of whether one war was worse than another is fundamentally unlike the question of whether one number is bigger than another. The latter question always has an answer. The former does not. ([Location 1160](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1160))
* Don’t talk about percentages of numbers when the numbers might be negative. ([Location 1178](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1178))
* It’s only after you’ve started to formulate these questions that you take out the calculator. But at that point the real mental work is already finished. Dividing one number by another is mere computation; figuring out what you should divide by what is mathematics. ([Location 1295](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1295))
* When you sink your savings into the incubated fund with the eye-popping returns, you’re like the newsletter getter who invests his life savings with the Baltimore stockbroker; you’ve been swayed by the impressive results, but you don’t know how many chances the broker had to get those results. ([Location 1445](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1445))
* The Baltimore stockbroker con works because, like all good magic tricks, it doesn’t try to fool you outright. That is, it doesn’t try to tell you something false—rather, it tells you something true from which you’re likely to draw incorrect conclusions. It really is improbable that ten stock picks in a row would come out the right way, or that a magician who bet on six horse races would get the winner right every time, or that a mutual fund would beat the market by 10%. The mistake is in being surprised by this encounter with the improbable. The universe is big, and if you’re sufficiently attuned to amazingly improbable occurrences, you’ll find them. Improbable things happen a lot. ([Location 1451](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1451))
* That doesn’t make each individual coincidence any less improbable. But here comes the chorus again: improbable things happen a lot. ([Location 1465](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1465))
* ‘one chance in a million’ will undoubtedly occur, with no less and no more than its appropriate frequency, however surprised we may be that it should occur to us.” ([Location 1471](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1471))
* Rather, McKay and Bar-Natan are making a potent point about the power of wiggle room. Wiggle room is what the Baltimore stockbroker has when he gives himself plenty of chances to win; wiggle room is what the mutual fund company has when it decides which of its secretly incubating funds are winners and which are trash. Wiggle room is what McKay and Bar-Natan ([Location 1491](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1491))
* used to work up a list of rabbinical names that jibed well with War and Peace. When you’re trying to draw reliable inferences from improbable events, wiggle room is the enemy. ([Location 1493](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1493))
* If the paper passed that test, shouldn’t we have accepted its conclusions, however otherworldly they might have seemed? Or, to put it another way: if we now feel comfortable rejecting the conclusions of the Witztum study, what does that say about the reliability of our standard statistical tests? ([Location 1513](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1513))
* point of Bennett’s paper is to warn that the standard methods of assessing results, the way we draw our thresholds between a real phenomenon and random static, come under dangerous pressure in this era of massive data sets, effortlessly obtained. We need to think very carefully about whether our standards for evidence are strict enough, if the empathetic salmon makes the cut. ([Location 1538](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1538))
* Getting excited about the fish voxels that matched the photos and ignoring the rest is as potentially dangerous as getting excited about the successful series of stock newsletters while ignoring the many more editions that blew their calls and went in the trash. ([Location 1549](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1549))
* We use probability even to talk about events that cannot possibly be thought of as subject to chance. What’s the probability that consuming olive oil prevents cancer? ([Location 1663](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1663))
* Maybe you think the randomization system is malfunctioning, and the numbers 4, 21, 23, 34, 39 are more likely to come up than others. Or maybe you think a corrupt lottery official is picking the numbers to match his own favorite ticket. Under either of those hypotheses, the amazing coincidence is not improbable at all. Improbability, as described here, is a relative notion, not an absolute one; when we say an outcome is improbable, we are always saying, explicitly or not, that it is improbable under some set of hypotheses we’ve made about the underlying mechanisms of the world. ([Location 1675](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1675))
* The “does nothing” scenario is called the null hypothesis. That ([Location 1682](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1682))
* is, the null hypothesis is the hypothesis that the intervention you’re studying has no effect. ([Location 1682](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1682))
* It’s not enough that the data be consistent with your theory; they have to be inconsistent with the negation of your theory, the dreaded null hypothesis. ([Location 1691](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1691))
* Run an experiment. Suppose the null hypothesis is true, and let p be the probability (under that hypothesis) of getting results as extreme as those observed. The number p is called the p-value. If it is very small, rejoice; you get to say your results are statistically significant. If it is large, concede that the null hypothesis has not been ruled out. ([Location 1717](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1717))
* The added risk posed by third-generation birth control pills, while significant in Fisher’s statistical sense, was not so significant in the sense of public health. The way the story was framed only magnified the confusion. The CSM reported a risk ratio: third-generation pills doubled women’s risk of thrombosis. That sounds pretty bad, until you remember that thrombosis is really, really rare. Among women of childbearing age using first- and second-generation oral contraceptives, 1 in 7,000 could expect to suffer a thrombosis; users of the new pill indeed had twice as much risk, 2 in 7,000. But that’s still a very small risk, because of this certified math fact: twice a tiny number is a tiny number. ([Location 1804](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1804))
* In the CUNY study, only a dozen or so babies a year died in accidents in family day cares, a tiny fraction of the 1,110 U.S. infants who died in accidents overall in 2010 (mostly by strangulation in bedclothes) or the 2,063 who died of sudden infant death syndrome. All things being equal, the results of the CUNY study provide a reason to prefer a day-care center to care in a family home; but all other things are usually not equal, and some inequalities matter more than others. What if the scrubbed and city-certified day-care center is twice as far from your house as the slightly questionable family-run in-home day care? Car accidents killed 79 infants in the U.S. in 2010; ([Location 1816](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1816))
    * **Note**: Whats not on the menu
* “statistically noticeable” or “statistically detectable” instead of “statistically significant”! That would be truer to the meaning of the method, which merely counsels us about the existence of an effect but is silent about its size or importance. ([Location 1831](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1831))
* It captures perfectly the spirit of the psychology that Skinner wanted to create. Where Freud had claimed to see what had previously been hidden, repressed, or obscured, Skinner wanted to do the opposite—to deny the existence of what seemed in plain view. ([Location 1883](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1883))
* A significance test is an instrument, like a telescope. And some instruments are more powerful than others. If you look at Mars with a research-grade telescope, you’ll see moons; if you look with binoculars, you won’t. But the moons are still there! ([Location 1885](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1885))
* A statistical study that’s not refined enough to detect a phenomenon of the expected size is called underpowered ([Location 1897](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1897))
* But what if the null hypothesis is wrong? The hot hand, if it exists, is brief, and the effect, in strictly numerical terms, is small. The worst shooter in the league hits 40% of his shots and the best hits 60%; that’s a big difference in basketball terms, but not so big statistically. What would the shot sequences look like if the hot hand were real? ([Location 1947](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1947))
* In more than three-quarters of those simulations, the significance test used by GVT reported that there was no reason to reject the null hypothesis—even though the null hypothesis was completely false. The GVT design was underpowered, destined to report the nonexistence of the hot hand even if the hot hand was real. ([Location 1952](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1952))
* The right question isn’t “Do basketball players sometimes temporarily get better or worse at making shots?”—the kind of yes/no question a significance test addresses. The right question is “How much does their ability vary with time, and to what extent can observers detect in real time whether a player is hot?” Here, the answer is surely “not as much as people think, and hardly at all.” ([Location 1959](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=1959))
* You can think of the null hypothesis significance test as a sort of fuzzy version of the reductio: Suppose the null hypothesis H is true. It follows from H that a certain outcome O is very improbable (say, less than Fisher’s 0.05 threshold). But O was actually observed. Therefore, H is very improbable. ([Location 2023](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=2023))
* It’s tempting to think of “very improbable” as meaning “essentially impossible,” and, from there, to utter the word “essentially” more and more quietly in our mind’s voice until we stop paying attention to it.* But impossible and improbable are not the same—not even close. Impossible things never happen. But improbable things happen a lot. That means we’re on quivery logical footing when we try to make inferences from an improbable observation, as reductio ad unlikely asks us to. ([Location 2061](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=2061))
* p-value of .05, or 1 in 20. Remember the definition of the p-value; this says precisely that if the null hypothesis is true for some particular experiment, then the chance that that experiment will nonetheless return a statistically significant result is only 1 in 20. If the null hypothesis is always true—that is, if haruspicy is undiluted hocus-pocus—then only 1 in 20 experiments will be publishable. ([Location 2210](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=2210))
* A low-powered study is only going to be able to see a pretty big effect. But sometimes you know that the effect, if it exists, is small. In other words, a study that accurately measures the effect of a gene is likely to be rejected as statistically insignificant, while any result that passes the p < .05 test is either a false positive or a true positive that massively overstates the gene’s effect. ([Location 2250](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=2250))
* But noise is just as likely to push you in the opposite direction from the real effect as it is to tell the truth. So we’re left in the dark by a result that offers plenty of statistical significance but very little confidence. ([Location 2263](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=2263))
* file drawer problem—a scientific field has a drastically distorted view of the evidence for a hypothesis when public dissemination is cut off by a statistical significance threshold. ([Location 2297](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=2297))
* Not “succeeds once in giving,” but “rarely fails to give.” A statistically significant finding gives you a clue, suggesting a promising place to focus your research energy. The significance test is the detective, not the judge. ([Location 2426](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=2426))
* They don’t have to know exactly what product you want; they just have to have a better idea than competing ad channels do. Businesses generally operate on thin margins. ([Location 2509](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=2509))
* First of all: when we call Bayes’s Theorem a theorem it suggests we are discussing incontrovertible truths, certified by mathematical proof. That’s both true and not. It comes down to the difficult question of what we mean when we say “probability.” When we say that there’s a 5% chance that RED is true, we might mean that there actually is some vast global population of roulette wheels, of which exactly one in twenty is biased to fall red 3/5 of the time, and that any given roulette wheel we encounter is randomly picked from the roulette wheel multitude. If that’s what we mean, then Bayes’s Theorem is a plain fact, akin to the Law of Large Numbers we saw in the last chapter; it says that, in the long run, under the conditions we set up in the example, 12% of the roulette wheels that come up RRRRR are going to be of the red-favoring kind. But this isn’t actually what we’re talking about. When we say that there’s a 5% chance that RED is true, we are making a statement not about the global distribution of biased roulette wheels (how could we know?) but rather about our own mental state. Five percent is the degree to which we believe that a roulette wheel we encounter is weighted toward the red. ([Location 2750](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=2750))
* Our priors are not flat, but spiky. We assign a lot of mental weight to a few theories, while others, like the RBRRB theory, get assigned a probability almost indistinguishable from zero. How do we choose our favored theories? We tend to like simpler theories better than more complicated ones, theories that rest on analogies to things we already know about better than theories that posit totally novel phenomena. ([Location 2775](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=2775))
* The combined theory, T + U, should start out with a smaller prior probability; it is by definition harder to believe than T, because it asks you to swallow both T and another theory at the same time. But as the evidence flows in, which would tend to kill T alone,* the combined theory T + U remains untouched. ([Location 2807](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=2807))
* It is not obvious. Rather, it is obvious if you already know it, as modern people do. But the fact that people who administered annuities failed to make this observation, again and again, is proof that it’s not actually obvious. Mathematics is filled with ideas that seem obvious now—that negative quantities can be added and subtracted, that you can usefully represent points in a plane by pairs of numbers, that probabilities of uncertain events can be mathematically described and manipulated—but are in fact not obvious at all. If they were, they would not have arrived so late in the history of human thought. ([Location 3016](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=3016))
* When you’re faced with a math problem you don’t know how to do, you’ve got two basic options. You can make the problem easier, or you can make it harder. ([Location 3307](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=3307))
* The unknown is a stone in the sea, which obstructs our progress. We can try to pack dynamite in the crevices of rock, detonate it, and repeat until the rock breaks apart, as Buffon did with his complicated computations in calculus. Or you can take a more contemplative approach, allowing your level of understanding gradually and gently to rise, until after a time what appeared as an obstacle is overtopped by the calm water, and is gone. Mathematics as currently practiced is a delicate interplay between monastic contemplation and blowing stuff up with dynamite. ([Location 3373](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=3373))
* if your government isn’t wasteful, you’re spending too much time fighting government waste. ([Location 3589](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=3589))
    * **Note**: Some amount of waste or false positives i good for efficiency
* mathematics is the extension of common sense by other means. ([Location 3757](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=3757))
* The “known unknowns” are like RED—we don’t know which ball we’ll get, but we can quantify the probability that the ball will be the color we want. BLACK, on the other hand, subjects the player to an “unknown unknown”—not only are we not sure whether the ball will be black, we don’t have any knowledge of how likely it is to be black. In the decision-theory literature, the former kind of unknown is called risk, the latter uncertainty. ([Location 3803](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=3803))
* The people who first figured out what was going on here were the people who needed to understand both how things are and how things look, and the difference between the two: namely, painters. ([Location 3967](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=3967))
* In fact, point R is not just the endpoint of the horizontal axis, but of any horizontal line. If two different lines are both horizontal, they are parallel; and yet, in projective geometry, they meet, at the point at infinity. ([Location 4019](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=4019))
* Mathematical elegance and practical utility are close companions, as the history of science has shown again and again. ([Location 4033](https://readwise.io/to_kindle?action=open&asin=B00G3L6JQ4&location=4033))