How to Make the World Add Up - Tim Harford

Better-Thinking  · 

In Three Sentences

Chapters

  1. Introduction - how to lie with statistics
  2. Rule one - search your feelings
  3. Rule two - ponder your personal experience
  4. Rule three - avoid premature enumeration
  5. Rule four - step back and enjoy the view
  6. Rule five - get the back story
  7. Rule six - ask who is missing
  8. Rule seven - demand transparency when the computer says “no”
  9. Rule eight - don’t take the statistical bedrock for granted
  10. Rule nine - remember that misinformation can be useful too
  11. Rule ten - keep an open mind
  12. The Golden Rule - be curious

Raw Notes

Chapter 1

How to lie with statistics (Huff, 1954) - a book that impressed Harford as a youngster, but now feels a touch wearied by the cynicism. The tone of the Huff book is that the bad folks already know all the tricks and are using them against you, therefore you should know them too so you can defend yourself. This feeds the current narrative that statistics are not to be trusted. The data and the discussion around them are used to manufacture doubt for motivated interests therefore let’s just throw it all out.

Harford take the opposite view - he wants to show the basic rules for thinking about statistics and data to help illuminate the world.

Chapter 2 - search your feelings

If our aim is to understand the world, then we should start with understanding ourselves - we need to try and see things as they are, not as we want them to be.

Our feelings can overpower our thinking.

Good example in the book to show how when presented with a neutral example - i.e. one that you’re unlikely to have strong feelings about - then you can ask sensible questions. Say someone said that Mars was 50m miles away. How would you assess the claim? What about if they then said 30m? There is unlikely any passionately held beliefs about it, there is no threat to our ego/identity, no challenge to our pride, no upsetting of our world view. Therefore we can ask some sensible questions. Is 30m miles a lot? How does it compare to the moon? What about orbits? Aren’t we on different ones so doesn’t it change all the time? Consequently we’re in a curious, open, inquiring frame of mind and we can try and see the world as it really is.

Other statements trigger different emotional responses. Consider “masks help prevent the spread of coronavirus and you must wear one.” Would you get the same kind of measured questions?

The Ostrich Effect (p25). Folks will go to considerable lengths to avoid hearing bad news. Example is the herpes study - experiment where bloods were taken from folks and they were then shown a video about herpes that was relatively hard hitting. They were told that their bloods would be tested. In theory this should be useful information to know since the disease is manageable. 1 in 5 (i.e. 20%) said they didn’t want to know and would pay to have their sample discarded. The didn’t want to face the anxiety of knowing.

Interesting to know if this is somehow the flip side of confirmation bias - i.e. we will go out of our way not to see information that contradicts the status quo.

The proposal is the we don’t become emotionless automatons churning through the data - but the value comes from understanding how unreliable we can be and how we can get led astray by our emotions. This will help us make better informed decisions.

Strong links to our mindsets and identity. Who do you want to be? Someone who sees the world as it is? Or someone who wants comforting fairy tales? (You brain will happily give you them!)

Not explicit in the book - but interesting to consider your motives when sharing an interesting fact/some story on social media. Why did you do it? For approval? For status? Want someone to be impressed? Because it exactly chimed with your passion and you must let people know so they can see the light?

Motivated reasoning (p29). Thinking about something with the aim of reaching a particular conclusion. Most important thing - this may be conscious or not. Consider a football match. The other side are dirty. We are robust.

Motivated reasoning is essentially reasoning that has not been filtered by a second step that asks basic questions. Have I made a simple mistake? Am I biased? (The answer will almost certainly be yes to some degree - so what impact does it have?). How do I feel about this topic and could that be influencing me?

Some interesting studies mentioned. One experiment split a group into defenders and prosecutors of a motorcycle accident case. Given modest financial incentives they were asked to argue the case persuasively. Then they were given a modest financial incentive to guess the actual result of the case. Their earlier experience coloured their secondary judgment so folks skewed to predicting what they had just argued.

Second experiment based on stock market wheat fluctuations. Two groups - one farmers and the other bakers. In the farmers interest if prices go up and in bakers interest for them to be lower. They were asked to predict what happened next in the market. They were given two modest financial incentives - first if they gave an accurate forecast of what happened next and second an extra incentive if it aligned with their characters (i.e. up for farmers, down for bakers). The prospect of the extra windfall influenced their “accurate” forecast - this is surprising since the first incentive should have been an emotionless view based on the data. It seems that our reasoning is swayed by our hopes.

Folks with more experience (i.e. experts) are even more likely to fall into the trap (p35). Generally they can find more reasons to back the idea that they want to believe.

More information doesn’t necessarily help due to biased assimilation (p36). Folks can dismiss information that challenges their beliefs - and the more detail is given the more there is to attack.

Identity plays a big part too - if you’re in a group that believes a certain thing, say climate change, then going with the evidence has profound social consequences but little effect in reality (for example one person’s actions are not going to avert climate change, but will mean you are ostracised from the group). Why question the orthodoxy? It’s better to fit in. Then once the beliefs become part of your identity then it’s even harder to change.

Obviously all these things are easier to spot in others than in your self. Nice footnote (p41) about how Harford is sure he is guilty of these things too but not sure how!

We can learn to control our emotions - indeed this is part of growing up. We aim not to be robots. Awareness of these issues (i.e. our unreliability and ease of leading ourselves astray) is often enough.

We need to think as well as feel. Imagine a spectrum between pure feeling and pure thinking. What would it be like to be in pure feeling? Probably being swept away by every emotion, being influenced by anything and everyone. What would it be like to be pure thinking? We’re not robots.

Many of these issues can go away with the mindset of trying to see the world as it is and getting into the habit of doing some basic fact checking.

The two classic examples of basic questions where people go with their gut and fail to do the most basic of self checking. First - bat and ball together cost £1.10. The bat costs £1 more than the ball. How much is the ball? The gut answer is the ball is £0.10. But obviously a simple check tells you that the answer is £0.05. Second - lily pads covering a pond double in size every day and takes 48 days to cover the pond. How long to cover half. Folks don’t tend to immediately see that it’s 47 days.

Chapter 3 - ponder your experience

Great example about TFL - for example the basic stats will say that on average there are 10 people on a bus (i.e. way below capacity). Folk’s personal experience will be very different from that since most people travel for work and therefore travel at the busiest times.

“The treatment worked” - but the question is whether it would have got better on it’s own. Where is the control group?

Naive realism (p57). The belief that we see the world objectively - with no bias, no filters, no prejudice. It is others that are biased, irrational and wrong.

Fast & slow statistics (p60). Similar to system 1 and 2 in Thinking Fast and Slow (Kahneman).

Fast statistics are visceral - news stories, shared posts etc. Almost like a worms eye view - they give the detail on a particular subject and could be from sources with agendas & motivations. Even the most impartial news still wants you to consume their news.

Slow statistics are broader, thoughtful, wider & needs careful digestion. These could be detailed reports or management metrics. This gives more of the birds eye view. Note that they have problems too - consider management metrics, they can fall prey to Goodharts Law which broadly says that when the measure becomes the target it is no longer a good measure since folks are incentivised to move it for it’s own sake. For example a surgeon refusing to treat the sickest of patients for fear it would bring down their success rate. Or wait times to see a doctor - the response was to remove all advanced bookings therefore bringing down the wait times, but arguably not the best outcome.

So the advice is be aware of naive realism and meld the fast and the slow. Try and be the bird and the worm at the same time. See the wide, tell stories about the detail and ask yourself what cold I have got wrong?

Chapter 4 - avoid premature enumeration

The definitions matter. If you don’t understand what is meant by a term, or how a particular term was arrived at then you can’t really evaluate it clearly.

It seems to me that there is an interesting phenomenon that we seem to readily accept a term (say something like the rate of self harm is x) without needing to understand what it means. Our brains seem to fill in the gaps and give us something fully formed in our heads, but unquestioned. It’s also interesting that that fully formed thing is probably different between each one of us. Brings to mind the “I’m glad we all agree” picture.

Example in the book of infant mortality. Clearly nothing could be more emotive and important. What the mortality rate means depends very much on the definition. Before 15 weeks it would be perhaps be counted as a miscarriage and after 24 weeks would be counted as a live birth followed by a death. But what about in-between? Comparing different hospital trusts and big disparities in the numbers would spark a big debate and lots of activity. Why are two trusts different? Who is to blame? Are the medical folks in one incompetent? Does the gov need to invest more?

It could be that no-one is to blame. Perhaps they have just used different definitions and no-one bothered to check.

Premature enumeration (p73) - Harford’s term for jumping too quickly into the detailed statistics without first really understanding the definitions and the context. And as we saw earlier - having detailed expertise in statistics makes you even more at risk - Stephen Pinker’s “the curse of knowledge” (p76).

Examples in book

Harford makes the point that not asking what a particular term actually means (and then accepting the complexity that sometimes comes with it) is a failure of empathy (p78).

Chapter 5 - step back and enjoy the view

From an idea by some Norwegians (p93), Johan Galtung & Mari Ruge, you can see that what counts as news very much depends on the timescale at which you look at it. Imagine a newspaper released….

One of the interesting things in thinking about these imaginary newspapers is what they would not include. The daily newspaper would never say something like “no nukes today” - it only really makes sense in the wider context.

Note that a headline like “the murder rate in London this month is higher than New York” while emotive and sure to draw folks in and gather clicks etc is essentially meaningless. What’s been happening to the trends in both? Why look at a particular month? That’s not at all representative of the essence of what matters to folks.

One way to help is to have a small number of “landmark numbers” in your head (after Andrew Elliot’s book “Is that a big number?”).

  1. Populations
    • US = 325m
    • UK = 65m
    • World = 7.5b
  2. Folk in UK - any age less than 60 there are approx 800k in that category
    • e.g. 800k three year olds
    • This surprises me! I’d have thought the demographic transition model isn’t so flat
  3. Earth dimensions - approx 40000km circumference. 25000 miles.
  4. 5000km across US (east/west)
  5. Length of a bed = 2m. Useful for sizing internal spaces etc
  6. GDP US
    • 20 trillion = 20,000 billion = 20,000,000 million
  7. 100k words in a novel
  8. Empire States Building is 381m high (100 storeys)

Kahneman - over-confidence is one of the most significant biases (our longevity, competence, career prospects etc). Interesting aside to coronavirus - arguably over-confidence and over-positivity are extra problematic in a pandemic.

Interesting thought experiement. Imagine the best thing that could happen to you today? Not that dramatic probably since everything generally ticking along ok. Now imagine the worst. There is probably a much, much richer set of things that come up now! Similar ideas used by the media - it’s much easier to grab attention with negativity, surprise, shock etc.

Interesting reflection from this chapter - I intend to change my news reading habits to go more weekly than constantly browsing the news app (Guardian/Twitter etc) as a automatic habit. Hope that the important stuff should be more readily noticeable.

Chapter 6 - get the back story

Survivorship bias - very significant bias. Note it can really be any “filter” - survived, newsworthy, got published etc.

You only see the things that survive. Try to remember this as it can easily lead to astray if you don’t. Kickstarter examples - you tend to hear about the ones that you hear about therefore skewing your views about chances of success etc. You see nothing of the vast majority that go nowhere.

Publication bias - a flavour of survivorship bias. Generally it’s the interesting and novel findings that get published (and therefore attention). The failures, non-replicants, non-findings either don’t get published or have a naturally higher barrier to get over.

It even works on an individual level. Say someone does a small study - if you get an interesting result, because of the pressures to keep publishing, why not publish it. If it’s not that interesting, why not bin it. In other words, flukes have a greater chance of being published (and then gaining attention) - i.e. they survive the filter. Or another example - say you get an interesting result, but not quite enough evidence, why not collect a little more until it supports the interesting result and then publish! Clearly the experiment is now deeply flawed.

It’s the same as the Derren Brown ‘flipping ten heads in a row trick’ - you just don’t see the nine hours of footage it took to get the chance result!

HARKing - Hypothesis After Results Known. This is fine to do, but then you need to collect more data (new) to test the hypothesis. Otherwise it’s meaningless.

The “interestingness” filter is very powerful.

There is replication crisis - i.e. many of the interesting and famous findings in psychology have been difficult to replicate. The “old person priming” (i.e. folks thinking about old age then walking slower) in Thinking Fast and Slow (Kahneman). The “power poses” when talking on the phone etc (Cuddy). Willpower being a limited resource (Baumeister). All 3 have been difficult to replicate. Therefore reasonable to consider have they just gone through the interesting filter.

Consider survivorship bias in business writing. “In search of excellence” (Peters & Waterman 1982) highlighted companies as examples of excellent leadership & strategy - the subsequent performance of these companies was anything other than excellent. The inference being that they could be another example of survivorship bias - since chance and good fortune as much as excellent leadership could have got them to where they were.

It is interesting to view the more recent trends through the same lense - e.g. Lean Startup etc.

A good self test after reading a claim that you want to shape your world view (or others by passing it on) is to see if you could explain it - what they did, why the results are reasonable etc. If all you have is “blueberries give you cancer” then you don’t really know anything.

Two good resources for trustworthy studies and data. Cochrane aims to be a high quality source of trusted, evidence based health studies. The Campbell Collaboration aims to do the same for social studies research.

Chapter 7 - ask who is missing

The Asch conformity experiments (lengths of lines, but with a rigged group to see what the subject would do - they conformed to the group). But clearly a big question to ask is - who where the folks involved in the study? Asch used what he had - 1950s American college students. So while the results are interesting - what was tested was not conformity in general - but conformity in a 1950s American monoculture.

Should be noted that there is a WEIRD problem with a lot of research - i.e. they use folks from Western, Educated, Industrial Rich Democracies as the subjects.

As it happens there are lots of follow up studies that back the original conformity findings so it seems that conformity is a thing that applies to humans generally - but that’s not a given from the original studies. Most accounts of the “interesting finding” recount the original tale and then immediately fast forward to the application to the whole of the human race. The next set of questions and the 2nd order implications don’t seem to occur to folks.

There are lots of similar examples in the most famous and highly quoted studies. Milgram’s electric shock experiments (p146) - only used men, how often is this mentioned when the study is quoted? The Thalidimide disaster (p147) resulted in women of child bearing age being excluded from studies - possibly a sensible precaution, but it’s a big assumption that you can extrapolate results from cohorts skewed to men. Sildenafil started life as an angina treatment - in the trials it had another effect on men (which became Viagra), but also showed some positive impact on period pain in women. But since the original angina studies were not representative (i.e. more men) the follow ups have been limited.

Another example is the the UK’s Universal Credit (p150) - given to the “head of the household”. This has the potential for clear gender biases baked into it.

A random sample of 1000 is generally enough to test a thing that you might be interested in - as long as that is really a random sample. There are two ways that it might not be (p156). Sampling error - this is the risk that the sample you have is not representative. Sampling bias - that the sample chosen is not random at all.

A nice thought experiment on sampling bias - imagine you were given a small budget (time and money) to go and get a random sample of people. Could you really say that you wouldn’t have been biased in any way? I don’t know exactly how my sample would be biased but I’m pretty sure it would be.

A famous example of these errors was with the Literary Digest predicting the US election result in the 30s - Roosevelt (D) vs Landon (R). 10m folks surveyed - some 2.4m responses. Massively impressive data set for the time. Should have been accurate result - but predicted it wrong. Obvious skews in the types of people who would respond etc. George Gallop got much closer to the answer with a much smaller study - i.e. that was more representative.

The problems still exist - consider how you would get a random sample now. What if you cold called people? Who would answer? At what times of day? Responses from door knocks have gone from 80% in 1963 to 55% in 2015.

Aside - these kind of biases can go the other way too. Consider voter registration - clearly this excludes certain folks and would skew a result. Those with agendas can use these ideas.

So is the answer speaking to everyone? Even the census is hard (p155) - 5% in the UK don’t respond. Sounds small, but given they could very well be a similar group to each other and very different from the folk who did respond then that’s certainly enough to skew the result.

Big data and the move to N = all is not a silver bullet either - the sampling problems still exist. Who filled in the form? Who’s account generated the click stream data? Also easy to forget that N != all, but N = those that signed up.

Algorithms that are trained on the data tend to be “pale and male” (p159). While it’s true that there are racist/sexist folks out there - generally the problems come from unexamined biases and hidden assumptions.

Chapter 8 - computers and transparency

Google’s Flu Trends tool - algorithm on the search data, the boast was that they didn’t need to have a theory, they could just use the data and predict flu outbreaks (p164). However - it didn’t really work - had they simply built a winter detector?

Another example was a cancer detector trained on photos of genuine skin cancers. The only problem was they had rulers in them. If we don’t have a theory are we trusting our lives to a ruler detector!

Point is that “theory free” is very fragile (p165).

There is the now classic story from Charles Duhigg about Target and pregnancy targeting (told in NY Times article and in The Power of Habit). Very briefly the story goes: angry dad into the store, why have you sent my teenage daughter these adverts for baby clothes, oh sorry turns out she is pregnant after all. It’s a very well written story - since the first reaction is one of wonder and amazement that they can predict these things. But it’s a good point that if you talked to a statistician it’s not all that remarkable, the data points for pregnancy are relatively mundane - if there is a new purchase of folic acid then probably pregnant. Also consider the potential for large numbers of false positives - i.e. folks who got the adverts and weren’t pregnant.

Two interesting takes (p166) - “Big Data” (Cukier/Mayer-Schönburger) vs “Weapons of Math Destruction” (Cathy O’Neil). First is pro - written from the point of view of the folks who are doing it, they are the creators. The second is anti - written from the point of view who have the big data algorithms done to them.

Good example of the being done to is the IMPACT algorithm. Was used in US to grade teachers - and therefore influence promote & fire decisions. Turns out it was very unreliable (too many factors influencing child outcomes) and arguably destructive rather than additive.

Alchemy died because the public arena in which scientific debate happened added much more value (p185).

When things are done in secret we miss too many opportunities.

The question about algorithms (p190) shouldn’t be “can we trust them” but rather “which ones can we trust and to do what”.

Chapter 9 - don’t take the bedrock for granted

Statistical bedrock - the wealth of public data from our governmental and non-governmental organisations. There is immense value in these orgs and the data they produce.

For example the CBO in US and the OBR in the UK. OBR formed in 2010 is there to forecast spending, tax base and other economic indicators. Independent of gov - before it was the Treasury performing the same function and therefore open to all kinds of pressure and bias.

Doesn’t mean they can’t be abused. Many examples by Trump. Tanzania has a law making it illegal to criticise government stats! (p207)

Differing attitudes to what data should be collected. In Hong Kong - Cowperthwaite/Friedmen argued that there should be very little collected, laissez faire - “the less you know, the less you’ll meddle”. Probably strong situational effects here - since the meddling would have been by UK gov over 2000 miles away, so beware to extrapolate too far.

Who owns the bedrock? (p217) If they are by and for the government then they own them and this has implications since will naturally be more secretive and less reliable. If by the gov but for everyone - then better decisions can be reached with folks being better informed. Science vs alchemy.

Public scrutiny is vital (p219).

Chapter 10 - beautiful disinformation

Florence Nightingale (p228). Aside - would be worth reading her bio as much more interesting life than “lady with the lamp” image. She became the first female fellow of the Royal Statistical Society and arguably produced the first data viz.

Vision is central to our lives (p229) - as per Atomic Habits (James Clear) of the 11m or so sensors in the body, 10m are dedicated to sight. (Although now thinking about it is that as impressive an idea as it first appears? - for example is hearing 1 sensor or 500k?).

“I see what you mean” - here “see” means understand. But we should always be aware of the trap that while it is easy to “see” it’s not always easy to “understand”. Reminds of the “I’m glad we all agree diagram”.

Big Duck Graphics (p230) - infographics that are in the shape of something relevant to the data.

Dazzle camouflage (p232) - designed to provoke misjudgments. Normal camouflage is designed to make something invisible. Dazzle is the opposite - it is highly visible, with bold, abstract shapes in high contrast colours - but the point was that it made it harder to determine the course and therefore harder to torpedo. Seems to be some debate about exactly how effective it was.

Maps with data seem to give strong sense of authority that folks accept (p238).

Robert Kosara (p241) - as data viz expert - suggests plotting things on a spiral as part of exploration since it may show up periodicities that are otherwise easily missed.

How a graph is presented can have a profound effect on the meaning (p246). Consider a graph of deaths in Iraq - with no other changes than flipping the vertical axis and adding colour it can be made to look like “Bloody Iraq” (i.e. the peak below the line looks like dripping blood if in red) vs “Deaths Decline” (i.e. there was a peak, but more recently a marked decline).

This chapter feels like a missed opportunity. In a book about making better sense of data and how it is presented there could have been a few practical tips on how to read graphs better. He mentions as an aside not to use pie charts - but doesn’t explain why (harder to read than alternatives, brain bad at angles, can’t show evolution easily etc.) Also nothing on abuse of scales, secondary axis etc.

Chapter 11 - keeping an open mind

Aside - interesting comparison to the idea of an open mind in How to Take Smart Notes (Sönke Ahrens). Ahrens point is that those that self-identify as being open minded - and therefore thinking themselves free from biases and prejudices - can be more prone to believing nonsense than those who have a framework to call bullshit against. This chapter and this idea actually align pretty well - although it’s a slightly different flavour of open-mind that Harford is talking about here. He is saying be open-minded about your own weaknesses, you might have got things wrong, so don’t be arrogant and be prepared to change your views when more solid information comes to light.

Study showing that you smell what you expect to smell (p260) - BO vs cheese when primed to smell it even though the actual smelly compound is the same.

We also selectively filter information - especially when painful.

Consider the background context (p268) - for example if at a wedding someone asked the (mildly inappropriate) question of how long do you think the couple will last - basing it on your knowledge of the couple and not including the base rates of divorce would be a mistake since you’re not using a framework. If the base rate of divorce is 90% then this is useful info to know.

“Super-forecasters” (p269) - essentially those folks who think their forecasts are hypothesis to be tested and not beliefs to die for. Be prepared to change your mind.

John Maynard Keynes (p270) - another bio worth reading, as sounds like he had an interesting life too! He was prepared to change his mind contributing to his success - in contract with Irving Fisher (another economist) who held fast to his beliefs and came unstuck.

Making a strong public commitment can make it hard to change your mind. This could be both positive and negative:

  1. “I predict this thing and I stake my reputation on it” can only end badly since you’ll probs be wrong and will be hard to row back from when it is, but more importantly shows an arrogance that misses the whole point. If the aim is to see the world how it is and take steps towards truth, the a statement like that is entirely the wrong attitude - it does not say that you want to learn
  2. Another side to public commitments is that it can keep you honest, as per the habit forming strategies in Atomic Habits (James Clear) - e.g. accountability partners.

In summary - don’t be arrogant and be prepared to change your mind.

Chapter 12 - golden rule: be curious

This is probably the most interesting chapter - perhaps because it summarises everything that comes before.

The rules

  1. Emotional awareness
  2. Birds eye view vs worms eye view
  3. Do you really understand the terms/labels/definitions
  4. Get the context
  5. Where did the data come from, what was the journey that the surviving data went through, what data didn’t make it
  6. Who is missing
  7. Big data is useful but not a silver bullet
  8. Take care of the bedrock
  9. Beauty != truth
  10. Open mind

“They saw a protest” (p282) - Kahan study at Yale where a protest was shown to folks. The answers people gave depended on their already held political outlooks. Some people saw well ordered demonstrators behaving well, others saw aggressions.

Cultural, political, social identity influences your decision making.

The more curious we are the less our tribalism matters (p283).

The brain deals with hearing facts that challenge us in the same way as threatening animals (p285). A curious frame of mind however turns it into a game to solve and neutralises the threat. How can you have your identity challenged on hearing new info if your identity is one that is curious about hearing new things!

The illusion of explanatory depth (p287) - ask folks to rate their knowledge of a particular subject from 1-7. For those topics where a person gives a high number, ask them to explain it to you. The act of explaining it to someone else very quickly shows whether you understand it. Note this links to some of the learning/self testing ideas in How to Take Smart Notes (Sönke Ahrens).

The illusion bit comes in since if you self rate at a 6 or a 7 on a particular topic (and no-one asks you to explain out loud), you think you know how it works - so why would you ever go deeper?

Interesting point about how this worked with political topics - the study showed that when was performed with people of differing views, the exercise reduced the polarisation as folks realised that they didn’t know everything.