How to be smart about science stories: It works for sports stat pieces too

Hat tip to B/R Lead Writers Editor Jake Leonard for pointing out this Vox piece headlined “15 ways to tell if that science news story is hogwash.”

As Jake points out, the wisdom in that piece, which is a guide to spotting the flaws in news reports about scientific studies, could be used when reading—and writing—sports stories based on statistics.

I would go so far as to say that the top 15 ways to tell if that stats-based sports story is hogwash are Numbers 4, 6 and 10 of Vox’s 15, repeated five times each. Here they are, in the words of Vox writer Susannah Locke:

4) Correlation and causation: Just because two things are correlated doesn’t mean that one caused the other. If you want to really find out if something causes something else, you have to set up a controlled experiment. (Chemical Compound’s infographic brings up the fabulous example of the correlation between fewer pirates over time and increasing global temperature. It’s almost certain that fewer pirates did not cause global temperatures to rise, but the two are correlated.)

This is a huge one in sports analysis. Probably the most famous example of mistaking correlation for causation is the idea that football teams have to “establish the ground game” to win. The evidence typically cited for this idea is that the winning team almost always outrushes the losing team. Of course it does: Teams that are leading run to kill the clock. Piling up rushing yardage doesn’t cause wins. It’s the reverse.

6) Small sample sizes: Did the researchers study a large enough group to know that the results aren’t just a fluke? That is, did they treat cancer in two people or in 200? Or 2,000? Was that brain scanning psych study on just seven people?

The next time you hear that a hitter “owns” a pitcher because he’s 3-for-5 lifetime against him, remember this one. For all you know, he’d be hitting .200 against the guy if he hadn’t benefited from a bad hop and a 12-foot squibber that the catcher had to take a bite out of.

10) “Cherry-picked” results: Ignoring the results that don’t support your hypothesis will not make them go away. It’s possible that the worst cherry-picking happens before a study is published. There’s all kinds of data that the scientific community and the public will never see.

In sports analysis, cherry-picking often appears in the form of selective endpoints. Here’s a piece from 2012 in which Hardball Talk’s Craig Calcaterra chides Jon Morosi of Fox Sports for arguing that Miguel Cabrera deserved the American League MVP award over Mike Trout because he’d been much better “since Aug. 24″:

Why August 24? Do games before that not matter? Or is it because on August 23 Mike Trout had a big game, going 3 for 6 and driving in a couple of runs and after that had an 0 for 5 while Miguel Cabrera ended an 0 for 10 stretch on August 24 and hit a homer? It has to have some other significance, does it not? Because it cannot be the case that Morosi felt it necessary to cut off things on that date simply because it bolsters his preconceived notions of the matter.

Here’s another piece, from Baseball Prospectus in 2011, in which Colin Wyers talks about both selective endpoints and small sample size to assure fans of the Boston Red Sox and Tampa Bay Rays that their teams could still make the playoffs even though they’d started 0-5. Only two playoff teams had ever started the season 0-5.

Playoff teams had lots of losing streaks of five games and longer, Wyers wrote. They just rarely happened in the first games of the season.

As it turned out, the Rays made the playoffs and the Sox missed by one game.