I have moved the blog to a new server at civilstat.com where I can host custom visualizations, maps, code, etc. more easily. The old posts are still archived here for now but new material will only appear on the new site.
The process of doing science, math, engineering, etc. is usually way messier than how those results are reported. Abstruse Goose explains it well:
In pure math, that’s usually fine. As long as your final proof can be verified by others, it doesn’t necessarily matter how you got there yourself.
Now, verifying it might be hard, for example with computer-assisted proofs like that of the Four Color Theorem. And teaching math via the final proof might not be the best way, pedagogically, to develop problem-solving intuition.
But still, a theorem is either true or it isn’t.
However, in the experimental sciences, where real-world data is inherently variable, it’s very rare that you can really say, “I’ve proven that Theory X is true.” Usually the best you can do is to say, “I have strong evidence for Theory X,” or, “Given these results it is reasonable to believe in Theory X.”
(There’s also decision theory: “Do we have enough evidence to think that Theory X is true?” is a separate question from “Do we have enough evidence to act as if Theory X is true?”)
In these situations, the way you reached your conclusions really does affect how trustworthy they are.
Some of their recommendations only make sense for limited types of analysis, but for those cases, it is sensible advice. I thought that the contrast between their two descriptions of Study 2 (“standard” on p. 2, versus “compliant” on p. 6) was very effective.
I’m not sure what to think of their idea of limiting “researcher degrees of freedom.”
For example, they discourage a Bayesian approach because “Bayesian statistics require making additional judgments (e.g., the prior distribution) on a case-by-case basis, providing yet more researcher degrees of freedom.”
I’m a bit hesitant to say that researchers should be pigeonholed into the standard frequentist toolkit and not allowed to use their best judgment!
If canned frequentist methods are unsuitable for the problem at hand, or underestimate uncertainty relative to a carefully-thought-out, problem-appropriate Bayesian method, you may not be doing better after all…
However, like the authors of this paper, I do support better reporting of why a certain analysis was judged to be the right tool for the job.
Ideally, more of us would know Bayesian methods and could justify the choice between frequentist and Bayes approaches for the given problem at hand, not by always saying “the frequentist approach is standard” and stopping our thinking there.
I see some kind of vague showy, wiggling lines — here and there an E and a B written on them somehow, and perhaps some of the lines have arrows on them — an arrow here or there which disappears when I look too closely at it. When I talk about the fields swishing through space, I have a terrible confusion between the symbols I use to describe the objects and the objects themselves. I cannot really make a picture that is even nearly like the true waves.
As it turns out, he probably did:
As I’m talking, I see vague pictures of Bessel functions from Jahnke and Emde’s book, with light-tan j’s, slightly violet-bluish n’s, and dark brown x’s flying around. And I wonder what the hell it must look like to the students.
The letter-color associations in this second quote are a fairly common type of synaesthesia. However, the first quote above sounds quite different, but still plausibly like synaesthesia: “I have a terrible confusion between the symbols I use to describe the objects and the objects themselves”…
I wonder whether many of the semi-mystical genius-heroes of math & physics lore (also, for example, Ramanujan) have had such neurological conditions underpinning their unusually intuitive views of their fields of study.
I love the idea of synaesthesia and am a bit jealous of people who have it. I’m not interested in drug-induced versions but I would love to experiment with other ways of experiencing synthetic synaesthesia myself. Wired Magazine has an article on such attempts, and I think I remember another approach discussed in Oliver Sacks’ book Musicophilia.
I have a friend who sees colors in letters, which helps her to remember names — I’ve heard her think out loud along these lines: “Hmm, so-and-so’s name is kind of reddish-orange, so it must start with P.” I wonder what would happen if she learned a new alphabet, say the Cyrillic alphabet (used in Russian etc.): would she associate the same colors with similar-sounding letters, even if they look different? Or similar-looking ones, even if they sound different? Or, since her current associations were formed long ago, would she never have any color associations at all with the new alphabet?
Also, my sister sees colors when she hears music; next time I see her I ought to ask for more details. (Is the color related to the mood of the song? The key? The instrument? The time she first heard it? etc. Does she see colors when practicing scales too, or just “real” songs?)
Finally, this isn’t quite synaesthesia but another natural superpower in a similar vein, suggesting that language can influence thought:
…unlike English, many languages do not use words like “left” and “right” and instead put everything in terms of cardinal directions, requiring their speakers to say things like “there’s an ant on your south-west leg”. As a result, speakers of such languages are remarkably good at staying oriented (even in unfamiliar places or inside buildings) and perform feats of navigation that seem superhuman to English speakers. In this case, just a few words in a language make a big difference in what cognitive abilities their speakers develop. Certainly next time you plan to get lost in the woods, I recommend bringing along a speaker of Kuuk Thaayorre or Guugu Yimithirr rather than, say, Dutch or English.
The human brain, ladies and gentlemen!
Scientific American has a short article on trends in undergraduate degrees over the past 20 years, illustrated with a great infographic by Nathan Yau. As a big fan of STEM (science, tech, engineering and math) education, I was pleased to see data on changing patterns among STEM degree earners.
However, there seemed to be a missed opportunity. The article mentioned that “More women are entering college, which in turn is changing the relative popularity of disciplines.” If the data were broken down by gender, readers could better see this fact for themselves.
I thought I could exploit the current graphic’s slight redundancy: the bar heights below and above the gray horizontal lines are exactly the same. Why not repurpose this format to show data on degrees earned by men vs. by women (below vs. above the horizontal line), in the same amount of space?
I could not find the gender breakdown for the exact same set of degrees, but a similar dataset is in the Digest of Education Statistics, tables 308 to 330. Here are my revised plots, made using R with the ggplot2 package.
Or see the STEM and non-STEM plots separately below.
So, what’s the verdict? These new graphs do support SciAm’s conclusions: women are largely driving the increases in psychology and biology degrees (as well as “health professions and related sciences”), and to a lesser degree in the arts and communications. On the other hand, increases in business and social science degrees appear to be driven equally by males and females. The mid-’00s spike in computer science was mostly guys, it seems.
I’d also like to think that my alma mater, Olin College, contributed to the tiny increase in female engineers in the early ’00s
Some of these degree categories are hard to classify as STEM vs. non-STEM. In particular, Architecture and SocialScience include some sub-fields of each type… Really, I lumped them under non-STEM only because it balanced the number of items in each group.
Many thanks to a helpful Learning R tutorial on back-to-back bar charts.
Shawn Cornally always has good ideas about how to keep high school useful:
“I want my student to be able to produce something from this study that lingers instead of just rots on a hard drive, because, like church, school shouldn’t be about the building.”
That also reminds me: I should make a list of my favorite simple-but-useful cooking science tips. For example, after I learned just a bit about the science of gluten in flour, it made so much more sense why you knead bread so thoroughly but you only mix muffin batter “just until combined” (lumps okay).
While we’re on the subject of statistics-related classroom activities with a “wow factor,” let me bring up my favorite: guessing whether a sequence of coin flips is real or fake.
For me, it really brought home the idea that math is an amazing BS detector. Sure, we tell kids to learn math so you can balance your checkbook, figure out the tip at a restaurant, blah blah blah. But consider these very reasonable counterarguments: (1) Yawn, and (2) Calculators/computers do all that for us anyway.
So you have to fire back: you wanna get screwed over? When you sign up for student loans at a terrible rate because the loan officer was friendly and you couldn’t even guesstimate the math in your head, you’ll be stuck with awful payments for the next 10 years. When your phone company advertises “.002 cents per kilobyte” but charges you .002 dollars per kilobyte instead, a hundred times as much, you should call them out on it.
You may never have the luck to acquire a superhero spider sense, but we mortals can certainly hone our number sense. People will try to con you over the years, but if you keep this tool called “math” in your utility belt I guarantee it’ll save your butt a few times down the line.
Anyway, the coin flip thing itself may be more of a cute demo than directly practical — but it’s really really cute. Watch:
You split the class into two groups. One is going to flip a coin 100 times in a row and write down the resulting sequence of heads and tails. The other is going to pretend they did this and write down a made-up “random” sequence of heads and tails. The teacher leaves the room until both groups are done, then comes back in and has to guess which sequence came from real coin flips and which is the fake. And BAM, like magic, no calculation required, the teacher’s finely-honed number-sense makes it clear which is which.
Can you tell from the pair below?
(example copied from Gelman and Nolan, 2002, Teaching Statistics)
Enterprising statisticians have noticed that, in a sequence of 100 truly random coin flips, there’s a high probability of at least one “long” streak of six or more heads in a row (and same for tails). Meanwhile, people faking the data usually think that long streaks don’t look “random” enough. So the fake sequence will usually switch back and forth from heads to tails and back after only 2 or 3 of each, while the real sequence will have a few long streaks of 5 or 6 or more heads (or tails) in a row.
So is your number sense tingling yet? In the example above, the sequence on the left is real while the right-hand data was faked.
(I’m not sure where this demo originates. I first heard of it in Natalie Angier’s 2007 book The Canon, but it’s also described in Gelman and Nolan’s 2002 book Teaching Statistics mentioned above, and in Ted Hill’s 1999 Chance magazine article “The Difficulty of Faking Data”. Hill’s article is worth a read and goes into more detail on another useful statistical BS detector, Benford’s Law, that can detect patterns of fraudulent tax data!)
Lesson learned: randomness may look non-random, and vice versa, to the untrained eye. Sure, this is a toy example, but let’s generalize a bit. First, here we have random data generated in one dimension, time. This shows that long winning or losing streaks can happen by pure chance, far more often than most people expect. Say the sports team you manage has been on a winning (or losing) streak — does that mean the new star player is a real catch (or dud)? Maybe not; it might be a coincidence, unless the streak keeps running much longer than you’d expect due to chance… and statisticians can help you calibrate that sense of just how long to expect it.
Or imagine random data generated in two dimensions, spatial data, like mapping disease incidence on a grid of city blocks. Whereas before we had winning/losing streaks over time, now we’ll have clusters in space. We don’t know where they’ll be but we are sure there’s going to be some clustering somewhere. So if neighborhood A seems to have a higher cancer rate than neighborhood B, is there a local environmental factor in ‘hood A that might be causing it? Or is it just a fluke, to be expected, since some part of town will have the highest rates even if everyone is equally at risk? This is a seriously hard problem and can make a big difference in the way you tackle public health issues. If we cordon off area A, will we be saving lives or just wasting time and effort? Statisticians can tell, better than the untrained eye, whether the cluster is too intense to be a fluke.
It’s hard to make good decisions without knowing what’s a meaningful pattern and what’s just a coincidence. Statistics is a crazy powerful tool for figuring this out — almost magical, as the coin flip demo shows.
In the last post I described a problem with Dan Meyer’s otherwise excellent expected-values teaching tool: you’d like to wow the kids by correctly predicting the answer a month in advance, but the given setup is actually too variable to let you make a safe prediction.
Essentially, if you’re saying “Let’s do a magic trick to get kids engaged in this topic,” but the trick takes a month to run AND only has a 30% chance of working… then why not tweak the trick to be more reliable?
spin it many more times?
Part of this unreliability comes from the low number of spins — about 20 spins total, if you do it once every weekday for a month. The “easy” fix is to spin Dan’s spinner many more times … but it turns out you’d have to spin it about 4000 times to be 90% confident your prediction will be right. Even if you have the patience, that’s a lot of individual spins to track in your spreadsheet or wherever.
use a MORE reliable spinner?
Another fix might be to change the spinner so that it works reliably given only 20 spins. First, we don’t want any of the sectors too small, else we might not hit them at all during our 20 spins, and then it becomes unpredictable. It turns out the smallest sector has to be at least about 1/9th of the spinner if you want to be 90% confident of hitting that sector at least once in those 20 spins.
(Let y ~ Binomial(p=1/9, n=20). Then prob(y>0) = 1-prob(y=0) = 0.905.)
If we round that up to 1/8th instead, we can easily use a Twister spinner (which has 16 equal sectors).
After playing with some different options, using the same simulation approach as the previous post, I found that the following spinner seems to work decently: 1/2 chance of $100, 3/8 chance of $150, and 1/8 chance of $1500. After 20 spins of this spinner, there’s about a 87% chance that the “$1500” will have been the winning bet, so you can be pretty confident about making the right bet a month in advance.
Unfortunately, predicting that spinner correctly is kind of unimpressive. The “$1500” is a fairly big slice, so it doesn’t look too risky.
spin just a few more times and use a safer spinner!
What if we spin it just a few more than 20 times — say 60 times, so two or three times each day? That’s not too much data to keep track of. Will that let us shrink the smallest slice, while keeping predictability high, and thus making this all more impressive?
Turns out that if we know we’ll have about 60 spins, we can make the smallest slice 1/25th of the spinner and still be confident we’ll hit it at least once. Cool. If we want to keep the Twister board, and have the smallest slice be 1/16th of the circle, we actually have a 90% chance of hitting it at least twice. So that makes things even more predictable (for the teacher), while still making it less predictable (to the kids) than the previous spinner.
More messing around led to this suggested spinner: 1/2 chance of $100, 5/16 chance of $200, 1/8 chance of $400, and 1/16 chance of $2500. The chance that “$2500” is the right bet after 60 spins of this spinner is about 88%, so again you can make your bet confidently — but this time, the “right” answer doesn’t look as obvious.
In short, I’d recommend using this spinner for around 60 spins, rather than Dan’s spinner for 20 spins. It’s not guaranteed to be “optimal” but it’s far more reliable than the original suggestion.
If anyone tries it, I’d be curious to hear how it went!