Be still my accuracy-seeking heart, Tetlock conducted a twenty-year study involving 284 political pundits, journalists, academics, and government types who toiled in making predictions about political and economic events. He asked a lot of questions over the years. Would Gorbachev be thrown out in a coup? Would apartheid in South Africa be ended in a non-violent manner? Would Canada break apart? Turns out these supposed experts did little better than chance at predicting these events. The experts did only slightly better than non-experts in their area of expertise. Crazy? Are you surprised?
I was so hepped up on all this I sought out more on this study as my copy of Phil's book (Expert Political Judgment) is stuck somewhere in the snow and ice (thanks, Amazon Prime). Here's a 2006 article from The New Yorker about the above study:
Tetlock also found that specialists are not significantly more reliable than non-specialists in guessing what is going to happen in the region they study. Knowing a little might make someone a more reliable forecaster, but Tetlock found that knowing a lot can actually make a person less reliable. “We reach the point of diminishing marginal predictive returns for knowledge disconcertingly quickly,” he reports. “In this age of academic hyperspecialization, there is no reason for supposing that contributors to top journals—distinguished political scientists, area study specialists, economists, and so on—are any better than journalists or attentive readers of the New York Times in ‘reading’ emerging situations.” And the more famous the forecaster the more overblown the forecasts. “Experts in demand,” Tetlock says, “were more overconfident than their colleagues who eked out existences far from the limelight.” Everybody's an Expert, The New YorkerPhil, have you even met any breastfeeding researchers?
Anyhow, Tetlock popped up in a chapter in my new favorite book, Thinking, one of the selections from my last post. Anyhow, after re-reading his chapter last week I got to thinking about his work. I wondered if he'd come across any research on the experts targeting parents and their chirrun. So I emailed him about the neglected state of parenting advice and claims. Although he's devoted a large portion of his career to the accuracy of political judgments and has teamed up with the federal government to track the accuracy of various forecasters (a newer project), he told me he hasn't run across any research on accuracy evaluations in the children's health or parenting spheres. He did, however, agree "THAT THE ASSERTION-TO-EVIDENCE RATIO IS LIKELY TO BE VERY UNFAVORABLE." Yes in ALL CAPS. Love it, makes me miss academia, truly.
I must also point out that he returned my email pronto, adding to my terribly unscientific opinion that the greater the expertise, the quicker the response to emails. Also, the fewer attempts at subterfuge or scientific jargon and arcane references to throw me off the trail. The last part is not so surprising. If a person really knows their stuff and is a true expert then they also recognize the limitations of their knowledge, the collected wisdom in the field, and the inherent uncertainty in the world not to mention your teenager's brain. But I'm not sure how to explain the impressive email response rate.
Moment of parenting zen brought to you by the true experts, i.e. The Onion:
Child-Safety Experts Call for Restrictions on Childhood Imagination.