Super Crunchers Read online

  CONTENTS

  Title Page

  Dedication

  Introduction: The Rise of the Super Crunchers

  1 Who’s Doing Your Thinking for You?

  2 Creating Your Own Data with the Flip of a Coin

  3 Government by Chance

  4 How Should Physicians Treat Evidence-Based Medicine?

  5 Experts Versus Equations

  6 Why Now?

  7 Are We Having Fun Yet?

  8 The Future of Intuition (and Expertise)

  Acknowledgments

  Notes

  Also by Ian Ayres

  Copyright

  For John Donohue and Peter Siegelman,

  who were there when I needed them most

  INTRODUCTION

  The Rise of the Super Crunchers

  Orley Ashenfelter really loves wine: “When a good red wine ages,” he says, “something quite magical happens.” Yet Orley isn’t just obsessed with how wine tastes. He wants to know about the forces behind great and not-so-great wines.

  “When you buy a good red wine,” he says, “you’re always making an investment, in the sense that it’s probably going to be better later. And what you’d like to know is not what it’s worth now, but what it’s going to be worth in the future. Even if you’re not going to sell it—even if you’re going to drink it. If you want to know, ‘How much pleasure am I going to get by delaying my gratification?’ That’s an endlessly fascinating topic.” It’s a topic that has consumed a fair amount of his life for the last twenty-five years.

  In his day job, Orley crunches numbers. He uses statistics to extract hidden information from large datasets. As an economist at Princeton, he’s looked at the wages of identical twins to estimate the impact of an extra year of school. He’s estimated how much states value a statistical life by looking at differences in speed limits. For years he edited the leading economics journal in the United States, the American Economic Review.

  Ashenfelter is a tall man with a bushy mane of white hair and a booming, friendly voice that tends to dominate a room. No milquetoast he. He’s the kind of guy who would quickly disabuse you of any stereotype you might have that number crunchers are meek, retiring souls. I’ve seen Orley stride around a classroom, gutting the reasoning behind a seminar paper with affable exuberance. When he starts out his remarks with over-the-top praise, watch out.

  What’s really gotten Orley in trouble is crunching numbers to assess the quality of Bordeaux wines. Instead of using the “swishing and spitting” approach of wine gurus like Robert Parker, Orley has used statistics to find out what characteristics of vintage are associated with higher or lower auction prices.

  “It’s really a no-brainer,” he said. “Wine is an agricultural product dramatically affected by the weather from year to year.” Using decades of weather data from France’s Bordeaux region, Orley found that low levels of harvest rain and high average summer temperatures produce the greatest wines. The statistical fit on data from 1952 through 1980 was remarkably tight for the red wines of Burgundy as well as Bordeaux.

  Bordeaux are best when the grapes are ripe and their juice is concentrated. In years when the summer is particularly hot, grapes get ripe, which lowers their acidity. And, in years when there is below-average rainfall, the fruit gets concentrated. So it’s in the hot and dry years that you tend to get the legendary vintages. Ripe grapes make supple (low-acid) wines. Concentrated grapes make full-bodied wines.

  He’s had the temerity to reduce his theory to a formula:

  Wine quality = 12.145 + 0.00117 winter rainfall + 0.0614 average growing season temperature–0.00386 harvest rainfall

  That’s right. By plugging the weather statistics for any year into this equation, Ashenfelter can predict the general quality of any vintage. With a slightly fancier equation, he can make more precise predictions for the vintage quality at more than 100 Châteaus. “It may seem a bit mathematical,” he acknowledges, “but this is exactly the way the French ranked their vineyards back in the famous 1855 classifications.”

  Traditional wine critics have not embraced Ashenfelter’s data-driven predictions. Britain’s Wine magazine said “the formula’s self-evident silliness invite[s] disrespect.” William Sokolin, a New York wine merchant, said the Bordeaux wine industry’s view of Ashenfelter’s work ranges “somewhere between violent and hysterical.” At times, he’s been scorned by trade members. When Ashenfelter gave a wine presentation at Christie’s Wine Department, dealers in the back openly hissed at his presentation.

  Maybe the world’s most influential wine writer (and publisher of The Wine Advocate), Robert Parker, colorfully called Ashenfelter “an absolute total sham.” Even though Ashenfelter is one of the most respected quantitative economists in the world, to Parker his approach “is really a Neanderthal way of looking at wine. It’s so absurd as to be laughable.” Parker dismisses the possibility that a mathematical equation could help identify wines that actually taste good: “I’d hate to be invited to his house to drink wine.”

  Parker says Ashenfelter “is rather like a movie critic who never goes to see the movie but tells you how good it is based on the actors and the director.”

  Parker has a point. Just as it’s more accurate to see the movie, shouldn’t it be more accurate to actually taste the wine? There’s just one catch: for months and months there is no wine to taste. Bordeaux and Burgundies spend eighteen to twenty-four months in oak casks before they are set aside for aging in bottles. Experts, like Parker, have to wait four months just to have a first taste after the wine is placed in barrels. And even then it’s a rather foul, fermenting mixture. It’s not clear that tasting this undrinkable early wine gives tasters very accurate information about the wine’s future quality. For example, Bruce Kaiser, former director of the wine department at auctioneer Butterfield & Butterfield, has claimed, “Young wines are changing so quickly that no one, and I mean no one, can really evaluate a vintage at all accurately by taste until it is at least ten years old, perhaps older.”

  In sharp contrast, Orley crunches numbers to find the historical relationship between weather and price. That’s how he found out that each centimeter of winter rain tends to raise the expected price 0.00117 dollars. Of course, it’s only a tendency. But number crunching lets Orley predict the future quality of a vintage as soon as the grapes are harvested—months before even the first barrel taste and years before the wine is sold. In a world where wine futures are actively traded, Ashenfelter’s predictions give wine collectors a huge leg up on the competition.

  Ashenfelter started publishing his predictions in the late eighties in a semiannual newsletter called Liquid Assets. He advertised the newsletter at first with small ads in Wine Spectator, and slowly built a base of about 600 subscribers. The subscribers were a geographically diverse lot of millionaires and wine buffs—mostly confined to the small group of wine collectors who were comfortable with econometric techniques. The newsletter’s subscription base was just a fraction of the 30,000 people who plunked down $30 a year for Robert Parker’s newsletter The Wine Advocate.

  Ashenfelter’s ideas reached a much larger audience in early 1990 when the New York Times published a front-page article about his new prediction machine. Orley openly criticized Parker’s assessment of the 1986 Bordeaux. Parker had rated the ’86s as “very good and sometimes exceptional.” Ashenfelter disagreed. He felt that the below-average growing season temperature and the above-average harvest rainfall doomed this vintage to mediocrity.

  Yet the real bombshell in the article concerned Orley’s prediction about the 1989 Bordeaux. While these wines were barely three months in the cask and had yet to even be tasted by critics, Orley predicted that they would be “the wine of the century.” He guaranteed th
at they would be “stunningly good.” On his scale, if the great 1961 Bordeaux were 100, then the 1989 Bordeaux were a whopping 149. Orley brazenly predicted that they would “sell for as high a price as any wine made in the last thirty-five years.”

  The wine critics were incensed. Parker now characterized Ashenfelter’s quantitative estimates as “ludicrous and absurd.” Sokolin said the reaction was a mixture of “fury and fear. He’s really upset a lot of people.” Within a few years Wine Spectator refused to publish any more ads for his (or any other) newsletter.

  The traditional experts circled the wagons and tried to discredit both Orley and his methodology. His methodology was flawed, they said, because it couldn’t predict the exact future price. For example, Thomas Matthews, the tasting director of Wine Spectator, complained that Ashenfelter’s price predictions “were exactly true only three times in the twenty-seven vintages.” Even though Orley’s “formula was specifically designed to fit price data” his “predicted prices are both under and over the actual prices.” However, to statisticians (and to anyone else who thinks about it for a moment), having predictions that are sometimes high and sometimes low is a good thing; it’s the sign of an unbiased estimate. In fact, Orley showed that Parker’s initial ratings of vintages had been systematically biased upward. More often than not Parker had to downgrade his initial rankings.

  In 1990 Orley went even further out on a limb. After declaring 1989 the vintage of the century, the data told him that 1990 was going to be even better. And he said so. In retrospect we can now see that Liquid Assets’ predictions have been astonishingly accurate. The ’89s turned out to be a truly excellent vintage and the ’90s were even better.

  How can it be that you’d have two “vintages of the century” in two consecutive years? It turns out that since 1986 there hasn’t been a year with below-average growing season temperature. The weather in France has been balmy for more than two decades. A convenient truth for wine lovers is that it’s been a great time to grow truly supple Bordeaux.

  The traditional experts are now paying a lot more attention to the weather. While many of them have never publicly acknowledged the power of Orley’s predictions, their own predictions now correspond much more closely to his simple equation results. Orley still maintains his website, www.liquidasset.com, but he’s stopped producing the newsletter. He says, “Unlike the past, the tasters no longer make any horrendous mistakes. Frankly, I kind of killed myself. I don’t have as much value added anymore.”

  Ashenfelter’s detractors see him as a heretic. He threatens to de-mystify the world of wine. He eschews flowery and nonsensical (“muscular,” “tight,” or “rakish”) jargon, and instead gives you reasons for his predictions.

  The industry’s intransigence is not just about aesthetics. “The wine dealers and writers just don’t want the public informed to the degree that Orley can provide,” Kaiser notes. “It started with the ’86 vintage. Orley said it was a scam, a dreadful year suffering from a lot of rain and not high enough temperatures. But all the wine writers of the day were raving, insisting it was a great vintage. Orley was right, but it isn’t always popular to be right.”

  Both the wine dealers and writers have a vested interest in maintaining their informational monopoly on the quality of wine. The dealers use the perennially inflated initial rankings as a way to stabilize prices. And Wine Spectator and The Wine Advocate have millions of dollars at stake in remaining the dominant arbiters of quality. As Upton Sinclair (and now Al Gore) has said, “It is difficult to get a man to understand something when his salary depends on his not understanding it.” The same holds true for wine. “The livelihood of a lot of people depends on wine drinkers believing this equation won’t work,” Orley says. “They’re annoyed because suddenly they’re kind of obsolete.”

  There are some signs of change. Michael Broadbent, chairman of the International Wine Department at Christie’s in London, puts it diplomatically: “Many think Orley is a crank, and I suppose he is in many ways. But I have found that, year after year, his ideas and his work are remarkably on the mark. What he does can be really helpful to someone wanting to buy wine.”

  The Orley Ashenfelter of Baseball

  The grand world of oenophiles seems worlds away from the grandstands of baseball. But in many ways, Ashenfelter was trying to do for wine just what writer Bill James did for baseball.

  In his annual Baseball Abstracts, James challenged the notion that baseball experts could judge talent simply by watching a player. Michael Lewis’s Moneyball showed that James was baseball’s herald of data-driven decision making. James’s simple but powerful thesis was that data-based analysis in baseball was superior to observational expertise:

  The naked eye was inadequate for learning what you need to know to evaluate players. Think about it. One absolutely cannot tell, by watching, the difference between a .300 hitter and a .275 hitter. The difference is one hit every two weeks…. If you see both 15 games a year, there is a 40 percent chance that the .275 hitter will have more hits than the .300 hitter…. The difference between a good hitter and average hitter is simply not visible—it is a matter of record.

  Like Ashenfelter, James believed in formulas. He said, “A hitter should be measured by his success in that which he is trying to do, and that which he is trying to do is create runs.” So James went out and created a new formula to better measure a hitter’s contribution to runs created:

  Runs Created = (Hits + Walks) × Total Bases/(At Bats + Walks)

  This equation put much more emphasis on a player’s on-base percentage and especially gives higher ratings to those players who tend to walk more often. James’s number-crunching approach was particular anathema to scouts. If wine critics like Robert Parker live by their taste and smell, then scouts live and die by their eyes. That’s their value added. As described by Lewis:

  In the scouts’ view, you found big league ball players by driving sixty thousand miles, staying in a hundred crappy motels, and eating God knows how many meals at Denny’s all so you could watch 200 high school and college baseball games inside of four months, 199 of which were completely meaningless to you…. [Y]ouwould walk into the ball park, find a seat on the aluminum plank in the fourth row directly behind the catcher and see something no one else had seen—at least no one who knew the meaning of it. You only had to see him once. “If you see it once, it’s there.”

  The scouts and wine critics like Robert Parker have more in common than simply a penchant for spitting. Just as Parker believes that he can assess the quality of a Château’s vintage based on a single taste, baseball scouts believe they can assess the quality of the high school prospect based on a single viewing.

  In both contexts, people are trying to predict the market value of untested and immature products, whether they be grapes or baseball players. And in both contexts, the central dispute is whether to rely on observational expertise or quantitative data.

  Like wine critics, baseball scouts often resort to non-falsifiable euphemisms, such as “He’s a real player,” or “He’s a tools guy.”

  In Moneyball, the conflict between data and traditional expertise came to a head in 2002 when Oakland A’s general manager Billy Beane wanted to draft Jeremy Brown. Beane had read James and decided he was going to draft based on hard numbers. Beane loved Jeremy Brown because he had walked more frequently than any other college player. The scouts hated him because he was, well, fat. If he tried to run in corduroys, an A’s scout sneered, “he’d start a fire.” The scouts thought there was no way that someone with his body could play major league ball. Beane couldn’t care less how a player looked. His drafting mantra was “We’re not selling jeans.” Beane just wanted to win games. The scouts seem to be wrong. Brown has progressed faster than anyone else the A’s drafted that year. In September 2006, he was called up for his major league debut with the A’s and batted .300 (with an on-base percentage of .364).

  There are striking parallels between the ways that Ash
enfelter and James originally tried to disseminate their number-crunching results. Just like Ashenfelter, James began by placing small ads for his first newsletter, Baseball Abstracts (which he euphemistically characterized as a book). In the first year, he sold a total of seventy-five copies. Just as Ashenfelter was locked out of Wine Spectator, James was given the cold shoulder by the Elias Sports Bureau when he asked to share data.

  But James and Ashenfelter have forever left their marks upon their industries. The perennial success of the Oakland A’s, detailed in Moneyball, and even the first world championship of the Boston Red Sox under the data-driven management of Theo Epstein are tributes to James’s lasting influence. The improved weather-driven predictions of even traditional wine writers are silent tributes to Ashenfelter’s impact.

  Both have even given rise to gearhead groups that revel in their brand of number crunching. James inspired SABR, the Society for American Baseball Research. Baseball number crunching now even has its own name, sabermetrics. In 2006, Ashenfelter in turn helped launch the Journal of Wine Economics. There’s even an Association of Wine Economists now. Unsurprisingly, Ashenfelter is its first president. By the way, Ashenfelter’s first predictions in hindsight are looking pretty darn good. I looked up recent auction prices for Château Latour and sure enough the ’89s were selling for more than twice the price of the ’86s, and 1990 bottles were priced even higher. Take that, Robert Parker.

  In Vino Veritas

  This book’s central claim is that the rise of number crunching in wine and baseball are not isolated events. In fact, the wine and baseball examples are microcosms of the larger themes of this book. We are in a historic moment of horse-versus-locomotive competition, where intuitive and experiential expertise is losing out time and time again to number crunching. In the old days, many decisions were simply based on some mixture of experience and intuition. Experts were ordained because of their decades of individual trial-and-error experience. We could trust that they knew the best way to do things, because they’d done it hundreds of times in the past. Experiential experts had survived and thrived. If you wanted to know what to do, you’d ask the gray-hairs.