SLOW Movement

NY Times: Local Governments Offers Data to Miners

December 7, 2009

Local Governments Offer Data to Miners

By CLAIRE CAIN MILLER

Eric Gundersen, a developer in Washington, D.C., has created an application showing people the safest route home from bars and another that tells people about bike routes.

SAN FRANCISCO — A big pile of city crime reports is not all that useful. But what if you could combine that data with information on bars, sidewalks and subway stations to find the safest route home after a night out?

In Washington, a Web site called Stumble Safely makes that possible. It is one example of the kind of creativity that cities are hoping to mobilize by turning over big chunks of data to programmers and the public.

Many local governments are figuring out how to use the Internet to make government data more accessible. The goal is to spawn useful Web sites and mobile applications — and perhaps even have people think differently about their city and its government.

“It will change the way citizens and government interact, but perhaps most important, it’s going to change the way elected officials and civil servants deliver programs, services and promises,” said Gavin Newsom, the mayor of San Francisco, which is one of the cities leading the way in releasing government data to Web developers. “I can’t wait until it challenges and infuriates the bureaucracy.”

Advocates of these open-data efforts say they can help citizens figure out what is going on in their backyards and judge how their government is performing.

But programmers have had trouble getting their hands on some data. And some activists and software developers wonder whether historically reticent governments will release data that exposes problems or only information that makes them look good.

It is too early to say whether releasing city data will actually make civil servants more accountable, but it can clearly be useful. Even data about mundane things like public transit and traffic can improve people’s lives when it is packaged and customized in an accessible way — a situation that governments themselves may not be equipped to realize.

A Web site called CleanScores, for instance, tracks restaurant inspection scores in various cities and explains each violation. After School Special combines data from San Francisco schools, libraries and restaurants so parents can plan after-school activities and see how children’s nutritional options compare by neighborhood. And Trees Near You, available for the iPhone, lets people identify trees on New York streets.

By releasing data in easy-to-use formats, cities and states hope that people will create sites or applications that use it in ways City Hall never would have considered.

San Francisco recently unveiled DataSF, a Web clearinghouse of raw government data that the public can download. The data sets include seismic hazard zones, street sweeping schedules and campaign finance filings. New York City’s Data Mine includes directories of sidewalk cafes, property values, horseback riding trails and historic houses.

Washington was a leader in releasing its data, and the federal government is doing it too at Data.gov.

Much of this data has always been publicly available, but until recently it has been almost impossible to find. Getting hold of it might have required tenacity, drive and endless phone calls.

The push to publicize government data goes as far back as the 1960s, but technology has made it possible for people to use the data in ways that would not have been possible even a year ago, said Eric Gundersen, president of Development Seed, the Washington company that created Stumble Safely. The company builds data and map applications for international development programs.

“The timing now with the open data movement is really critical because there are a lot of open-source tools that really make that data usable,” Mr. Gundersen said. These include the mapping tool he used to build Stumble Safely and also a site for the United States Agency for International Development that maps public health clinics.

Some activists are skeptical that governments will release politically risky data that could show that people are not doing their jobs. Mayor Newsom said he wants to release all kinds of data, and said he would not be surprised if “people who love to hate their mayor” create an application that maps his public schedule, to bolster their cases about which parts of town he neglects.

There is also the concern that people might misinterpret what the data is telling them.

“In the most basic of forms, with regard to crime stats and unemployment numbers, these kinds of bulletin boards are very useful,” said Saul Bloom, executive director of Arc Ecology, an environmental nonprofit group, who has been an activist in San Francisco for three decades. “But on detailed data dealing with very complicated material, you really have to know what you’re looking for in order to distinguish between good data and junk data.”

Mr. Bloom also worries that cities could manipulate data to gloss over things like unemployment rates by neighborhood.

Governments are trying to make data openness a more open process itself by asking people to vote for data sets they want to be released. In New York, for example, people have requested data on school violence, public restroom locations and bicycle accidents.

Still, asking for the data is often not enough. Software developers in New York have been unsuccessful in getting data feeds of pedestrian and bicyclist injuries and fatalities from the Police Department, said Noel Hidalgo, who is director of technology innovation for the New York State Senate and has been working with developers on building city-data applications. He envisions applications that overlay accident information on city bike maps.

Paul J. Browne, a deputy commissioner of the New York City Police Department, said it releases information about individual accidents to journalists and others who request it, but would not provide software developers with a regularly updated feed. “We provide public information, not data flow for entrepreneurs,” he said.

There have been other scuffles over who has the right to data. Routesy, an iPhone application, uses data from San Francisco’s Municipal Transportation Agency to show train and bus schedules and locate stations on a map. It stopped working for a while because a private contractor working with the agency wanted to charge a licensing fee for the information. The agency now requires its contractor, NextBus, to make the data freely available.

There is evidence that governments’ attitude toward publicizing data is changing. Two years ago, when a Web design and research firm called Stamen Design started a Web site, Crimespotting, that mapped crime data for Oakland, Calif., the city cut off access to the data a week after the site went up.

Bob Glaze, the city’s chief technology officer, said the frequent data requests from the site were disrupting the city’s own crime site. The city eventually changed its mind. And in August, Stamen’s designers unveiled a San Francisco version of Crimespotting with Mayor Newsom at their side.

Some government leaders are making data disclosure an official policy. Mayor Newsom signed an executive order saying city data should be released, and the White House is about to publish a directive expected to give similar instructions to federal agencies.

San Francisco, New York and Washington have all organized contests to encourage software developers to create applications with their data. And the developers are using the data to build businesses. Stamen, for example, uses Crimespotting to show potential clients what it could create for them. Other firms are selling the iPhone apps they have built.

The cities, meanwhile, are to some degree using developers to provide citizens with a service so they do not have to.

“We are increasingly governing in a time when the demand for services exceeds our resources,” said Aneesh Chopra, chief technology officer of the United States. If the contests “spur dozens of innovative applications,” he said, “then we’ve essentially achieved a policy objective at virtually no cost.”

===

Stamen Design put together the San Francisco Crimespotting site using information from the city's police department.

DC Bikes, which shows bike paths in the Washington area, and Stumble Safely, which shows the safest way to get home from bars at night there, were both developed using government data.

Copyright 2009 The New York Times Company

TIME: The Four-Day Workweek Is Winning Fans

Monday, Sep. 07, 2009

The Four-Day Workweek Is Winning Fans

By Bryan Walsh

In an era when most of us seem to be working more hours than ever (provided we're still lucky enough to have jobs), 17,000 people in Utah have embarked on an unusual experiment. A year ago, the Beehive State became the first in the U.S. to mandate a four-day workweek for most state employees, closing offices on Fridays in an effort to reduce energy costs. The move is different from a furlough in that salaries were not cut; nor was the total amount of time employees work. They pack in 40 hours by starting earlier and staying later four days a week. But on that fifth (glorious) day, they don't have to commute, and their offices don't need to be heated, cooled or lit.

After 12 months, Utah's experiment has been deemed so successful that a new acronym could catch on: TGIT (thank God it's Thursday). The state found that its compressed workweek resulted in a 13% reduction in energy use and estimated that employees saved as much as $6 million in gasoline costs. Altogether, the initiative will cut the state's greenhouse-gas emissions by more than 12,000 metric tons a year. And perhaps not surprisingly, 82% of state workers say they want to keep the new schedule. "It's beneficial for the environment and beneficial for workers," says Lori Wadsworth, a professor at Brigham Young University who helped survey state employees. "People loved it." Those who didn't tended to have young children and difficulty finding extended day care. (See 10 ways your job will change.)

Managers from around the world have gotten in touch with Utah officials, and cities and towns including El Paso, Texas, and Melbourne Beach, Fla., are following the state's lead. Private industry is interested as well — General Motors has just instituted a workweek of four 10-hour days at several of its plants. "There is a sense that this is ready to take off," says R. Michael Fischl, an associate dean at the University of Connecticut's law school, which is organizing a symposium on four-day weeks.

The advantages of a so-called 4-10 schedule are clear: less commuting, lower utility bills. But there have been unexpected benefits as well, even for people who aren't state employees. By staying open for more hours most days of the week, Utah's government offices have become accessible to people who in the past had to miss work to get there in time. With the new 4-10 policy, lines at the department of motor vehicles actually got shorter. Plus, fears that working 10-hour days would lead to burnout turned out to be unfounded — Wadsworth says workers took fewer sick days and reported exercising more on Fridays. "This can really make a difference for work-life balance," says Jeff Herring, Utah's executive director for human resources.

Of course, in the age of the BlackBerry, fewer days in the office may not make much of a difference in terms of workload. But as energy prices start rising again, it makes sense to be flexible and find savings where we can. 10-4, 4-10.

Find this article at:
http://www.time.com/time/magazine/article/0,9171,1919162,00.html

Copyright © 2009 Time Inc.

FT: Superfreakonomics

Superfreakonomics

Review by Tim Harford

Published: October 17 2009 00:37 | Last updated: October 17 2009 00:37

Superfreakonomics: Global Cooling, Patriotic Prostitutes and Why Suicide Bombers Should Buy Life Insurance
By Steven D. Levitt and Stephen J. Dubner
Allen Lane £20, 288 pages
FT Bookshop price: £16

For fans of the multimillion-selling pop-economics book, Freakonomics, all that needs to be said is that the sequel’s title is an accurate description. This book is a lot like Freakonomics, but better.

The original, a runaway hit, had its genesis in Stephen Dubner’s masterful New York Times Magazine profile of “rogue economist” Steven Levitt. “Rogue” may be stretching it a bit, because Levitt is, in fact, a garlanded and hugely influential professor at the University of Chicago.

He has applied his statistical techniques, now much emulated, to unconventional topics such as the link between abortion laws and crime, or whether sumo wrestlers cheat (they do, he concludes). The 2005 book that resulted was wide-ranging, fascinating and above all, likeable – however, it showed signs of haste, and it was never clear whether it was supposed to be a book by Steven Levitt or about him.

SuperFreakonomics offers much the same range and amiability, but is more polished. The book’s chapters cover prostitution; data analysis in healthcare and counter-terrorism; altruism; innovation; and geo-engineering. The reader may not guess the central topic from the chapter titles or the opening pages, however, which betray a fondness for springing surprises and putting twists in the storytelling.

Detours are all part of the style; an afternoon reading SuperFreakonomics is like one of those thrilling and occasionally frustrating conversations where ideas tumble out so quickly that they keep interrupting each other. In short, the book’s organisation is deliberately on the freaky side, but if you simply resolve to read it from cover to cover you are guaranteed a good time.

My favourite chapter describes the research of John List, a colleague of Levitt’s, as he zaps some of the most famous results in behavioural economics. In the “dictator” game, well-known in economic circles, player A is given $10 by the experimenter and told they can keep it all. Alternatively they can give some to anonymous player B. Many players do, indeed, hand over money, a finding that troubles conventional economic theory.

List thinks many researchers have embraced this finding too easily, however. “What is puzzling”, he comments, “is that neither I nor any of my family of friends (or their families and friends) have ever received an anonymous envelope stuffed with cash”. The lab experiments, in which large numbers of students display a preference for sending cash to anonymous strangers, need to be questioned more closely. Yet List showed that with small modifications to the dictator game, experimental subjects could be persuaded not only to curb their generosity, but to confiscate cash from others.

There is much more here, and all is told with verve and care. Levitt and Dubner have a gift for explaining precisely how a researcher discovers something. Their epilogue, on Keith Chen’s attempts to introduce currency to a monkey society, is a model of how to tell a gripping story of scientific research without compromising on accuracy.

The most eye-catching chapters in the book are the first, on prostitution, and the last, on global warming. The chapter on prostitution flits from academic research into street prostitution, carried out by Levitt and sociologist Sudhir Venkatesh, to an engaging profile of a high-end escort and various digressions into the economics of gender and other topics.

One of these asides provides the book’s best moment: when the authors demonstrate that a prostitute gets more money through the use of a pimp than a homeowner gets through the use of a realtor, or estate agent. The financial impact of a pimp is greater than that of a realtor, “Or, for those who prefer their conclusions rendered mathematically, PIMPACT > RIMPACT.”

Those with a prurient curiosity (I am guilty), will find some of the descriptions of what prostitutes do all day rather coy. Those with an interest in the economic angle (guilty, again), will find some questions unanswered.

“The real puzzle isn’t why someone like Allie becomes a prostitute, but rather why more women don’t choose this career,” Levitt and Dubner write. Do tell, thinks the reader, but they don’t, even though literature on the puzzle does exist. The leading research on the question is written by two women, not two men, a fact that some people will find relevant.

The analysis of street prostitution is based on careful academic work. The account of high-end prostitution is merely a journalistic profile of a single successful and intelligent woman. But Levitt and Dubner seem to have decided that while data-driven discoveries are generally wonderful, you can have too much of a good thing.

The authors claim to prefer data to “individual anecdote”, but part of the secret of their success is that they like a good story more than anyone.

As for the final chapter on global warming, it is a striking discussion of geo-engineering, surveying various schemes for cooling down the planet rather than trying to prevent climate change by cutting carbon emissions. This is a strong story, but it is also one-sided, portraying the geo-engineers as brilliant iconoclasts, dismissing the objections to geo-engineering as the knee-jerk reaction of the unreflective, and failing to convey the views of a single credible geo-engineering sceptic. A well-deserved swipe at Al Gore does not really count.

According to this chapter, the only reason everyone is making so much fuss about carbon dioxide is that they’ve never heard of geo-engineering, or are the kind of stubborn Luddites who think technology never solved anything. I have some sympathy with that view but the section nevertheless needed more balance.

In the end, a book such as SuperFreakonomics stands or falls on its entertainment value. And on that count, there’s no doubt: it’s a page-turner.

More revealing, though, was that I’d folded over at least a dozen pages, resolving to go back, follow up the references, and find out more. This is a book with plenty of style; underneath the dazzle, there is substance too.

Tim Harford is the FT’s ‘Undercover Economist’

Copyright The Financial Times Limited 2009.

FT: Do not discount what you cannot measure

Do not discount what you cannot measure

By John Kay

Published: September 22 2009 20:56 | Last updated: September 22 2009 20:56

“When you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.” This observation scores high on the list of silly remarks made by clever men, to which its author, Lord Kelvin, made several sterling contributions. The knowledge possessed by Plato and Shakespeare, Austen and Darwin was neither meagre nor unsatisfactory.

Bogus quantification attempts to compress complex problems and analyses into single observations. The annual report of the United Nations Human Development Programme is an entirely admirable endeavour. The agency focuses on three broad dimensions of development – health, education and material standard of living – and reports on progress towards these goals among UN members.

Then the curse of Kelvin strikes. Health is measured easily enough. Take life expectancy at birth, subtract 25 years, then divide by 60. This score has a one-third weight in the final total. Educational development is measured as a combination of attainment – adult literacy – and opportunity – enrolment. Material standard of living is measured by gross domestic product per head at purchasing power parity. But differences in GDP are much larger than differences in life expectancy, so you prevent this measure from swamping the whole calculation by using the logarithm of GDP rather than the level. Finally, you give equal weight to each of the three components and come up with your ratings. Iceland, ironically, comes top, and Sierra Leone bottom.

If you must undertake an exercise of this kind, then what the UNDP does is sensible enough. But suppose I thought that more weight should be given to health and less to per capita income? Or that a measure of human development should include freedom of speech, or democracy, or religious tolerance, or religious observance? The problem is not just that reasonable people would have different views on these things, but that there are no objective criteria by which such disagreement could ever be conducted, far less resolved.

If Iceland did not come near the top of the ranking and Sierra Leone near the bottom, we would not change our assessments of Iceland or Sierra Leone: we would change the index. If we were to revise our view of Iceland, it would not be because we were told that its HDI score had fallen to 0.96: it would be because we were told it was a poorer country than we had previously thought. The index of human development is not telling us anything we have not already told the index of human development.

The French government has just published a report, supervised by a distinguished panel of economic experts, on the measurement of national income. President Nicolas Sarkozy is miffed by repeated criticism of French economic performance, since the quality of life in France is plainly very high. You can find evidence of this, for example, in the number of foreigners who visit the country every year. Much of the difference between France and the US in GDP per head is explained by the French penchant for longer lunches, longer holidays and earlier retirement.

It is not difficult to invent an index that would represent these facts. Such measures do not, however, add anything to the knowledge contained in the preceding paragraph. If anything, they reduce it. Just as the statement “Iceland has an HDI score of 0.99” tells us less, not more, than the sentence “Icelandic people are, by global standards, rich, healthy and well educated”.

In Peter Weir’s film Dead Poets Society, Robin Williams portrays a charismatic teacher obliged to teach from a text by J. Evans Pritchard. Mr Pritchard explains that “if a poem’s score for perfection is plotted along the horizontal of a graph, and its importance is plotted on the vertical, then calculating the total area of the poem yields the measure of its greatness”. Williams tells his pupils to tear these pages from the book, and goes on to inspire them with a genuine love of literature. We should approach bogus quantification in the same way.

johnkay@johnkay.com

More columns at www.ft.com/johnkay

Copyright The Financial Times Limited 2009.

Online Metrics Insider: The Lighter Side of Metrics Today

The Lighter Side Of Metrics Today  by Pat LaPointe , Tuesday, October 13, 2009 In the pressure of the current business environment, humor isn't lost. Here are a few ideas for metrics and measurement tools I've come across in my travels recently that you might like to hear about: Aleanment Index  - how you manage expectations for what will get done with fewer headcount than last year. Bland Awareness: A measure of how uninspiring your brand is. Brand Vasequity: A calculation of brand value taken after your marketing budget has been neutered. Customer Satisfiction: A metric that skews customer feedback in the most favorable terms for your next presentation to senior management. Dashbored: A comprehensive set of marketing performance metrics guaranteed to make your CEO's eyes glaze over. Dumbographics: Statistical data describing the consumer segment stupid enough to purchase your competitor's overpriced, poor-quality product or service. Mantric - the one metric that looks so good you hear about it over and over and over. Misery Mix Model - a statistical methodology for ascertaining exactly why your plans failed to achieve objectives. ROE (Return on Everything): What your CEO is demanding. Six Stigma: A set of visible marks on your forehead signify a failed attempt at improving the quality of the creative-development process. If you've got others, I'd love to add them to our collection - post below. This commentary is insightful. I recommend it to others. Post your response to the public Metrics Insider blog. See what others are saying on the Metrics Insider blog. Pat LaPointe is Managing Partner at MarketingNPV -- specialty consultants on marketing measurement and metrics, and publishers of MarketingNPV Journal, available online free athttp://www.MarketingNPV.com

WSJ: Can a Ballclub's Record Justify Its Beer Prices?

THE COUNT

SEPTEMBER 11, 2009, 10:01 A.M. ET

Can a Ballclub's Record Justify Its Beer Prices?

In an ideal world, beer prices at the ballpark would be based solely on the quality of the team. Only the very best ballclubs would jack up the prices, while the mediocre teams would offer bargains ... and the Washington Nationals would give beer away for free. Regrettably, we don't live in an ideal world.

 According to data collected by Team Marketing Report for the 2009 season, beer prices vary dramatically among big-league teams. A 21-ounce beer costs $4.75 in Pittsburgh, but you'll shell out $8.75 for a 20-ounce brew at San Francisco's AT&T Park. This led us to wonder: Does quality have anything to do with beer prices?

Surprisingly, it does. A team with a .600 winning percentage charges, on average, about $1.30 more for a 16-ounce beer than does a team with a .400 percentage.

So which ballclub offers the best value based on its team's winning percentage? At Citizens Bank Park in Philadelphia, you'll pay a modest $6.75 for a 21-ounce beer while watching one of the best teams in the National League. And for all their losing seasons, at least the Pirates allow their fans to drown their sorrows at a decent price. Even after factoring in their dismal record this season, that 21-ounce beer they sell at PNC Park is still 32% less expensive than it should be by this measure.

On the other end of the spectrum there's Nationals Park where, in exchange for watching baseball's worst team, fans get to spend $7.50 for a 20-ounce beer. Of course, nothing compares to Boston's Fenway Park. There, you'll pay $7.25 for just 12 ounces—a rate that is, ounce for ounce and win for win, the worst beer value in baseball.

—Justin Merry

Copyright 2009 Dow Jones & Company, Inc. All Rights Reserved

FT: How economists are tackling sports injuries

The high cost of injuries – not just for players, but for the success of teams – has inspired a new breed of statisticians to number-crunch the best solutions

===

How economists are tackling sports injuries

By Robert Hudson

Published: September 18 2009 13:25 | Last updated: September 18 2009 13:25

Tom Brady: Make-or-break quarterback of the New England Patriots

Tom Brady is like a cartoon superhero. The tall, comically handsome quarterback for the New England Patriots is one of the richest and most glamorous sports stars on the planet (he’s married to a Brazilian supermodel, for starters). And he throws a football like Superman throws a punch: powerfully, effortlessly, and at the very moment the enemy thinks he’s been disabled by Kryptonite. On ­Monday night in Foxborough, Massachusetts, basking in the bright lights and raucous adoration of 70,000 fans packing Gillette Stadium, Brady faced almost certain defeat and refused to accept it. Twelve points behind with two minutes to play, he led his team to a fabulous victory. It’s what superheroes do. But a generation earlier, it would have been a different story. The same player would have spent the evening pottering round the house, doing what he could to avoid the game on TV – a reluctant ex-jock at the age of 32.

Ten minutes into the 2008 NFL season, on September 7 last year, the football was snapped into play and two lines of giants collided in exquisitely choreographed violence. Kansas City were playing the Patriots, ­perennial favourites for the Super Bowl. The ball was in Brady’s hands. Kansas City’s Bernard Pollard, nicknamed “The Bonecrusher” at university, was trapped on the ground under one of Brady’s huge guardians. Just as the quarterback cocked his arm to throw, Pollard squirmed free and leapt headlong. Helmet hit knee, and that was the end of the Patriots’ season.

Could Brady really be so vital? Yes. He had led the Patriots to four Super Bowls and won three of them. In 2007, he was named the league’s Most Valuable Player. Bill ­Barnwell, managing editor at Football Outsiders, which produces an eye-watering almanac of American football statistics, says that the observed impact on a team of an injury to a superstar is about 6.75 times that of the observed impact of an injury to a reserve player. And Brady isn’t just a superstar, he’s a superstar quarter­back. He leads the offence, where injuries have a statistically higher impact than they do on defensive units. ­Barnwell explains that the injury rate of a team’s offensive players year on year has a 0.42 correlation with team success (where 1 is a perfect correlation and 0 is no correlation). With defensive players, that drops to a 0.29 ­correlation. And the quarterback is crucial – his individual correlation is 0.3, twice that of any other position. An NFL offence is designed intricately around the quarterback’s strengths and weaknesses. Change the central cog, and the whole machine jams.

Medically, Brady is not so interesting. When Pollard’s helmet crashed into his left knee and he fell twisting to the ground, Brady tore his medial collateral and anterior cruciate ligaments. These are cords of tissue on the inside of the knee and running through the centre of the knee, respectively. But 30 years ago, before modern surgical techniques were available, such injuries ended professional sportsmen and women’s careers.

With teams matched as evenly as they are in the NFL, the loss of Brady had pundits guessing that the Patriots, who had won every regular season game in 2007, wouldn’t make the ­playoffs. They didn’t.

Brady’s story shows vividly how injury can affect a team, and in particular injury to a star player, whose performance is doubly critical. And since sport is increasingly obsessed by Freakonomics-style analysis, where economists and statisticians (and mere stats nuts) hunt for inefficiencies in the system, his case raises questions, too. Most Valuable Player isn’t some airy-fairy construct in the big-money world of US sport. The NFL uses a salary cap to ensure more balanced competition between the teams: each can spend $116m on player salaries this season, which they divide between 53 active players. New England spends $14m of this – more than 12 per cent – on Brady.

Sport is not a conventional business (Barnwell says that getting to the playoffs can cost a team more than they earn from it) but an NFL team’s raison d’être is to win, and one of the biggest constraints they face is the salary cap. In this context, the $14m spent on Brady represents a massive opportunity cost – a cost made painfully clear by his season on the sidelines as he convalesced. With so much at stake in the big money world of professional sport, the questions become more urgent. Do teams and their owners understand the economics of injury? How are ­decisions surrounding sports injury made? And is that changing?

. . .

Simon Kemp is head of sports medicine at the Rugby Football Union in England, an organisation striving to incorporate stats-based economic insights into its approach to injury. He says that the way professionals are treated when they are injured differs from the way ordinary people are treated. “We’re dealing with people whose priorities are very different from the man on the street. While I might advise an amateur to try conservative treatment, which will still let him lead a pretty active life, professionals can’t get by with that. Besides, they’re used to injury.” Kemp says that between 2002 and 2004, professional rugby players in England spent an average of 69 days a year injured.

How long players – and especially star players – spend recovering, therefore, becomes a matter of vital concern. Team doctors, for better or worse, are focused on helping players get back on the field as quickly as possible: in professional sport, the difference between 10 days and 12 days can be the ball game. When Jason Robinson, the talismanic England winger and full back, pulled his hamstring at the 2007 World Cup, the ­medical team threw everything at the problem. “We’re on the cutting edge and we have a very small window,” says Kemp. “We’ll try new things, so long as the science suggests they’ll help, and we think they’re not dangerous.” Kemp had found some success with homeopathic injections of Traumeel – a combination of various botanical bits and pieces. He told ­Robinson that the potential benefits exceeded the risks, and Robinson agreed to be injected. “I’m not saying it was the Traumeel,” Kemp says. “Jason was getting great rehab and placebos can be powerful. But we had him back on the pitch for the quarter-final.”

In 2004 Richard Hill, one of England’s World Cup-winning rugby heroes of the year before, faced a similar injury to Brady’s. He says now: “I just wasn’t ready to stop.” Hill was told about risks of infection and rates of success of surgery, which for this kind of procedure are 85 per cent and upwards. He had the operation. When he needed another one a year later, he was told it would probably mean he’d need a knee replacement in his forties. He had that operation, too. As it happened, the replacement took place only a year after Hill stopped playing. “It was worth it,” he says. “The last two games I played [the European Cup quarter- and semi-finals last year] are two of my best rugby memories.”

It’s clear that most serious sportsmen will risk long-term health ­difficulties in order to play.

. . .

Carson Palmer: Cincinnati Bengals star who returned to form after surgery

But what if Tom Brady had never been injured? The treatment he received and his successful recovery tell only part of the story. In fact, the sports science decisions best informed by statistics and micro­economics increasingly focus on preventing injuries rather than curing them. The process begins with information-gathering. For seven years, the RFU has been logging every injury in the professional game in an effort to gather enough data to identify trends reliably. The ­England and Wales Cricket Board and the English Institute of Sport, which looks after Olympians, now have similar rolling audits, although they are at an earlier stage of development.

The numbers are already being crunched – and proving useful in planning which treatments might best serve not just the player, but the team. A recent study of hamstring injuries found that every new hamstring injury costs the team an average of 14 playing days; an average recurrence costs 25 days. Furthermore, almost all the recurrences took place during matches in the first month after return, and after an hour of play. It quickly became clear that players who had sustained hamstring injuries should be replaced after an hour during their first few games back. Moreover, the two clearest risk factors for hamstring injury were age and a previous injury, and players who performed specific strengthening exercises reduced the incidence, severity and ­recurrence of hamstring injuries.

It’s work like this that has helped the RFU halve injury recurrence rates between 2003 and 2007. Ian ­Beasley, senior doctor at the Football Association, looks at the data sets with envy. “I can’t really understand why we don’t run a proper audit,” he says. One difficulty is getting clubs to agree: each team, after all, needs to devote resources to collecting the information and entering the stats, at the very least – let alone finding people to analyse the data. “Hopefully,” says ­Beasley, “we can get someone attached to do that.”

It is eyebrow-raising that rugby – and cricket – can afford audits but the FA cannot. According to Stefan Szymanski, an economist at Cass Business School who recently co-authored, with FT columnist Simon Kuper, Why England Lose & Other Curious Football ­Phenomena Explained, “You have to look at where the money is and why it’s being spent. With the England and Wales Cricket Board and English Institute of Sport, the money is being spent by people whose target is national success.” The ECB pays each county £20,000 a year to improve their medical practices. “The Rugby Football Union is more balanced between the clubs and England, but ­England money is still very important, which helps England set the agenda. Also, rugby is a dangerous game, so there’s a responsibility to the players. Therefore, these organisations can run sport-wide programmes.”

Some information is published – more by the RFU than the others, echoing its wider concerns and more balanced financial structure – but these audits primarily exist to help English sport maintain a competitive edge over its rivals.

The financial situation in football (and the NFL in the US) is radically different. What might be feasible for Chelsea’s sophisticated ­medical team is not feasible for a lower league club with one physio and a part-time doctor. And even within the Premier League, Chelsea have vastly more money than Hull. The FA can’t demand compliance because, points out ­Szymanski, “[the] England [team] is an economic sideshow. The clubs have the muscle. It might be great for Leyton Orient to pool medical information – the losses in competitive edge would be more than ­outweighed by efficiency savings – but the same isn’t true for Chelsea.”

AC Milan’s famously secretive players’ lab, which routinely coaxes great players into their mid- and late-30s, gives the club a big advantage. Why would it share? It is safe to assume the same holds true in the NFL. For all their money, the 32 clubs don’t have access to the sheer quantity of data that less wealthy sports are starting to gather.

. . .

The defining text for the new breed of sports statisticians is ­Moneyball, Michael Lewis’s tale of how a new understanding of scoring efficiency helped an underfunded US baseball team punch above its financial weight. In a forthcoming piece for the British Journal of Sports Medicine, John Orchard speculates that football medics might be facing their Moneyball moment.

Orchard says that differences in skill between the top clubs’ players are so slight that he’s surprised teams don’t seem to appreciate “what seems obvious from the outside: injury outcomes will be a key factor in determining who wins the Premiership each year”. It has become almost ­hackneyed to say that squad depth is crucial and that Liverpool don’t score if Gerrard and Torres aren’t playing, and yet still there is no published study exhaustively trying to link injury with success, and seeing if medical treatment makes a difference.

According to the RFU’s injury report for 2002 to 2004, the teams finishing in the top four places of the league lost an average of about 600 player days per season. The next four lost something like 800. The bottom four lost closer to 1,000. The situation is hideously complex – better managed, better funded teams, which you would expect to do well, tend to have better medical programmes as well – but it is hard to imagine that causation doesn’t run both ways.

Back in the US, Bill Barnwell and Football ­Outsiders crunch the numbers to show an even more graphic link: of the 10 healthiest teams in 2008’s NFL, seven made the playoffs. A huge 26 per cent of a team’s win rate from year to year can be chalked up to a change in injury rate. In spite of this, says Barnwell, “teams rarely credit the absence of injury for their success”.

Someone has to do this kind of analysis on football. Orchard argues that a team spending £100m on wages with an injury rate of 22 per cent will probably beat a team paying £25m with a rate of 12 per cent, but statisticians can control for that. If better medical teams really do keep players on the pitch, then management (and fans) might finally feel that £1m invested in better medical care has more marginal utility than £4m spent on another spare midfielder, “just in case”.

If injury incidences and outcomes were blind luck, then there would be nothing to do except focus on treatment. But they clearly aren’t. The RFU’s success with hamstrings is just one example. Simon Kemp knows rugby will always be dangerous, but he hopes that better conditioning might even help players come through some of the collisions which tear anterior ­cruciate ligaments.

“How?” I ask.

“We don’t know yet,” he replies.

Knees will never be impervious to helmets, but audits can protect future Tom Bradys in other ways. The RFU data have demonstrated so clearly that illegal tackles disproportionately cause injuries that they have helped persuade referees to apply the law more strictly. The NFL is tightening up on knee-level tackles post-Brady – but perhaps an ongoing data collection programme would have caught the problem earlier.

If so, then going by Brady’s salary – the crudest measure of his worth – it would have been at a fraction of the cost of his absence, let alone those of the other quarterbacks whose recent seasons might have been saved. As Tom Brady runs blinking into the floodlights this season, he won’t be thinking about value-for-money risk and resource management; it’s not his job. But somebody should be.

Robert Hudson is author of ‘The Kilburn Social Club’, published by Jonathan Cape

....................................................................

Rugby Union’s most common injuries

The data illustrated here, collected by the RFU (and featuring Gloucester and England’s Olly Morgan), show the highest-risk training injuries, measured in days missed, as suffered by rugby backs. Injuries are also broken down by the type of problem most associated with a particular match event, eg the commonest injury for a forward to suffer as a result of a scrum is calf strain, while a back is most susceptible to a thigh haemotoma as a result of a collision

....................................................................

He stats, he scores

Statistics have been popular among sports fans and coaches long before Freakonomics-style analysis took off, writes Alex Cardno. Walks, hits, runs … every player comes with a compendium of facts and figures. About eight years ago, however, Nate Silver, a Detroit Tigers baseball fan, began using these figures differently. Rather than judge players on batting average alone, for example, Silver developed a series of complex algorithms to predict the outcome of a player’s career and the success of various teams. His predictions, including that the usually dismal Tampa Bay Rays would win more than 80 games in the 2008 season, proved correct time and again. This consistency soon gained him a large following of baseball fans.

In 2003, Silver sold his main algorithm to Baseball Prospectus, publisher of a website devoted to detailed analysis and advanced statistics on the game. Silver received a stake in the site and stayed on as a writer and consultant until late 2007.

Then in March last year, he launched a statistics-driven politics blog, FiveThirtyEight.com – named after the number of votes in the US electoral college. Silver built a database of political polls and weighted their results against historical fact. He noticed patterns emerging and applied the techniques he had used with baseball statistics to predict elections.

The website’s profile rose dramatically as he accurately predicted Democratic primary elections in state after state. When it came to the presidential poll, he predicted the vote of 49 out of 50 states, and was the first to declare Obama’s victory, beating professional pollsters.

Silver, now 31, continues to publish FiveThirtyEight, applying statistical analysis to major policy issues. “There are some things [where] you can’t be quite as robust as you can be in forecasting an election, but, for example, there was a climate change bill that was passed, barely, by the House [of Representatives], where we can look at the characteristics of which House members voted for it and which didn’t, and try to predict what is going to happen in the Senate.”

Silver is a regular on US news channels and made Time magazine’s 100 Most Influential People List in April. Last November he signed a book deal with Penguin (owned by Pearson, the FT’s parent company). His first book will focus on his predicting methods, picking up where Freakonomics left off. “It should be a book that’s both serious and fairly rigorous but also fun,” he says. It’s due for publication late next year, by which time Silver’s name may be known round the world: as this magazine went to press he wouldn’t give details, but he did reveal that his next website venture would focus on the 2010 FIFA World Cup.

To read Alex Cardno’s interview with Nate Silver, go to www.ft.com/natesilver

Copyright The Financial Times Limited 2009.

NY Times: Wall Street's Math Wizards Forgot a Few Variables

September 13, 2009

Unboxed

Wall Street’s Math Wizards Forgot a Few Variables

By STEVE LOHR

IN the aftermath of the great meltdown of 2008, Wall Street’s quants have been cast as the financial engineers of profit-driven innovation run amok. They, after all, invented the exotic securities that proved so troublesome.

But the real failure, according to finance experts and economists, was in the quants’ mathematical models of risk that suggested the arcane stuff was safe.

The risk models proved myopic, they say, because they were too simple-minded. They focused mainly on figures like the expected returns and the default risk of financial instruments. What they didn’t sufficiently take into account was human behavior, specifically the potential for widespread panic. When lots of investors got too scared to buy or sell, markets seized up and the models failed.

That failure suggests new frontiers for financial engineering and risk management, including trying to model the mechanics of panic and the patterns of human behavior.

“What wasn’t recognized was the importance of a different species of risk — liquidity risk,” said Stephen Figlewski, a professor of finance at the Leonard N. Stern School of Business at New York University. “When trust in counterparties is lost, and markets freeze up so there are no prices,” he said, it “really showed how different the real world was from our models.”

In the future, experts say, models need to be opened up to accommodate more variables and more dimensions of uncertainty.

The drive to measure, model and perhaps even predict waves of group behavior is an emerging field of research that can be applied in fields well beyond finance.

Much of the early work has been done tracking online behavior. The Web provides researchers with vast data sets for tracking the spread of all manner of things — news stories, ideas, videos, music, slang and popular fads — through social networks. That research has potential applications in politics, public health, online advertising and Internet commerce. And it is being done by academics and researchers at Google, Microsoft, Yahoo and Facebook.

Financial markets, like online communities, are social networks. Researchers are looking at whether the mechanisms and models being developed to explore collective behavior on the Web can be applied to financial markets. A team of six economists, finance experts and computer scientists at Cornell was recently awarded a grant from the National Science Foundation to pursue that goal.

“The hope is to take this understanding of contagion and use it as a perspective on how rapid changes of behavior can spread through complex networks at work in financial markets,” explained Jon M. Kleinberg, a computer scientist and social network researcher at Cornell.

At the Massachusetts Institute of Technology, Andrew W. Lo, director of the Laboratory for Financial Engineering, is taking a different approach to incorporating human behavior into finance. His research focuses on applying insights from disciplines, including evolutionary biology and cognitive neuroscience, to create a new perspective on how financial markets work, which Mr. Lo calls “the adaptive-markets hypothesis.” It is a departure from the “efficient-market” theory, which asserts that financial markets always get asset prices right given the available information and that people always behave rationally.

Efficient-market theory, of course, has dominated finance and econometric modeling for decades, though it is being sharply questioned in the wake of the financial crisis. “It is not that efficient market theory is wrong, but it’s a very incomplete model,” Mr. Lo said.

Mr. Lo is confident that his adaptive-markets approach can help model and quantify liquidity crises in a way traditional models, with their narrow focus on expected returns and volatility, cannot. “We’re going to see three-dimensional financial modeling and eventually N-dimensional modeling,” he said.

J. Doyne Farmer, a former physicist at Los Alamos National Laboratory and a founder of a quantitative trading firm, finds the behavioral research intriguing but awfully ambitious, especially to build into usable models. Instead, Mr. Farmer, a professor at the interdisciplinary Sante Fe Institute, is doing research on models of markets, institutions and their complex interactions, applying a hybrid discipline called econophysics.

To explain, Mr. Farmer points to the huge buildup of the credit-default-swap market, to a peak of $60 trillion. And in 2006, the average leverage on mortgage securities increased to 16 to 1 (it is now 1.5 to 1). Put the two together, he said, and you have a serious problem.

“You don’t need a model of human psychology to see that there was a danger of impending disaster,” Mr. Farmer observed. “But economists have failed to make models that accurately model such phenomena and adequately address their couplings.”

When a bridge over a river collapses, the engineers who built the bridge have to take responsibility. But typically, critics call for improvement and smarter, better-trained engineers — not fewer of them. The same pattern seems to apply to financial engineers. At M.I.T., the Sloan School of Management is starting a one-year master’s in finance this fall because the field has become too complex to be adequately covered as part of a traditional M.B.A. program, and because of student demand. The new finance program, Mr. Lo noted, had 179 applicants for 25 places.

In the aftermath of the economic crisis, financial engineers, experts say, will probably shift more to risk management and econometric analysis and concentrate less on devising exotic new instruments. Still, the recent efforts by investment banks to create a trading market for “life settlements,” life insurance policies that the ill or elderly sell for cash, suggest that inventive sales people are browsing for new asset classes to securitize, bundle and trade.

“Good or bad, moral or immoral, people are going to make markets and trade via computers, and this is a natural area of financial engineers,” says Emanuel Derman, a professor at Columbia University and a former Wall Street quant.

Copyright 2009 The New York Times Company

FT: The man who invented exercise

The man who invented exercise

By Simon Kuper

Published: September 11 2009 16:24 | Last updated: September 12 2009 03:43

In the early years after the second world war, health researchers in Britain noticed a curious epidemic: people had begun dying of heart attacks in unprecedented numbers. Nobody knew why, and so a scientist in London named Jerry Morris set up a vast study to examine the heart-attack rates in people of different occupations – schoolteachers, postmen, transport workers and more.

Morris is today a neat, bird-like man who uses a cane to get around. He turns 100 next May but still regularly makes his way to his poky little office at the London School of Hygiene and Tropical Medicine. It is here that he describes the day he saw the data that changed medicine. “The very first results we got were from the London busmen,” says Morris, in ­Glaswegian tones undimmed by seven-odd decades in London. “And there was a striking difference in the heart-attack rate. The drivers of these double-decker buses had substantially more, age for age, than the conductors.”

The data were so telling because drivers and conductors were men of much the same social class. There was only one obvious difference between them. “The drivers were prototypically sedentary,” explains Morris, “and the conductors were unavoidably active. We spent many hours sitting on the buses watching the number of stairs they climbed.” The conductors ascended and descended 500 to 750 steps per working day. And they were half as likely as the drivers to drop dead of a sudden heart attack.

Morris found that bus conductors had fewer heart attacks than sedentary drivers

Today, almost everyone understands that physical exercise can help prevent heart disease, as well as cancer, diabetes, depression and much else besides. But on that day in 1949 when Morris looked at the bus data, he was the first person to see the link. He had inadvertently – “mainly luck!” – stumbled on a great truth about health: exercise helps you live longer.

That’s a truth even more relevant in 2009 than it was in 1949, because so few of us now perform vigorous physical tasks as part of daily life. If present generations are going to live longer, we must exercise. But if that depends on us getting on rowing machines in our spare time, it won’t happen.

Morris is not simply a man with a mission – he is a man with several missions. But since reading those numbers about busmen, he has wanted to help everyone exercise more. Then we might live lives something like his.

Morris grew up in the lost world of the Southside of ­Glasgow “in a two-bedroom-and-kitchen house, father, mother and three boys”, he recalls. “We were next door to a slum street. I still remember the screaming women on Friday night and Saturday night, when they were beaten by their drunk husbands. As you can imagine, I have a contact with social inequalities and with poverty that most of my colleagues know from the literature.”

He began exercising in early childhood. “My father used to take me on a four-mile walk from Glasgow once a week, when I was a schoolboy. We used to aim to do the four miles in an hour. If we did that OK, I got an ice-cream. If we did it in even a minute less, I got a choc-ice.

“Many years later, when I became interested in these things, I challenged him, and he had no idea where he got this four miles an hour from. I was eager for a quote, you see. And he couldn’t produce it. Except, ‘It’s always been a habit.’”

Morris warbles a line from a song: “‘What comes naturally…’ This is from one of my favourite musicals. ‘Na-tu-rally.’ It’s probably from Guys and Dolls.”

In Morris’s own words, he always was a “do-gooder”. He once went to his local Labour MP to volunteer for campaign work, but was rejected on the grounds of being 12 years old. He joined Labour aged 16 in 1926, and remains a due-paying member, despite breaking with the party over the Iraq war. “I remember writing to Blair: ‘How many of your cabinet have even seen the Blitz?’ None, is the answer. I was in London during the Blitz. Since 2003, I have voted Green.”

But he remains obsessed with the old Labour issue of inequality in health. From the neat paper trays that cover his office, he ferrets out one of his recent reports: “Minimum income for healthy living: Older people.” Morris’s office features none of the awards or pictures of handshakes that you see in many old men’s playrooms. Other than some obit­uaries of dear colleagues on his noticeboard, this is a working ­person’s office. He has even organised our lunch himself: a cheese-and-tomato sandwich, a slice of cake and an orange juice await me.

When Morris went into medicine, he took a novel approach: he wanted to know why different kinds of people got different diseases. Epidemiology, as this science is now called, barely existed then. In the 1930s, he became obsessed with juvenile rheumatism. His boss, a fashionable Harley Street doctor who also worked at Eton College, remarked that he never saw a case at Eton. Morris dryly calls this “a good clinical observation”. Only the poor got juvenile rheumatism.

He spent most of the war as a military doctor in India. “We ran a first-class hospital in the swamps of Assam in the middle of the summer, where you got boiled alive the moment you woke up.” Demobbed afterwards in London, he landed in a golden age of British medical research. During the war, researchers in Oxford had helped develop penicillin. After it, Morris’s political hero, Nye Bevan, created the National Health Service, the world’s first completely free healthcare system. And what epidemiology there was in the world was led by Britain. In 1950, another London health researcher, Richard Doll, published a study showing that smoking – and not ­Tarmac or car fumes, as many had suspected – was causing the new epidemic of lung cancer.

Only occasionally in conversation do you get a hint that Morris is a man from a bygone age. Usually it’s when he uses extinct language, such as “refectory” for restaurant, or “pumping ship” for going to the toilet. Otherwise, he keeps up so well that he could probably win current-affairs quizzes. Yet he says: “I feel like a displaced person, in many ways. On the one hand, I’m a 2009 person. On the other hand I’m a 1940s person.”

. . .

“The 1940s was the generation that said ‘Yes we can.’ You need a national health service? You go out and do it. Eating fruit and vegetables – in the 1940s, they would have got the best brains in nutrition, in retail, in agriculture, and said: ‘Go away to the Hebrides and don’t come back until you’ve got us a national programme that’s easy for everybody every day.’”

Morris’s own mission of the late 1940s was heart disease. He spent “interminable hours” reading the wonderful postmortem folios of the London Hospital in the East End for 1907 to 1949. But he still couldn’t understand why heart attacks were increasing.

“We were in the fortunate situation,” he says, “that very ­little research had been done on it. It might be hard for you to imagine a time when heart attack wasn’t a major preoccupation of everybody.” Today, heart disease is the most common cause of death in western countries.

“The only hunch I had was that this might be related to occupation. It was commoner in men than women, it was commoner as middle age advanced, and there were some hints in the national statistics of mortality that it might be connected in some way to occupation.” The busmen’s data were fascinating, and the sample size was thousands of men. But Morris didn’t treat it as proof of anything. In what he has called “one of the tensest moments of my professional life”, he had to wait for data to arrive for other occupations. Finally, he got the figures for postal workers. “It was strikingly similar!” he says. The postmen who delivered the mail by bike or on foot had fewer heart attacks than sedentary men who served behind counters or as telephonists and clerks. It was true: exercise prevented heart disease.

And yet Morris sat on his data for years. If there were flaws in his theory, he was determined to find them before anyone else could. “We set about destroying this observation,” he says. “We brought in outside people with no blood in their veins, no interest, to destroy it.” But they couldn’t. His paper (“Coronary heart-disease and physical activity of work”) finally appeared in The Lancet in 1953. His hypothesis, as he still called it, was greeted with general disbelief. What could exercise possibly have to do with heart attacks? True, there had always been a vague belief that exercise was good for the soul. Mens sana in corpore sano (“a healthy mind in a healthy body”), the Roman poet Juvenal had written nearly two millennia before Morris, possibly with satirical intent, and the Victorians fetishised team spirit and muddy playing fields. But before Morris, nobody knew that exercise stopped people dying.

A 1950s office worker smoking at work

Even he had no idea how exactly the mechanism worked. Only after his paper appeared did the physiologist Henry Taylor sit him down for a solid day in a Washington hotel room and, in Morris’s words, “schoolboy-taught me the physiology of exercise”. Morris thinks the essential story is simple: “Exercise normalises the workings of the body.” Humans were meant to keep active.

With hindsight, his London bus drivers inhabited one of the first societies on earth where exercise was ceasing to be part of daily life. Technology was letting people grow slothful. Even in the 1950s, Morris foresaw that when poor countries developed, they would have the same problems. He remembers warning then: “Their time will come to develop these diseases, and not to make the mistakes that we made, eg a lack of exercise, eg smoking, eg our lousy diets. Of course, nobody paid any attention.” Today, India has the world’s worst diabetes epidemic, and in parts of rural Kenya, the prevalence of angina has reached levels found in inner-city Glasgow.

. . .

Clearly, if modern humans were going to exercise, it would have to be in their spare time. But would they? After his initial studies of occupations and heart attacks, Morris embarked on a large-scale study of British civil servants, to find out whether they did.

This was in the days before computers. Morris remembers: “I think of a room in this school, with the floor consisting of piles of documents. Men of this age, men of that age, men doing this kind of exercise or that. Going through all of these documents to extract the cyclists, then going through all the cyclists to extract those who cycle to work. Three very respectable ladies would spend days and days, and another lady would check they were not cutting corners. Changing a hypothesis now, in the computer age, is a matter of a twiddle on the knobs. Changing a hypothesis when we were doing important work was a major enterprise.”

Because the civil servants in his study were middle-class British males, 91 per cent were gardeners. “It’s what keeps us sane,” they repeatedly told Morris’s team. Morris had thought gardening would protect them from heart disease. It turned out not to. Only vigorous exercise, such as swimming or playing football, was enough.

These findings were made in an era when many British adults got no exercise beyond lighting cigarettes. Morris, however, applied all the new epidemiological findings to himself. Long a “stress smoker”, he read Doll’s studies and quit cigarettes even before Iain Macleod, the health minister, held the famous press conference in which he confirmed Doll’s findings while chain-smoking throughout. Morris also began to jog. “I was the first person to run on Hampstead Heath, in the 1960s. Every Sunday morning, if the weather was at all possible, I took off my coat, and my little boy carried my coat, I took off my jacket and my little girl carried my jacket, and I ran for 20 minutes. People thought I was bananas.”

Morris has been a habitual exerciser for many decades. “It’s the only way in which I feel entitled, as an old buffer, to give advice to people. I’m constantly being asked: ‘Your long life, what would you advise?’ and so forth. To start telling other people what to do – I’m very reluctant. Except on exercise, where to a large extent I feel it’s what I’ve done myself that’s contributed to longevity. For decades I had a pulse-rate of 52.” Even today, Morris is often to be found shuffling up and down stairs at the London School.

Since he discovered exercise, medical researchers have established that it is good not just for the heart but for almost everything. “Exercise is a universal for health,” Morris likes to say. By now, most people have got the message. Some of them even exercise. Did Morris improve the world by encouraging the habit? “That’s a very generous qualification.”

Surely he got people to exercise more?

“I suppose so. And failed in many respects with my own friends.”

Morris: ’I’m a work addict. It’s an obsession, a compulsion. Now, that’s a happy way of getting into old age’

Morris is not a man who likes to waste time celebrating himself. When friends planned a great academic party for his 90th, he was very anxious that it should be a proper scientific conference, and not just a bunch of people in smart clothes standing up and saying what a good chap he was. But when he discusses the limited take-up of exercise he is being frank, not humble. It frustrates him hugely. The key issue is not individual will, he insists. “It’s got to be a joint effort between the government and yourself.”

He uses his own example: until his mid-90s, he was a habitual swimmer (he stopped in part because he was embarrassed by people rushing over to help him out of the pool). And, “well, swimming means there must be pools”. Governments need to build pools and bike paths, and pedestrianise city centres. In 1994, Morris wrote a paper arguing that “physical activity could be today’s ‘best buy’ in public health for the west.” Hardly anybody listened. Now western countries spend growing fortunes curing their obese and inactive citizens. Already 10 per cent of the NHS’s budget goes towards diabetes.

“For the first time in history,” says Morris, “the mass of the population has deliberately got to take exercise. It’s a new phenomenon, which is not appreciated.” For decades he has tried to persuade governments to make exercise easier. He was involved in the pioneering English National Fitness Survey of 1990, which found that half of women aged 55 to 64 could not comfortably walk a mile. These people were in effect disabled. The government ignored the report. Since then, British exercise levels haven’t changed much. His voice becomes high-pitched with outrage: “Just imagine, what historians in the future are going to say about the way we’ve allowed this epidemic of childhood obesity. ‘Disgrace’ is a sort of mild word.”

He’s not about to stop pushing governments now. “Unfortunately,” he tries to look sorrowful, “I’m a work addict. It’s an obsession, a compulsion. I don’t think it’s a disease. Now, that’s a happy way of getting into old age. It’s better than an alcohol addict or – are there womanising addicts in old age? They must have a tough time.”

He has two big ambitions for government: “On the one hand, for the first time, to have a serious programme on health inequalities. Not a large programme, or a complicated programme or an ambitious programme, I mean the word ‘serious’.” Second, he wants government to help the new, giant cohort of old people to exercise.

To Morris, none of this is utopian. All you need to achieve it is the spirit of the 1940s. He’s not sure any politician has it now, though – perhaps not even his new hero, Barack Obama. “This phrase, ‘Yes we can’, comes up much less nowadays, doesn’t it?” he grumbles.

He ushers me out. A car – his family won’t allow him to use public transport any more – is arriving to take him home. There is work to be done, experts to call, e-mails to send, an exercise bike to ride, books to order, pupils to mentor (he micromanaged my eating of my sandwich), governments to push, novels to read.

In a short memoir published this year, Morris quotes Philip Roth’s Everyman: “Old age is not a battle but a massacre.” The line was widely received as a beautiful restatement of an ancient truth. But Morris writes: “Is he not plain wrong?”

Simon Kuper is an FT writer based in Paris

Copyright The Financial Times Limited 2009.

Online Metrics Insider: Visualising Data

Visualizing Data

Jodi McDermott, Sep 04, 2009 12:45 PM

When an engineering colleague of mine (95% of my immediate surroundings) emailed me a link this past week, I thought for sure it was for a YouTube video.  This link was different though, in that it introduced me to Anscombe's quartet, a set of four datasets that share the exact same statistical properties.  Each one of these datasets has the same mean, variance and mean of each y variable, variance and mean of each x variable and the exact same correlation between the x and y variables.   A little geeky, eh? What's my point, you might ask?  When each one of these datasets is plotted out visually, they have completely different appearances (just click on the link above and you'll see what I mean).  There are outliers where one would not expect to see them -- identifying both opportunities and risks in your data depending on what you are analyzing.  However, one would never see the variance in data patterns if it was not plotted in a chart or graph (or analyzed data point by data point). Looking at your data is just as important as reading your data.  Not all of us can see the obvious by just looking at numbers, even if we don't consider ourselves "visual" people.  I've learned this quite extensively in my current job when presenting data findings to a combination of both finance and product teams.  Some people are "table" folks and others are "chart" folks.  Regardless, the combination of the two data presentation methods jogs the brain and forces you to see the data in new ways and patterns. Here are three reasons why you need to visualize your data: 1.     Visualizing data allows you to segment elements within your results set.  For example, when analyzing campaign data, the average click-through rate or cost per acquisition might meet your campaign goals.  However, deeper segmentation via visualization can help you determine where further optimization opportunities lie via eliminating waste and exploiting upside outliers. 2.     Ability to see relationships and correlations within the data.  Scatterplots and grids can help sift out opportunities where you might have huge upside if you can optimize your key trigger points (whether it is behavior metrics like CTR or margin management). 3.     Trend analysis over time is more obvious when it is visualized. Trends in data can shift dramatically or slowly over time. Visualizing trends can allow you to see the slope of a metric or group of metrics to identify seasonal and market trends that may not pop out at you when you are just staring at a table of numbers. Segmentation, correlation and trending are just three of the reasons for data visualizations -- but there are so many more.  As analysts, we sometimes do not see the forest through the trees.  Visualizing data forces you to stop and look at the results and ponder the bigger picture.  In this case, perhaps, a picture is worth a thousand data points.

Jodi McDermott is the Director of Data Strategy for Clearspring Technologies and blogs at http://widgetanalytics.wordpress.com. Contact her here

Washington Post: Blue Chip, White Cotton - What Underwear Says About the Economy

Blue Chip, White Cotton: What Underwear Says About the Economy

By Ylan Q. Mui
Washington Post Staff Writer
Monday, August 31, 2009

For one answer to the nation's most pressing economic question -- when will the recession end? -- just take a peek inside the American man's underwear drawer.

There may be some new pairs there, judging by recent reports from retailers and analysts, and that could mean better days ahead for everyone.

Here's the theory, briefly: Sales of men's underwear typically are stable because they rank as a necessity. But during times of severe financial strain, men will try to stretch the time between buying new pairs, causing underwear sales to dip.

"It's a prolonged purchase," said Marshal Cohen, senior analyst with the consumer research firm NPD Group. "It's like trying to drive your car an extra 10,000 miles."

The growth in sales of men's underwear began to slow last year as the recession took hold, according to Mintel, another research firm. This year, Mintel expects sales to fall 2.3 percent, the first drop since the company started collecting data in 2003.

But the men's underwear index -- or, conveniently, MUI -- may also have a silver lining. Mintel predicts that next year, men's underwear sales will fall by 0.5 percent, and as with many economic indicators, a slowing of a decline can be welcomed as a step in the right direction. Retailers are reporting encouraging signs in the men's underwear department. Sears spokeswoman Amy Dimond said stores are beginning to see more sales. At Target, spokeswoman Jana O'Leary said sales of men's underwear have been stronger over the past two months and multi-pair packs are moving.

No less an oracle than former Federal Reserve chairman Alan Greenspan has given this theory credence, as described in a report on NPR two years ago. But you don't have to take his word for it. Just ask Kenneth Sanford, 59, of Capitol Heights, about his underwear. He said he usually buys new boxers every three months or so in maroon, black or white. But he had to stop working for medical reasons, and now he's having a hard time finding a new job.

To save money, he doesn't go out for ribs with his friends and family as much anymore. And when he indulges, he gets one piña colada instead of two. He hasn't bought a new pair of underwear in at least eight months.

"It's been a while now," Sanford said. "I just don't ever go shopping."

Of course, there are more conventional indicators of the nation's economic health. The gross domestic product fell 1 percent during the second quarter. Consumer spending and consumer confidence have been on a roller coaster this year. Home sales show some signs of bottoming out. But sometimes it is the little things that can be the most telling.

Leonard Lauder, chairman of the cosmetics company Estee Lauder, famously looked to lipstick sales as a barometer of consumers' mind-set during the last downturn. He believed that women were looking for small indulgences to lift their spirits during a tough economic time, though that theory has not held up in this recession, as sales of lipstick at mass retailers fell 8 percent over the past year, according to the research firm Information Resources.

Others look to a reported rise in prescriptions of anti-depressants and sleep aids last year as a sign of consumers' fragile state.

But perhaps no other purchase is as intimate as underwear. Few, if any, other people see it, so it's an easy place to skimp. According to Mintel, men buy an average of 3.4 pairs of underwear in a year. But from 2004 to 2008, the proportion of men buying single pairs at a time increased from 5 percent to 8 percent, while the share of men opting for packs of four or more fell slightly, from 68 to 66 percent -- indicating that shoppers may be trying to save money by buying only when necessary.

"People still need underwear," said Michael Kleinmann, president of FreshPair.com, an online underwear retailer. "They just have less money to spend."

The company sells high-end men's underwear that can run as much as $30 per pair, along with brands that cost less than $10. Kleinmann said that such less expensive pairs have had double-digit-percentage sales growth recently, while demand for pricier pairs is slowing.

Cohen, of NPD, said he hopes the recent positive signs in men's underwear will spill over into other need-based purchases. With the recession nearing two years, shoppers are at the stage where their stuff is simply beginning to wear out, providing an incentive to return to the stores.

"The consumers may be down, but they're not out," said Cohen, who is bullish on an economic recovery. "If this were a true, deep, long, embedded recession, they wouldn't even be buying underwear."

See also: Economist: Lip Reading (Lipstick Index)

Fast Company: The Gripping Statistics - How to Make Your Data Matter

August 10, 2009

The Gripping Statistic: How to Make Your Data Matter

By Dan Heath & Chip Heath

Imagine that the federal government announces a second-stage bailout in the amount of 703,000 hectoshekels. (We've gone metric!) You're probably not sure how to feel about this. (Though if you're already incensed, you should probably cut down on the Fox News.)

To assess the bailout, you'd ask: How much money is that, exactly? Is it too much or not enough? (Also, in this crazy metric world, how many centiliters do I weigh? And do I look skinnier?)

For all practical purposes, an $800 billion stimulus package is as opaque as a 703,000-hectoshekel package; we have no real grasp of what it means. Big numbers fuzz our brains, and that is just as true in business as it is in public policy. Speaking in "millions" and "billions" is like your second year of Spanish: You've memorized the vocabulary, but it's hard to think in the language. The challenge of communicating the significance of numbers -- and acting on them -- is to find ways to bring them closer to people's day-to-day experience.

Take the $800 billion stimulus package. Some commentators have tried various ways to put the figure in perspective -- if you laid those bills end to end, how many times would they circle the earth? (If there's one thing people have a keen intuition about, it's the earth's circumference, right?) How can you relate to this monstrous figure in the daily-life zone?

Well, there are roughly 112 million households in the United States, with a median household income of about $50,000. So an $800 billion stimulus works out to be the rough equivalent of seven weeks' income for an American household. Is that worth it? By way of comparison, we already work three or four months a year just to pay our federal, state, and local taxes.

So maybe this seems like a no-brainer to you: seven weeks' worth of work to stave off a potential depression. Or maybe you're appalled. Regard-less, we can finally have a real argument, because we have a better idea of what we're arguing about.

Putting a number in a day-to-day context is critical. For instance, years ago, Cisco Systems was contemplating whether to install a wireless network for its employees (a "duh" decision today but not at the time). The company had calculated that it would cost roughly $500 per year, per employee to maintain the network. Was that worth it? Hard to say since we don't have much intuition about $500 yearly expenses.

One employee brought the number into daily life, computing that given what Cisco paid its average employee, if the wireless network could save that worker one to two minutes per day, it would be a good investment. Suddenly, our intuition is activated. Can we imagine a situation where the network might save someone two minutes? Almost certainly yes. (Whereas if the network had required 52 minutes of daily savings to pay off, that would have been a hard sell.)

Building intuition about numbers is different from shocking people with numbers. We've all heard stats like this one (which is real): 27 billion disposable diapers are used each year in the United States -- enough to stretch all the way to the moon and back seven times. What to say about this? For starters, it would be a funny joke to play on the astronauts.

But notice that the astronomical analogy blocks any useful intuition. Would we feel better, for instance, if the diapers only stretched to the moon and back once? That would be just as gross, yet it would mean that six out of every seven families had given up disposables.

It's possible to create intuition without losing shock value. In the film Super Size Me, documentarian Morgan Spurlock mentions a media campaign that encourages kids to eat five fruits and vegetables per day. Its ad budget is $2 million. Meanwhile, McDonald's annual ad budget for the United States is around $750 million. That's a ratio of 375 to 1. That may help explain why your daughter is more likely to beg for a Happy Meal than a fruit salad.

Spurlock could have gone one step further. His ratio is good -- better than millions and billions -- but we still haven't pulled it inside the frame of daily life. So suppose your 5-year-old daughter watches three hours of car-toons every Saturday morning and sees two McDonald's commercials per hour. Every Saturday, then, Ronald McDonald engages her six times.

How long will it be before your daughter sees a fruits-and-veggies com-mercial? She'd wait about 14 months to see the first one, and she'd have a driver's license before she saw 10 of them (the same number of McDonald's ads she'd see in two Saturdays).

A good statistic is one that aids a decision or shapes an opinion. For a stat to do either of those, it must be dragged within the everyday. That's your job -- to do the dragging. In our world of billions and trillions, that can be a lot of manual labor. But it's worth it: A number people can grasp is a number that can make a difference.

Dan Heath and Chip Heath are the best-selling authors of Made to Stick: Why Some Ideas Survive and Others Die. Want to share a Made to Stick column with your team? Go to fastcompany.com/madetostick [1].

---

Article location:http://www.fastcompany.com/magazine/138/made-to-stick-the-gripping-statistic.html

NY Times: For Today's Graduate, Just One Word: Statistics

  

August 6, 2009

For Today’s Graduate, Just One Word: Statistics

By STEVE LOHR

MOUNTAIN VIEW, Calif. — At Harvard, Carrie Grimes majored in anthropology and archaeology and ventured to places like Honduras, where she studied Mayan settlement patterns by mapping where artifacts were found. But she was drawn to what she calls “all the computer and math stuff” that was part of the job.

“People think of field archaeology as Indiana Jones, but much of what you really do is data analysis,” she said.

Now Ms. Grimes does a different kind of digging. She works at Google, where she uses statistical analysis of mounds of data to come up with ways to improve its search engine.

Ms. Grimes is an Internet-age statistician, one of many who are changing the image of the profession as a place for dronish number nerds. They are finding themselves increasingly in demand — and even cool.

“I keep saying that the sexy job in the next 10 years will be statisticians,” said Hal Varian, chief economist at Google. “And I’m not kidding.”

The rising stature of statisticians, who can earn $125,000 at top companies in their first year after getting a doctorate, is a byproduct of the recent explosion of digital data. In field after field, computing and the Web are creating new realms of data to explore — sensor signals, surveillance tapes, social network chatter, public records and more. And the digital data surge only promises to accelerate, rising fivefold by 2012, according to a projection by IDC, a research firm.

Yet data is merely the raw material of knowledge. “We’re rapidly entering a world where everything can be monitored and measured,” said Erik Brynjolfsson, an economist and director of the Massachusetts Institute of Technology’s Center for Digital Business. “But the big problem is going to be the ability of humans to use, analyze and make sense of the data.”

The new breed of statisticians tackle that problem. They use powerful computers and sophisticated mathematical models to hunt for meaningful patterns and insights in vast troves of data. The applications are as diverse as improving Internet search and online advertising, culling gene sequencing information for cancer research and analyzing sensor and location data to optimize the handling of food shipments.

Even the recently ended Netflix contest, which offered $1 million to anyone who could significantly improve the company’s movie recommendation system, was a battle waged with the weapons of modern statistics.

Though at the fore, statisticians are only a small part of an army of experts using modern statistical techniques for data analysis. Computing and numerical skills, experts say, matter far more than degrees. So the new data sleuths come from backgrounds like economics, computer science and mathematics.

They are certainly welcomed in the White House these days. “Robust, unbiased data are the first step toward addressing our long-term economic needs and key policy priorities,” Peter R. Orszag, director of the Office of Management and Budget, declared in a speech in May. Later that day, Mr. Orszag confessed in a blog entry that his talk on the importance of statistics was a subject “near to my (admittedly wonkish) heart.”

I.B.M., seeing an opportunity in data-hunting services, created a Business Analytics and Optimization Services group in April. The unit will tap the expertise of the more than 200 mathematicians, statisticians and other data analysts in its research labs — but that number is not enough. I.B.M. plans to retrain or hire 4,000 more analysts across the company.

In another sign of the growing interest in the field, an estimated 6,400 people are attending the statistics profession’s annual conference in Washington this week, up from around 5,400 in recent years, according to the American Statistical Association. The attendees, men and women, young and graying, looked much like any other crowd of tourists in the nation’s capital. But their rapt exchanges were filled with talk of randomization, parameters, regressions and data clusters. The data surge is elevating a profession that traditionally tackled less visible and less lucrative work, like figuring out life expectancy rates for insurance companies.

Ms. Grimes, 32, got her doctorate in statistics from Stanford in 2003 and joined Google later that year. She is now one of many statisticians in a group of 250 data analysts. She uses statistical modeling to help improve the company’s search technology.

For example, Ms. Grimes worked on an algorithm to fine-tune Google’s crawler software, which roams the Web to constantly update its search index. The model increased the chances that the crawler would scan frequently updated Web pages and make fewer trips to more static ones.

The goal, Ms. Grimes explained, is to make tiny gains in the efficiency of computer and network use. “Even an improvement of a percent or two can be huge, when you do things over the millions and billions of times we do things at Google,” she said.

It is the size of the data sets on the Web that opens new worlds of discovery. Traditionally, social sciences tracked people’s behavior by interviewing or surveying them. “But the Web provides this amazing resource for observing how millions of people interact,” said Jon Kleinberg, a computer scientist and social networking researcher at Cornell.

For example, in research just published, Mr. Kleinberg and two colleagues followed the flow of ideas across cyberspace. They tracked 1.6 million news sites and blogs during the 2008 presidential campaign, using algorithms that scanned for phrases associated with news topics like “lipstick on a pig.”

The Cornell researchers found that, generally, the traditional media leads and the blogs follow, typically by 2.5 hours. But a handful of blogs were quickest to quotes that later gained wide attention.

The rich lode of Web data, experts warn, has its perils. Its sheer volume can easily overwhelm statistical models. Statisticians also caution that strong correlations of data do not necessarily prove a cause-and-effect link.

For example, in the late 1940s, before there was a polio vaccine, public health experts in America noted that polio cases increased in step with the consumption of ice cream and soft drinks, according to David Alan Grier, a historian and statistician at George Washington University. Eliminating such treats was even recommended as part of an anti-polio diet. It turned out that polio outbreaks were most common in the hot months of summer, when people naturally ate more ice cream, showing only an association, Mr. Grier said.

If the data explosion magnifies longstanding issues in statistics, it also opens up new frontiers.

“The key is to let computers do what they are good at, which is trawling these massive data sets for something that is mathematically odd,” said Daniel Gruhl, an I.B.M. researcher whose recent work includes mining medical data to improve treatment. “And that makes it easier for humans to do what they are good at — explain those anomalies.”

Andrea Fuller contributed reporting.

Copyright 2009 The New York Times Company

Strategy & Business: The Statistician Who Ate Humble Pie

The Statistician Who Ate Humble Pie

by Nassim Nicholas Taleb

6/17/09
Nassim Nicholas Taleb, author of The Black Swan: The Impact of the Highly Improbable, introduces an engaging lesson in business forecasting from Dance with Chance: Making Luck Work for You, by Spyros Makridakis, Robin Hogarth, and Anil Gaba.

Just about every human decision about the future is tainted by a gap — the difference between what we think we know and what we actually know. The more expert we are, the wider the gap is likely to be. The story below, an excerpt from the book Dance with Chance, is a classic example of an expert-busting enterprise. The experts in this case are highly sophisticated statisticians and professors who are using up-to-date models. Most of us assume that sophistication helps us understand the future. But in fact, sophistication makes things worse; it invites misplaced focus on the complicated. In the end, if we expect to make better decisions, we would do well to internalize the lesson in this passage. That doesn’t mean relying on common sense. It might mean looking for experts who understand the limits of what they know and the relative value of simpler methods.

— Nassim Nicholas Taleb 

---

Excerpted from chapter 9 of
Dance with Chance: Making Luck Work for You

---

As an expert in statistics, working in a business school during the 1970s, one of the authors…couldn’t fail to notice that executives were deeply preoccupied with forecasting. Their main interest lay in various types of business and economic data: the sales of their firm, its profits, exports, exchange rates, house prices, industrial output…and a host of other figures. It bugged the professor greatly that practitioners were making these predictions without recourse to the latest, most theoretically sophisticated methods developed by statisticians like himself. Instead, they preferred simpler techniques which — they said — allowed them to explain their forecasts more easily to senior management. The outraged author decided to teach them a lesson. He embarked on a research project that would demonstrate the superiority of the latest statistical techniques. Even if he couldn’t persuade business people to adopt his methods, at least he’d be able to prove the precise cost of their attempts to please the boss.

Every decent statistician knows the value of a good example, so the professor and his research assistant collected many sets of economic and business data over time from a wide range of economic and business sources. In fact they hunted down 111 different time series, which they analyzed and used to make forecasts — a pretty impressive achievement given the computational requirements of the task back in the days when computers were no faster than today’s calculators. They decided to use their trawl of data to mimic, as far as possible, the real process of forecasting. To do so, each series was split into two parts: earlier data and later data. The researchers pretended that the later part hadn’t happened yet and proceeded to fit various statistical techniques, both simple and statistically sophisticated, to the earlier data. Treating this earlier data as “the past,” they then used each of the techniques to predict “the future,” whereupon they sat back and started to compare their “predictions” with what had actually happened.

Horror of horrors, the practitioners’ simple, boss-pleasing techniques turned out to be more accurate than the statisticians’ clever, statistically sophisticated methods. To be honest, neither [were] particularly great, but there was no doubt that the statisticians had served themselves a large portion of humble pie.

One of the simplest methods, known as “single exponential smoothing,” in fact appeared to be one of the most accurate. Indeed, for 61.8% of the time it was more accurate than the so-called Box-Jenkins technique, which represented the pinnacle of theoretically based statistical forecasting technology back in the 1970s. The academic journals of the day had proven that the Box-Jenkins method was more accurate than large econometric models where predictions were based on hundreds of equations and impressive volumes of data. So, by extension, single exponential smoothing was also more accurate than the grand-scale econometric models that cost hundreds of thousands of dollars to develop and use!

At first, the professor and his assistant were so taken aback by their results that they suspected they’d made a mistake. They made extensive checks on their own calculations but could find no errors at all. The initial shock now over, they began to cheer up. If there’s one thing that makes up for an academic proving himself wrong, it’s the opportunity to show that other eminent authorities are wrong too. So the professor submitted a paper on his surprising and important findings to a prestigious, learned journal and waited for the plaudits to start rolling in. This in itself turned out to be another forecasting error. The paper was rejected on the grounds that the results didn’t square with statistical theory! Fortunately, another journal did decide to publish the paper, but they insisted on including comments from the leading statisticians of the day. The experts were not impressed. Among the many criticisms was a suggestion that the poor performance of the sophisticated methods was due to the inability of the author to apply them properly.

Undaunted, the valiant statistician and his faithful assistant set out to prove their critics wrong. This time around they collected and made forecasts for even more sets of data (1,001 in total, as computers were much faster by this time), from the worlds of business, economics and finance. As before, the series were separated into two parts: the first used to develop forecasting models and make predictions; and the second used to measure the accuracy of the various methods. But there was a new and cunning plan. Instead of doing all the work himself, the author asked the most renowned experts in their fields — both academics and practitioners — to forecast the 1,001 series. All in all, fourteen experts participated and compared the accuracy of seventeen methods.

This time, there were no bad surprises for the professor. The findings were exactly the same as in his previous research. Simpler methods were at least as accurate as their complex and statistically sophisticated cousins. The only difference was that there were no experts to criticize, as most of the world’s leading authorities had taken part.

That was way back in 1982. Since then, the author has organized two further forecasting “competitions” to keep pace with new developments and eliminate the new criticisms that academics have ingeniously managed to concoct. The latest findings, published in 2000, consisted of 3,003 economic series, an expanding range of statistical methods, and a growing army of experts. However, the basic conclusion — supported by many other academic studies over the past three decades — remains steadfast. That is, when forecasting, always use the KISS principle: Keep It Simple, Statistician.

— Spyros Makridakis, Robin Hogarth, and Anil Gaba

This article is from
Click HERE to subscribe to strategy+business
 

strategy+business is published by the global commercial consulting firm Booz & Company.

WSJ: The Triump of the Random

LIFE & STYLE
JULY 5, 2009, 5:26 A.M. ET

The Triumph of the Random

From banking to baseball, winning streaks owe much to the laws of chance

By LEONARD MLODINOW

It was the summer of 1945, and World War II had ended. Former soldiers, including famous baseball stars, streamed back into America and into American life. Yankee slugger Joe DiMaggio was trying to be Yankee fan Joe DiMaggio, sneaking into a mezzanine seat with his 4-year-old son, Joe Jr., before rejoining his team. A fan noticed him, then another. Soon throughout the stadium people were chanting “Joe, Joe, Joe DiMaggio!” DiMaggio, moved, gazed down to see if his son had noticed the tribute. He had. “See, Daddy,” said the little DiMaggio, “everybody knows me!”

Baseball Hall of Fame

What role did chance play in Joe DiMaggio’s epic 56-game hitting streak?

We all interpret the events around us according to our own worldview. By adulthood we’ve either gotten beyond the me-me-me context of 4-year-olds, or gone into politics. But drawing conclusions about data we encounter in sports, business, medicine and even our personal lives, we often make errors as significant as that of Joe Jr.

This holiday weekend—the Fourth of July—kicks off the home stretch of a two-month period that made Joe DiMaggio Sr. an icon of American culture. In 1941, a few months before Joe Jr. was born, and sandwiched between the day Hitler’s insane deputy Rudolf Hess parachuted into Scotland on an unauthorized peace mission and the day a secret British report concluded that the Allies could complete an atomic bomb ahead of Germany, there was a period of 56 successive Yankee games in which Joltin’ Joe had at least one hit.

DiMaggio’s hitting streak was an inspiration in troubled times. The pursuit of any record comes with pressure—Roger Maris lost some of his hair during his attempt to break Babe Ruth’s home-run record in 1961—but most records forgive you an off day as long as you compensate at other times. Not so with a streak, which demands unwavering performance. And so DiMaggio’s streak has been interpreted as a feat of mythic proportion, seen as a heroic, even miraculous, spurt of unrivaled effort and concentration.

But was it? Or was this epic moment simply a fluke?

NBAE/Getty Images

Kobe Bryant of the Los Angeles Lakers scored 81 points in a 2006 game.

Recent academic studies have questioned whether DiMaggio’s streak is unambiguous evidence of a spurt of ability that exceeded his everyday talent, rather than an anomaly to be expected from some highly talented player, in some year, by chance, something like the occasional 150-yard drive in golf that culminates in a hole in one. No one is saying that talent doesn’t matter. They are just asking whether a similar streak would have happened sometime in the history of baseball even if each player hit with the unheroic and unmiraculous—but steady—ability of an emotionless robot.

That randomness naturally leads to streaks contradicts people’s intuition. If we were to picture randomness, we might think of a graph that looks jerky, not smooth like a straight line. But random processes do display periods of order. In a toss of 100 coins, for example, the chances are more than 75% that you will see a streak of six or more heads or tails, and almost 10% that you’ll produce a streak of 10 or more. As a result a streak can look quite impressive even if it is due to nothing more than chance.

A few years ago Bill Miller of the Legg Mason Value Trust Fund was the most celebrated fund manager on Wall Street because his fund outperformed the broad market for 15 years straight. It was a feat compared regularly to DiMaggio’s, but if all the comparable fund managers over the past 40 years had been doing nothing but flipping coins, the chances are 75% that one of them would have matched or exceeded Mr. Miller’s streak. If Mr. Miller was really merely the goddess of Fortune’s lucky beneficiary, then one would expect that once the streak ended there would be no carryover of his apparent golden touch. In that expectation Mr. Miller did not disappoint: in recent years his fund has significantly lagged the market as he bet on duds like AIG, Bear Stearns, Merrill Lynch & Co. and Freddie Mac.

Of course a 10- or 15-game streak is a far cry from one of 56 games. This is where DiMaggio’s great ability plays a role, for if we are to compare DiMaggio’s performance to a coin, it must be a weighted coin. With a lifetime batting average of .325, DiMaggio had a better-than-75% chance of getting a hit in a game, while a balanced coin has but a 50% chance of success. Moreover, each year for over a century, hundreds of players have sought to achieve a streak such as DiMaggio’s. All those factors increase the odds that such a streak could have occurred by chance alone.

It’s not just the statisticians who wonder whether our heroes achieve records more often than coins. Psychologists, and, increasingly, economists, also puzzle over the seemingly discrete worlds of chance and perception. The fusion of those worlds was sanctified when half of the 2002 Nobel Prize in economics was awarded to psychologist Daniel Kahneman “for having integrated insights from psychological research into economic science, especially concerning human judgment and decision-making under uncertainty.”

Associated Press

In 1988 the Baltimore Orioles lost their first 21 games.

Research into why people misinterpret streaks dates to 1985, and a paper co-authored by Mr. Kahneman’s regular collaborator, the late Amos Tversky, in the journal Cognitive Psychology. (No one doubts that, had he lived, Mr. Tversky would have shared in the prize). The paper was titled “The hot hand in basketball: On the misperception of random sequences.” Everyone who has ever played basketball knows the feeling of being “in the zone.” Your hand is on fire. You can’t miss. But are you feeling a true increase in ability, or is your mind inferring it because you just took a bunch of shots that, for whatever reason, went in?

If a person tossing a coin weighted to land on heads 80% of the time produces a streak of 10 heads in a row, few people would see that as a sign of increased skill. Yet when an 80% free throw shooter in the NBA has that level of success people have a hard time accepting that it isn’t. The Cognitive Psychology paper, and the many that followed, showed that despite appearances, the “hot hand” is a mirage. Such hot and cold streaks are identical to those you would obtain from a properly weighted coin.

Why do people have a hard time accepting the slings and arrows of outrageous fortune? One reason is that we expect the outcomes of a process to reflect the underlying qualities of the process itself. For example, if an initiative has a 60% chance of success, we expect that six out of every 10 times such an initiative is undertaken, it will succeed. That, however, is false. In order to warrant confidence that results reflect a deeper truth, you need many more trials than 10. In fact, one of the most counterintuitive features of randomness is that for a small number of trials, the results of a random process typically do not reflect the underlying probabilities.

For example, suppose we undertake an analysis of the resources, effort and ability of all the companies in the Fortune 500 and determine that every company has the same 60% chance of success in any given year. If we observe all the companies over a period of five years and the underlying probability of success were reflected in each company’s results, then over the five-year period every company would have three good years.

The mathematics of chance indeed dictate that in this situation the odds of a company having zero, one, two, four or five good years are lower than the odds of having three. Nevertheless it is not likely that a company will have three out of five good years—because there are so many of those misleading outcomes, their combined odds add up to twice the odds of having exactly three. That means that of the 500 companies, two-thirds will experience results that belie their underlying potential. In fact, according to the rules of randomness, nearly 50 of the companies will have a streak of either five good years, or five bad years, even if their corporate capacities were no better or worse than their counterparts’. And so if you judged the companies by their five-year results alone, you would probably over- or underestimate their true value.

In sports, the championship contenders are usually pretty evenly matched. But in baseball, even if one assumes that the better team has a lopsided 55/45 edge over the inferior one, the lesser team will win the seven-game World Series 40% of the time. That might seem counterintuitive, but you can look at it as follows. If you play a best-of-one game series, then, by our assumption, the lesser team will win 45% of the time. Playing a longer series will cut down that probability. The problem is that playing a seven-game series only cuts it down to 40%, which isn’t cutting it down by much. What if you demand that the lesser team win no more than 5% of the time—a constraint called statistical significance? The World Series would have to be the best of 269 games, and probably draw an audience the size of that for Olympic curling. Swap baseball for marketing, and you find a mistake often made by marketing departments: assuming that the results of small focus groups reflect a trend in the general population.

We find false meaning in the patterns of randomness for good reason: we are animals built to do just that. Suppose, for example, that you sit a subject in front of a light which flashes red twice as often as green, but otherwise without pattern. After the subject watches for a while, you offer the subject a reward for each future flash correctly predicted. What is the best strategy?

A nonhuman animal in this situation will always guess red, the more frequent color. A different strategy is to match your proportion of red and green guesses to the proportion you observed in the past, that is, two reds for every green. If the colors come in some pattern that you can figure out, this strategy will enable you to be right every time. But if the colors come without pattern you will do worse. Most humans try to guess the pattern, and in the process allow themselves to be outsmarted by a rat. (Those trying to time the market lately might wish they had let the rat take charge.) Looking for order in patterns has allowed us to understand the patterns of the universe, and hence to create modern physics and technology; but it also sometimes compels us to submit bids on eBay because we see the face of Jesus in a slice of toast.

Another reason we reject the power of randomness is our need for control. DiMaggio’s streak affects us because we all appreciate struggle and effort, triumphing despite huge odds. The notion that we might not have control over our environment, on the other hand, causes us to shudder.

Many studies illustrate how this basic aspect of human nature translates to a misperception of chance. For example, a group of Yale students were asked to predict the result of a series of coin tosses. The tosses were secretly rigged so that each student would have some success initially, but end up with a 50% success rate. The students were obviously aware of the random nature of their task. Yet when asked whether their performance would be hampered by distraction, and whether it would improve with practice, a significant number indicated that it would. Their deep-seated need for control trumped their intellectual understanding of the situation.

What about DiMaggio’s streak? There are many subtleties in randomness. For example, do you model a player as having a fixed batting average—that is, probability of a hit—or do you allow for the average to vary within the season, or even game to game? How do you treat the variation in at-bats, walks, etc.? The analyses can get long and the number of data needed unwieldy, so the jury is still out, but one of the studies, by Samuel Arbesman and Stephen H. Strogatz of Cornell University, attacked the problem by having a computer generate a mock version of each year in baseball from 1871 to 2005, based on the players’ actual statistics from that year. The scientists had the computer repeat the process 10,000 times, generating in essence 10,000 parallel histories of the sport.

The researchers found that 42% of the simulated histories had a streak of DiMaggio’s length or longer. The longest record streak was 109 games, the shortest, 39. In those 10,000 universes, many other players held the record more often than DiMaggio. Ty Cobb, for example, held it nearly 300 times.

DiMaggio’s streak, for better or worse, defined his life. Decades later, constantly hounded by autograph seekers, he wrote “If I thought this would be taking place due to the streak, I would have stopped hitting at 40 games.” He died just 10 years ago, at age 84. Joe Jr., who had trouble coping with his father’s fame, fell to a history of drug and alcohol abuse. He died five months after his father.

People are remembered—and often rewarded—not for their usual level of talent or hard work, but for their singular achievements, the ones that stand out in memory. It does no harm to view those achievements as heroic. But it does harm us to make investments or other decisions on a basis of misunderstanding. And it can be sad or even tragic when we interpret as failures plans or people simply because they did not succeed. Extraordinary events, both good and bad, can happen without extraordinary causes, and so it is best to always remember the other factor that is always present—the factor of chance.

—Leonard Mlodinow teaches randomness at Caltech. His most recent book is “The Drunkard’s Walk: How Randomness Rules Our Lives.”

Copyright 2009 Dow Jones & Company, Inc. All Rights Reserved

This copy is for your personal, non-commercial use only. Distribution and use of this material are governed by our Subscriber Agreement and by copyright law. For non-personal use or to order multiple copies, please contact Dow Jones Reprints at 1-800-843-0008 or visit

www.djreprints.com

NY Times: Math and the City

May 19, 2009, 8:26 pm

Guest Column: Math and the City

Thanks again to Leon Kreitzman for four fascinating articles about biological clocks in everything from peonies to people. My sabbatical is rapidly drawing to a close — but it isn’t over yet! My guest for the next three weeks is Steven Strogatz, a professor of applied mathematics at Cornell University and the author of “The Calculus of Friendship: What a Teacher and a Student Learned about Life While Corresponding about Math,” to be published in August.
Please welcome him.
— Olivia

By Steven Strogatz

As one of Olivia Judson’s biggest fans, I feel honored and a bit giddy to be filling in for her. But maybe I should confess up front that, unlike Olivia and the previous guest writers, I’m not a biologist, evolutionary or otherwise. In fact, I’m (gasp!) a mathematician.

One of the pleasures of looking at the world through mathematical eyes is that you can see certain patterns that would otherwise be hidden. This week’s column is about one such pattern. It’s a beautiful law of collective organization that links urban studies to zoology. It reveals Manhattan and a mouse to be variations on a single structural theme.

The mathematics of cities was launched in 1949 when George Zipf, a linguist working at Harvard, reported a striking regularity in the size distribution of cities. He noticed that if you tabulate the biggest cities in a given country and rank them according to their populations, the largest city is always about twice as big as the second largest, and three times as big as the third largest, and so on. In other words, the population of a city is, to a good approximation, inversely proportional to its rank. Why this should be true, no one knows.

Even more amazingly, Zipf’s law has apparently held for at least 100 years. Given the different social conditions from country to country, the different patterns of migration a century ago and many other variables that you’d think would make a difference, the generality of Zipf’s law is astonishing.

Keep in mind that this pattern emerged on its own. No city planner imposed it, and no citizens conspired to make it happen. Something is enforcing this invisible law, but we’re still in the dark about what that something might be.

Many inventive theorists working in disciplines ranging from economics to physics have taken a whack at explaining Zipf’s law, but no one has completely solved it. Paul Krugman, who has tackled the problem himself, wryly noted that “the usual complaint about economic theory is that our models are oversimplified — that they offer excessively neat views of complex, messy reality. [In the case of Zipf’s law] the reverse is true: we have complex, messy models, yet reality is startlingly neat and simple.”

After being stuck for a long time, the mathematics of cities has suddenly begun to take off again. Around 2006, scientists started discovering new mathematical laws about cities that are nearly as stunning as Zipf’s. But instead of focusing on the sizes of cities themselves, the new questions have to do with how city size affects other things we care about, like the amount of infrastructure needed to keep a city going.

For instance, if one city is 10 times as populous as another one, does it need 10 times as many gas stations? No. Bigger cities have more gas stations than smaller ones (of course), but not nearly in direct proportion to their size. The number of gas stations grows only in proportion to the 0.77 power of population. The crucial thing is that 0.77 is less than 1. This implies that the bigger a city is, the fewer gas stations it has per person. Put simply, bigger cities enjoy economies of scale. In this sense, bigger is greener.

The same pattern holds for other measures of infrastructure. Whether you measure miles of roadway or length of electrical cables, you find that all of these also decrease, per person, as city size increases. And all show an exponent between 0.7 and 0.9.

Now comes the spooky part. The same law is true for living things. That is, if you mentally replace cities by organisms and city size by body weight, the mathematical pattern remains the same.

For example, suppose you measure how many calories a mouse burns per day, compared to an elephant. Both are mammals, so at the cellular level you might expect they shouldn’t be too different. And indeed, when the cells of 10 different mammalian species were grown outside their host organisms, in a laboratory tissue culture, they all displayed the same metabolic rate. It was as if they didn’t know where they’d come from; they had no genetic memory of how big their donor was.

But now consider the elephant or the mouse as an intact animal, a functioning agglomeration of billions of cells. Then, on a pound for pound basis, the cells of an elephant consume far less energy than those of a mouse. The relevant law of metabolism, called Kleiber’s law, states that the metabolic needs of a mammal grow in proportion to its body weight raised to the 0.74 power.

This 0.74 power is uncannily close to the 0.77 observed for the law governing gas stations in cities. Coincidence? Maybe, but probably not. There are theoretical grounds to expect a power close to 3/4. Geoffrey West of the Santa Fe Institute and his colleagues Jim Brown and Brian Enquist have argued that a 3/4-power law is exactly what you’d expect if natural selection has evolved a transport system for conveying energy and nutrients as efficiently and rapidly as possible to all points of a three-dimensional body, using a fractal network built from a series of branching tubes — precisely the architecture seen in the circulatory system and the airways of the lung, and not too different from the roads and cables and pipes that keep a city alive.

These numerical coincidences seem to be telling us something profound. It appears that Aristotle’s metaphor of a city as a living thing is more than merely poetic. There may be deep laws of collective organization at work here, the same laws for aggregates of people and cells.

The numerology above would seem totally fortuitous if we hadn’t viewed cities and organisms through the lens of mathematics. By abstracting away nearly all the details involved in powering a mouse or a city, math exposes their underlying unity. In that way (and with apologies to Picasso), math is the lie that makes us realize the truth.

***********

NOTES:

For Zipf’s law see:

Zipf, G. K. (1949) “Human Behavior and the Principle of Least Effort.” Addison-Wesley, Cambridge, MA.

Gabaix, X. (1999) “Zipf’s law for cities: An explanation.” The Quarterly Journal of Economics 114, 739–767.

For Paul Krugman quote:

Krugman, P. (1996) “Confronting the mystery of urban hierarchy.” Journal of the Japanese and International Economies 10, 399–418.

The new laws of infrastructure for cities are reported in:

Bettencourt, L. M.A., Lobo, J., Helbing, D., Kühnert, C, and West, G. B. (2007) “Growth, innovation, and the pace of life in cities.” Proceedings of the National Academy of Sciences 104, 7301–7306.

For an overview of Kleiber’s law and the theory of West, Brown and Enquist, see:

Whitfield, J. (2006) “In the Beat of a Heart: Life, Energy, and the Unity of Nature.” Joseph Henry Press, Washington DC.

For the tissue culture results about mammalian cells, see:

Brown, M. F., Gratton, T. P., and Stuart, J. A. (2007) “Metabolic rate does not scale with body mass in cultured mammalian cells.” Am J Physiol Regul Integr Comp Physiol 292, R2115–R2121.

Copyright 2009 The New York Times Company

FT: Of couples and copulas

How the broken-heart syndrome led to a mathematical formula so brilliant that it sparked a market boom and so flawed that it is blamed for bringing down Wall Street.

===

Of couples and copulas

By Sam Jones

Published: April 24 2009 22:04 | Last updated: April 24 2009 22:04

Johnny Cash and June Carter met backstage at the Grand Ole Opry. It was a little like a country song: he was married, she recently divorced, and an affair ensued; both singers had young children, and Cash would have three more with his first wife before she left him in 1966, citing his drinking and carousing. Two years later, he proposed to Carter on stage and, despite having turned him down numerous times before, she accepted. They’d each found a life match.

It ended like a country song, too. In 2003, Carter died in Nashville of complications from heart surgery, and Cash followed her to the grave four months later. The heart complication for him, it seemed, was that it was broken: “It hurts so bad,” he told the audience at the last concert he would give. The pain, he said, tuning his guitar, close to tears, was “the big one. It’s the biggest.”

EDITOR’S CHOICE

More from Reportage - Nov-24

Cash was speaking for many a bereaved partner – and well before Johnny ever met June, scientists had noticed that cases of spouses dying in rapid succession were not at all unusual. By the 1980s, medical researchers had started writing about “stress cardiomyopathy”, or “apical ballooning syndrome”, the ungainly name for the peculiar condition whereby an individual’s brain, following an intense emotional trauma, would inexplicably release chemicals into the bloodstream that weakened the heart – in some cases, causing it quite literally to break.

The medical community was interested because it offered a chance, potentially, to intervene and prolong life. Another industry was interested in the phenomenon, too – but less to stop it and more to understand it. These were the actuaries working in life assurance. Actuarial science is the study of the statistics surrounding life and death – and the statistics surrounding the broken heart phenomenon were striking. Pages and pages of death records showed the same marked trend: that in human couples, the death of one partner significantly increases the chances of the death of the other. Dying of a broken heart – in the most general sense, not necessarily from stress cardiomyopathy – was not a rare occurrence, but something of a statistical probability. So much so that life assurers, in order to conduct their business, needed to incorporate it into their models. In a March 2008 study, Jaap Spreeuw and Xu Wang of the Cass Business School observed that in the year following a loved one’s death, women were more than twice as likely to die than normal, and men more than six times as likely. “This implies … that joint life annuities [in which payments continue at the same price until both partners die] are underpriced while last survivor annuities [in which payments increase after one partner dies] are overpriced,” concluded the authors.

Even before the definitive Cass study, however, actuaries had begun to incorporate the broken heart trend into their mathematical models calculating the chances of clients dying. How could such an ephemeral relationship be reliably captured? The actuaries, of course, relied on probability. While they could not hope to devise a model that predicted the likelihood of death from a broken heart for a specific couple, they could use statistical science to devise a fairly accurate picture across a group of people.

They borrowed from physics and devised a formula based on something called a Markov chain: a way of expressing a series of statistical events whose outcomes are dependent upon one another. In physics, Markov chain processes underlie our most basic understanding of the world around us, from the way liquid turns to gas to the way a drop of vivid ink might slowly diffuse through a glass of water. If you treated people like atoms, the actuaries reasoned, you could apply the same maths.

. . .

In the autumn of 1987, the man who would become the world’s most influential actuary landed in Canada on a flight from China. Neither Xiang Lin Li nor the handful of fellow junior academics with whom he was travelling – all from the University of Nankai – had ever been abroad before, yet they had come at the behest of the Chinese and Canadian governments to do something most unusual: study capitalism. The small band of mathematicians and statisticians would be taking business degrees at Quebec’s Laval University.

For Li, going to Canada was just the latest in a series of unlikely opportunities that had shaped his life up to then. Decades earlier, his family had suffered at the blunt end of Mao’s cultural revolution: his father, a mid-ranking police official, was precisely the kind of lowly bureaucrat that the red guard mob was intent on re-educating, and the family was uprooted to a small village in southern China. In the countryside, the chances of young Xiang Lin going to school – let alone university – were slim. But he was talented and driven, and made it not only into school, but on to ­Nankai, one of the country’s most prestigious institutions. Li studied economics, and had just passed his master’s examinations when the Canadians came calling. Determined to be among those sent to Quebec, he learnt French in four months – as much at home studying the language, it seemed, as he was crunching numbers.

Li’s drive did not abate abroad. Four years after arriving in Canada, he’d earned his MBA; by this stage, he had no intention of returning home. In the few years he had been away, China’s mini-glasnost period had withered. Hu Yaobang, general secretary of the Communist party and a pro-democratic reformist, had been ousted, and Chinese leader Deng Xiaoping was now wary of the liberalism genie that had been let out of the bottle. In 1989, the world had looked on as students were mowed down in Tiananmen Square. Universities such as Nankai were not exactly the safest places for ambitious young students – especially not ones returning from MBA courses in the west.

As if to make clear the break, Xiang Lin changed his name and became David Li. After graduation from Laval, he enrolled at a new university, Waterloo, near Toronto. He would now be studying actuarial science. And this wasn’t the only change: the move from genteel, francophone Montreal to the more worldly and business-oriented Toronto was profound – and deliberate. According to Jie Dai, a fellow immigrant from China and a classmate at Laval, “I clearly remember [Li] mention that if you are an actuarial guy, you can earn a lot of money.”

. . .

The big money in the 1990s, of course, was not to be found at Waterloo but in Silicon Valley, on Wall Street and in the City of London. The first of these might have been an obvious destination for a talented mathematician, but the latter two were also becoming magnets for the likes of Li. In 1984, Robert Rubin, who a decade later would become US treasury secretary, made a bold decision for his employer at the time, the investment bank Goldman Sachs. Rubin decided to hire Fischer Black, an economist and academic at the Massachusetts Institute of Technology’s Sloan School of Management. Prior to 1983, a few academics had toyed with economics and markets, but as intellectual curiosities; Black was the first of his kind – a serious academic, with publications under his belt and a tenured position to boot – to make the move to Wall Street, putting theory into practice and risking the scorn of his ivory tower colleagues.

Rubin’s bet earned Goldman many multiples of Black’s salary. At the bank, the professor pioneered the use of mathematics in pursuit of money. He was one half of the duo that came up with the Black-Scholes formula, which revolutionised Wall Street by promising to determine a rational price for market risks – a principle that would become the founding doctrine of a new field, quantitative finance. Quantitative finance’s practitioners were trying to outwit the markets, using maths to eliminate risk by first using maths to calculate it. And the numbers of those practitioners grew quickly. With the collapse of the Soviet Union, the end of the arms race and, in 1993, the cancellation by the US congress of the superconducting super collider – intended to be the world’s greatest physics experiment – particle physicists, experts in quantum mechanics and computing engineers were twiddling their thumbs. For the younger generation of newly qualified grads and PhDs, applying their expertise to finance was the obvious alternative to fighting it out for the dwindling number of jobs strictly in their fields.

Emanuel Derman – a particle physicist – was such a convert. He joined Goldman in 1985, working under Black before eventually taking over from his mentor, and recalls members of the “quant” influx being referred to as “POWs” – physicists on Wall Street. Equally accurate was the acronym used by Andrew Lo, another Wall Street quant and now a lecturer at MIT. What Wall Street was really after, said Lo in a recent lecture, was not PhDs, but PSDs: people who were “poor, smart and with a deep desire to get rich”. At Waterloo, Li fitted that description to a T. He was studying for his PhD in actuarial science, but no one expected him to go on to a career in academia. Instead, in 1997, after earning his doctorate, he took a job at one of Canada’s largest banks, Canadian Imperial Bank of Commerce (CIBC).

For graduates such as Li, joining the rough and tumble world of business could be something of a shock, even when armed with MBAs. At best, the mathematicians were semi-disparagingly referred to as quants; if they were lucky, trader colleagues might slap them on the back and call them, half in flattery, “rocket scientists”. Emanuel Derman recalls a time at Goldman Sachs when he and another quant colleague were standing on the trading floor, on either side of a central aisle, and a senior trader passed between them. The trader “winced, clutching his head with both hands as though in excruciating pain, and exclaimed, ‘Aaarrggh-hhh! The force field! It’s too intense! Let me out of the way!’”

And yet by the time Li got to New York, in 1998, the quants had taken over the asylum. In the summer of that year, Long Term Capital Management, a hedge fund run by the finest minds in quantitative finance, required a massive bailout from the federal government. But far from serving as a warning that mathematical models could get investors into serious trouble, LTCM exploded the notion of quantitative finance as a geeky, back-office support task. The fund’s might before its fall – and the fact that its failure might have left a trillion-dollar hole in the financial system – discounted the notion that traders’ instinct and experience counted more than numerical intelligence.

The quants weren’t exactly out on the trading floor, however. The best of them still spent their days writing papers, crunching numbers, applying their academic expertise to the world of business. Li had come to New York to work for a consultancy called the RiskMetrics Group, which had been spun out of JP Morgan, but he was still thinking about life, death and love. In 2000, he published a paper in the prestigious Journal of Fixed Income that gained some serious attention. In it, Li performed a most elegant trick. Borrowing from his work in actuarial science and insurance and his knowledge of the broken-heart syndrome, he attempted to solve one of Wall Street quants’ most intractable problems: default correlation.

Markets do not function in laboratory-like isolation. They are linked, correlated. It isn’t enough for any quant to try and know the probability of each individual company in his bank’s portfolio going bust; he has to know how the bankruptcy of one company – or several – might increase (or decrease) the likelihood that other companies will default. Suppose, for example, that a bank loans money to two outfits – a dairy farm and a dairy. The farm, according to ratings agencies, has a 10 per cent chance of going bust and the dairy a 5 per cent chance. But if the farm does go under, the chances that the dairy will follow will rise above 5 per cent – quickly and steeply – if the farm was its main milk supplier.

And it gets more complicated from there. How correlated are the default probabilities on bonds issued by our Irish dairy farm and those issued by a software company in Malaysia? Not at all, you might think: the businesses not only provide totally different products and services, they’re also geographically remote from each other. Suppose, though, that both companies have been lent money by the same troubled bank that is now calling in its loans.

In fact, this is exactly what sank LTCM. How correlated are Russian government bonds and those in Mexico? Not at all, according to LTCM’s model, which, it should be noted, crunched data going back a hundred years. And yet it turned out for the hedge fund that both markets were dominated by the same few investors. The 1998 financial crisis in Russia, when Boris Yeltsin’s government defaulted on its bonds, caused panic selling in Mexico as investors rushed to de-risk their portfolios.

Li realised that his insight was groundbreaking. Speaking to The Wall Street Journal seven years later, he said: “Suddenly I thought that the problem I was trying to solve [as an actuary] was exactly the problem these guys were trying to solve. Default [on a loan] is like the death of a company.” And if he could apply the broken hearts maths to broken companies, he’d have a way of mathematically modelling the effect that one company’s default would have on the chance of default for others.

. . .

When mathematicians and physicists want to describe the chances of events occurring, they often rely on a curve called a copula. The Latin root is a noun meaning a “link or tie”, and indeed, copulas connect variables in such a way that their interdependence can be plotted. Throughout his PhD at Waterloo, and at CIBC, Li had been interested in how he could use copulas to develop existing actuarial models of the ­broken heart syndrome. The problem with relying on Markov chains was that they painted a far too mechanical, physical – atomic, even – picture of the human lifespan. Li reasoned that with a copula that showed a probable distribution of outcomes, a more accurate, encompassing picture of the broken heart or, for that matter, the broken company, could be devised.

He decided to use a very standard type of curve – the Gaussian copula, which is better known as a bell curve, or normal distribution – to map and determine the correlation on any given portfolio of assets. In the same way that actuaries could tell their employers the chances of Johnny Cash dying soon after June Carter without knowing anything about Cash other than the fact of his recent widowhood, so quants could tell their employers the effect one company defaulting might have on another doing the same – without knowing anything about the companies themselves. From this point on, it really could be, would be, a number-crunching game.

By 2003, Li’s paper had made his name on Wall Street. By now he was director and global head of derivatives research at Citigroup, and on a bright Tuesday morning in November, he arrived at the annual Quant Congress to bask in the glory with a presentation about his work. In front of a room of hundreds of fellow quants (“not a million miles from some kind of science fiction convention”, one person who was there recalls) he ran through his model – the Gaussian copula function for default.

The presentation was a riot of equations, mathematical lemmas, arching curves and matrices of numbers. The questions afterwards were deferential, technical. Li, it seemed, had found the final piece of a risk-management jigsaw that banks had been slowly piecing together since quants arrived on Wall Street.

. . .

By 2001, correlation was a big deal. A new fervour was gripping Wall Street – one almost as revolutionary as that which had struck when the Black-Scholes model brought about the explosion in stock options and derivatives in the early 1980s. This was structured finance, the culmination of two decades of quants on Wall Street. The basic idea was simple: that banks no longer had to hold on to risks. Instead they could value them, using complex maths and modelling, then package and trade them like any other, ordinary security.

Mortgages were the prime example. Rather than make a mortgage loan and gradually collect interest over its lifespan, banks began to bundle the loans together and sell them into specially created off-balance-sheet shell companies. These companies in turn issued bonds to raise cash. And by using the modelling and maths being cranked out by quants, banks were able to tailor the structure of mortgage portfolios to ensure that bonds of varying risks could be issued to investors. The problem, however, was correlation. The one thing any off-balance-sheet securitisation could not properly capture was the interrelatedness of all the ­hundreds of thousands of different mortgage loans they owned. As a consequence, structured finance had remained a niche and highly bespoke practice throughout the 1990s.

On August 10 2004, however, the rating agency Moody’s incorporated Li’s Gaussian copula default function formula into its rating methodology for collateralised debt obligations, the structured finance instruments that subsequently proved the nemesis of so many banks. Previously, Moody’s had insisted that CDOs meet a diversity score – that is, that each should contain different types of assets, such as commercial mortgages, student loans and credit card debts, as well as the popular subprime debt. This was standard investing good practice, where the best way to guard against risk is to avoid putting all your eggs in one basket. But Li’s formula meant Moody’s now had a model that enabled it to gauge the interrelatedness of risks – and that traditional good practice could be thrown out of the window, since risk could be measured with mathematical certainty. No need to spread your eggs across baskets if you knew the exact odds of your one basket being dropped. A week after Moody’s, the world’s other large rating agency, Standard & Poor’s, changed its methodology, too.

CDOs built solely out of subprime mortgage debt became the rage. And using the magic of the Gaussian copula correlation model, and some clever off-balance-sheet architecture, high-risk mortgages were re-packaged into triple-A-rated investor gold. The CDO market exploded. In 2000, the total number of CDOs issued were worth somewhere in the tens of billions of dollars. By 2007, two trillion dollars of CDO bonds had been issued. And with so many investors looking to put their money in debt, that debt became incredibly cheap, fuelling a massive boom in house prices and turbo-charging the world’s economies.

. . .

The unwinding started, as we all now know, in the US subprime housing market. Defaults started to increase in late 2006. The banks weren’t worried at first. Their models assumed that the pinprick default points all over the US were not correlated. But the defaults kept coming. By early 2007, it was clear that the US subprime market had a problem and by that summer, homeowners all over the US were defaulting on their mortgages. The cheap debt made available by the finance revolution was so cheap, in fact, that the loans should never have been made. And the correlation model was still mapping the housing market as it had been in 1990s, not the grossly inflated monster it had become. The development of the model had, ironically, changed the nature of the reality it was modelling.

The losses the banks began to take against their holdings of CDOs were staggering. And as the institutions grew fearful about one another’s solvency, they stopped lending to each other. Global liquidity dried up. The rot spread from asset class to asset class, and the banks’ pain spread to the real economy. Suddenly, everything was highly correlated.

How had Li’s formula failed to anticipate this? The problem was that it assumed events tended to cluster heavily around an average – a “normal” state. In actuarial science, Li’s formula could adequately capture binary outcomes such as life or death, but in the messy world of mortgages and economics, it faltered. The range of possible outcomes here was more complicated, and indeed, random, than those facing an insurance company’s clients. Markets – particularly the mortgage market – were far more prone to extreme correlation scenarios than were insurers. Death from a broken heart, for all its poetic associations, is far easier to predict than the more prosaic, but ultimately unknowable, interrelatedness of markets.

Why did no one notice the formula’s Achilles heel? Some did. Nassim Nicholas Taleb, author of the bestselling The Black Swan – a book about the importance of considering outliers when looking at copulas – was a voluble critic of quantitative finance and Li’s formula. “The thing never worked,” he says. “Anything that relies on correlation is charlatanism.”

In 2007, David Li left Wall Street and moved back to China. He could not be contacted for this story. But two years earlier, before the financial system blew up, he did warn: “Very few people understand the essence of the model.” Harry Panjer, a professor of statistics and actuarial science who was Li’s mentor at Waterloo, strikes a balance between Taleb’s accusations and the stance of Li. Earlier this year, Panjer told The Toronto Star newspaper: “We have a saying in statistics, ‘All ­models are wrong, but some are useful.’” And David Li’s model was, for a period, profoundly useful.

Sam Jones is a reporter for FT Alphaville.

Copyright The Financial Times Limited 2009

FT: From the fat cats to long tails: when all is not normal

From the fat cats to long tails: when all is not normal

By John Kay

Published: March 31 2009 19:35 | Last updated: March 31 2009 19:35

Like many people, I found James Joyce’s masterpiece, Ulysses, tough going. So I am full of admiration for George Zipf, who sat down 50 years ago to count Joyce’s words. The main reason for admiration, however, is not his accomplishment of a tedious task. What the professor discovered changes the way we think about business and finance, and illuminates both the commercial strategy of Amazon and the way banks messed up their risk models.

The most common word in the English language is “the”. Zipf’s thesis was that this word appears twice as often as the second most common word, “of”; and 10 times as often as the 10th most common word, “was”, and so on. We know now that Zipf was misled by the exceptional diversity of Joyce’s vocabulary, but the basic idea was right.

Those who are not very interested in Zipf’s, or any, critical study of Ulysses may prick up their ears when they learn that his analysis helps determine how many items a retailer should stock or what to expect when they buy derivative securities. His theory of the distribution of word use was an early example of what we call a power law. The language of other authors also follows a power law, but a different one. So does the incidence of earthquakes.

More than two centuries ago, mathematicians discovered that a small group of statistical distributions – the normal, or bell, curve is the most famous – had very wide application. These distributions are now used almost everywhere risk is quantified in business and finance – in quality and inventory control, in building investment portfolios, in calculating value at risk. Some people talk about “six sigma management” without realising that “sigma” is a parameter of the normal distribution.

But – and this is where Zipf’s contribution was important – we now know that this group of distributions is drawn from a larger family that includes power laws. However, the power law distributions, as Zipf showed, have different properties from those modellers have become used to.

In classical statistics, the extremes generally do not matter very much to the aggregate outcome. In many power law distributions, they do. Benoit Mandelbrot, the mathematical scientist, argues that security price movements follow a power law rather than the bell curve. That explains why it is so damaging to performance to be out of the market even for a few days, if they happen to be the wrong days. And, more relevant in current circumstances, why it can be so damaging to be in the market when things fall apart. “Fat tails” proved the downfall of fat cats. Extreme events, especially extreme adverse events, happen much more often than they are supposed to in the world of classical statistics.

Fat tails are rarely long tails, and vice versa. But power laws are relevant to both. If book sales are governed by a power law, then if 10 American books sell 1m copies in a year, and 400 sell more than 100,000, then about 16,000 titles will sell more than 10,000 copies. As with Zipf’s conjecture that the power law could be applied to Ulysses, this seems to be roughly true.

But in practice, the power law runs out in the “long tail”. The rule would predict there would be 640,000 books selling more than 1,000 copies. There are not, and for an obvious reason. Most titles that might sell 100,000 books get published but most titles that would only sell 1,000 do not. Chris Anderson’s acclaimed book, The Long Tail, explores the hypothesis that technological change in distribution has lowered the threshold at which publication becomes viable. This dynamic makes previously unsustainable strategies feasible.

Many distributions – such as book sales – are truncated. The long tail is docked. Companies that would have only a few thousand pounds of sales do not continue to exist: people who would have incomes below a certain level are supported by social benefits. To choose appropriate models you need to understand both the maths and the business environment. Media industries and financial institutions have both been unsuccessful in marrying these two skills.

John Kay’s most recent book is ‘The Long And The Short Of It: A Guide To Finance And Investment For Normally Intelligent People Who Aren’t In The Industry’

johnkay@johnkay.com

Post and read comments at www.ft.com/kay

Copyright The Financial Times Limited 2009

BusinessWeek: Bad Marriages Harder on Women's Health

Bottomline: "Choose your partner carefully. A strained marriage is bad for your health."

===

Executive Health March 05, 2009, 09:00 EST text size: TT

Bad Marriages Harder on Women's Health

More likely than mates to show signs of metabolic syndrome in strained unions, study finds

By Kathleen Doheny
HealthDay Reporter

THURSDAY, March 5 (HealthDay News) -- The cardiovascular damage wrought by an unhappy marriage may be greater for women than men, a new study shows.

While both men and women in "strained" unions, those marked by arguing and being angry, were more likely to feel depressed than happier partners, the women in the contentious relationships were more likely to develop high blood pressure, high cholesterol, high blood sugar and other markers of what's known as "metabolic syndrome," said study author Nancy Henry, a doctoral candidate in clinical healthy psychology at the University of Utah.

Metabolic syndrome is known to boost the risk of heart disease, stroke and diabetes.

While many studies have linked poor marriages with poor health, Henry said she believes her is the first to tie in depression as a possible route through which the strain boosts the risk of metabolic syndrome. "The negativity triggers the depression, which is associated with the metabolic syndrome," said Henry. This was found true, she said, only for the women in her study.

For the study, she interviewed 276 couples, median age 54, by questionnaires, asking about positive aspects of marriage quality such as mutual support and sharing, and negative aspects such as arguing, feelings of hostility and disagreeing over important issues such as kids, sex, money and in-laws. She asked about depressive symptoms.

Couples were married, on average, 27.5 years, most in their original marriage.

"For the most part, you could say, these were happily married couples," Henry said. About 20 percent of the men and 12 percent of the women in the study had metabolic syndrome (diagnosed when three of the five risk factors were present).

The men were as likely as the women to become depressed with marital strain, but the link between negativity, depression and metabolic syndrome only applied to women, she said. The depression in women accounted for the metabolic syndrome, she said.

Exactly why isn't known, but Henry speculated that women may take the negativity more to heart and ruminate about it more than men.

Henry can't say specifically how much risk of metabolic syndrome is attributed to the negativity. Earlier research has linked negativity in marriage with an increased risk of heart disease for both men and women.

She was expected to present her findings Thursday at the American Psychosomatic Society annual meeting, in Chicago.

Another researcher in the field called the findings interesting, especially the new focus on depression as a possible mechanism through which the strain influences the metabolic syndrome.

"The study raises the importance of increasing our understanding of how depression influences biological processes that result in metabolic syndrome -- and why these processes might be stronger for women than men," said Debra Umberson, a professor of sociology at the University of Texas at Austin.

The findings, Umberson said, fit in with her research finding a strong effect of marital strain on partners' overall health. But the gender difference finding differs from her research. "Basically, we find that marital strain undermines the health of men and women," she said, adding that perhaps the men in Henry's study had their health influenced in a different way.

More research is needed, Henry said, to figure out how the pieces fit together.

Meanwhile, Umberson said: "Choose your partner carefully. A strained marriage is bad for your health." If it's already strained, she said, focus on reducing conflict.

More information

To learn about improving a marriage, visit the American Psychological Association.

SOURCES: Nancy Henry, Ph.D., candidate, clinical health psychology, University of Utah, and clinical psychology intern, Salt Lake City VA Healthcare System; Debra Umberson, Ph.D., professor, sociology, University of Texas at Austin; March 5, 2009, presentation, American Psychosomatic Society annual meeting, Chicago

Copyright © 2009 ScoutNews, LLC. All rights reserved.

FT: Investment and the crisis - an error-laden machine

Investment and the crisis: an error-laden machine

By John Plender

Published: March 2 2009 20:08 | Last updated: March 2 2009 20:08

As stock markets everywhere continue their slide, global equities have in effect now shed all of the gains they had notched up between the Asian economic crisis of 1997-98 and the onset of the credit crisis in 2007.

Below: Credit rating agencies

But that seemingly remorseless retreat – which apart from anything else has pushed pension funds seriously into deficit – is only one part of a litany of investor woe. Consider also these other aspects of what is happening in the investment world:

First, along with the equity market collapse, the fall in value of complex structured credit products increasingly puts a question mark over many insurance companies’ solvency.

EDITOR’S CHOICE

Candover falls into private equity trap - Mar-02

BIS seeks warning signals of crises - Mar-02

Asean vies to find crisis response - Mar-02

Insight: Bolder moves are needed - Mar-02

Eurozone central banks weather financial storm - Mar-02

Wolfgang Münchau: Shortcomings will be costly - Mar-01

Second, the population of hedge funds is expected to shrink by more than half, as shaky business models are torpedoed by the bad market conditions.

Third, in private equity, industry experts reckon that most of the $85bn (£60bn, €67bn) to $100bn invested in transactions since 2005 has been wiped out. According to a Boston Consulting Group paper, potential losses from defaults on leveraged buy-out debt could reach $300bn in a market with $1,000bn of debt outstanding.

Fourth, a move by institutions into alternative asset categories this decade failed to deliver the expected benefits of diversification, as prices for many assets have plunged simultaneously.

The message of all this misery is summed up by Michael Lewitt of Harch Capital Management, a fund manager who was quick to identify the risks in the credit bubble. “Virtually every strategy institutional investors followed, or were advised to follow by their consultants or funds of funds”, he says, “turned out to be a complete disaster”. Even if that verdict errs on the sweeping side, it is clear that mainstream investment strategies failed to deliver. Why – and what needs to change to prevent a repetition?

‘Virtually every strategy institutional investors were advised to follow was a complete disaster‘: Michael Lewitt

A good diagnostic starting point is the phenomenon that academics call “disaster myopia” – the tendency to underestimate the probability of disastrous outcomes, especially for low-frequency events last experienced in the distant past. The risk of falling victim to this syndrome was particularly acute in the recent period of unusual economic stability known as the “great moderation”. Investors were confronted by falling yields against a background of declining volatility in markets. Many concluded that a new era of low risk and high returns had dawned. Their response was to search for yield in riskier areas of the market and then try to enhance returns through leverage, or borrowings.

Equally popular were trading strategies such as carry trades, which involved borrowing at low interest rates and investing at higher rates, especially via the currency markets. Favourite trades included borrowing in Japanese yen to invest in Australia or New Zealand, and borrowing in Swiss francs to invest in Icelandic assets.

This was dangerous because the interest rate spread could be wiped out in short order by volatile currency movements. Yet because volatility remained low for so long, disaster myopia prevailed. Carry traders were lulled into a false sense of security, while more sceptical competitors joined in for fear of underperforming.

In due course, markets turned and myopic traders were burned – confirming the wisdom of Warren Buffett, the sage of Omaha, who declared that “nothing sedates rationality like large doses of effortless money”. Yet even this most admired of investors admitted at the weekend to having lost billions of dollars after failing to anticipate the fall in energy prices.

‘Nothing sedates rationality like large doses of effortless money’: Warren Buffett

The sedative was exacerbated in the bubble, according to a recent paper by Andrew Haldane, director for financial stability at the Bank of England, by badly flawed risk models. “With hindsight, the stress-tests required by the authorities over the past few years were too heavily influenced by behaviour during the golden decade” of 1998-2007, he says. So many risk management models were pre-programmed to induce disaster myopia. The input into the models was based on highly unusual macroeconomic circumstances that differed materially from longer-term historical experience. Risk was thus mispriced on a dramatic scale because of model-enhanced myopia.

Among hedge funds, disaster myopia is more cynically entrenched by a poor alignment of interests between managers and their investors. Hedge fund fee structures rarely allow investors to claw back fees if years of profits are wiped out by a single year’s giant loss. Research by Harry Kat, professor of risk management at the Cass Business School in London, confirms just what this would lead one to suspect. Many hedge fund managers take on “tail” risks in derivatives markets, which produce a positive return most of the time as compensation for a very rare negative return. In effect, the funds have been writing catastrophe insurance. Then the catastrophe happened. Arbitrage strategies that took market liquidity for granted also foundered.

Equally unfortunate has been a botched approach to portfolio diversification. This powerful tool allows investors to achieve higher rewards for a given degree of risk, or the same reward for a lower level of risk. Yet in alternative asset categories it has failed to do that, despite the use of sophisticated mathematical modelling of correlations between asset classes. Hedge funds, private equities and commodities have underperformed in unison.

John Kay, a fellow Financial Times columnist, points out in The Long And The Short Of It, a new book on investment, that the endowments of Harvard and Yale did well in hedge funds and private equity in the 1990s. But asset classifications can change their meaning. As the sector grew, hedge funds became less a bet on an individual’s skills, more a conventional run-of-the-mill fund.

‘Stress tests required by the authorities were too heavily influenced by the golden decade from 1998’: Andrew Haldane

At the same time, private equity firms, bloated on credit, turned into a highly borrowed play on the stock market. Returns became increasingly correlated with other investments. The endowments, along with other investors who accepted consultants’ conventional wisdom on alternative assets, have suffered in consequence. Prof Kay’s message is that diversification is a matter of judgment, not statistics, and that a model will tell you only what you have already told the model. It can never replace an understanding of market psychology and the factors that make for successful business.

The fact that some strategies are more profitable if others do not adopt them is illustrated in When Markets Collide, by Mohamed El-Erian of Pimco, the bond fund manager. He tells the tale of Harvard Management Company’s investment in timber. This produced attractive risk-adjusted returns, which in due course were boosted by a herd-like migration of other investors into timber. Goodhart’s Law, named after the economist Charles Goodhart, then applied: recognisable statistical relationships change as economic agents’ behaviour adapts. So the expected benefits were eroded. The resulting closer correlation of timber to other asset classes is, Mr El-Erian concludes, an inevitable outcome in a competitive financial industry.

A more fundamental point is simply that diversification cannot work well in a credit bubble because virtually all asset categories are driven up by leverage. Then when the bubble bursts, deleveraging affects asset categories indiscriminately. Equally fundamental is that fund managers tend to move in herds because that reduces the risk of their losing client mandates. Minimising business risk takes priority over the interests of beneficiaries.

Many of these investment failures, including an excessive reliance on rating agencies (see above left), have a common feature in their unquestioning acceptance of models or methodologies. This “black box” approach to investing has been encouraged by the increasing complexity and opacity of a financial world where many assets have migrated to a shadow banking system that spawned structured products such as collateralised debt obligations, or to less regulated hedge funds.

As in private equity, many investors failed to grasp the penal nature of hedge fund charges. Prof Kay illustrates this by reference to the 20 per cent average compound rate of return earned by Mr Buffett at Berkshire Hathaway. If the normal hedge fund charges of an annual 2 per cent of funds under management and 20 per cent of profits had been applied to the resulting $62bn, no less than $57bn would have been absorbed in fees.

If big mistakes have been made in investment strategy, it does not follow that the remedy should be more regulation. The problems of disaster myopia, poor modelling, mismanaged diversification and excessive reliance on rating agencies stem more from failures of judgment by consultants, investment committees and pension fund trustees than systemic flaws.

So despite the complexity of today’s markets, the lessons in all this are oddly homespun. Mathematical models should not be relied on without a proper understanding of the economic conditions and behaviour that fed them. It is foolish to put blind faith in credit rating agencies. Do not invest in what you cannot understand. Shun arbitrage strategies that assume permanent access to liquidity. Avoid investment vehicles that inflict swingeing charges in exchange for what in most cases will amount to market performance or worse. Treat leverage with due care. Recognise that the conventional wisdom of the consulting fraternity is not conducive to contrarian behaviour, one of the keys to successful investing. Above all, beware what Charles Mackay, the 19th-century historian, called the madness of crowds.

  

CREDIT RATING AGENCIES: ‘HIGHLY PAID PROFESSIONALS LET A THIRD PARTY DO THE WORK’

Many of the biggest losses incurred by investors after the bursting of the credit bubble were in structured products such as collateralised debt obligations. This was a failure of due diligence, since investors left it to the credit rating agencies to assess the quality of underlying assets such as subprime mortgages.

According to Christopher Whalen, managing director of Institutional Risk Analytics, an advisory firm, “one of the dirty little secrets of Wall Street is that fund managers for years have been compelled and content to utilise ratings by the big three agencies to make asset allocation decisions.

“Simply put”, he adds, “these highly paid professionals let a third party do the hard work and failed to validate the fact that the work was done.”

The investors’ mistake was compounded by a failure to recognise a subtle shift in the nature of the credit rating agencies’ role in turning mortgage loans into complex securitised products. The agencies have long been paid by the companies that they rate, prompting questions about the independence of their judgments. But with instruments such as CDOs, they also advised banks on how to structure the product to enhance its rating and saleability.

Critics say this “mission creep” resulted in a more intense potential conflict of interest than with conventional credit – much as the big auditors’ move into consulting gave rise to acute conflicts of interest before the collapse of Enron. For some investors, such as pension funds and charities, the rating agencies’ writ is law because legislation, trust deeds and other governing instruments often stipulate that investments must carry certain ratings.

This seemingly prudent requirement inflicts underperformance on investors, since it condemns them to buy high and sell low. For smaller investors who lack the resources to do their own due diligence on complex products, there is no alternative to relying on rating agencies short of shunning the investments they rate.

Copyright The Financial Times Limited 2009