Tuesday, February 9, 2010

One More Reason Why High-Stakes Tests Are Not a Good Idea

Have you heard of Campbell's Law?

On Bridging Differences, an educational policy blog, Diane Ravitch writes:
Numbers don't lie, do they?

Well, yes, they do. A major front-page story in The New York Times on February 6 described a major study conducted by criminologists who found that the numbers do lie. More than 100 retired, high-ranking police officers in New York City told them that intense pressure to produce improved crime statistics had led to manipulation of the data. For the past 15 years or so, the city boasted that its data system, known as CompStat, had brought about a major reduction in crime. But the survey said that the data system had encouraged supervisors and precinct commanders to relabel crimes to less serious offenses. The data mattered more than truth. Some, for example, would scout eBay and other Web sites to find values for stolen items that would reduce the complaint from a grand larceny (over $1,000 in value) to a misdemeanor. There were reports of officers who persuaded crime victims not to file a complaint or to change their accounts so that a crime's seriousness could be downgraded.

This is not only a major scandal, it is a validation once again of Campbell's Law, which holds that: "The more any quantitative social indicator is used for social decisionmaking, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."

Anyone who wants to learn more about Campbell's law and how it applies to education should read Richard Rothstein's Grading Education and Daniel Koretz's Measuring Up. Or Google Rothstein's "Holding Accountability to Account," if you want to see what happens when data becomes our most important goal.


  1. i don't guess i *have* heard of it;
    good to have a handle to grab onto
    when we turn the conversation
    to touchy topics like "just how
    compromised *are* our sacred
    institutions?". somebody having
    "given it a name" can make it
    a little less scary to talk about.
    which *is* the problem: talking
    about it. everybody *knows*
    where the booze *is*...

  2. ps. sherman dorn
    (&, e.g. accountability frankenstein).

  3. I don't see what you're referring to in that link...

  4. oh, he's just done a book
    and a bunch of posts
    on "accountability"... the doctrine
    of data-uber-alles. i look in on
    his blog from time to time & he's
    usually well worth reading.
    i didn't intend anything too
    specific; just "here's another
    good source on this topic".

  5. I think that this is an interesting point. When we think of mathematics we often think of one correct answer or one correct method. Knowing that numbers are not always telling the "truth" is just another excellent reason as to why high-stakes testing should be reconsidered.


Math Blog Directory