Repeatability and Truth

In the excellent article in The New Yorker’s December 13, 2010 edition , “The Truth Wears Off”, Jonah Lehrer summarizes the widespread phenomena that provocative experiments commonly fail to stand the test of time by being unable to be repeated. This analysis, assumes, however that the “gold standard” of science is repeatability. While there is considerable value in repeating experiments, an inability to repeat a result may not only be the result of subtle psychological tendencies on the part of scientists, but in some instances may reflect how the world works.

In our newly issued edited book, The Ecology of Place (Chicago Press), Mary Price and I note that variability is a fundamental part of ecological systems—Darwin’s “tangled bank”; an experiment run in one meadow may yield different results when run simultaneously in an adjacent meadow simply because the meadows are different. Yet good field scientists are able to develop general insights, sometimes not despite of the variability, but because of the variability.

Through sustained investigation of single locations, some scientists develop general insights by building a detailed understanding of a single ecosystem. False results are revealed not because other scientists fail to find similar results in other ecosystems, but because they are eventually undermined by other findings for the same system. For scientists committed to this approach, the incentives are not to produce flashy, false results that do not hold up, but to develop a rich understanding of a system based upon stitching together a large body of experimental results, observations and measurements.

One of the take away lessons of Lehrer’s article, in addition to the very real psychological phenomena which he describes, is that there is no one way to do science. All too often the general public, and even scientists themselves, fall back on overly simplistic descriptions of how scientists work. In reality, the richness of the world we live in demands that we be devious and subtle, and that we take a “no holds barred” approach to understanding.

The New Yorker, December 13, 2010