Posted by: Alexandre Borovik | February 17, 2009

Down with evidence-based research!

I had already had a chance to rant against evidence-based educational research  which leads to replacement individual studies by faceless statistics. It is a sensitive issue for me since I am writing a book based entirely on exceptional individual cases. But today I touch on a similar and earlier development in medical research:  indidual cases disappeared from medical literature in 1960-s, being replaced be statistics. And not everyone is happy with that.

I quote David Corfield and Darian Leader (Chapter 2 of ‘Why do people get ill?’): 

It was quite shocking after wading through pages and pages of mathematical equations used in statistical calculations, with not one single piece of reported speech from a patient, to come upon Swiss conference proceedings where there wasn’t a single graph, chart or number in sight, just detailed case reports of individual patients. Such major differences in scientific style almost guarantee that the Swiss research would not be taken seriously, if read at all, by an Anglo-American audience. In a recent survey of psychosomatic literature on the popular topic of pain research in the field’s premier journal ‘Psychosomatic Medicine’, it was found that after the publication of many case studies in the 40s, and a handful in the 50s and early 60s, not a single clinical case had been published.

In most Anglo-American contexts, it is the production of statistics that matters more than the individual patient and the listening process. In a recent cardiology study, we can read that “One patient (0.7%) developed somatic contraindications for a heart transplantation”, but apart from knowing that they were 0.7%, we learn nothing more. Not who the patient was, not what they said, nor what the somatic contraindications were. It reminds us of the ironic comment made by a doctor in the mid-50s : “The patient recovered and lived to be a statistic in a published report”. The philosopher Henri Bergson once said that it is unlikely that anyone analysing thousands of buckets of water from the ocean will ever learn very much about the tides. 

About these ads

Responses

  1. Absolutely! Let’s stick with non-evidence-based speculation!

  2. [...] Down with evidence-based research! « Mathematics under the Microscope [...]

  3. Matthew: I understand your irony, and perhaps I have to explain that I was brought up in strictest traditions of scientific rigour and learned one important principle: never try to seek rigour where it could not be achieved.

    A teacher (and a doctor) deals with individual cases; mass statistics is needed for pharmaceutical companies and regulatory authorities; “evidence” in the sense of “massed, statistically treated evidence” has its legitimate uses. Unfortunately, such evidence is (a) routinely misused by policymakers; (b) in most case is useless for a practitioner of the art of teaching or healing.

    From a point of view of a mathematician, the use of statistics in educational research and especially educational policy is shocking; forming arithmetic means of order statistics is just a lesser crime routinely committed by education policymakers.

    For a house buyer, the most usable statistics of house prices is the median price of a house in a particular neighbourhood, because he or she buys one house and wants to understand how his/her future house relates to other houses in the neighbourhood; for a bank that provides dozens of mortgages in the same neighbourhood the “right” statistics is arithmetic mean.

    A larger part of medicine still can be called a science, since it is at least partially rooted in firm factual knowledge, such as human anatomy, for example. In teaching, there is no factual foundation comparable to that of medicine. Teaching is not a science, it is art, and should be treated as such.

  4. I think your Bergson quote misses the mark. The tides are an emergent phenomenon, and can only be understood by stepping back and ignoring the individual nature of all those buckets of water. If anything, it advocates ignoring more and more of the personal stories — which serve to differentiate patients — and focusing on large statistical trends.

  5. I really don’t understand how you can say that the work of pharmaceutical companies are “in most cases useless for a practitioner of the art of healing”. I’m no medic, but aren’t effective drugs pretty useful for doctors?

    Similarly, aren’t well-designed curricula and tasks pretty useful for teachers? The evidence suggests that they are. Have a look at, for example, the work of the Carnegie Mellon group:

    http://pbr.psychonomic-journals.org/content/14/2/249.abstract

    The CMU work can also, I think, be seen as a refutation of the claim that “in teaching, there is no factual foundation comparable to that of medicine”.

  6. Matthew: drugs are useful, good curricula are useful. What I say is that there is no need for doctors to read detailed statistical accounts of double blind field trials of miriads of early versions of a drug (which never even reach hospitals). There is no need for a teacher to read statistical accounts of school trials of early variants of a particular curriculum. Doctors and teachers need good case studies. Teachers and doctors work with individual pupils and individual patients, not with statistical mass. Yes, there are people who have to make decisions about the statistical mass of humanity. In medicine, such decision makers appear to be better trained, better selected and bound by a more rigorous code of practice than in education. But in both cases the number of real decision makers is relatively small; on the other hand, bureaucracy that pretends to digest huge amounts of information is disproportionally huge.

    The principal issue of evidence based research is the ownership of its results. Gert Biesta instantly struck a cord with me because he places evidence based research in the context of functioning of the democracy in the modern society.

    A teacher has to be returned to the centre, to the focal point of educational process — and we need good papers and good books written for a teacher, not for educational bureaucracy.

  7. Two comments:

    1. Medical hypotheses have to come from somewhere. The most common source, still, are the individual experiences of individual doctors presented with individual patients. Such cases need to be published or else we risk losing ideas for testing. William McBride’s identification of thalidomide as a possible cause of birth defects, for instance, arose from a handful of cases of pregnant women he treated. If he had had to wait until he had had a large sample of cases before publication of his speculations, we may never have known of the possibility (the hypothesis) of the ill-effects of thalidomide. (The drug had already been tested thoroughly, although not on pregnant subjects.)

    2. Research publications are a form of signalling. If reporting on scientific experiments, these signals may be noisy (ie, what is reported as true may not be true, due to experimental error). Using standard statistical methods, with a standard level of significance, enables a receiver of the signal to estimate an upper bound for the extent of the noise in the signal.

  8. peter: I find myself is strange position when nothing contradicts my thesis: it does matter for the nature of research who owns the results of research. I am not against statistics — it is a fantastically powerful tool. But I feel that a majority of research in education should be done with understanding that its results should be useful to individual teachers.

    “Evidence based” educational research is a classical cargo cult in the sense of Feynman; for one reason, i am unaware about anyone checking evidence. as soon as “evidence” is claimed, it is treated as absolute truth. Secondly, statistical methodology is frequently very fishy. Finally, how are the findings used?

  9. I don’t really understand your argument.

    I would probably agree with you that teacher’s (or doctor’s) don’t need to read the majority of educational (or medical) research papers. (We might also agree that physicists don’t need to read the majority of mathematical research papers). As with all academic subjects, the primary audience of an educational research paper is always going to be other educational researchers. But no one is going to dispute the value of education/medicine/mathematics researchers spending some time writing accessible reviews for non-specialists.

    I don’t see how any of these considerations imply that evidence-based research is bad.

    The issue of how to communicate research findings to more applied colleagues is an interesting topic, but I don’t think it has much to do with how that research should be conducted.

  10. About the only way to convince your intended audience is to
    find a real-life showcase where (statistical) evidence based
    research is either useless or misleading. It should not be too hard, should it? …


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

Follow

Get every new post delivered to your Inbox.

Join 84 other followers

%d bloggers like this: