I had already had a chance to rant against evidence-based educational research which leads to replacement individual studies by faceless statistics. It is a sensitive issue for me since I am writing a book based entirely on exceptional individual cases. But today I touch on a similar and earlier development in medical research: indidual cases disappeared from medical literature in 1960-s, being replaced be statistics. And not everyone is happy with that.
I quote David Corfield and Darian Leader (Chapter 2 of ‘Why do people get ill?’):
It was quite shocking after wading through pages and pages of mathematical equations used in statistical calculations, with not one single piece of reported speech from a patient, to come upon Swiss conference proceedings where there wasn’t a single graph, chart or number in sight, just detailed case reports of individual patients. Such major differences in scientific style almost guarantee that the Swiss research would not be taken seriously, if read at all, by an Anglo-American audience. In a recent survey of psychosomatic literature on the popular topic of pain research in the field’s premier journal ‘Psychosomatic Medicine’, it was found that after the publication of many case studies in the 40s, and a handful in the 50s and early 60s, not a single clinical case had been published.
In most Anglo-American contexts, it is the production of statistics that matters more than the individual patient and the listening process. In a recent cardiology study, we can read that “One patient (0.7%) developed somatic contraindications for a heart transplantation”, but apart from knowing that they were 0.7%, we learn nothing more. Not who the patient was, not what they said, nor what the somatic contraindications were. It reminds us of the ironic comment made by a doctor in the mid-50s : “The patient recovered and lived to be a statistic in a published report”. The philosopher Henri Bergson once said that it is unlikely that anyone analysing thousands of buckets of water from the ocean will ever learn very much about the tides.
Mathematics under the Microscope 
Absolutely! Let’s stick with non-evidence-based speculation!
By: Matthew on February 17, 2009
at 8:55 am
[…] Down with evidence-based research! « Mathematics under the Microscope […]
By: Down with evidence-based research! « Mathematics under the Microscope | ukmed.org on February 17, 2009
at 9:21 am
Matthew: I understand your irony, and perhaps I have to explain that I was brought up in strictest traditions of scientific rigour and learned one important principle: never try to seek rigour where it could not be achieved.
A teacher (and a doctor) deals with individual cases; mass statistics is needed for pharmaceutical companies and regulatory authorities; “evidence” in the sense of “massed, statistically treated evidence” has its legitimate uses. Unfortunately, such evidence is (a) routinely misused by policymakers; (b) in most case is useless for a practitioner of the art of teaching or healing.
From a point of view of a mathematician, the use of statistics in educational research and especially educational policy is shocking; forming arithmetic means of order statistics is just a lesser crime routinely committed by education policymakers.
For a house buyer, the most usable statistics of house prices is the median price of a house in a particular neighbourhood, because he or she buys one house and wants to understand how his/her future house relates to other houses in the neighbourhood; for a bank that provides dozens of mortgages in the same neighbourhood the “right” statistics is arithmetic mean.
A larger part of medicine still can be called a science, since it is at least partially rooted in firm factual knowledge, such as human anatomy, for example. In teaching, there is no factual foundation comparable to that of medicine. Teaching is not a science, it is art, and should be treated as such.
By: Alexandre Borovik on February 17, 2009
at 12:22 pm
I think your Bergson quote misses the mark. The tides are an emergent phenomenon, and can only be understood by stepping back and ignoring the individual nature of all those buckets of water. If anything, it advocates ignoring more and more of the personal stories — which serve to differentiate patients — and focusing on large statistical trends.
By: John Armstrong on February 17, 2009
at 1:28 pm
I really don’t understand how you can say that the work of pharmaceutical companies are “in most cases useless for a practitioner of the art of healing”. I’m no medic, but aren’t effective drugs pretty useful for doctors?
Similarly, aren’t well-designed curricula and tasks pretty useful for teachers? The evidence suggests that they are. Have a look at, for example, the work of the Carnegie Mellon group:
http://pbr.psychonomic-journals.org/content/14/2/249.abstract
The CMU work can also, I think, be seen as a refutation of the claim that “in teaching, there is no factual foundation comparable to that of medicine”.
By: Matthew on February 18, 2009
at 9:44 am
Matthew: drugs are useful, good curricula are useful. What I say is that there is no need for doctors to read detailed statistical accounts of double blind field trials of miriads of early versions of a drug (which never even reach hospitals). There is no need for a teacher to read statistical accounts of school trials of early variants of a particular curriculum. Doctors and teachers need good case studies. Teachers and doctors work with individual pupils and individual patients, not with statistical mass. Yes, there are people who have to make decisions about the statistical mass of humanity. In medicine, such decision makers appear to be better trained, better selected and bound by a more rigorous code of practice than in education. But in both cases the number of real decision makers is relatively small; on the other hand, bureaucracy that pretends to digest huge amounts of information is disproportionally huge.
The principal issue of evidence based research is the ownership of its results. Gert Biesta instantly struck a cord with me because he places evidence based research in the context of functioning of the democracy in the modern society.
A teacher has to be returned to the centre, to the focal point of educational process — and we need good papers and good books written for a teacher, not for educational bureaucracy.
By: Alexandre Borovik on February 18, 2009
at 10:28 am
Two comments:
1. Medical hypotheses have to come from somewhere. The most common source, still, are the individual experiences of individual doctors presented with individual patients. Such cases need to be published or else we risk losing ideas for testing. William McBride’s identification of thalidomide as a possible cause of birth defects, for instance, arose from a handful of cases of pregnant women he treated. If he had had to wait until he had had a large sample of cases before publication of his speculations, we may never have known of the possibility (the hypothesis) of the ill-effects of thalidomide. (The drug had already been tested thoroughly, although not on pregnant subjects.)
2. Research publications are a form of signalling. If reporting on scientific experiments, these signals may be noisy (ie, what is reported as true may not be true, due to experimental error). Using standard statistical methods, with a standard level of significance, enables a receiver of the signal to estimate an upper bound for the extent of the noise in the signal.
By: peter on February 18, 2009
at 9:15 pm
peter: I find myself is strange position when nothing contradicts my thesis: it does matter for the nature of research who owns the results of research. I am not against statistics — it is a fantastically powerful tool. But I feel that a majority of research in education should be done with understanding that its results should be useful to individual teachers.
“Evidence based” educational research is a classical cargo cult in the sense of Feynman; for one reason, i am unaware about anyone checking evidence. as soon as “evidence” is claimed, it is treated as absolute truth. Secondly, statistical methodology is frequently very fishy. Finally, how are the findings used?
By: Alexandre Borovik on February 23, 2009
at 10:55 am
I don’t really understand your argument.
I would probably agree with you that teacher’s (or doctor’s) don’t need to read the majority of educational (or medical) research papers. (We might also agree that physicists don’t need to read the majority of mathematical research papers). As with all academic subjects, the primary audience of an educational research paper is always going to be other educational researchers. But no one is going to dispute the value of education/medicine/mathematics researchers spending some time writing accessible reviews for non-specialists.
I don’t see how any of these considerations imply that evidence-based research is bad.
The issue of how to communicate research findings to more applied colleagues is an interesting topic, but I don’t think it has much to do with how that research should be conducted.
By: Matthew on February 23, 2009
at 7:35 pm
About the only way to convince your intended audience is to
find a real-life showcase where (statistical) evidence based
research is either useless or misleading. It should not be too hard, should it? …
By: Anonymous on March 3, 2009
at 9:35 pm