Experimenting with Students? An exchange with Eric Hanushek

Following are two email messages sent to Professor Eric Hanushek in April/May, 2001, in regard to an essay "Experimenting with Students?" that he published in a Hoover Institute journal (see citation below). Professor Hanushek was at that time member of an NAS/NRC Committee on Scientific Principles of Education Research, and my correspondence (cc'ed to other members of the committee) was also an attempt to influence the efforts of that group. The attempt was unsuccessful, as became apparent when the committee's report "Scientific Research in Education" was published (National Academy Press, 2002).

See the Education Page of Bas Braams - Links, Articles, Essays, and Opinions on K-12 Education - for related matter.

Dear Professor Hanushek,

Re: http://www.hoover.org/pubaffairs/we/current/hanushek_0301.html

Your recent Hoover Weekly Essay "Experimenting with Students?" is interesting. You write, and I concur, "correcting the lack of information about program effectiveness should be a national priority". You then write approvingly of using "a random-assignment experiment, where students are placed into a program by chance", and you recommend: "We should experiment more systematically across a range of reform areas so that we can improve our knowledge base.".

I doubt the wisdom of this recommendation in the form in which you present it. I apologize if I mis-read your essay, but it reads to me as if you advocate an extensive set of experiments in which pupils, and I suppose teachers too, are randomly assigned to one or another kind of program. This strikes me as morally dubious, dangerous, and gratuitously inviting conflict with every party in sight -- and these negative qualities will only be amplified by the kind of poor judgment that the education research community has often demonstrated in their choice of programs for investigation.

I have paid some attention to mathematics education research, and have no doubt that, in terms of quality and integrity, that field is at the low end of social policy research and fails miserably by the standards of science and engineering. The great majority of work is trite, anecdotal, self-serving and of no use as a guide to policy. It is painful to observe that the Federal Government, and especially the National Science Foundation throught its Education and Human Resources Directorate, has supported such educational research at the lowest levels of quality and integrity.

By itself, your recommended research policy will do nothing to improve matters -- in fact, the contrary may be true. The recommendations in your essay are easily read and will be read as a license to expand the kind of research that is today dumbing down the curriculum and lowering pupils' intellectual skills and attitudes in thousands of schools nationwide courtesy of the NSF/EHR's Systemic Initiatives and similar Federal programs.

You invoke a comparison with medicine and agriculture, but the comparison with medical research is severely flawed -- in medical research, patients have a genuine choice whether to participate in an experiment or not. The comparison with agricultural research also seems off the mark. A more appropriate point of comparison for your proposal as I read it would be the forced school busing in Boston in the 1970's, and I don't mean that as an endorsement.

I have a pragmatic and constructive view of the proper direction of change in education research. Research should piggy-back on the natural diversity that exists in pedagogical and curricular practices, and the researcher should live with the associated difficulties in data analysis. These difficulties are, I think, quite surmountable, as is indicated in the research sketch reproduced below, which you may have seen already in an earlier email.

- Large scale longitudinal study. Many pupils from many schools followed over multiple years. The primary unit of analysis is the pupil, not the classroom or the school. Performance data are obtained at least once a year using broad tests -- sufficiently broad to avoid teaching to the test. Of course one has some personal data on the subjects (date of birth and gender to begin with, and one may appreciate to have things like ethnicity and social economic status), and then one has a classification of each pupil's educational history -- the effect of aspects of this history is what one wants to assess. Information on the pupils' educational history would include at least: what curricular material, what teaching style, what class-level environment, and what homework policy; there will be many other items.

- The study looks at pupils' progress over the years. One first studies the data in order to obtain best predictions for pupils' performance at later times based on their performance at earlier times (and, maybe, on certain extracurricular factors), and then one studies how the addition of data on pedagogical practices modifies these predictions. Basically one asks, does the information that the XYZ pedagogical practice was used change materially the pupils' predicted performance.

- The study must be managed by a group that is independent of any particular educational strategy and any particular set of curricular materials, or the study must be jointly owned by different groups.

Absent from that research sketch, because of the many impediments that it would place in the way of the study, is the feature of randomized assignment of pupils to programs. A model of the kind of data collection effort involved is the Tennessee Value Added Assessment System (TVAAS), although in that context the data is used for teacher assessment rather than for comparison of the effectiveness of different classroom practices. Another model is the Texas Schools Project, and I know that you have analyzed data of that project for an assessment of the effect of teacher quality upon pupil performance.

I think that your stated aim of "correcting the lack of information about program effectiveness" would be well served if the Federal Government would promote research programs of the kind that I outlined above. Moreover, a policy of that nature would avoid the dubious morality of large-scale forced social experimentation, and would help eliminate the present practice of imposing very flawed pedagogical practices and curricula on our pupils and teachers under the guise of scientific research.

Yours Cordially,

Bas Braams
Courant Institute
New York University

Professor Hanushek replied to my email. He wrote that (1) he has, of course, no problems with the kinds of research that I proposed; and (2) he does not see a moral problem with random assignment experimentation, since such experimentation is implicitly done all the time, although we don't call it that and we neglect to learn from it.

My second email follows below.

Dear Professor Hanushek,

I agree that experimenting takes place all the time without analysis of the outcomes. Different teachers or sections make different choices of textbook material or curricular practice, and they may change textbook series every few years.

My proposal is that education researchers should, unobtrusively and without raising a stink and making everybody their enemy, attach their research to this natural variability. The researchers would acquire data of the kind that you know from the TVAAS or the Texas Schools Project plus data on classroom practices (esp. choice of textbooks) and then crunch their data much in the way that this is done in the context of "value added" assessment of teacher quality. The principal difference is that the researchers are now assessing the quality of curricular practices rather than of individual teachers.

Random assignment of pupils (and, I have to assume, teachers) to different educational programs will, in inevitable practice, mean that teachers will have to follow practices that they don't support. "Morally dubious", the phrase that I used in the earlier email, is about the least of the opprobrium that will be attached to such research by teachers, pupils, parents, teachers' unions, PTA's, school boards, principals, the press, and anyone else that gets wind of it.

And for what purpose? A teacher that wishes guidance in choice of textbook material is likely to care more for a comparison of outcomes involving other teachers that used the material more or less by free choice rather than a comparison of outcomes involving teachers that had the material imposed upon them against their best judgment.

Yours Cordially,

Bas Braams