Review of PISA Sample Science Unit 1: Stop That Germ

By Bas Braams, November 29, 2004

PISA is the Programme for International Student Assessment, done by the OECD. It involves testing of 15-year olds in about 30 countries in the domains of Reading Literacy, Mathematical Literacy, Scientific Literacy, and (starting in 2003) Problem Solving. The test is carried out every three years starting in 2000. Reading was the principal domain in 2000, Mathematical Literacy in 2003, and it will be Scientific Literacy in 2006, but all domains are tested each time.

When the results of PISA 2000 were released by the OECD in December 2001 it garnered much attention in Europe - and something approaching mass hysteria in Germany, whose pupils had not done well on the test. Absent from all the reporting was a critical look at the content of the test. I provided a critical review, Mathematics in the OECD PISA Assessment, and it rests in peace on the Web.

Now that the results of PISA 2003 are in the press it is time to look again at the content of the exam. I intend to do so in a piecemeal fashion as time permits, starting with some of the sample questions provided in the PISA 2003 Assessment Framework, and focussing at first on the Science component. There are three sample Science Units in the Framework, each Unit being composed of several questions. The present contribution concerns Unit 1, which is on pp. 141-142 of the Framework.

[Addendum: Please see also my reviews of Science Unit 2: Peter Cairney and Science Unit 3: Corn from the PISA 2003 Framework.]

Science Unit 1: Stop That Germ.


As early as the 11th century, Chinese doctors were manipulating the immune system. By blowing pulverised scabs from a smallpox victim into their patients' nostrils, they could often induce a mild case of the disease that prevented a more severe onslaught later on. In the 1700s, people rubbed their skins with dried scabs to protect themselves from the disease. These primitive practices were introduced into England and the American colonies. In 1771 and 1772, during a smallpox epidemic, a Boston doctor named Zabdiel Boylston tested an idea that he had. He scratched the skin on his six-year-old son and 285 other people and rubbed pus from smallpox scabs into the wounds. All but six of his patients survived.

Science Example 1.1

(Item type: open constructed response)

What idea might Zabdiel Boylston have been testing?

Scoring and comments on Science Example 1.1

Full Credit: Answers with reference both to

Partial Credit: Answers which refer to either of the above points.

No Credit: Other responses.

Science Example 1.2

(Item type: open constructed response)

Give two other pieces of information that you would need to decide how successful Boylston's approach was.

Scoring and comments on Science Example 1.2

Full Credit: Answers that provide the following TWO pieces of information:

Partial Credit: Answers that provide either of the above points.

No Credit: Other answers.

(end of Science Unit 1 from the PISA 2003 Assessment Framework)

Comments on Science Unit 1

The PISA 2003 Framework does not provide sample student answers and their scores, so we have no idea how these scoring guidelines would be applied in practice. Many students will hand in long rambling answers in which, with good will, one may recognize the requested items, among much else. How the PISA staff thinks they are going to achieve uniformity in scoring with this kind of open question over 30 countries is a mystery to me, but that is only a part of all that is wrong with this question.

Question 1.1 - What idea might Zabdiel Boylston have been testing? I would say he might have been testing the idea that his treatment provided some protection against smallpox - either against getting the disease in the first place or against the severity of it if one did get the disease. The question doesn't ask for more than one idea, and in any case I don't like to try to get into Boylston's mind too much, so I would leave it at that.

That answer would presumably be awarded partial credit, because the PISA staff wants to hear also that Boylston might have been testing the idea that by breaking the skin, the smallpox was introduced into the bloodstream. Give me a break! Boylston, in 1771/1772, was going to pull out his electron beam microscope and look for the smallpox virus in the bloodstream? Boylston may of course have philosophized that the blood plays some role in transmitting either the disease or some agent opposing the disease through the body, but it would have been pure speculation and outside the reach of his scientific inquiry. For the purpose of this exam we should credit Boylston with asking only questions that were within his scientific reach.

Even the first answer offered by the PISA staff is poor, I think, although one should give full marks for it if it is the answer of a 15-year-old. It is poor because Boylston is a pioneer in immunization and is working without the benefit of a microbial theory of disease. He can't know if he is "infecting someone with smallpox". Maybe he is, as far as he knows, but maybe he is introducing into his subjects some agent that causes only some symptoms of the disease, or maybe some agent opposing the disease, or maybe more than one of the above. Fortunately he doesn't need to know what is the case - he just has to look at the results.

Question 1.2 - Two other pieces of information that we would need to decide how successful Boylston's approach was. There is one piece of information that I absolutely need and several other items that would be just very good to know. I definitely need to know the death rate due to smallpox in the general Boston population or in some other control group during the epidemic. With regard to the six that died, I want confirmation that it is meant that they died of smallpox during that epidemic; otherwise they all died. Then I want to know how many of the subjects got the disease and recovered, and I want to know the corresponding rate for the general population in Boston during that same time. For the six subjects that died I also like to know a timeline: did they develop the disease and die soon after the treatment, in which case perhaps Boylston overdosed them or perhaps they had acquired the disease already before the treatment; or did they develop the disease later in the general course of the epidemic, in which case the treatment apparently failed to protect them? The same timeline information would be valuable with regard to any of Boylston's subjects that developed smallpox and survived.

This answer too would have gotten partial credit, because I don't say we need to know whether the patients were exposed to smallpox apart from the treatment. Well, that is a can of worms. We might like to know if our subjects were exposed to smallpox, but no-one is going to tell us and we're never going to find out, so there is no point in asking for this information. At a more sophisticated level information related to exposure would be valuable (were any close relatives infected?), but it requires a lot more than just one or two items of information to analyze this and it doesn't cross my mind when I'm putting myself in Boylston's or his contemporaries' shoes, or in those of the PISA pupil. We will just have to assume that, except for the treatment, the subjects were exposed to smallpox at the same intensity as the general population.

There are also grave, inexcusable technical errors in the wording of this item. In question 1.1 they ask "What idea", but the scoring guidelines want to see two ideas. And in question 1.2 they ask for two pieces of information, but we really shouldn't have to go through a field trial to discover that many good students will want to know (1) the size of the Boston population at the time and (2) the number of deaths due to smallpox during the epidemic - a fine answer that is not captured by the scoring guidelines.

The misery of this PISA sample science item doesn't end with the bad questions and answers and the apparent inability of the PISA science staff to think like a competent scientist in the given context. They even got the story wrong! Boylston did his experiment in 1721 and not 1771. See Zabdiel Boylston and Smallpox Inoculation, by Ira M. Rutkow. Another Web reference dishes up the 1771 year, but they have the epidemic in 1721, they later correctly place Boylston's experiments in 1721, and they have Boylston's year of death as 1766, so the misprint would have been noticed by a careful reader.


I may add links to other reviews of PISA sample items and released items here in the future. Please note already my reviews of Science Unit 2: Peter Cairney and Science Unit 3: Corn, also from the PISA 2003 Framework. Note also England's Education, what can be learned by comparing countries?, by Alan Smithers (May, 2004), which includes a valuable comparison of the nature of science and mathematics items in TIMSS 1999 and PISA 2000.

(Return to Links, Articles, Essays, and Opinions on K-12 Education or to BJB Essays and Opinions or to New York City HOLD.)