Following up on reviews of the PISA 2003 Sample Science Unit 1 and Sample Science Unit 2 I now look at the third and final science unit that is provided in the OECD PISA 2003 Assessment Framework. Whereas PISA Science Units 1 and 2 each have multiple egregious errors and Unit 2 has an offensive anti-science component as well, the questions in this third sample science unit are at least well-intentioned, and of the three questions that make up the unit only the first one is severely flawed. I also take issue with the verbosity of the presentation.
The following three items are from a unit entitled Corn. The stimulus material is a newspaper report about a man, Auke Ferwerda, who burns corn on his stove as a fuel.
...Ferwerda points out that corn, in the form of cattle food, is in fact a type of fuel too. Cows eat corn to get energy out of it. But, Ferwerda explains, the sale of corn for fuel instead of for cattle food might be much more profitable for farmers.
Ferwerda knows the environment is receiving increasing attention and government legislation to protect the environment is becoming increasingly elaborate. What Ferwerda does not quite understand is the amount of attention being focused on carbon dioxide. Carbon dioxide is regarded as the cause of the greenhouse effect. The greenhouse effect is said to be the main cause of the increasing average temperature of the Earth's atmosphere. In Ferwerda's view, however, there is nothing wrong with carbon dioxide. On the contrary, he argues, plants and trees absorb it and convert it into oxygen for human beings.
He says: ``This is an agricultural area and the farmers grow corn. It has a long growing season, absorbs a lot of carbon dioxide and emits a lot of oxygen. There are many scientists who say that carbon dioxide is not the main cause of the greenhouse effect''.
(Item type: complex multiple choice)
Ferwerda compares corn used as fuel to corn used as food.
The first column of the table below contains a list of things that happen when corn burns as fuel.
Do these things also happen when corn acts as fuel in an animal body?
Circle Yes or No for each.
|When corn burns:||Does this also happen when corn acts as fuel in an animal body?|
Oxygen is consumed.
|Carbon dioxide is produced.||Yes/No|
|Energy is produced.||Yes/No|
Scoring and comments on Science Example 3.1
Full Credit: Answers that specify Yes, Yes, Yes, in that order. (All parts have to be answered correctly, since any one error would indicate some failure in understanding the process of using food in an animal body).
No Credit: Answers which specify any other combination of responses.
(Item type: Open constructed response)
In the article a conversion of carbon dioxide is described: ``...plants and trees absorb it and convert it into oxygen ...''.
There are more substances involved in the conversion than carbon dioxide and oxygen only. The conversion can be represented in the following way:
|carbon dioxide + water --> oxygen + [box]|
Write in the box the name of the missing substance.
Scoring and comments on Science Example 3.2
Full Credit: Answers that mention any of the following: glucose; sugar; carbohydrate(s); saccharide(s); starch.
No Credit: Other responses.
(Item type: multiple choice)
At the end of the article Ferwerda refers to scientists who say that carbon dioxide is not the main cause of the greenhouse effect.
Karin finds the following table in which research results about the four most important gases causing the greenhouse effect are listed.
|Carbon dioxide||Methane||Nitrous oxide||Chlorofluorocarbons|
From this table Karin concludes that carbon dioxide is not the main cause of the greenhouse effect. However, this conclusion is premature. The data in the table need to be combined with other data to be able to conclude whether or not carbon dioxide is the main cause of the greenhouse effect.
Which other data does Karin need to collect?
A. Data about the origin of the four gases.
B. Data about the absorption of the four gases by plants.
C. Data about the size of each of the four types of molecules.
D. Data about the amounts of each of the four gases in the atmosphere.
Scoring and comments on Science Example 3.3
There is a close relationship between the scientific knowledge that the concentration of a substance affects the extent of its action, and the recognition that a valid conclusion cannot be drawn without this extra information.
Full Credit: Response D: Data about the amounts of each of the four gases in the atmosphere.
No Credit: Other responses.
(end of Science Unit 3 from the PISA 2003 Assessment Framework)
At the outset let us agree that the questions appear well-intentioned and suitable in principle to measure knowledge and skills that plenty of 15-year-olds will have had the opportunity to acquire in their biology or general science studies. I have quarrels with example 3.1, and further objections to the verbosity of it all.
With regard to science example 3.1, comparing corn burning as fuel to corn acting as fuel in an animal body, my answer would indeed be (Yes, Yes, Yes) as requested by the PISA staff, but I would have misgivings and would like to answer the question "under protest".
My first, minor, misgiving is that Oxygen is an ambiguous term - it can mean the molecular or the atomic substance. It is only indirectly through context that we can understand that the molecular substance is meant here, and then we must agree that it is consumed. Oxygen atoms, of course, are preserved no matter how the corn is burned. So, on the left hand side of the table I would have preferred to see "Oxygen (O2) is consumed" and then, for stylistic consistency, "Carbon dioxide (CO2) is produced".
My major quarrel with science example 3.1 is the third item, ``Energy is produced''. Colloquially, yes, that may be said to happen when fuel is burned. But in this question we are looking at the chemistry and then it is really better to say that energy is converted or is released. Moreover, whatever verb we choose, there are profound differences between the two uses of the corn. When fuel burns, the energy is released in the form of heat and radiation. When it is consumed in an animal then some energy comes out as heat, but that is a side-show: the principal issue is that chemical energy stored in the fuel is converted into chemical energy stored in ATP. With regard to that conversion process the colloquialism "energy is produced" is really not appropriate. A student who understands the physics and biochemistry of it all fully well may find that the straightforward (Yes, Yes, Yes) answer does not do justice to the profound differences in the nature of the energy release or conversion. Certainly it would have been better to elicit the student understanding of that third point, the energy budget, via a separate question.
The content of questions and answers 3.2 and 3.3 is fine, I think. Not fine in these questions, or in question 3.1 or in the other PISA Science Units that I've seen, is the immense wordiness of the presentation. The whole introduction via farmer Auke Ferwerda is irrelevant: questions 3.1 - 3.3 could all be asked in a straighforward manner without this human interest story. In question 3.3 the PISA authors are not satisfied with the level of authenticity brought by farmer Ferwerda and the scientists who say that carbon dioxide is not the main cause of the greenhouse effect; they need to introduce Karin as well for the pupils to identify with. It represents an extreme case of a philosophy of assessment in which one cannot ask a straightforward question "Q", but instead one must always embed it in some story. In some attempted authentic assessments the embedding can be as trivial as: "Bob asks Alice 'Q'. What should Alice tell him?", but obviously the PISA staff operate at a higher level of sophistication in their view of what constitutes authenticity.
How harmful is this verbosity, or is it perhaps a positive feature? I don't know the psychometric literature enough to have an authoritative answer, but I'm very sceptical of the value of the wordiness in the PISA assessments of scientific and mathematical literacy. Certainly it mixes up a straightforward assessment of scientific or mathematical competence with an assessment of language skills, which in PISA are already extensively assessed in the Reading Literacy and Problem Solving domains. This severe dilution of the science and mathematics components with language skill components makes PISA, I think, a much more nearly one-dimensional and a less valuable assessment of performance of an educational and social system than it could have been. It makes it entirely inappropriate to draw from the results of PISA far-reaching conclusions about restructurings needed in a country's educational system. PISA provides primarily a view of pupils' language skills with focus on the lower levels of ability. PISA is all but irrelevant for a more refined assessment of the quality of an educational system.
I plan to add links to other reviews of PISA sample items and released items here in the future.
(Return to Links, Articles, Essays, and Opinions on K-12 Education or to BJB Essays and Opinions or to New York City HOLD.)