Review of PISA Sample Science Unit 2: Peter Cairney

By Bas Braams, November 29, 2004

The OECD PISA 2003 Assessment Framework provides sample questions for each of the four domains. In a companion article I reviewed the Sample Science Unit 1 and found plenty wrong with it. In the present contribution we look at Science Unit 2, and it turns out to be even worse than Unit 1. Question 2.2 in this PISA unit is even of an offensively anti-science nature.

Please note in addition my review of Science Unit 3: Corn from the PISA 2003 Framework. Unit 3 is well-intentioned and much better than units 1 and 2, but seriously flawed just the same.

Science Unit 2: Peter Cairney


The following four items are part of a unit for which the stimulus material is a passage about Peter Cairney, who works for the Australian Road Research Board. The stimulus material is presented below.

...Another way that Peter gathers information to improve road safety is by the use of a TV camera on a 13 metre pole to film the traffic on a narrow road. The pictures tell the researchers such things as how fast the travel is going, how far apart the cars travel, and what part of the road the traffic uses. Then after a time, lanes are painted on the road. The researchers can then use the TV camera to see whether traffic is now different. Does the traffic now go faster or slower? Are the cars close together or further apart than before? Do the motorists drive closer to the edge of the road or closer to the centre now that the lines are there? When Peter knows these things he can give advice about whether or not to paint lines on narrow roads.

Science Example 2.1

(Item type: complex multiple choice)

If Peter wants to be sure that he is giving good advice, he might collect some other information as well beyond filming the narrow road. Which of these things would help him to be more sure about his advice concerning the effect of painting lines on narrow roads?

A. (Yes/No) Doing the same on other narrow roads.

B. (Yes/No) Doing the same on wide roads.

C. (Yes/No) Checking the number of accidents in a certain time period before and after painting the lines.

D. (Yes/No) Checking the number of cars using the road before and after painting the lines.

Scoring and comments on Science Example 2.1

Full Credit: Answers that specify Yes, No, Yes, No, in that order.

Partial Credit: Answers that specify Yes, No, No, No, in that order.

No credit: Any other combination of answers.

Science Example 2.2

(Item type: open constructed response)

Suppose that on one stretch of narrow road Peter finds that after the lane lines are painted the traffic changes as below.

Speed: Traffic moves more quickly.

Position: Traffic keeps nearer edges of road.

Distance apart: No change.

On the basis of these results it was decided that lane lines should be painted on all narrow roads. Do you think this was the best decision? Give your reasons for agreeing or disagreeing.

Scoring and comments on Science Example 2.2

Full Credit: Answers that agree or disagree with the decision for reasons that are consistent with the given information. For example:

No Credit: Answers that agree or disagree without specifying the reasons, or provide reasons unrelated to the problem.

Science Example 2.3

(Item type: open constructed response)

Drivers are advised to leave more space between their vehicles and the ones in front when they are travelling more quickly than when they are travelling more slowly because faster case take longer to stop.

Explain why a faster car can take more distance to stop than a slower one.

Scoring and comments on Science Example 2.3

Full credit: Answers that mention that:

Partial Credit: Answers that mention only one of the above points.

No Credit: Other responses, or repetition of the statement, e.g. that it takes longer to stop because of its speed.

Science Example 2.4

(Item type: multiple choice)

Watching his TV, Peter sees one car A travelling at 45 km/h being overtaken by another car B travelling at 60 km/h. How fast does car B appear to be travelling to someone in car A?

Scoring and comments on Science Example 2.4

Full credit: Response B: 15 km/h

No Credit: Other responses.

(end of Science Unit 2 from the PISA 2003 Assessment Framework)

Comments on Science Unit 2

As I wrote in my Review of PISA Sample Science Unit 1, the PISA 2003 framework does not provide sample student answers and their scores, so we cannot judge how these scoring guidelines would be applied in practice when students hand in their long, rambling, badly worded answers. But just as with Science Unit 1 there is plenty wrong with the questions and answers as given.

Question 2.1: What other information? I would say (A), the same on other narrow roads; (C), the number of accidents before and after; and (D), checking road use before and after. The scoring guidelines want to see (A) and (C) but not (D). But (D) is important. I don't really expect that painting lines on a road will change the intensity of traffic, but there may have been extraneous circumstances of which the researcher is unaware that caused such a change. Maybe some other road is under repair, or opened up, or lights were installed somewhere else, or whatever. If the intensity of traffic on the experimental narrow road has changed then this messes up the measurements and the researcher should want to be aware of it. Besides, the information comes essentially for free if he has the films anyway.

Question 2.2. This is a real howler. We are told that traffic moves faster and retains the same distance apart, and the PISA staff appears to be oblivious to the consequent that apparently more cars are using the road. Something fishy is going on, and we had better understand the reasons for the higher traffic intensity before drawing conclusions.

Question 2.2 is not only a howler, it is an offensive anti-science question. Forget about the issue of the apparent increase in traffic. The student is given some information that is quite clearly insufficient for a firm advice one way or another, and the PISA staff admits as much by allowing full credit for advice either way provided the reasons are consistent. It is offensive that the student who understands that his information is inadequate is forced into a choice one way or another. The PISA staff is testing whether the student can wage an argument, not whether he can behave as a scientist.

Question 2.3. The answer key suggests that they want long, rambling answers. A concise answer at the level of a competent 15-year old would offer their second reason: it takes longer to reduce speed to zero from a greater speed, so the car will travel further. Their first suggested answer is contained in this one and should not be required as an independent contribution for full credit. Of course, some students can do better than the PISA staff: The total stopping process involves a reaction time followed by the actual breaking time. Reaction times being equal, during that time the faster car will travel further. And during the actual breaking time, the faster car has to cover the additional distance that it needs to slow down to the initial speed of the slower one. I suppose this would earn only partial credit, as the magic word "momentum" is not used.

Question 2.4. How fast does car B appear to be travelling to someone in car A? Why bring psychology into this? If I'm in a car travelling at 45 km/h being overtaken by a car travelling 60 km/h then that other car will appear to me to be travelling at about 60 km/h. Its relative velocity I might estimate at 15 km/h, but the question doesn't ask for that. And even the false answer 105 km/h should perhaps not be held against a student: it probably indicates that the student is not a native speaker and is unsure about the meaning of "to overtake".

I plan to add links to other reviews of PISA sample items and released items here in the future.

(Return to Links, Articles, Essays, and Opinions on K-12 Education or to BJB Essays and Opinions or to New York City HOLD.)