OECD PISA: Programme for International Student Assessment

The results of the first OECD PISA study were released in December, 2001. There was lots of reporting in the press around that time, in the United States and also in Europe. Remarkably, nothing that I ever saw in the press paid any attention to the content of the PISA exam. The press reports were hand-wringing (Germany), neutral (USA), or smug (Netherlands), but devoid of content analysis. (Note that the Netherlands does not occur in the official tabulations, as its sampling did not meet the PISA rules.)

The following is a condensation of three postings that I contributed on Jan 3, 2002, to the NYC HOLD and Kto15 mailing lists about the content of the PISA exam. In the postings I said I would undertake a more interpretative essay later, but I did not get around to it until several months later. That more interpretive essay is Mathematics in the OECD PISA Assessment. Please see the Education Page of Bas Braams - Links, Articles, Essays, and Opinions on K-12 Education - for related matter.

[Addendum, Dec 2004: On the occasion of the release of the results from the 2003 PISA Assessment I've reviewed PISA Scientific Literacy questions from the 2003 Assessment Framework. See: Review of PISA Sample Science Unit 1: Stop That Germ and Review of PISA Sample Science Unit 2: Peter Cairney. My conclusion based on review of PISA sample mathematics and science questions is that the PISA project cannot be taken seriously as an attempt to study the quality of educational systems.]


From braams Thu Jan 3 10:28:32 2002
Subject: PISA language assessment sample

Dear All,

The PISA (Programme on International Student Assessment) study, by the OECD, has been in the news. One can read more about PISA and find sample questions and their scoring guidelines by following the links from the PISA home page, www.pisa.oecd.org.

Only a rather small set of sample questions are posted there. Here I reproduce two questions from the language assessment. I've selected these two based on two criteria. First, the text on the which the questions are based is rather short. Second, the question is of the "constructed response" type. I interpret the "feedback" following each question as the scoring guideline.

I defer my commentary to a separate posting. Here follow the two questions. Text bracketed by [...--BJB] is mine.

[Preamble for a sequence of questions, only one of which is reproduced here --BJB]

This letter appeared in a newspaper in 1997. Refer to the letter to answer the questions that follow.

IN POOR TASTE

from Arnold Jago

Did you know that in 1996 we spent almost the same amount on chocolate as our government spent on overseas aid to help the poor?

Could there be something wrong with our priorities?

What are you going to do about it?

Yes, you.

Arnold Jago,

Mildura

Source: The Age Tuesday 1 April 1997

Question 6: The Chocolate Letter

What kind of response or action do you think Arnold Jago would like his letter to prompt?

Feedback for Question 6

This question is not marked automatically.

To obtain full credit, the answer must either provide a statement or sentence indicating that the government or individuals should spend more on (overseas) aid.

Example answers include: - People donate more money to overseas aid. - Donate money to charities. - People should spend less on chocolate and more on the poor.

or provide a statement or sentence indicating that Government or individuals should change their priorities or awareness. - Change our priorities. - He would like people to raise their awareness about how we spend our resources.

Question 19: Warranty

[This question is one of a sequence relating to a warranty receipt. I don't reproduce the receipt here, as it is not relevant to this particular question. --BJB]

The words "Thank you for your business" are printed on the bottom of the receipt. One possible reason for this is simply to be polite. What is another possible reason?

Feedback for Question 19

This question is not marked automatically.

A correct answer would refer either explicitly or implicitly to development of the business-customer relationship. Example responses include: - It's good for business to be nice to you. - To create a good relationship with the customer. - They want you to come back.

From braams Thu Jan 3 10:52:24 2002
Subject: PISA mathematics assessment sample

Here follow two sample mathematics questions from the OECD PISA study (Programme on International Student Assessment), from their web site at www.pisa.oecd.org . Only a few sample questions are posted there, and most of them are of the constructed response variety. I interpret the "feedback" following each question as the scoring guideline.

[The two questions, questions 5 and 6 of the sample, refer to a drawing of three shapes in the plane. The shapes are labeled (A), (B) and (C). Figure B is a circle. Figures A and C look like squid; these two figures have each a very jagged edge, with lots of indents, and their overall size is such that they would just fit inside or on the circle, figure B. I don't think I'm giving much away if I say that figures A and C have very obviously smaller area and larger circumference than figure B. --BJB]

Question 5: Shapes

Which of the figures has the largest area? Give explanations for your answer.

Feedback for Question 5

This question is not marked automatically.

To obtain full credit the answer B, must be supported with plausible reasoning.

B. It doesn't have indents in it which decreases the area. A and C have gaps.

B, because it's a full circle, and the others are like circles with bits taken out.

B, because it has no open areas.

Question 6: Areas

Describe a method for estimating the area of figure C.

Feedback for Question 6

This question is not marked automatically.

To obtain full credit a reasonable method must be given.

Draw a grid of squares over the shape and count the squares that are more than half filled by the shape.

Cut the arms off the shape and rearrange the pieces so that they fill a square then measure the side of the square.

Build a 3D model based on the shape and fill it with water. Measure the amount of water used and the depth of the water in the model. Derive the area from the information.

You could fill the shape with lots of circles, squares and other basic shapes so there is not a gap. Work out the area of all of the shapes and add together.

Redraw the shape onto graph paper and count all of the squares it takes up.

Drawing and counting equal size boxes. Smaller boxes = better accuracy (Here the student's description is brief, but we will be lenient about student's writing skills and regard the method offered by the student as correct)

Bas
--
Bastiaan J. Braams - braams@math.nyu.edu
Courant Institute, New York University
251 Mercer Street, New York, NY 10012-1185


Dear All,

Earlier this morning I posted sample questions for reading and mathematics from The OECD PISA study (Programme on International Student Assessment). Here is a description of the general nature of PISA and then, specifically, the focus of the mathematics assessment component. It is all taken from the PISA web site, http://www.pisa.oecd.org . I'm posting this without editorial commentary, as I have not yet found quite the right words.

Bas

What PISA assesses

Skills and knowledge that prepare students for life, and for lifelong learning

PISA is based on a dynamic model of lifelong learning in which new knowledge and skills necessary for successful adaptation to a changing world are continuously acquired throughout life.

Previous international assessments have concentrated on "school" knowledge. PISA aims at measuring how well students perform beyond the school curriculum.

Although the first assessment domains (reading literacy, mathematical literacy and scientific literacy) are closely related to subjects learned at school, PISA concentrates on the value of the skills acquired, beyond the school gates through applying literacy in a broader sense. It tests performance in real-life situations.

Competencies across disciplinary boundaries have a growing importance in PISA as it develops over time. PISA 2000 analysed the students' approaches to learning and beliefs in their own abilities, motivation and engagement and other aspects of student attitudes, under the heading "self-regulated learning". In 2003, PISA will also specifically assess students' ability to solve problems.

Mathematics Literacy

"Mathematics literacy is an individual?s capacity to identify and understand the role that mathematics plays in the world, to make well-founded mathematical judgements and to engage in mathematics, in ways that meet the needs of that individual?s current and future life as a constructive, concerned and reflective citizen."

Mathematical literacy entails the use of mathematical competencies at several levels, ranging from performance of standard mathematical operations to mathematical thinking and insight.

It also requires the knowledge and application of a range of mathematical content that is drawn from areas such as chance, change and growth, space and shape, quantitative reasoning, uncertainty and dependency relationships. This includes specific areas of the mathematics curriculum, such as algebra, numbers and geometry.

PISA will assess mathematical literacy in three dimensions:

First, the content of mathematics, as defined mainly in terms of broad mathematical concepts underlying mathematical thinking (such as chance, change and growth, space and shape, reasoning, uncertainty and dependency relationships), and only secondarily in relation to "curricular strands" (such as numbers, algebra and geometry). The PISA 2000 assessment, in which mathematics is a minor domain, will focus on two concepts: change and growth, and space and shape. These two domains will allow a wide representation of aspects of the curriculum without giving undue weight to number skills.

Second, the process of mathematics as defined by general mathematical competencies. These include the use of mathematical language, modelling and problem-solving skills. The idea is not, however, to separate out such skills in different test items, since it is assumed that a range of competencies will be needed to perform any given mathematical task. Rather, questions are organised in terms of three "competency classes" defining the type of thinking skill needed: The first class of mathematical competency consists of simple computations or definitions of the type most familiar in conventional mathematics assessments, The second class requires connections to be made to solve straightforward problems, The third competency class consists of mathematical thinking, generalisation and insight, and requires students to engage in analysis, to identify the mathematical elements in a situation and to pose their own problems.

Third, the situations in which mathematics is used, ranging from private contexts to those relating to wider scientific and public issues.

Bas
--
Bastiaan J. Braams - braams@math.nyu.edu
Courant Institute, New York University
251 Mercer Street, New York, NY 10012-1185


Addendum, December 2004. I point the reader again to my reviews of some PISA sample items: Mathematics in the OECD PISA Assessment; Review of PISA Sample Science Unit 1: Stop That Germ; and Review of PISA Sample Science Unit 2: Peter Cairney. In 2003 I moved to Emory University. Current address:

Department of Mathematics and Computer Science
Emory University
Mathematics and Science Center Suite W401
400 Dowman Drive
Atlanta, GA 30322