Looking for Content Reviews of Standardized Assessments

(NAEP, SAT-9, CTB Terra Nova, ITBS, CTBS, STAR, TIMSS, PISA, PIRLS/IGLU; CAHSEE, FCAT, MCAS, TAAS/TAKS, NY Regents, etc.)


Introduction

At the top of the "BJB's Most Wanted" category in my web page: Links, Articles, Essays, and Opinions on K-12 Education is a request for links to content reviews of the various popular standardized school assessments. Any help and advice will be much appreciated.

The tests that I care for include national and multi-state tests such as NAEP, the Stanford-9 (SAT-9, SAT9), the CTB/McGraw-Hill Terra Nova, the Iowa Test of Basic Skills (ITBS), and the older CTBS. I also care for the tests that individual states have developed for their assessment, in any grade up to the high school exit exam. This category includes the California STAR and CAHSEE, the Texas TAAS and TAKS, the Florida FCAT, the New York Regents, and the Massachusetts MCAS. For international focus I am interested in TIMSS, PISA, and PIRLS (aka IGLU).

The kind of review that I would like to see focusses on the content of the test, describing the nature of the test items, the level of difficulty, and the level of education that is being tested. I especially like it if the article or report would provide a comparative review of several of the popular tests.

I have found very little in the way of content reviews of any of the earlier mentioned tests. In many cases the tests are treated as confidential, and sample tests may not be publicly available. The situation appears somewhat better for high school exit exams, where it is usually possible to find complete sample past exams. Even in the case of high school exit exams, however, I would like to find comparative content reviews. For the case of the SAT-9, ITBS, Terra Nova, and other such achievement tests, I can't even find a decent content description, let alone a careful comparative content review.

For the record, here are some useful sources of information that provide a partial answer to my questions.

General Information

Buros Yearbook of Mental Measurements. Richard Phelps and also the good folks at CRESST adviced me about this. It is a big fat book; the 13th edition is from 1998. It contains decriptions of hundreds (perhaps 500) different tests. Much of the review focusses on psychometrics, and there is not much of a focus on the test content.

State Education Indicators with a Focus on Title I. Report by the US Department of Ed which is useful as a reference source on standards and tests used by states. Within that report, especially this sub-page

Data from the 2000-2001 Annual Survey of State Student Assessment Programs, by Council of Chief State School Officers. A valuable reference on state assessment policy and practice.

The Marketplace for Educational Testing, by Marguerite Clarke, George Madaus, Catherine Horn, and Miguel Ramos (NBETPP, Apr 2001). A useful description of the commercial market for educational testing.

Description of Standardized Tests Used by Various Jurisdictions. Brief descriptions of the CAT, CTBS, ITBS, KIRIS, SAT/8, SAT/9, TAP, and TAAS.

Student Performance Standards of the National Assessment of Educational Progress. By NAGP, November 2000. Describes States' performance standards.

State Tests. A Web page from the National Association of Private Schools with links to State testing pages containing sample or released questions.

Assessment and Accountability Systems in the 50 States: 1999-2000, by Margaret E. Goertz, Mark C. Duffy, with Kerstin Carlson LeFloch. A CPRE Report of March 2001. Looks descriptive and comprehensive about what states are using.

The New Generation of Standardized Testing, by Christine Lynd, CER, Oct 2000. An overview of high-stakes testing in some of the States.

Organizations and Individuals to Contact for Testing Information, by Richard P. Phelps. A large list of links.

General Commentary

IllinoisLoop.Org Tests and Assessment. Many links here, including articles on the Illinois tests, the SAT, performance assessments, standardized tests and classroom grades, and other issues.

Kimberly Swygert has a good Weblog, Number 2 Pencil, with focus on testing and assessment.

Filling In the Blanks: Putting Standardized Tests to the Test, by Gregory J. Cizek (Fordham Report, Oct 1998). A primer on standardized testing. Identifies key terms and concepts, provides information about the most widely used standardized achievement tests, and discusses some controversies.

A Measure of Knowledge. Article in the American School Board Journal, February 2002, about standardized assessment.

The Tests We Need and Why We Don't Quite Have Them, by E.D. Hirsch Jr. (from EdW, Feb 2, 2000).

What is the Appropriate Role for Student Achievement Standards, by John H. Bishop (draft, 020615). The article looks at the impact of curriculum based external exit exam systems, such as the NY Regents Exams, on school policies, teaching, and learning. The article is a useful source of information, but does not offer a content review of the relevant assessments.

National Board on Educational Testing and Public Policy. See especially their Reports section. They are part of CSTEEP; upon brief perusal of their web site I do not know what to make of this organization at Boston University.

Uncensored! Secret evidence demonstrating achievement gains from testing with stakes, by Richard Phelps. A research bibliography.

NAEP, TIMSS, PISA, PIRLS/IGLU

A Comparison of NAEP, TIMSS-R, and PISA, by David Nohara. NCES Working Paper 2001-07, June 2001. Looking only at the title this would be just the kind of report that I've asked for, but it is shallow on review of content.

NAGB and NAEP publications. A collection of publications from the National Assessment Government Board. In particular the NAEP frameworks for all subjects are found here, and also achievement level reports. Comparing NAEP, TIMSS, and PISA in Mathematics and Science. A briefing paper from NCES (Dec 2004). It is based on two forthcoming (2005) reports: one by T. S. Neidorf, M. Binkley, K. Gattis, and D. Nohara on the Mathematics assessments and on by T. S. Neidorf, M. Binkley, and M. Stephens on the Science assesments.

Regarding NAEP mathematics there is the Testimony on the Draft 2004 Mathematics Framework, by John Hoven on behalf of the Center for Education Reform (Sep 24, 2001). Hoven finds that the "hard" 8th grade NAEP problems are at a level similar to Singapore's grade 5.

Regarding TIMSS there is Bill Quirk's review, Searching for Truth About the TIMSS 4th Grade Math Test.

Regarding PISA there are my content reviews: Mathematics in the OECD PISA Assessment (2002), Review of PISA Sample Science Unit 1: Stop That Germ (2004), and Review of PISA Sample Science Unit 2: Peter Cairney (2004).

For PISA and TIMSS together there is a valuable report by Alan Smithers of the British CEEB: England's Education, what can be learned by comparing countries? (May, 2004).

CTB Terra Nova; CTBS; ITBS; STAR; SAT-9

These are important tests, used directly or in customized variants in many states for standardized assessment in grades 2-8 approximately.

Comparison of the Stanford Achievement Test Series and The Iowa Test of Basic Skills, by Erika Stewart and Dr. Lutjemeier, November 12, 2001. A useful and very readable description of the two tests, also looking at content.

A Comparison of the MEAP, SAT-9, and ITBS, by Kirk A. Johnson. A Policy Brief for the Mackinac Center for Public Policy.

State-Specific Assessments

This section contains pointers to the various State testing Web sites, especially to sites where past tests or sample questions may be found, for states that develop their own tests. It would be very useful to have comparative reviews, but I am not aware of any.

The statewide high school exit exams of which I know are in each case developed by the state. Many states use some national test in the earler grades, but some states also have their own test for the grades 3-8 or so. Texas and Massachusetts both have the excellent policy to make all their past tests available. Does any other state do that? Quite a few make their high school exit exams available, but the grades 3-8 tests are often kept confidential or only sample items are released.

Texas TAAS and TAKS

Released Tests, Answer Keys, and Scoring Guides for the Texas TAAS, TAKS, and other statewide examinations.

Statewide Mathematics Assessment in Texas, by Paul Clopton, Wayne Bishop, and David Klein (1999?). From the abstract: "This report evaluates the mathematics assessments in use in Texas - the Texas Assessment of Academic Skills (TAAS) mathematics exams and the end-of-course algebra 1 exams - with respect to their mathematics content and grade level expectations as a way of understanding their impact on mathematics achievement in Texas. Several means of evaluating these assessment tools suggest that only low levels of achievement are being measured. Indeed, the high school exit exam seems more appropriate to sixth-grade achievement"... [more]

Adding It All Up: What does it really mean to pass the TAAS?, by Shaila Dewan (Houston Press, March 4, 1999). Dewan describes many independent reviews of TAAS. She concludes: "Instead of learning algebra and geometry, high school students review eighth-grade math. Teachers go to special TAAS training sessions; administrators check practice tests, buy test strategy books and spend their summers coming up with plan after plan designed specifically to raise scores. Bonuses and careers hang on the outcome of the TAAS - not students' careers, but educators' careers. And that, perhaps, is the shameful part: not so much that children can't pass the test (although that's distressing enough), but that so much effort, time, energy and money is poured into preparing for a test so easy that when a student does succeed on the TAAS, it means hardly anything at all"... [more]

Texas Mathematics Education In Transition, by Paul Clopton (2000). The report addresses the Texas Assessment of Academic Skills (TAAS) as it was from 1995 to 1998, the transition to the more recent version tied to the Texas Essential Knowledge and Skills (TEKS), and the future TAAS II. From the conclusion: "There is every indication that an emphasis on minimal competence will continue. It will take a concerted effort on the part of all involved to incorporate a broader range of achievement into the sensitivity range of the new exams. Without doing so, Texas will continue to be blind to higher achievement levels"... [more]

California CAHSEE

California High School Exit Examination (CAHSEE). The main California Department of Education (CDE) Web page for the California High School Exit Exam. There are only some sample questions from recent tests; not complete previous tests.

Recent CST Released Test Questions. Sample test questions for all subjects and all grades.

Demystifying the CA High School Exit Exam, by the Education Committee of the Los Altos - Mountain View League of Women Voters. This well-organized site offers information about the high school exit exam, the content standards and framework on which it is based, the CA instructional materials adoption process, the earlier grades 2-11 STAR assessment, and California's API school ratings.

Florida FCAT

Florida's Comprehensive Assessment Test. This Web site of the Florida Department of Education gives links to sample test items for grades 3-10. I understand that the grade 10 tests do duty as well as a high school exit requirement.

New York State grades 3-8 testing

New York State Assessment. General testing information and links to sample tests for all subjects. For mathematics there are some complete actual tests here.

New York Regents

New York State Regents Examinations. Past tests and scoring keys for all recent exams.

New York State Regents Exams from the New York State Library. Exams in all subjects going back to 1957.

Critique of the New York Regents Math A Exam (June, 2003), by Bas Braams. The Regents Math A exam that was given on June 17, 2003, has received much negative press attention. Here is a review of the latest exam and also of the Aug 2002 and Jan 2003 instances.

Massachusetts MCAS

Massachusetts Comprehensive Assessment System Test Items. This Web page provides a complete release of all past tests.

Illinois ISAT

Illinois Standards Achievement Test (ISAT). This is the main ISAT Web page of the Illinois State Board of Education. There are links to to some sample questions, but I don't believe that any complete past tests have been released.

The following links and more, including annotations, are found on the Illinois Loop Tests and Assessment page.

What Do State Tests Measure? A cautionary tale from Illinois by Dawn Earl.

ISAT Tests: Worth the time? Worth the Money? by Dawn Earl, Education Policy Analyst, IFI.

Standardized Writing Assessments May Be Harmful To Children's Learning, University of Chicago Chronicle, July 11, 2002, Volume 21, No. 18.

Kentucky KIRIS and CATS

Kentucky Department of Education Accountability System. CATS is the (KY) Commonwealth Accountability Testing System. KIRIS is the earlier Kentucky Instructional Results Information System. Testing in Kentucky provides a glossy and very brief description for parents.

Kentucky Education Data. Test results and other indicators in tabular and graphical forms, compiled and edited by Richard G. Innes, a critical observer.

Kera Update. A newsletter (74 issues between 1995 and Sep 2003) from Richard G. Innes with focus on the Kentucky Education Reform Act and the associated testing and accountability systems.

Miscellaneous related works

IGOs and Stanford Testing in Preston County (WV) Schools, by Timothy C. Miller, M.D., April 10, 2000. The author is a member of the local Board of Education.

Conclusion

As I said at the top, it all doesn't add up to an authoritative set of content reviews of the most popular tests. Further pointers will be much appreciated.

There seems to be a real problem, though, in that perhaps content reviews are truly not possible because of confidentially. I would think it scandalous if it were impossible, and not merely difficult, to arrange for academic researchers to perform comparative content reviews of states' tests. But maybe it really cannot be done.

Testing the Testers reports on what was supposed to be a comparative review of all the states' assessment systems, by the Princeton Review company. They quote Daniel M. Koretz as saying the study doesn't deserve any credence because the company never reviewed the tests themselves or their psychometric properties. Could they have reviewed the tests if they had wanted to?

Bastiaan J. Braams
Courant Institute of Mathematical Sciences
New York University
251 Mercer Street
New York, NY 10012-1185

Email: braams@math.nyu.edu