Flaws In The New School Tests by Andrew Beveridge, Jun 10, 2004 TwitterHigh-stakes testing of schoolchildren has been causing high anxiety not just among students and their parents, but also for teachers, principals, superintendents, even public officials - all of whose fates, in a very real way, are now determined in part by these tests. Some of the latest headlines warned that, thanks to the mayor's new policy to end "social promotion," the results of the tests for third graders in New York City public schools indicated that some 10,000 of them may be forced to repeat third grade next year. As more provisions of the No Child Left Behind Act take effect, more and more such testing will occur. Is it any wonder that the press has started to publish charts that track these scores in every single school? But what do these tests actually measure? The truth is something that those who produce the data from the tests admit only privately: The entire process is seriously flawed. Even sustained increases over time may tell us much less about the academic performance of a student or a school than about the testing itself. Educational researchers know that there are better ways to assess the progress of both individual students and entire school systems than the process now mandated by New York. The current testing process is fundamentally flawed in at least three ways. UNRELIABLE TEST RESULTS Fundamental to the use of these tests is the assumption that they measure the same set of skills and knowledge from year to year, and for the same sort of students. But in fact a score from one year may not be equivalent to the same score the following year. It is impossible to know whether a change in a test score from one year to the next is due to a change in actual student performance or to an unintended change in the make-up of the test. Obviously, one cannot use the same questions from year to year, and even though much effort goes into making sure that the test given in 2004 yields the same standards as the test form in 2003, it is a difficult task to be absolutely sure this is so. TEACHING TO THE TEST No one who is working in any of the hard pressed urban schools in New York State can doubt the advantages of getting higher test scores on the English and Language Arts (ELA) and mathematics test. If a school does not improve, it may find itself on one of the several "bad school" lists prepared by the State Education Department. The school may be reconfigured; and the principal may lose his or her job or be reassigned. So, once the new test was established in 1999, it is not surprising that districts and schools realigned their curriculum. However, as with any standardized test, there are specific things about the test itself that one needs to learn to do well on it. James Traub's article "The Test Mess," in the April 7, 2002 New York Times Magazine follows the efforts (or lack of efforts) of three districts to prepare for the exam. Such test prep can work, but it costs time, effort and resources for drill and practice. Now with the No Child Left Behind Act, there are complaints that schools are sacrificing additional parts of the curriculum to prepare the students for the test material only. Companies now market test prep approaches to school districts. State grants, in part based upon federal money are now available for supplementary educational services that include test preparation. No matter how effective such programs are, they certainly must displace time and resources that could be used for other classroom activities, so improved scores may come at the expense of other learning. PLAYING GAMES WITH TEST SCORES Some commentators noted that Mayor Michael Bloomberg's efforts to end social promotion starting with third graders should lead immediately to an improvement in fourth grade test results next year, since many of the low-performing students will no longer be slated to take the test. Such "gaming" of high stakes test standards is rampant throughout the US. (See my August Column on counting dropouts) A host of such approaches are being used to make the results look better, from suggesting the poor-achieving students stay home, to reclassifying them as special education or English As A Second Language students. One year New York City suppressed the test results for seventh grade, when they indicated a drop, after trumpeting them the year before when they indicated an increase. New York City is not alone. A national study found evidence of just this sort of Enron-style accounting among school systems throughout the United States. BETTER METHODS OF ASSESSMENT AND TRACKING To assess educational progress at the state or large city level, the United States Department of Education developed the National Assessment of Educational Process. It is a test based upon knowledge determined to be needed at various grade levels. This is not a high stakes test, since it is only given to a sample of students, and the results are used to track overall educational performance change and have no consequences for a specific school or student. One does not learn anything about the effect of schools on individual students from the current testing regime, nor would one from a test like the National Assessment of Education Process. An alternative approach that overcomes many of these difficulties is the so-called "value added" approach. One tracks the gains for each particular student, not the raw test score averages for each school, district, county or state. In this way one can relate the student gains to specific schools, teachers and educational practices. This testing approach was mandated as a remedy to civil rights violations in a Tennessee court case and is being used in some other states as well. The value added method goes well beyond simply grading schools on whether the percent of students who passed a state-mandated exam increased or decreased from one year to next. Rather it takes into account where the student starts and the extent to which growth occurs while the student is enrolled in a specific school. New York State and New York City it seems, rely upon an inferior and invalid method of tracking student progress even though they use the information it provides for many decisions affecting schools and students. Perhaps, given the brouhaha over the recent release of test scores, more realistic methods of tracking and assessment will be adopted - especially if parents and educators demand them. Andrew A. Beveridge has taught sociology at Queens College since 1981, done demographic analyses for the New York Times since 1993, and provides expert testimony on a range of cases, including housing discrimination. The opinions expressed are his alone. Tweet Editor's Choice Everything Dot NYC Council 2.0: Rules Reform Outlines Next Steps in Open Data Overdue & Over Budget, Capital Projects Roll On Comments Comment(s) The comments section is provided as a free service to our readers. Gotham Gazette's editors reserve the right to delete any comments. Some reasons why comments might get deleted: inappropriate or offensive content, off-topic remarks or spam.