Scandal Haunts Atlanta’s School Chief—NYTimes 8/7/2010
_____
Given the importance of test scores, it is not surprising that teachers and school officials have devised various ways of gaming the testing system: that is, tricks and shortcuts to achieve the desired results, without improving education. When the purpose of testing is informational and diagnostic, there is no reason for teachers and administrators to alter the results except through improved instruction. But when the purpose of testing is accountability, then teachers and administrators understand that there are real consequences if the scores in their classroom or school change. If scores go up, they may get a handsome bonus; if they go down, their school will be stigmatized, and they may lose their jobs. The intense pressure generated by demands for accountability leads many educators and school officials to boost the scores in ways that have nothing to do with learning.
The most reprehensible form of gaming the system is plain old-fashioned cheating. There have been many news stories about a teacher or principal who was fired for correcting students’ answers before handing in the tests or leaking the questions in advance to students. In some instances, the cheating is systematic, not idiosyncratic. The Dallas Morning News analyzed statewide scores in Texas on the state’s high-stakes TAKS test—which determines schools’ reputations and teachers’ rewards—and found evidence that tens of thousands of students cheated every year without being detected or punished. The cheating was especially pervasive on eleventh-grade tests, which students must pass to graduate. Most of the cheating uncovered by reporters was in Houston and Dallas and was more common in low-achieving schools, “where the pressure to boost scores is the highest.” Cheating was found in charter schools at almost four times the rate of traditional public schools. In response to the story, Dallas school officials beefed up their school system’s testing security, but Houston school officials slammed the newspaper’s study as an effort “to dismiss the real academic progress in Texas schools.”
Many ways of gaming the system are not outright illegal, yet they are usually not openly acknowledged. Most principals know that the key to getting higher test scores is to restrict the admission of low-performing students, because they depress the school’s test scores. As choice becomes more common in urban districts, principals of small schools and charter schools—both of which have limited enrollments—may exclude the students who are most difficult to educate. They may do it by requiring an interview with parents of applicants, knowing that parents of the lowest-performing students are not as likely to show up as the parents of more successful students. They may do it by requiring that students write an essay explaining why they want to attend the school. They may ask for letters of recommendation from the students’ teachers. They may exclude students with poor attendance records, since poor attendance correlates with poor academic performance. They may limit the number of students they admit who are English-language learners or in need of special education. All such requirements tend to eliminate the lowest performers.
__
Ravitch, Diane. The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education. New York: Basic, 2010. 154-155. Print.
The most toxic flaw in NCLB was its legislative command that all students in every school must be proficient in reading and mathematics by 2014. By that magical date, every single student must achieve proficiency, including students with special needs, students whose native language is not English, students who are homeless and lacking in any societal advantage, and students who have every societal advantage but are not interested in their schoolwork. All will be proficient by 2014, or so the law mandates. And if they are not, then their schools and teachers will suffer the consequences.
The term “proficiency”—which is the goal of the law—is not the same as “minimal literacy.” The term “proficiency” has been used since the early 1990s by the federal testing program, the National Assessment of Educational Progress, where it connotes a very high level of academic achievement. The federal assessment refers to four levels of achievement. The lowest is “below basic,” which means a student who is unable to meet the standards for his or her grade. The next level is “basic,” which means that a student has partially mastered the expectations for the grade. Then comes “proficient,” indicating that a student has fully mastered the standards for the grade. And at the very top of the performance levels is “advanced,” which represents truly superior achievement. On the 2007 NAEP for fourth-grade reading, 33 percent of the nation’s students were below basic; 34 percent were basic; 25 percent scored proficient; and 8 percent were advanced. Now, in a nation where only one-third of students meet the federal standard for proficiency, we are expected to believe that fully 100 percent will meet that standard by 2014. It will not happen. Unless, that is, the term “proficiency” is redefined to mean functional literacy, minimal literacy, or something akin to a low passing mark (say, a 60 on a test with a 100-point scale, a score that once would have merited a D, at best).
The goal set by Congress of 100 percent proficiency by 2014 is an aspiration; it is akin to a declaration of belief. Yes, we do believe all children can learn and should learn. But as a goal, it is utterly out of reach. No one truly expects that all students will be proficient by the year 2014, although NCLB’s most fervent supporters often claimed that it was feasible.* Such a goal has never been reached by any state or nation. In their book about NCLB [No Remedy Left Behind], [Chester E.] Finn [Jr.] and [Frederick M.] Hess acknowledge that no educator believes this goal is attainable; they write “Only politicians promise such things.” The law, they say, is comparable to Congress declaring “that every last molecule of water or air pollution would vanish by 2014, or that all American cities would be crime-free by that date.” I would add that there is an important difference. If pollution does not utterly vanish, or if all cities are not crime-free, no public official will be punished. No state or municipal environmental protection agencies will be shuttered, no police officers will be reprimanded or fired, no police department will be handed over to private managers. But if all students are not on track to be proficient by 2014, then schools will be closed, teachers will be fired, principals will lose their jobs, and some—perhaps many—public schools will be privatized. All because they were not able to achieve the impossible.
* including Margaret Spellings in a Forbes editorial.
Ravitch, Diane. The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education. New York: Basic, 2010. 102-103. Print.
The Committee on Appropriate Test Use of the National Research Council stated in an authoritative report in 1999 that “tests are not perfect” and “a test score is not an accurate measure of a student’s knowledge or skills.” Because test scores are not an infallible measure, the committee warned, “an educational decision that will have a major impact on a test taker should not be made solely or automatically on the basis of a single test score.” ….
Psychometricians are less enthusiastic than elected officials about using tests to make consequential judgments, because they know that test scores may vary in unpredictable ways. Year-to-year changes in test scores for individuals or entire classes may be due to random variation. Student performance may be affected by the weather, the student’s state of mind, distractions outside the classroom, or conditions inside the classroom. Tests may also become invalid if too much time is spent preparing students to take them.
Robert Linn of the University of Colorado, a leading psychometrician, maintains there are many reasons why one school might get better test scores than another. NCLB, he says, assumes that if school A gets better results than school B, it must be due to differences in school quality. But school A may have students who were higher achieving in earlier years than those in the other school. Or school A might have fewer students who are English-language learners or fewer students with disabilities than school B. School A, which is presumably more successful, may have a homogeneous student body, while the less successful school B may have a diverse student body with several subgroups, each of which must meet a proficiency target. Linn concludes, “The fact that the school that has fewer challenges makes AYP [adequate yearly progress] while the school with greater challenges fails to make AYP does not justify the conclusion that the first school is more effective than the second school. The first school might very well fail to make AYP if it had a student body that was comparable to the one in the second school.”
State testing systems usually test only once each year, which increases the possibility of random variation. It would help, Linn says, to administer tests at the start of the school year and then again at the end of the school year, to identify the effectiveness of the school. Even then, there would be confounding variables: “For example, the students at the school with the higher scores on the state assessment might have received more educational support at home than students at school B. The student bodies attending different schools can differ in many ways that are related to performance on tests, including language background, socioeconomic status, and prior achievement.” The professional organizations that set the standards for testing—such as the American Psychological Association and the American Educational Research Association—agree that test results reflect not only what happens in school, but also the characteristics of those tested, including such elusive factors as student motivation and parental engagement. Because there are so many variables that cannot be measured, even attempts to match schools by the demographic profile of their student body do not suffice to eliminate random variation.
___
Ravitch. Diane. The Death and Life of the Great American School System. New York: Basic/Perseus, 2010: 153-4.
The problem with using tests to make important decisions about people’s lives is that standardized tests are not precise instruments. Unfortunately, most elected officials do not realize this, nor does the general public. The public thinks the tests have scientific validity, like that of a thermometer or a barometer, and that they are objective, not tainted by fallible human judgment. But test scores are not comparable to standard weights and measures; they do not have the precision of a doctor’s scale or yardstick. Tests vary in their quality, and even the best tests may sometimes be error-prone, because of human mistakes or technical foul-ups. Hardly a testing season passes without a news story about a goof made by a major testing company. Sometimes questions are poorly worded. Sometimes the answers are wrongly scored. Sometimes the supposedly “right” answer to a question is wrong or ambiguous. Sometimes two of four answers on a multiple-choice question are equally correct.
All tests have a margin of error, like opinion polls, and the same student could produce different scores when taking the same test on different days. The scores might not be wildly different, but they might be different enough to nudge the student’s rating across the line from “not proficient” to “proficient,” or drop her down a notch. So, a student who failed a test on Monday might pass if she took the same test on Wednesday. Maybe the student got a good night’s sleep one day, but not the next; maybe she was distracted by a personal crisis—a spat with her best friend—one day, but not the next. Tests themselves differ from one another, even when they are designed to be as similar as possible. So a student could pass one test and fail another that was designed to be of equal difficulty. Testing experts frequently remind school officials that standardized test scores should not be used in isolation to make consequential decisions about students, but only in conjunction with other measures of student performance, such as grades, class participation, homework, and teachers’ recommendations. Testing experts also warn that test scores should only be used for the purpose for which the test was designed: For example, a fifth-grade reading test measures fifth-grade reading skills and cannot reliably serve as a measure of the teacher’s skill.
Ravitch. Diane. The Death and Life of the Great American School System. New York: Basic/Perseus, 2010: 152-3.
