Behind the Standardized Test Curtain

Standardized tests. It’s hard nowadays to think about education without them—they’ve become the metric, the substance, the rationale for schooling.  We use them to identify success and failure, to tell us which schools to shut down, which teachers to fire, and which students to let graduate. I think it’s safe to say standardized tests carry more weight today than literally any other aspect of our education system.

But what do these tests actually consist of? I recently got a window into this issue after reading the Kafkaesque account of a man who spent 15 years of his life working in the K-12 testing business for some of the biggest players in the industry (Pearson Education, Educational Testing Service, American Institutes of Research, etc.—names any teacher or school administrator would be very familiar with). The man, Todd Farley, wrote, “While I did enjoy the career (good money, nice people, fun trips), it also left me completely convinced of the utter folly of entrusting decisions about American students, teachers, and schools to the for-profit industry that long employed me. I don’t know how anyone who’s seen what I’ve seen could feel any differently.”

Farley did not develop this position out of any pedagogical or ideological concerns; he’s not an educator and he doesn’t seem to be invested in this debate one way or the other. Rather, he came to his position for the same reason a guy who works in the kitchen of a not-so-reputable restaurant might choose to never frequent that eatery during off-hours—he knows what goes on back there and will be damned before he’ll put that food in his own mouth.

Farley’s not talking about the major disasters that get publicity, like the recent cheating scandals in Atlanta or the one currently being investigated in D.C.. In Farley’s words, “Any Google search will result in many similar testing tragedies, but I’d say the scandals that make the news are only the tip of the iceberg. In fact, I’d say there aren’t scoring problems on some standardized tests—my experience suggests there are scoring problems on all of them.”

Those problems come from a few different sources, according to his account. First are the scorers themselves, whom he described as “a motley crew of temporary employees earning low hourly wages…while many of those people are earnest and conscientious employees, many others are not.” He goes on to tell a number of recollections about the folks he’s worked with at test-scoring centers, many of whom only take the jobs because they can’t get real employment elsewhere. From those who come in and leave drunk to those who clearly don’t care what they’re doing (he tells one story of a scorer who gave every student response the score of two one day just for the hell of it), he does not paint a flattering picture of the folks we are allowing to shape the futures of students, teachers, and schools.

But even more disturbing are the stringent scoring rules that are utilized unwaveringly in order to get these large groups of temporary employees to score the tests in a standardized way. Take, for example, the rubric for the very first standardized test Farley scored—if the fourth-grade students gave an example of bicycle safety in a drawing (wearing a helmet, stopping at a red light, etc.) they would get a point; if not, they’d get zero. Farley picked up the first test and saw the student had indeed included one example: the rider in the drawing was wearing a helmet. He was also doing an Evel Knievel leap over a chasm of fire. Puzzled, Farley consulted his supervisor; he was told the rider was wearing a helmet and that was enough to indicate the child understood the basics of bike safety. He was also told to credit any bike at a stop sign, “and soon I was being instructed to give full credit to a poster showing a bike flying through the air in front of a stop sign; a bike in the back of a pick-up truck in front of a stop sign; and a bike busted into pieces in front of a stop sign.” On the other hand, when Farley read a beautifully moving and well-constructed essay by one high school girl about “A Special Place,” he could only give it three out of four points because her piece did not include the words “a special place.”

That’s simply how the rules of the game are set up, according to Farley. “When I was a supervisor and trainer in charge of 10, 20, 100 people, the last thing I needed was for each scorer to give a meticulous and earnest review to every student response. All I really needed was for them to quickly slap down a score and move on to the next answer. How else do people imagine those tens of millions of students responses get scored?”

But there’s an even darker side to Farley’s account: “Perhaps most importantly,” he writes, “the test-scoring industry cheats. It cheats on qualification tests to make sure there is enough personnel to meet deadlines/get tests scored; it cheats on reliability scores to give off the appearance of standardization even when that doesn’t exist; it cheats on validity scores and calibration scores and anything else that might be needed. I don’t want to just point fingers here, because I am guilty too, and over the years I fudged the numbers like everyone else. Statistical tomfoolery and corporate chicanery were the hallmark of my test-scoring career, and while I’m not proud of that, it is a fact. Remember, I was never in the testing business for any reason other than to earn a pay check, just like many of the testing companies are in it solely to make a buck.”

I understand that the bigger-picture critiques being adopted by many educators and community members across the country do not resonate with everyone. For me, standardized tests are dangerous because I don’t think they can actually measure a student’s ability to think, to be a critical learner, to work well with others, to have emotional intelligence—in short, all the things that actually matter in life and that one would hope an adequate education would provide. But for those readers who don’t share this pedagogical perspective, I hope Farley’s nuts-and-bolts condemnation means something. After all, he was behind the curtain. If my own experience in food retail taught me anything, it’s that when a waiter tells you not to get a particular item on the menu, it’s a big mistake to get that item. And it’d be an even bigger mistake to base your education system around it.