Hot Seat: Test Security Expert Tells All
On the Hot Seat, test security expert John Fremer talks about how big, how complicated, how dangerous cheating prevention efforts can be (people have been killed!), who cheats (everyone but state agencies, apparently), and high-tech tactics that might be used to prevent cheating on the "common core" assessments" (can you say "Linear On The Fly Testing"?). A 35-year ETS veteran who's now Executive Vice President of Caveon Testing Security, Fremer is an increasingly familiar name in news coverage of the testing industry -- including most recently the hotly disputed instances of cheating in the Atlanta Public School system, in which Governor Sonny Perdue and others believe that cheating was much more widespread than Caveon and other have found it to be (see AJC's Get Schooled for some background). Caveon works with 15 states and the DoD.
JF: The testing industry is tens of billions of dollars annually with many facets - test development, test administration, test scoring, test program management, test preparation, etc. The test security industry is much smaller and much of the expenditure is buried within other contracts. Overall it is probably in the tens of millions of dollars.
Q: What would you consider a normal, average amount of cheating each year?
JF: My guess is that one to two percent of educators have some involvement in cheating when they play a major role in high stakes testing. This is a very low rate compared to many other fields, although some fields have even lower levels of cheating.
Q: Education's not the worst? What fields have higher rates of cheating, and which have lower?
JF: The Information Technology area has very significant problems, in part because a substantial number of their test takers take exams in other countries around the world. The legal testing arena has relatively low rates of testing misbehavior, in good measure because they exercise very firm control of the testing process.
Q: What did you find when you looked into cheating allegations in Atlanta?
JF: We confirmed that there were some schools and some classes within schools where the results strongly suggest testing misbehavior on the part of educators. In Atlanta, looking at the school with the greatest number of inexplicable results, the probability of those results occurring by chance is one over ten to the 52nd power. I liken this to flipping two coins and having the first land and stay on its edge and the second landing and staying on the edge of the first.
Q: What if anything do you think you or the state or district could have done to prevent the dispute over your findings that's taking place with the Governor?
JF: I think things would have gone more smoothly if I had arranged to meet directly with the Governor's Office of Student Achivement to be absolutely certan that our data requests and the reasons for them where spelled out in the clearest possible way. As to the Atlanta Public Schools, I have recommended that they get permission to run their own cheating detection analyses to give them more ability to focus on what specific kinds of problems are occurring and where.
Q: How's your relationship with Governor's Office of Student Achievement ED Kathleen Mathers these days?
JF: People seem intrigued by the disagreement that I have with Kathleen Mathers about Caveon not getting all the data we asked for and with the outcome of Caveon and KPMG's analyses and with our conclusions. Still, I truly applaud her for carrying out cheating analyses and then calling for a thorough review of what accounted for the results that her vendor came up with. Kathleen's activities are quite commendable and in the forefront of how a state testing leader should address possible cheating.
Q: Is most cheating done by kids, teachers, school-level staff, or district or state staff?
JF: Cheating is found in all of educational testing, including teacher-made tests which account for the majority of all testing in education. In these situations cheating by students is the most common; there's no reason for teachers to cheat.
Q: What about in school- district- or state-wide testing situations?
JF: Student cheating may still be the most prevalent, but attention tends to focus on educator cheating, of which most occurs at the school level, some at the district level, and almost none to my knowledge at the state level.
Q: Is most cheating done on high-stakes tests that affect graduation or on lower stakes testing that affects school ratings?
JF: Cheating occurs on all high stakes tests, regardless of the purpose. The degree of cheating is very much impacted by the way the program is managed. Very careful monitoring, clear testing rules, explicit consequences, systematic cheating analyses, etc. when done carefully and thoroughly can keep cheating to a quite low level. It is extremely difficult to completely avoid cheating and very expensive to come close.
Q: Are there more cheating problems now or is it just that there’s more testing and more coverage in the media under NCLB?
JF: I believe that there is more cheating now. There is also more media coverage, but I think that is partly because there is more cheating to cover. It is true that the public's appetite for cheating stories seems very strong.
Q: What's the worst test security breach or cheating scenario you've ever seen or heard about?
JF: We track stories about test cheating from around the world and proctors have been killed while trying to maintain test security.
Q: What's the worst breach over the past year or two?
JF: In a certification testing program outside of education, an entire computer-delivered test item bank was stolen before even one person had taken the test for real.
Q: What field did that happen in, and was it reported to the public?
JF: It was Information Technology. I don't think that there was a public report about it.
Q: Will the common core of standards and the new tests coming with them make things worse or better in terms of cheating?
JF: The incentive to compromise the tests will go up because of the widespread use of common test items so the return to "test pirates" will go up. Also the same test items will be given in many different places. Unless the cooperating states prepare conscientiously and follow through with great skill and focus, cheating will go up substantially. Fortunately we have time to get this right.
Q: What will be required to keep the common tests secure?
JF: In some content areas, test items can be composed as they are being administered, sometimes called "Linear on the Fly Testing." In this approach there are no test items to steal in advance and the ones administered will not be used again exactly as is. This is a powerful anti-cheating approach where it can be applied.