TESTING AND MEASUREMENT ISSUES
Psychometric Matters: our Biggest (Unspoken) Fear, Part I
by Leon Gross
Cheating? Loss of test material? Scoring inaccuracies? Litigation? Sure, each of these matters rank near the top of our top ten list of fears.
Cheating has rightfully become a huge concern. Perhaps the best indicator of its significance is Cizek’s comprehensive book (1999) devoted exclusively to this subject. Among his many concerns, Cizek forewarned that age-old copying has been supplemented by high technology to communicate answers.
This was demonstrated in November 2002, as a GRE candidate transmitted item content to a wired partner sitting in a van in the test center parking lot. The partner then transmitted answers back to the candidate. They got away with it, at least for a while.
On day two, they switched roles and experienced transmission difficulties. The balky transmission, evoking unusual test-taking behavior in the frustrated candidate, led to the eventual exposure of both candidates. In case you were wondering, this was a computer-based test. Switching to computer-based testing (CBT) from paper-and-pencil has many advantages, but it did not prevent this near-successful cheating episode.
Are cheating techniques versus prevention and detection methods entering an arms race? I think so. If you are doubtful, browse the Internet for sites related to fraud and spying. Cizek’s book has many suggestions. The miniaturization and application capability of the displayed devices will erase those doubts forever.
Now consider the emerging new generation of cell phones. They can browse the Web, have built-in word processing and spreadsheet capability, an internal digital camera, and much more. (Don’t ask why they’re still called cell phones. Some new name underscoring their technological convergence is needed.) The challenge of preventing access to this type of device (e.g., if ‘planted” in a lavatory) will be enormous. As the cheating arms race evolves, the test center of the future may require candidates to pass through a metal detector.
Loss of test content can destabilize a certification program, particularly if it is relatively new. If a test booklet is stolen before the test is administered, the results may have to be invalidated, at least for some candidates. If widespread exposure is suspected, the test results of innocent candidates may be invalidated also. In addition, the lost items may have to be retired permanently. If the loss occurs after the test administration, at least the scoring can proceed without any impact of “insider information.”
For this reason, loss prevention is the fundamental rationale for the surge in CBT among the large certification programs. With virtual transmission, no paper is stolen or lost. However, this deterrent should not lead to complacency, for if determined, skilled computer hackers are able to penetrate the host network, the magnitude of loss may be greater than with paper-and-pencil, the loss detection may be delayed, and the audit trail may be more difficult to follow.
Scoring inaccuracies are a nightmare. Despite being preventable, when one considers the tremendous number of examinations and data bits per examination per year, it is amazing that there are so few inaccuracies.
Litigation often begins with two words. (Hint: They do not include any expression of gratitude.) When the candidate’s attorney calls, expect to hear the dreaded words, “my client.” Yet, the risk of litigation is a professional fact of life that we must accept and attempt to minimize, simply because our examinations serve in a gatekeeping capacity.
Our justifiable fear of cheating, loss of test content, scoring inaccuracies, and litigation has led to the development of proactive systems and approaches. We have highly refined, sophisticated methods to prevent cheating and to detect and implement consequences if it nevertheless occurs. Scoring inaccuracies are preventable. Certainly, they are not the candidates’ fault. Although litigation presents an inherent degree of risk, if we are appropriately deferential to laws and statutes, and candidate due process, we reduce risk. As noted, CBT is expected to eliminate loss of content.
Each of our cited fears has significant commonalities - we are familiar with them and understand them. Consequently, we feel able to exert some mitigating effect.
What then is our biggest fear? Consider memorization of test content. I am not referring to inevitable, isolated recall of a particularly easy, difficult, or personally resonating item. Rather, I am referring to the systematic, concerted, organized memorization of test items by groups or classes of test takers for the specific purpose of reproduction and distribution, whether for sale or as a free study aide for future candidates.
Organized memorization combines our worst fears about cheating and loss of content. It is an invisible, silent adversary of test security and integrity that leaves no audit trail and is low-tech, simple to execute, and undetectable while in progress. Furthermore, it is immune to prevention by CBT. Worse still, the reproduction and distribution of the captured content can attain a scale previously unimaginable. Organized memorization occurs despite participant awareness of ethical standards and intellectual property law.
A determined, well-organized cadre of examinees, particularly a large student class, is capable of capturing and reproducing complete tests. This level of “success” may not have been achieved yet, at least in the United States. If so, let’s regard that as good fortune; however, let’s not be in denial about the potential. As a confluence of test characteristics forms a perfect psychometric storm (e.g., high stakes with a relatively high failure rate, combined with little perceived preparation guidance), a complete reproduction of a test or large portions of an item bank is realistic and perhaps inevitable.
What can or should be done about systematic memorization? Certainly, as candidate memories cannot be zapped at the conclusion of a test (scenes from the movie, Men in Black, notwithstanding), how can memorization of test content be prevented? If memorization occurs, how can its presence, breadth, and magnitude be detected? Can its impact be mitigated in a manner similar to how other cheating is handled?
Part II of this column in the Summer 2003 issue of the CLEAR Exam Review will examine this insidious problem. The column will explore the root causes of organized memorization, which, if understood, can position us to develop some preventive mechanisms. We look forward to this important ongoing dialogue.
References
Cizek, G.J. (1999). Cheating on tests: How to do it, detect it, and prevent it. Mahwah (NJ): Lawrence Erlbaum Associates.
“Columbia students accused of high-tech exam scam.”
http://www.wnbc.com/news/1795475/detail.html, November 19, 2002.
Reprinted with permission from the Clear Exam Review (CER), Volume XIV, Number 1, Winter 2003, pp. 15-16.