Data Forensics. CSI? No, It’s the NPTE

Forum, Volume 21, Number 2

Last edition of the Forum included a fairly detailed article on detecting aberrant patterns in test takers. While an aberrant pattern in and of itself is not necessarily an indication of cheating, cheaters often demonstrate unusual testing behavior, so it can be one of several clues. This aberrant behavior, if discovered, may warrant further investigation. Data forensic studies are one more thing the Federation of State Boards of Physical Therapy is participating in within the overall efforts to maintain the security of the National Physical Therapy Exam (NPTE).

Braindumps

Data forensics involves the detection of unusual test behavior mentioned in the previous article. It includes statistical analyses of test-response data to identify patterns indicative of cheating and piracy. In addition to the analysis of individual item responses, data forensics may evaluate the responses from all possible pairs of examinees within and across test administration sites to assess the extent to which examinee response patterns are so similar that they may be indicative of organized cheating, or collusion. This type of analysis may pick up areas where candidates have received “test prep” materials obtained from a “braindump” site or test prep course. “Braindump" is a term used in the testing industry to describe websites where test questions are posted. These questions are often submitted from memory by test takers who have recently taken the actual exam. The Federation’s internet cheating episode of 2002 was a good example of an actual braindump.

Piracy

This past year, the Federation contracted with a test security firm to conduct a forensic study. This study reviewed individual test patterns to determine if there were any unusual response patterns. Were some item responses inconsistent with the candidate’s overall ability? For example, did the candidate get large number of difficult questions correct along with a large number of easy questions incorrect? This type of forensic analysis can help detect a candidate who may be taking the test not to pass the test but to pirate items from the exam. Item piracy is the stealing of items in order to sell them or provide them to others. Other unusual response patterns are also looked at. Were there test responses that followed a specific pattern? Patterned test responses may indicate the candidate reviewing and memorizing the items and not concerned whether he or she is answering the question correctly. Another indication of potential memorization of items is extreme amounts of time spent on a few questions and little to no time spent on the rest of the questions. These candidates typically focus only the number of questions he or she can memorize.

Geographic Studies

Another part of the item forensic study done for the Federation included a geographical study. This study looked at unusual exam response patterns that are focused around specific geographical areas in the country or specific test delivery centers. This type of study is helpful in further identifying any areas where candidates may be colluding and where piracy or braindumping may be occurring. An unusually high pass rate in a regional area or at a particular test center may warrant further investigation.

Fortunately, the forensic study that was completed last year for the Federation did not show any obvious aberrant patterns in NPTE test takers. It did, however, identify one region that warranted some further investigation. The Federation is currently pursuing this investigation.

Drift in Item Difficulty Over Time

Another aspect of the Federation’s data forensic plan is to look for drift in item difficulty over time. An item that becomes easier over time is a cause for concern related to the exposure of that item and the security of the exam. The Federation has been working on an “item drift analysis” over this past year.

The Federation plans to continue using item forensics on an ongoing basis as part of its overall security plan. It is unfortunate that testing companies have to go to such lengths to detect and prevent cheating. And we believe that those that participate in these cheating behaviors are by far the minority. However, we are confident that taking these pro-active steps along with carrying out the entire FSBPT security plan is critical for the ongoing viability and quality of the NPTE. After all, we know what one bad apple can do.