By Sandra Neustel

With just 1,400 students in every graduating class, optometry is a very, very small community. There are20 schools across the United States, of which three are so new they do not yet have a graduating class.Individual class size ranges from 26 to 150 students.

In 1951, a National Board of Examiners in Optometry (NBO) was funded by the regulatory state boardsas well as by schools and colleges in optometry. It is focused entirely on creating exams. Of the eightboard members, four are from the regulatory boards, three from the schools and colleges and one fromthe public. There are also 10 staff members. Until 2010, there were no known incidents of cheating.Similar to the medical field, the optometry examination has a three-part structure, with twopaper-and-pencil written exams and a performance exam.

• Part one is a 500-item, multiple-choice exam administered in four sessions, with 125 items in each session. It is available to be taken twice a year. The 17 testing sites are in the cities of the schools, either in a gymnasium or a hotel ballroom. NBO hires its own proctors and does its own testing.
• Part two is a multiple-choice, paper-and-pencil exam, administered in two sessions in one day. It is presented as a medical record based on 60 patient cases, with five or six items related to each case, including use of images and photographs.
• Part three is a performance evaluation, and is administered by 500 volunteer examiners who rate the critical skills performance of candidates as they do a refraction and various exams on volunteer patients.

Our pass rate in 2009 was between 80% and 90% for each part of the exam. There is also a fourth,separate, breakout score that focuses on disease treatment and management. It is administered withinone of our other exams. In 2009, 76% of the class passed all three exams without repeating any of theexams. Ultimately, 92% pass all exam parts before graduation; the ultimate pass grades for eachindividual exam were also reasonably high.

Our cheating incident focused on the March 2010 administration of part one of the exam. One school alerted us of a piracy event involving one student. There had been a similar incident in 2009, but it wasrectified before the exam. The 2010 incident, however, was after the fact.

We noticed and were concerned about a shift in performance for this particular group. There was a 5%differential from 2009 to 2010 which could have meant the exam was a bit easier, but the shift wasmysterious. Some items became incredibly easy while other items became more difficult. Ultimately, wegot a crash course in security and in cheating. We had to consider whether there was pre-knowledge ofour items or some other kind of breach.

Our preliminary fact finding determined that something indeed had happened at this particular school, sowe decided to hold the score for all candidates. The vast majority of students taking this test were still inschool; we didn’t prevent anyone from practicing.

NBO hired a private investigation firm to deal with the piracy and it came back with three different sets ofinterviews. Caveon compiled some data for NBO and did not find evidence of a wider breach; it couldn’teven find a statistical difference among the group that had the piracy scheme. It did find a pocketed,isolated set of collusions - people who copied. It identified four candidates in multiple sessions that hadvery similar patterns to their neighbors, and based on our previous exam agreement and policy, wecanceled the scores of those candidates.

The four candidates came forth and were extremely cooperative. The excuse was, “Everybody is doing itacross the country. We have to do it in order to stay competitive.” The students were permitted to retakethe test.

What was really disappointing was that the students told us a faculty member was orchestrating thecheating, which was later confirmed. The staff member that may have been involved is still working. Thenaive question is whether money is an issue. Optometry schools do compete.

We finally released the student scores, but told them the scores were provisional. Since then, we havenot found links to other institutions in the optometry field. However, the NBO board decided to invalidatescores for all students at that particular school, which has never been publicly named. Students wereoffered a free re-take and most did very well.

NBO believes it lost 165 total items from its item bank to piracy. Twenty-six students were distinctivelyidentified and banned from taking part one for two years. Effectively, they will be eliminated frombeginning active practice for one year after graduation day. The students may still appeal to our judicialboard, however.

We are going to continue with the data forensic as it is a very important step for protecting our exams.We are more actively monitoring test preparation courses. NBO is also increasing communication withinstitutions, emphasizing the application statement and making it a multi-step process. We are viewingthis incident as an opportunity for increasing communication within the optometric field.

Additionally, NBO is introducing new item types, specifically questions where students may have to selectthree or four correct responses rather than a single one. We are also systematically re-examining thestandard on our cut score for the part one exam to make certain we do have the bar set appropriately.

We would like very much to go to a computer-based test, but have not done so for several reasons. Themain reason is that the test is so long. We would have to get our testing time down. The other reason isbecause our candidates are so localized. We have more than 120 candidates that would need to testwithin a reasonably short window in Memphis, Tennessee and there are not enough seats in Memphis todo that.

Any organization that says it doesn’t have any security breaches may have its head in the sand. We allneed to be vigilant. Cheating may not be rampant or widespread, but it occurs in many different shapesand forms. We must protect our product and the integrity of our scores.

Sandra Neutsel, PhD, is Director of Psychometrics and Research for the National Board of Examiners inOptometry.