Susan Layton, FSBPT COO and Lorin Mueller, FSBPT Managing Director of Assessment

Note: The following article was developed from an educational session at the 2012 FSBPT annual meeting.

Alarming data on security breaches – item harvesting - has warranted a quick response. Last year, PT NPTE testing transitioned to fixed-date events and this year, PTA NPTE testing did the same. But while we could immediately get exams out there for fixed-date testing, there remain administrative items that need to be addressed.

Communication

Our website has a fixed-date testing page, and visitors are encouraged to link to that page. On another page is the information for each date, including when the jurisdiction has to approve the registration, when reservations for seats must be made and when scores will be available.

We just added new language explaining what occurs if you are not able to test through no fault of your own. For instance, when we tested July 2, 2012, 90 examinees were displaced due to power outages at Prometric centers. So we explained how we would reschedule the test as soon as possible and most of the July 2 examinees tested within two weeks.

Scheduling reports are updated weekly. Unfortunately, they do not contain a complete list of available seats, and the partial list is removed from the website 21 days before the exam, as that’s when the seats are released. (In high-density areas, seats are hard to come by). This report is what we were able to do quickly, but we know we can do better. To that end, we will be updating these reports in 2013.

Here are some other improvements we’ve made.

• ATT letters, which are still being mailed, now contain a color insert urging scheduling as soon as the letter is received.
• We improved our candidate handbook online to make it easier to search and we have added webinars.
• We’ve also improved our candidate satisfaction surveys. In the past, we’ve asked questions concerning processing, security procedures and testing environment. Now we are also asking how far they had to travel to test, their thoughts on our customer service and whether they’ve read the candidate handbook.
• We have enhanced the online system for jurisdictions. The move from continuous to fixed-date testing has been very challenging from an administrative perspective. Now that people register six months in advance, we need to find a way to tell administrators what testing day they have chosen.
• Candidates can also now withdraw their testing request online. We are also working on a system to withdraw and re-register or switch dates without going through the whole process again. Why have deadlines at all?

Deadlines maximize the likelihood that all candidates who are approved to test have a seat and allow FSBPT, jurisdiction licensing boards and Prometric time to process all candidates’ records. They reduce issues with guessing how many people will test andthey allow adjustments at Prometric centers to accommodate more candidates when possible. They also ensure forms are assigned correctly and help make certain ADA accommodations are ready.

Score reporting

We now have what we call “almost immediate” score reporting, which means the reporting will occur within five business days after testing. To understand the delay, you must first understand the process. We first receive test records from the Prometic centers, which now number in the thousands instead of hundred of records because of a reduced number of testing dates. Then we reconcile and verify the records as well as confirm the scores. The most difficult aspect is to identify and investigate any testing anomalies, such as missing records, potential security breaches and missing biometric data.

All of this is an intense effort of active investigation to meet the reporting window. The consequences of making errors during this phase would be extremely problematic for our stakeholders and FSBPT. So at this point, it is not likely that this process will move any quicker.

As for testing dates, we are still trying to determine when candidates want to test – before they graduate, just after or later. FSBPT now allows candidates to register for an examination within 90 days of graduation, a policy that 20 states accept. In 2013, there will be 4 PT and 4 PTA test dates, one in each quarter. By far, July appears to be the most popular choice for testing.

Our goal is use an item just once, but we are not there yet. However, when we reach that goal, and are able to create more items, we may increase the number of tests per year.

As a footnote, candidates can test three times in a one-year period.

A fixed-date disadvantage?

We are attempting to determine the impact of fixed-date testing, and, of course, the data is somewhat limited.

Of those who took test prior to graduation date, 89.4% of the 827 passed. Of those who took the test on or after the graduation date, 88.1% of the 7,817 passed. The percentage difference is minimal.

About 90% of examinees take their test within 90 days of graduation. We found that there are slightly lower scores for those who wait more than 120 days to take the test, and that the score decreases the longer the student is removed from the graduation date.

The probability of failing increases significantly for those who take the test more than 200 days after graduation. That also explains why foreign-educated students tend to do worse on the tests; they are taking the test many months, if not years, after their graduation.

New guidelines

New content guidelines have been developed, and FSBPT is offering new services to examinees as well, including updates to the Practice Exam and Assessment Tool (PEAT) to reflect those new guidelines. They include extended availability to 60 days and revised performance feedback reports to offer scores by sections, raw scores, percent of correct answers and scale scores. The revised reports came about because of significant feedback to tweak them.

We also will be making individual score reports available for 30 days free of charge.

Behind the scenes, we are improving item fairness review procedures for U.S. and non-U.S. examinees, developing enhanced item writing tools and templates, researching new security analyses such as item harvesting and collusion and doing research on eligibility criteria.

		Susan Layton, Chief Operating Officer, Federation of State Boards of Physical Therapy Susan Layton is the Chief Operating Officer for the Federation of State Boards of Physical Therapy. Her areas of responsibility include Assessment, Continuing Competence, Exam Services, Information Systems and Meeting Planning. Susan has a Master’s of Science in Management from the London Business School where she was a Sloan Fellow and a Bachelor’s of Science in Business Administration from the University of Mary Washington.
		Lorin Mueller, PhD Managing Director of Assessment, Federation of State Boards of Physical Therapy Lorin joined the FSBPT’s Assessment Department in November of 2011. Prior to joining the FSBPT, Lorin spent 10 years as Principal Research Scientist at the American Institutes for Research in Washington, D.C. He has contributed his expertise in statistics, research design, and measurement to projects in a wide variety of areas, including high-stakes test development, work disability assessment, K-12 assessment, assessing students with cognitive disabilities, teacher knowledge, teacher performance evaluation, and school climate. He is a nationally-recognized expert in the field of setting standards for occupational assessments and has published or presented in nearly all of the areas he has worked. Lorin received his Ph.D. in Industrial and Organizational Psychology with a specialization in statistics and measurement in 2002 from the University of Houston.