FSBPT is now conducting the practice analysis annually in order to better monitor trends in the field that may affect the exam and other regulatory issues. The NPTE is also working toward including video and scenario-based items. This article is based on a presentation by Kelly King, PT, DPT, Senior Clinical Manager, Rehab Department, Banner Desert Medical Center & Cardon Children's Medical Center; Rebecca E. Porter, PhD, PT, Emeritus Faculty, Indiana University Department of Physical Therapy; Teresa Briedwell, PT, DPT, OCS, Item Writing Task Force Member; Lorin Mueller, PhD, Managing Director of Assessment; and Richard Woolf, PT, DPT, FSBPT Assessment Content Manager, Federation of State Boards of Physical Therapy, at the 2018 FSBPT Annual Meeting.

Practice Analysis

Presented by Kelly King

As the developer of the National Physical Therapy Examination, FSBPT has a responsibility to ensure the validity of the results as a measure of entry-level knowledge and skills of the PT and the PTA. Therefore, we must conduct a practice analysis to know what skills are needed and to answer other pressing questions.

• How has the physical therapy profession changed in the past five to ten years?
• How do we identify emerging areas of practice as safe and effective treatment techniques and interventions?
• What are the areas of practice that are becoming obsolete?
• What do entry-level PTs and PTAs do on the job?
• What do entry-level PTs and PTAs need to know to do their jobs safely and effectively?

To accomplish this, we undertake a formal systematic process referred to as the analysis of practice, or the practice analysis survey. This process begins with the identification of the work activity requirements for entry-level practitioners and ends with the development of a formal systematic survey that has questions that identify the knowledge and skills related to safe and effective entry-level practice.

Two volunteer committees, the Oversight Committee and the Practice Analysis Taskforce, work on this initiative. In February 2016, the Oversight Committee, of which I was involved with, met to identify the process for the survey format and the criteria for survey questions. We also discussed whether any areas of the practice analysis survey needed to either be included or excluded since the last analysis in 2011. The 2016 analysis survey was disseminated to PTs and PTAs across the country.

The results were interesting. There were 240 work activities on the PT survey. Respondents considered 94 percent of those as critical for entry-level practice. Two work areas were flagged for omission from the exam as a result of the survey: hydrotherapy using contrast baths and paraffin bath thermotherapy.

For the PTA exam there were 197 work activities on the survey. Respondents considered 87 percent of those critical for entry-level practice. Interestingly, there were actually nine areas that were omitted from the exam based on the results of the actual work activities and knowledge and skills required. Those included the use of ultraviolet light as well as training patients in intermittent pneumatic compression.

There was also a major structural change: separating the Cardiovascular/Pulmonary & Lymphatic System into two distinct content domains. Knowledge related to the Lymphatic System, including examination, evaluation, differential diagnosis and prognosis, and interventions, is now covered as a stand-alone content domain.

As we reviewed the current NPTE content outline, trends regarding specific emerging or declining PT practice interventions (e.g., dry needling, diathermy) were noted to be outside of the threshold of importance or frequency of use. Therefore, we wondered if every five years was actually frequent enough to assess the practice. After discussing this, we decided to do it annually. The updated frequency provides us with a better ability to monitor trends and ensure that the exam represents actual practice for entry-level PTs and PTAs.

The Rationale behind Alternative Item Types

Presented by Becky Porter

Before we look forward, it may be helpful to look back at the history of the NPTE to establish that it has been continually evolving over the time that I have been involved. When the recent discussion of the concept of scenario-based multiple-choice items began, I recalled the scenario-based questions that were used in the 1990s. The items posed logistical problems with formatting the examination and they faded from appearing on the exam.

In the late 1990s, the conversation started on using visual images (graphics) to enhance the questions. So much of what we do in our professional practice involves the interpretation of visual information, using photographs and diagrams to enrich items and eliminate lengthy descriptions was very appealing. After determining how to best present the images, in 2000 the use of graphic items became live. Thinking about how to make changes didn’t stop there.

Educators have been using scenario-based questions on their classroom exams and have been dealing with the challenges of presenting the array of information necessary to set up the scenario and the difficulty of avoiding the problem of writing questions so that getting the correct answer to the second question hinged on answering the first question correctly. In 2008, when the Practice Review Tool was being conceptualized, we wondered how to present a scenario that didn’t have a large reading load. We realized that following the organizational structure of charts could be the solution. We could emulate charts and establish standardized sections for information instead of worrying about using a flowery paragraph. This experience provided the background to move forward with developing the structure for scenario-based questions on the NPTE.

Before making any changes, the critical question is why change the NPTE? It is a well-established examination with highly defensible psychometrics. Students and educators know what is on it in terms of the framework and content analysis. But the continuing feedback from educators was that using videos and scenarios would better reflect current practice.

A couple of points continued to be clear. If the goal of physical therapy is to optimize movement, we should seek to evaluate how potential PTs and PTAs visually assess movement. In clinical practice, I have never treated a patient based solely on a written description of the gait deficit. Shouldn’t we seek a standardized assessment of clinical decision making based on the interpretation of visual information? And if the treatment of our increasingly complex patients requires us to integrate multiple pieces of information, shouldn’t we be testing that ability rather than basing a decision on the information in a three or four sentence stem?

There seemed to be a strong case for changing.

Of course, none of these changes could be pursued until it was determined that Prometric test centers could accommodate these formats into the tests they deliver. Once it was established that these types of items were feasible, developing the concepts could move forward.

The Board of Directors supported moving forward and making the significant investment required to alternative item types for our high-stakes exam. One development is the formulation of the Item Writing Task Force composed of highly skilled item writers serving a three-year term. They are providing the expertise to develop the items in the new formats and the guidelines for developing future items.

What are the parameters for scenario-based items? The detailed clinical scenario presents the information on the patient history and current status in a standardized format so the information can be easily located by the candidate. Information in a patient’s chart that typically appears in a graphic format like test results could be included. One of the challenges provided to the item writers was that in addition to each question being independent of the others, specific information in the scenario would be required to answer each question. For example, if a patient had a hip arthroplasty and the question hinged on standard information about the site of the surgical incision, it is likely the question could be answered without information in the scenario. Constructing the items has many challenges.

Video items contain very short video clips of a patient in motion. Videos have their own considerations. For example,

• How many times can the candidate watch the clip?
• How much additional time will this video item take as opposed to a regular item and how does that change the number of questions?
• How do we make sure that we aren't compromising candidates' time to take the exam?

Additionally, how does the addition of these new item types affect the item development and approval processes? We need more than 3,000 items generated each year to keep the item bank healthy. What’s the impact of adding these new formats in terms of the time it’s going to take to write the items, and what’s going to be involved in continuing to feed the item bank?

The Exam Development Committees and the Technical Advisory Panels are considering all of these questions and concerns as well as others to assure that the development of these item types continues moving in the right direction.

Implementing Alternative Item Types

Presented by Teresa Briedwell

I’m representing the Item Writing Task Force. They’ve been giving us credit for being skilled item writers. Well, we were skilled at writing multiple choice questions. And then all of a sudden they throw this extra weight on us. We did some practice questions in 2017 and now we’re officially writing questions that we hope will go out and be tested.

There are two new types, video and scenario-based to better reflect the clinical reality—our patients are complex. When’s the last time you had a patient without medications or comorbidities? That’s why we will build these really complex situations and we can see if the people taking the exam can pull the pertinent information and make good decisions.

We need to write fifteen questions per scenario. Of course, we would never want to have fifteen questions with a scenario. However, if you’ve been part of the process, you know that we write a lot of questions that don’t actually end up on the exam as a scored item. So we need many questions for each scenario.

There are multiple things to keep in mind while writing scenario items. We need to ensure that each question could stand alone, meaning it isn’t dependent on another question associated with the scenario, but each question also needs to be dependent on the scenario. You shouldn’t be able to answer the question if you haven’t seen the scenario. And, of course, we need to ensure that one scenario question doesn’t give you an advantage in answering another question from that scenario. And, finally, we also consider the reader burden.

Video questions are important because we’re movement specialists. We need to be able to assess the candidates’ observational skills. When we first thought of what kinds of videos we would want, we had many ideas such as abnormal movement patterns, substitutions, wound care measurements, gait deviations, pediatric reflexes, wheelchair movements and techniques, etc. There are many things that could be included. Just like with the scenarios and with photographs, they have to be needed to answer the question.

It’s a really exciting time to be an item writer and it’s great to be a part of this, helping to figure out how we can take this exam to the next level.

Steps Moving Forward

Presented by Lorin Mueller and Richard Woolf

We are going to continue to collect data annually on the practice analysis. We’d like your thoughts and feedback. If an assistive movement device is trending upward in use, is there a reason that might be a blip? Is there something worth following up on? What other questions should we ask to really get at the heart of what is entry-level versus what is not?

Our next step for the scenario and video items is to continue to look at the feasibility. We want to make sure that scenario-based and video questions add value to the assessment of entry-level competence. We also want to make sure that we have a large library of videos that we can draw from to ensure that any one type of video isn’t associated with just one question and answer.

We also need to consider field testing. When we put new items in the field, we have a process where we pre-equate the test. However, as we introduce new items types, candidates may know or suspect those aren’t being scored. We have to take some precaution to make sure that candidates are answering those with the same diligence that they would answer a potentially scored question. Fortunately, we’ve got a great Technical Advisory Panel to help us with these issues.

We will also ensure that everyone has plenty of notice before the changes are implemented. We will update PEAT and other resources to ensure candidates can use those to prepare for the new item types. Any changes to the content outline will be based on data.

We really want to recognize the volunteers. Truly, none of this would be possible without our volunteers. We talk about volunteers as our life blood and they really are. We couldn’t generate 3,000 good questions a year without a huge contingent of fantastic volunteers, who are conscientious and hard-working and very knowledgeable.

In a lot of ways, the volunteers are our leaders, too. In the practice analysis, it was the Oversight Committee who encouraged more frequent surveys. Volunteers were the ones saying that our field is about movement, we need to have video items.

And that input and advice is really powerful. But we needed to think about what would be feasible and how we’re going to ensure the changes don’t have a negative impact on the candidates in terms of the cost of the exam. We have many creative thinkers, both on staff and among our volunteers, who have helped us flesh out and frame the problems and approaches in front of us to help us move forward.