The Board Assessment Task Force is hard at work creating a new tool that will help boards conduct self-assessments and provide resources to improve performance and advance their missions. This article is based on a presentation by Kathy O. Arney, PT, MA, Executive Director, North Carolina Board of Physical Therapy Examiners and Charlotte Martin, MPA, CBA Chair and Executive Director, Louisiana Physical Therapy Board, at the 2018 FSBPT Annual Meeting.

We decided to call this presentation Functional Goals, similar to physical therapy functional goals, because we thought that was a good analogy. You look at your patient and you establish both where they are and where they need to be—that’s how we’re thinking of licensing boards.

It’s really important that we have input from administrators because the administrators will likely be completing the self-assessment. We need to ensure that the amount of work required of them to fill out the tool would be reasonable and feasible.

For a historical perspective, FSBPT developed a similar tool in 2001 that was used among the jurisdictions that included productivity and process data points to allow a board to compare itself nationally with other jurisdictions. After the tool was discontinued in 2007 due to lack of use and complaints about the onerous process to fill out the tool, there wasn’t much discussion about other options until the 2017 Leadership Issues Forum. At that time, there were valuable discussions around the climate of deregulation and increased scrutiny of regulatory boards. The membership started to discuss how useful it would be to have a tool where they could say, “Yes, this is where I am in terms of my effectiveness with public protection, and here is where I stand nationally.” In response to this membership interest, the FSBPT Board of Directors formed a task force to explore potential development of a new tool.

2018 Board Assessment Task Force Members

Charlotte F. Martin, MPA, LA, Chairman
Kathy Arney, PT, NC, Member
Jason Kaiser, CA, Member
Jeffrey Vinzant, AL, Member
Deborah Richardson-Peter, MPA, VI, Member
Michelle Sigmund-Gaines, OR, Member

The task force started working early in 2018 and presented at the 2018 Leadership Issues Forum where functional areas and operational metrics were discussed.  The dialogue and feedback from the small group discussions were very helpful. Not only were an additional five pages of metrics suggested, but there were new interesting perspectives and recommendations:

1.     What are the characteristics of effective boards?

• Responsiveness to emerging issues
• Open-mindedness
• Disseminating information
• Engagement with the profession/outreach to the public

2.     How do boards demonstrate public protection?

The task force took all of the recommendations and realized that there were certain themes emerging in terms of how to demonstrate public protection and the committee determined that we needed to rethink our entire approach.

To start, the task force met in-person after the LIF meeting, reviewed previous versions of tools and discussed why the old tool was discontinued. Realizing that the old tool metrics were very similar to what the task force was proposing, as well as hearing that the old tool was onerous to fill out, the task force determined the entire discussion needed to be reframed. The new tool should be a board self-assessment; this is a best practice, something that boards should be proactively doing for themselves. The tool needs to properly assess where a board is and give actionable items to help move the board to where they want to be.

Although there were some components to the old tool that were helpful or informative, the task force basically went back to the drawing board. First, we decided it needed to be a toolkit, not just one tool. We divided the toolkit into three sections:

• Question Bank Dashboard
• Resources and Data
• Supporting Documents

Question Bank and Dashboard

When the task force came to appreciate the new task, we decided we needed to start with a phase one. It’s important to do whatever we do really well, so it’s better to start off small. The toolkit could continue to be built out after that as needed, but we needed to start with something manageable. Four main themes were identified as a starting point: 

• Board Performance

Do you have a strategic plan? Do you do orientation for staff and your board? Do you do a periodic rules review?

• Outreach and Education

Do you have outreach initiatives? Do you have customer surveys? Is there access to your board?

• Licensure

Do you collect workforce data? Is your board prioritizing portability? Do you have a system where consumers can verify licenses?

• Complaint Resolution

Do you publicize resolutions? Do you have guidelines that you follow? Do you have preventative outreach for your licensees?

The task force devised a few ways to numerically answer those questions; continuing to use the analogy to patient care with the amount of assistance required:

• 0 - Dependent / There is no system in place
• 1 - Max Assist / There is a system in place
• 2 - Mod Assist / Collecting
• 3 - Min Assist / Analyzing
• 4 - Stand By Assist / Interpreting
• 5 - Independent / Making Changes

Further, the task force decided that the questions should be in a yes/no question format rather than open-ended.  All questions would fit this format.  For every question the user assesses where they are on the spectrum. Depending on how the user answered the questions, the system would generate different advice and resources.

Example:

Does your board have a strategic plan?

• Level 0 - No.
• Level 1 - Yes.
• Level 2 - Yes and collecting data.
• Level 3 - Yes and analyzing data.
• Level 4 - Yes, and interpreting data.
• Level 5 - Yes, and using data to implement change.

If you answered that you didn’t have a system for strategic planning in place, then it would provide an opportunity for you to look at resources on how you could get started on or orient your board to the strategic planning process.  However, if the answer was, “Yes, we have a system in place but we don’t use it,” then it would have recommendations on how to build upon what you have or what you are doing to ultimately end up a Level 5. If, as in this example, the board performance is at Level 0, the board has nothing in place regarding strategic planning, then when they complete the self-assessment, a recommendation would be generated based on that. The toolkit would have resources on the back-end that would help the board implement the recommendation.

Resources and Data

Realizing FSBPT does not have unlimited resources, the task force looked at leveraging established resources and tools such as the performance evaluation tool, score reports, and the website wizard created for foreign applicants looking at foreign education standards. It is possible existing technology can be leveraged for this toolkit.

Supporting Documents

When considering supporting documents, we knew that we needed a purpose, common definitions, and a rationale for each question.

Supporting Documents

1. Purpose of the Toolkit
• Provide a resource for boards to self-assess function in terms of effectiveness and efficiency in fulfilling their mission to protect the public and provide data on areas for improvement.


2. Common Definitions
• Provide a resource for boards to self-assess function in terms of effectiveness and efficiency in fulfilling their mission to protect the public and provide data on areas for improvement.
• Effectiveness: Producing the intended or desired result
• Efficiency: Using resources judiciously and appropriately to accomplish the desired result
• Desired Result: Public Protection - Allowing the consumer to access safe and competent physical therapy services while preventing and reducing real or potential harms.


3. Rationale for Questions
• Rationale ties to the desired result.

For example, if you’re looking at the category of complaint resolution and we are saying, “Does your board publish complaint resolutions?” that would be the yes or no question. The rationale for having this question on the dashboard is that by publishing complaint resolutions, a licensing board is accomplishing the intended results of allowing consumers of PT services to access safe and competent PT services by providing a means to access information related to the risks of harm of individuals who have practiced in a way that was found to be unprofessional, unethical, etc.

Next Steps

The task force made significant changes from LIF to the 2018 Annual Meeting, it has taken a giant leap from where the task force was headed prior to LIF, but there’s much more to be done. As a task force, we need to continue to work on definitions and finish completing a rationale for each question. We also want to finesse the levels one through five for each question and create a library of resources for recommendations.

We initially thought this would be a quick project, but now that it’s fleshed out, the task force feels this is more realistic as a three-year project. Initially, the task force will gather data, clarify things, and see what this really looks like. In 2019, the task force will finalize the specifics and look at IT requirements. The goal is to build the self-assessment tool and have it operational for boards in 2020.

Tool Validation: UNC Public Policy Capstone Project

The North Carolina Board of Physical Therapy Examiners (NCBPTE) reached out to the University of North Carolina’s public policy graduate students to gauge their interest in examining board performance and assessment for their required capstone project. The advisors of the capstone program were very interested and asked NCBPTE to submit a proposal for students explaining the project, which was to examine board assessment and define what that looks like in terms of public protection. Students would examine how a board is effective and efficient in its role in public protection and what metrics should be collected in order to demonstrate those qualities. As part of the project, the students will also help validate the board self-assessment tool being created by the task force.  

The task force knows that other professions are battling the same question: how do you assess that a board is protecting the public? Hopefully, with the help of UNC, the task force will be able to create and validate the toolkit and other health care occupational licensing boards can benefit from the work as well.