Introduction
State medical regulatory boards (SMBs) share a common mandate to safeguard patient safety through the effective oversight of licensed medical professionals. Across all U.S. jurisdictions in 2020 SMBs executed more than 7,100 adverse regulatory actions related to 3,342 licensed physicians.1 Yet, to date, SMBs have had no objective methods for determining that the regulatory actions they impose are, in fact, effective at rehabilitating licensees and preventing reoccurrences of similar conduct.
This is concerning given the relatively high instance of repeat offenses or “recidivism.” Review of the FSMB’s SMB regulatory action data during the ten-year period from 2011–2020 finds that, on average, 45% of physicians disciplined nationally were first-time recipients of regulatory action.1 More than half of physicians (55%) disciplined during the same 10-year period had previously received at least one public regulatory action. It is unclear how many actions may have arisen from the same instance of misconduct, since the FSMB’s count of actions includes reciprocal actions.
In 2019, in what is believed to be the first study of its kind, the North Carolina Medical Board (NCMB) undertook a comprehensive analysis of its disciplinary data. Its goals included gaining insight into the efficacy of its regulatory interventions, with an eye towards reducing recidivism. This work was supported by a generous grant from the FSMB Foundation. NCMB chose to focus its study on cases involving inappropriate opioid prescribing, which is an active enforcement area for North Carolina and most medical regulatory boards.
What the project team discovered, ultimately, is that successful data analysis requires a level of forethought, planning and data infrastructure investment that few SMBs — including NCMB — have completed. As a result, the project was unable to complete its objectives. But the process of data collection and analysis during the study period positioned NCMB for more meaningful analysis of enforcement data in the future.
We offer a summary of our experience here with the intention of providing lessons learned that might prove valuable for the broader state medical board community. The concept of “data-driven regulation” is an aspiration for many medical regulatory bodies, based on the idea that a better understanding of data can help boards better gauge the efficacy of their interventions, but the implementation of data-driven strategies may be easier said than done.
Structure and Methods of the NCMB study
NCMB engaged the expertise of Blaze Advisors, a health care consulting firm specializing in the strategic, tactical and technology services necessary to improve outcomes based in Wilmington, North Carolina, to conduct the data analysis for its study. Blaze looked at 39,286 NCMB licensees (MDs and DOs) with one or more prescribing cases during calendar years 2014–2018, using data exported from NCMB’s database.
The firm identified 462 individuals (1.2 %), whose conduct resulted in 538 enforcement cases and 600 total actions. Data analyzed included cases closed with no action, as well as those that resulted in either private or public action. A “case” may be based on a single report of inappropriate prescribing or it may be based on multiple reports from different sources (e.g., patients or family members, pharmacy staff, other prescribers, etc.) that are consolidated into a single enforcement case. Research was initiated in March 2019 and results were reported to NCMB in August 2019.
Study objectives included:
Create a profile of licensees at risk of prescribing issues
Determine which regulatory actions are effective in reducing recidivism
Identify gaps in education and training
Evaluate the disciplinary process to ensure evidence-based regulation
The recidivism project was not able to fulfill these objectives, beyond creating a basic demographic profile of licensees with one or more regulatory actions related to prescribing. Factors that limited analysis included missing data fields, mismatched data, large numbers of free text fields, and duplicated entries in NCMB’s exported data.
On further analysis it was determined that NCMB’s method for entering case information during the period studied, 2014–2018, did not allow researchers to easily identify enforcement cases that specifically involved opioids. The reason for this is that most information about each enforcement case is captured in written reports, summaries and notes that cannot be queried or used to generate data reports. These issues limited researchers’ ability to complete fine-detail profiling and meaningful analysis of the efficacy of regulatory actions at addressing deficiencies with opioid prescribing.
The larger issue, however, and the ultimate reason the project could not complete desired study objectives, is that NCMB did not build its database with the idea of exporting quality data in mind. Rather, the database was built simply as a means to an end — an administrative tool to allow medical board staff to store and process licensing and enforcement data. It should be noted that the same is likely true for many, if not most, other SMBs, suggesting that other regulatory bodies would face similar challenges if attempting to analyze their data.
Fortuitously, NCMB was in the process of planning its transition to a new database vendor when it received the results of its recidivism study from Blaze Advisors. The timing enabled NCMB to incorporate several suggestions to improve data collection into the planning and implementation of its new database system. Principles of sound data collection are explored in the Discussion section of this article.
Study Results
As noted, Blaze Advisors identified 462 licensees (374 men and 88 women) with one or more prescribing cases. Blaze determined that 406 (88%) licensees had a single prescribing case during the studied five-year period. Forty (8.7%) licensees had two prescribing cases between 2014–2018 and 12 (3.6%) of licensees had three cases. Less than 1% of the sample (4 individuals) had four prescribing cases during the period studied.
Demographic analysis of the data identified the most common traits of licensees with one or more prescribing cases as a male prescriber over the age of 50. Some 81% of all prescribing cases analyzed involved male physicians (males make up 67% of the total licensed physician population in North Carolina) and just over 65% of all cases involved prescribers aged 50 or older. The highest number of prescribing cases was observed in U.S.-born white male physicians between the ages of 65–79. It must be noted that these data do not necessarily represent an accurate profile of licensees most at risk of engaging in substandard prescribing, but more likely simply reflect the current demographics of the medical profession in North Carolina. NCMB has no plans to use demographic information gathered through this analysis to target specific licensee groups within the state.
Our analysis found that prescribing cases were nine times more likely among licensees in urban practice settings than among licensees in rural areas. This was especially striking because the population of the state of North Carolina is distributed at a rate of roughly 2:1 in urban vs. rural areas. The study was not able to establish any direct causal relationship to provide insight as to why prescribing cases were so much more prevalent in urban areas during the studied period.
The project had limited ability to assess the impact of regulatory actions, in part because of the relatively small number of actions that occurred during the study period and in part because NCMB’s data did not capture enough information to facilitate study of this topic.
Lessons Learned and Advice for the Medical Regulatory Community
It is not enough to simply want to practice data-driven regulation or aspire to evidence-based remediation and discipline. SMBs must also have the data infrastructure to harness the information needed to ask the right questions and devise the right strategies to achieve these aims.
To effectively mine enforcement data for insights that can drive improvements to a state medical board regulatory program, it is critical to first assess whether the SMB is set up to capture the right range and depth of information needed to allow meaningful analysis. NCMB’s recidivism project was invaluable because it confirmed that the agency was not yet prepared to engage in the sort of meaningful data-analysis that is necessary to practice evidence-based regulation. NCMB offers this observation to any other SMBs that may be interested in data-driven regulation: In short, it is easier said than done, and substantial planning and investment in a board’s data collection strategy will likely be needed.
Building a Data strategy
In September 2021, following its initial study, NCMB engaged Blaze Advisers to collaboratively devise a comprehensive data strategy for the agency. This strategy will be customized to NCMB’s specific needs, goals, and data maturity level.2 The strategy encompasses the following principles:
Data is an organizational asset. An agency must understand that its data is not just a byproduct of business activities but an asset with independent value and, potentially, a transformational tool that can support organizational and stakeholder objectives. Embracing this principle is a critical first step for any organization interested in using data to guide and inform regulatory or business activities.
Quality data starts at data entry. The process of generating meaningful, usable data starts with ensuring that each raw data point is captured in a standard way. This cannot be emphasized enough. To facilitate data analysis, entered data must be standard, consistent, validated and trusted. Organizations must design software systems and processes to ensure this, beginning with identifying what data points must be captured. Use of free text fields should be kept to the bare minimum and use of pre-populated dropdown menus or checklists should be used wherever possible to ensure accuracy and consistency.
These basic foundations are essential and should result in the ability for NCMB or other regulatory organizations to generate meaningful information that can be trusted and used for further inquiry and dimensional analysis. These activities are the basis for data-informed knowledge, not knowledge based on hunch or anecdotal experience, and subsequently leveraged in prescriptive actions to produce or drive toward a desired outcome.
Staff training is key. Software and processes are only as good as the people using them. In order to generate quality data, staff who enter raw data must understand the value of accuracy and consistency and should know that their work is helping to create a valuable resource — the organization’s data asset — for their employer. Some may be familiar with the saying, “garbage in, garbage out.” It applies here. Staff who are not properly trained or who are unaware of the value of their work are more likely to make mistakes or introduce inconsistencies to the data.
Patience yields results. Designing and implementing a system to collect data and training staff to use and maintain it is just the beginning of becoming a data-driven organization. It takes time to collect enough data to enable the types of analysis that help identify patterns, trends, outliers and other useful information. For example, most new licensee-reported data comes to NCMB through its annual renewal process. Since licensees renew annually during their birth month, it takes a full year to gather updated information from the entire active licensee population. It will take several years to collect enough data to facilitate some types of research.
Therefore, embracing data-driven regulation takes time and a commitment to the agency’s unique data governance needs, as well as internal stakeholder involvement.
By adopting the principles stated above, NCMB hopes to develop a blueprint that will set the organization up to conduct meaningful analysis of enforcement activity in the future. An additional goal for this initiative is to build a rich and robust culture around reporting and thinking about data collection processes at all levels of the organization.
Having gained valuable insights from our 2019 Recidivism Project, NCMB remains optimistic about the potential of data-driven regulation and looks forward to enhancing its ability to harness enforcement data in the service of more effective regulation.
References
- 1.↵Federation of State Medical Boards, “Medical Regulatory Trends and Actions, 2020.” Accessible at: https://www.fsmb.org/u.s.-medical-regulatory-trends-and-actions/u.s.-medical-licensing-and-disciplinary-data/physician-discipline/.
- 2.↵Adapted from Gartner (2017). Source: https://www.gartner.com/en.
