The Wicked Problem of Regulation of Artificial Intelligence in Health Professionals' Work: Whose Job is it Anyway?

HiL-AI vs HoL-AI

Recently, the categorization of AI has evolved by emphasizing the degree to which humans control or direct activities and outputs. ³ "Human in the loop" AI ("HiL-AI") requires a human in the AI process (or "loop") to actually make a final decision or judgement, with AI playing a supportive, informational role (for example, decision support systems in diagnosis). In contrast, "human-out-of-the-loop" AI ("HoL-AI") systems delegate final decision making and judgement to AI itself, without need for human involvement or oversight (for example, autonomous self-driving vehicles).

This distinction between HiL-AI and HoL-AI is especially useful in examining the "wicked problem" ⁴ of AI in the context of healthcare systems and the regulation of health professionals. Wicked problems are those that are rooted in inherent complexities that defy straightforward analysis or solutions; indeed the "solution" to a wicked problem frequent triggers unanticipated new problems of expanded complexity. It is sometimes noted that there are no "right answers" for wicked problems—only "least worst alternatives" to be considered. ⁴

Today, AI is being used in diverse medical contexts, including diagnosis, medical imaging, and pharmacotherapeutic decision making in personalized medicine. ⁵ Even when HiL-AI is being used for these clinical tasks, there are significant concerns regarding "de-skilling" of the human workforce. ⁶ Further, there are risks that HiL-AI may codify and make semi-permanent existing implicit biases: the way in which AI systems are trained relies upon existing documents and practices which may have been produced with unconscious bias. ^6,7 Where human "acceptance" of HiL-AI recommendations is required (through, for example, inputting a keystroke), there are concerns that this leads to an uncritical, unquestioning, lazy, and automatized/performative human response because of outsized faith in the power of AI and lack of understanding of how AI-powered decisions are actually made. ⁶ Similar to the collapse in every-day mental mathematical abilities (such as estimation of costs or purchases at a grocery store) that has occurred with children who have learned basic numeracy skills by using a calculator, ⁶ there are concerns that deskilling of the human health workforce will follow, whether human professionals are officially in the loop or out of the loop while using AI. ^6,7 Of course, there are also concerns regarding the well-documented phenomenon of "hallucinations," a response generated by AI that contains false or misleading information presented as fact, ⁸ for example, labelling an ambiguous skin lesion as benign rather than malignant based on insufficient training of the system. Newer language model coding systems currently being pioneered may mitigate this problem in the future, so concerns persist regarding the impact of these hallucinations in the present. ⁹

Is regulation necessary—or desirable?

As AI's presence in healthcare expands, there are questions as to whether the regulators of health professionals (ie, State Boards) should be taking a more proactive and interventional approach to safeguard the public interest. In this issue of the Journal, ¹⁰ we report on recent regulatory research focused on understanding perspectives and priorities of regulators with respect to AI in the work of healthcare professionals. The work of regulators is complex, and there are ongoing calls for regulators to do more, engage more broadly, and take on ever more complicated issues that are of societal importance. While no one disputes the epochal significance of an issue such as AI, many regulators wonder whether they have a legitimate role to play in addressing this issue—and whether health professional regulatory bodies could even have any kind of meaningful impact if they tried. The term "scope creep" is frequently invoked to describe the concern that regulators should focus their limited energies and resources on profession-specific imperatives such as competency assessment, entry-to-practice registration, and managing complaints from the public. These activities are important, sufficiently complex, and entirely relevant to the mandate of health professions' regulatory bodies. They also consume considerable time and attention, leaving little bandwidth to properly, thoroughly, and methodically address other important issues. Regulatory culture is not rooted in risk-taking, experimentation, or trial-and-error. Regulatory practice must be consultative, deliberative, methodical, and measured—which means it takes time, concentration, and consensus building.

In contrast, the culture of technology entrepreneurs and those who are generating the revolutionary breakthroughs in AI today is often characterized by the mantra "move fast and break things." ⁹ It is a culture that prizes creative destruction, experimentation, and high-risk high-reward enterprise. Of course, such a culture is also characterized by failure, unforeseen consequences, and unanticipated harm.

Many regulators today are being asked by their registrants for guidance, rules, or regulations to guide responsible adoption of AI in daily practice. In most professions, there has been a rapid proliferation of both HiL and HoL AI-enabled technologies, and many of these address immediate workplace concerns such as operational/administrative efficiency, skills shortages, and cost-effective delivery of care. Overworked professionals may see AI as an answer to the daily grind of practice—a practical lifeline to allow them to do "more with less" and still meet their professional and ethical responsibilities for provision of quality care to patients. In many cases, AI may not even be visible—it may be embedded in systems and technologies used by professionals in ways that are not obvious. Increasingly, AI is being used by employers and organizations to address needs for cost containment or to deal with human resources shortages. They too are asking regulators of health professionals for guidance on how to ensure the investments they make in these technologies are responsible, appropriate, and aligned with regulatory expectations.

Given the clash of regulatory and entrepreneurial cultures, it is understandable that the pace of regulatory change with respect to AI has been considerably slower. As reported in the research published in this edition, few regulators in any health profession have developed or instituted regulatory change to respond to the proliferation of AI in practice, though many have begun discussing it.

What kind of regulation for AI is needed—and useful?

The question of whether and how regulators should be responding to AI in professional practice is complicated by the reality that "regulation" is a broad term covering a diverse array of independent agents in different organizations with different tools, roles, and powers. Those regulators who oversee the professional practice of healthcare workers (eg: State Boards) are a relatively small part of the system of regulation that currently exists and therefore have a relatively small amount of power with respect to what can and should be done with AI.

Those who advocate for a more modest role for health professions' regulators in the AI arena may begin by noting that there are already many different regulatory tools in place to safeguard public interests. For example, the tort law system exists to ensure that an individual harmed by the actions of another person can use civil litigation to redress the situation. In the context of either HoL AI or HiL AI, this could be interpreted as saying that a patient harmed by AI could individually or collectively sue the manufacturer/entrepreneur and if the case is proven in court, they would be entitled to compensation. The tort law system is well established, but it is slow, cumbersome, inefficient, and is only invoked after a problem arises. A core objective of health professions' regulation is to try to prevent problems from occurring in the first place. Further, navigating the civil litigation process is expensive, time consuming and requires a level of social capital and legal literacy that may disadvantage many individuals who might suffer harm. This asymmetry raises questions of equity and justice and suggests that reliance on tort law to "regulate" AI in health professions practice, while necessary, may be insufficient.

Another type of regulation involves reliance on industry standards associations. ⁸ Entrepreneurs and producers of AI have a strong interest in ensuring their products are safe, effective, and reliable as a way of insulating these businesses from negative publicity and the risk of lawsuits. One way of addressing this is for companies involved in AI to develop a self-policing system in which the industry itself defines quality standards as a way of proactively preventing their products from causing harm. Industry standards associations are a well-established mechanism that still prioritizes innovation and competitive capitalism, but in a way that established guardrails for everyone's interest—including those who profit from the sale of a product. While industry standards associations may be part of the regulatory framework for AI in healthcare, it is unlikely this approach alone (or in tandem with tort law) would be sufficiently acceptable by either the public or health care professionals. Concerns regarding profit incentives, public distrust of large corporations (particularly in the technology industries), and the size and wealth of these companies may all undermine well-intentioned efforts to allow the industry to set its own standards.

In some parts of the world, elected governments have taken a more muscular and direct approach in trying to regulate technologies and the companies that produce them. Currently, the European Union has advanced some of the most formal types of legislation to ensure responsible adoption of AI, not just in health care but by society as a whole. ¹¹ The EU approach is aligned with the European philosophy of the role of government in general—activist, interventional, and a counterweight to for-profit corporations. ¹¹ The legislation is focused on different risks posed by different kinds of AI and requires manufacturers and vendors of products to prove the AI is safe and meets acceptable standards for public protection. The legislation has been decried as heavy-handed by many technology entrepreneurs, who claim that legislative intervention of this sort will stifle innovation and suppress the full potential of AI to address all manner of problems, in healthcare and beyond. In contrast, the American government has adopted a stance towards de-regulation, aligning itself with those who believe that regulation can, in some cases, do more harm than good, especially if it interferes with the evolution and growth of a transformational innovation such as AI. ¹¹

Preferring to rely upon marketplace mechanisms, competition, the courts, and other existing regulatory safeguards is seen as a way of supporting innovation and entrepreneurship, even if it gives the appearance of a "wild west" approach.

To regulate—or not?

Those who are supportive of a more formalized, interventional approach to regulation of AI in healthcare may refer to "the Facebook problem". At the dawn of the social media era, tools such as Facebook were curiosities, toys used by bored college students that provoked little concern but much interest. Since its development and explosive growth, the realities of Facebook and other forms of social media have raised significant concerns regarding psychosocial, political, economic, and other harms on a society-wide scale. ¹² These concerns have prompted even the United States government to consider intervention to safeguard public interests. As has been seen however—the scale, power, and ubiquity of social media is so vast that no force on the planet can constrain it at this point. Given the technological superiority of AI over "simple" technologies such as Facebook or Google, the concern that a similar pattern to Facebook will replicate with AI (but at an even larger scale and more rapid pace) raises the urgent need to regulate now, before it is too late to do so in the future. While it is true that regulators of health professionals are a relatively small group with limited power, within the sphere of health care work, they have considerable influence, and thus an opportunity to at least do something. From this perspective, there is a time pressured urgency for health professions' regulators to act in developing the regulatory guardrails necessary to prevent "the Facebook problem" from erupting with AI-driven health care. Failure to use existing regulatory tools today will make it impossible to do so in the future as AI becomes bigger and more ensconced. Admittedly, regulators of health professions acting alone cannot take on the corporate might of technology companies and innovators. Regulators working with other regulators, partnering with end-user professionals, organizations and employers, and most importantly with the public could potentially create a regulatory framework that would still encourage innovation but within a perimeter of acceptable and safe professional practice that still prioritizes human and professional values. For example, the use of HoL-AI could be limited to specific circumstances in healthcare work or could be subject to audit and outcome measurement on a periodic basis to ensure it is safe and effective. Health professions' regulatory bodies could use their existing networks and tools to craft rules that may be important to safeguard public interests.

This debate as to whether health professions' regulators should engage in regulation of AI in health care work highlights important existential questions for health professions' regulation itself. There are legitimate concerns that attempting to regulate AI—especially HiL-AI is an example of scope creep. It is not unreasonable to claim that attempts to regulate HoL-AI will be futile, given public acceptability and the power of big technology companies. Further, and in the United States in particular, the current social and political climate may be averse to expansive regulation that is seen to stifle innovation, increase cost, or result in more bureaucracy. Still, the potential harms and risks of AI are substantial and speak directly to the role of a health professions' regulatory body. The risk of deskilling that may occur with reliance on either HiL-AI or HoL-AI raises concerns regarding competencies of health professionals—and what happens should technology fail for prosaic reasons such as a power failure. Well-documented concerns regarding AI hallucinations or algorithmic bias raise concerns of system inequities becoming even more entrenched than they are now. The risk that the expense of AI may further exacerbate socioeconomic disparities is also real: will the best health care only be available for the wealthiest, with substandard healthcare on offer for everyone else? While regulation cannot be expected to address all these issues, health professions' regulators may have some opportunities to steer the technology in a more equitable direction, but only if they firmly and act now—before the Facebook problem surfaces. Table 1 presents a summary "pros-and-cons" table highlighting some factors regulators may need to consider in making decisions with respect to regulation of AI in professional work.