Introduction: Few dispute that communication skills are essential for physicians; physicians must both know and do. In the practice of medicine, knowledge is applied in the context of patient encounters, and physicians must have the skills necessary to gather information from (and transmit information to) patients, families, and colleagues. Research studies and practical experience indicate that some physicians have an adequate knowledge base, but display unsatisfactory clinical and communication skills. Such physicians pose a potential threat to the public, at least as great as that posed by clinicians with inadequate knowledge.
BACKGROUND
Physicians, educators, specialists in evaluation design, representatives of the licensure community, and lay persons associated with the National Board of Medical Examiners (NBME) have repeatedly acknowledged the need for a test of clinical skills since the bedside oral examination was dropped from NBME Part III in 1961. In 1973, the Committee on Goals and Priorities (GAP), in a report entitled “Evaluation in the Continuum of Medical Education” (CME), recommended that “the NBME, in cooperation with the Federation of State Medical Boards, should develop an examination to evaluate performance characteristics requisite for providing patient care in a supervised setting,” and that “the examination should be administered at or near the end of undergraduate medical education.”
In 1976, in response to the recommendations of the GAP report, research was begun to assess skills that were not assessed well with multiple choice question (MCQ) examinations. This work received significant funding from The Robert Wood Johnson Foundation; the Josiah Macy, Jr. Foundation; the W.T. Grant Foundation; the Commonwealth Fund; the Department of Health, Education, and Welfare, Health Resources Administration, Bureau of Health Manpower, Division of Medicine; and the American Board of Family Practice. The largest field trials of evaluation of clinical skills using standardized patients up to that time were conducted during 1975 and 1977.
In September 1985 an NBME Task Force on Clinical Skills chaired by George E. Miller, MD, concluded that the NBME must play a leading role in the development of Clinical Skills Assessment (CSA), with the expectation that it would ultimately become an integral part of the certification procedure. Further support for the proposal was received from the joint FSMB/NBME Task Force to Study Pathways to Licensure in 1989. The Task Force supported a test of clinical skills as either a component of the licensure examination or as an eligibility requirement for entry into graduate medical education, and anticipated including a test of clinical skills as part of the licensure examination once valid and reliable methods became available. In 1995, the Composite Committee of the United States Medical Licensing Examination (USMLE) adopted a strategic plan for the enhancement of USMLE that included implementation of a test of clinical skills using Standardized Patients (SPs). Responding to this recommendation, the FSMB House of Delegates and the members of the NBME endorsed the inclusion of a test of clinical skills using standardized patients in USMLE.
The NBME has devoted considerable expertise and financial resources to investigations of the theoretical and practical implications of large-scale evaluation of clinical skills using standardized patients. Prototype examinations have been delivered to thousands of medical students at more than 20 cooperating US medical schools since the GAP report. In 1998, the NBME’s Standardized Patient Implementation Advisory Committee (SPIAC) reviewed the results of this research and concluded that valid and reliable methods for evaluating clinical skills were available.
On the basis of the SPIAC report, the Composite Committee approved a test of clinical skills using SPs in the USMLE in January 1999. At their annual meetings in the following March and April respectively, the NBME and FSMB governing bodies again endorsed the plan to incorporate a test of clinical skills in USMLE. The governing bodies delegated oversight of the process to the NBME Executive Board and the FSMB Board of Directors, with advice from the Composite Committee.
In June 2001, the NBME Executive Board directed the staff of the NBME to proceed with final tests of logistics of large-scale evaluation of clinical skills using the delivery model described below. The Executive Board directed that NBME seek the collaboration of the Educational Commission for Foreign Medical Graduates (ECFMG) to accomplish these tests efficiently and expediently. At the direction of its Executive Board, the NBME has entered into agreements with the ECFMG to combine the experience, expertise, and resources of the 2 organizations in the final tests prior to implementation of the examination.
The NBME Executive Board and the FSMB Board of Directors are committed to prompt inclusion of a test of clinical skills using standardized patients in the USMLE. At a joint meeting in February 2002, both boards reaffirmed that an examination of clinical skills in the United States Medical Licensing Examination is desirable and will improve the protection of the public.
RATIONALE
A separate measure of clinical and communication skills would be unnecessary if existing MCQ examinations adequately identified both candidates with insufficient cognitive knowledge and those with inadequate clinical or communication skills. Research conducted by NBME and the experience of other groups conducting large-scale tests of clinical skills in North America indicate convincingly that this is not the case.
The Medical Council of Canada currently administers a 2-part certification examination. The first section (MCCQE Part I) is composed of multiple choice and short answer items, which assess cognitive and clinical reasoning skills. The second component, MCCQE Part II, is a 14-station Objective Structured Clinical Examination (OSCE) using standardized patients. Successful completion of Part I is a prerequisite for Part II.
Approximately 11% of the more than 10,000 candidates who attempted the Part II examination between 1992 and 1999 failed on their first attempt. More than 4% of examinees who successfully passed MCCQE Part I have not passed the OSCE portion of the licensing examination, even after several attempts.
ECFMG has administered a clinical skills examination as part of its certification process for approximately 4 years. Most candidates for the ECFMG test of clinical skills have passed Steps 1 and 2 prior to attempting the clinical skills examination. Despite this, the first-time failure rate for the ECFMG exam is approximately 20%, and to date, more than 5% of candidates have not passed the examination on repeated attempts.
NBME research studies indicate that the degree of correlation between performance on clinical skills examinations and other USMLE components is moderate. Based on field studies conducted during the past 10 years, NBME estimates that approximately 3 to 5% of students in US medical schools will be unsuccessful on initial attempts. Ultimate failure rates will be lower, possibly in the 1 to 1.5% range. Using these estimates, approximately 160 to 300 candidates from each annual US cohort will not be licensed based on failure to demonstrate adequate communication or clinical skills. This is approximately the same number of candidates from each cohort who fail to pass the cognitive components of the USMLE.
LOGISTICS
The NBME clinical skills examination has been designed to certify that a candidate possesses the requisite clinical and communication skills necessary to practice medicine in the supervised setting of postgraduate training. Candidates for the examination from accredited US medical schools would most likely take the examination at the end of their third year or during their fourth year of medical school. If the examination is incorporated into the USMLE, ECFMG candidates would also be required to take the examination prior to entry into postgraduate training.
The “live” examination will consist of 10 to 12 SP case encounters. Each encounter lasts approximately a half hour, and the overall duration of the exam is 5 to 6 hours. SPs are selected to represent a broad range of age, racial, and ethnic backgrounds. For practical and ethical reasons, no pediatric patients are included, although some encounters deal with standardized patients acting as parents. Examination forms are balanced by gender. Each encounter consists of a 15-minute interview with an SP, followed by a 10-minute interval during which the candidate records pertinent history and physical examination findings, lists diagnostic impressions, and outlines plans for further evaluation, if necessary. Candidates are expected to establish rapport with patients, elicit pertinent historical information from the patients, perform focused physical examinations, answer questions from the standardized patients, and provide counseling where appropriate.
Approximately 17,000 US and Canadian first-time candidates and 6,000 ECFMG first-time candidates are expected to take the examination annually. The size of the candidate pool for the USMLE precludes a one-day administration, as currently employed in Canada. Continual administration will be required, regardless of the final choice of a delivery model. Extrapolation from current and predicted failure rates indicate that the annual volume of first-time takers and initial repeaters will probably range from 24,000 to 26,000 candidates. Excess capacity will be required to permit scheduling flexibility; current projections call for a system with the ability to test 30,000 candidates per year.
The NBME has explored a variety of delivery networks. Initially, the NBME hoped to deliver the examination at as many as 50 medical school sites. The NBME has also considered a narrower network, combining 4 or 5 small semi-permanent sites with up to 12 medical school sites. Unfortunately, it is extremely difficult to adequately assure standardization and security in these models, and the facility, equipment, and training costs are prohibitive. After careful analysis of logistics, costs, and exam security, the NBME has concluded that a delivery system based initially at 5 to 7 regionally located fixed sites operated year-round is the least expensive, most equitable, and most sound method for delivery of the clinical skills examination.
FINANCIAL PROJECTIONS
The costs associated with a high-quality standardized test of clinical skills are high. The fee for the current Canadian examination and the ECFMG clinical skills is $1,200 or more. NBME has been able to estimate some of the costs of inclusion of a clinical skills examination in the USMLE from past experience with other NBME exam programs, as well as its own standardized patient program. Other costs have been estimated using information provided by the ECFMG.
Using a combination of available information and careful estimates, the NBME has estimated that the “cost” per examination in the fixed-site delivery model under consideration will be approximately $950, exclusive of travel expenses. The costs of delivery at medical schools would be much higher. Estimates of the cost per student for delivery of the examination in either of the medical school models considered by the NBME range from $1,600 to $2,000. The NBME is now testing these projections prior to final implementation. This will be the primary focus of field trials in 2002. It is important to remember that the price of the examination has not been set, but it is likely to be as close to the actual cost as fiscally prudent.
2002 PLANS
The NBME and the ECFMG have finalized cooperative agreements for 2002–03 field tests, which provide for sharing information regarding examination design and delivery, including information related to case development and training of standardized patients. In addition, the NBME and ECFMG have agreed to open a center for administration of a clinical skills examination in Atlanta, Georgia. The Atlanta site and the current ECFMG clinical skills center in Philadelphia will be used for field tests in 2002. The NBME and ECFMG have agreed to administer an exam that will simultaneously serve as a “live” examination for purposes of ECFMG certification, and as a pilot test for NBME research studies.
CONCLUSION
The findings of the NBME, the ECFMG, and the Medical Council of Canada indicate that clinical skills examinations measure competencies different from those measured by traditional tests. The NBME and the FSMB believe that a SP exam can be successfully incorporated into the USMLE sequence. If final tests scheduled for 2002 meet expectations, it is likely that a test of clinical skills will become part of the USMLE in 2004 or 2005. A fixed-site delivery system is most practical, and the NBME anticipates opening 5 to 7 testing centers at strategic locations in the US.
The ECFMG clinical skills examination and the MCCQE Part II are hurdles that a substantial number of candidates are unable to meet. A clinical skills component of the USMLE will probably have the same results.
