Implementing Strategic Changes to the USMLE

In Support of the Licensing Decision

While the use of USMLE results to inform the licensing decisions by medical boards has been the primary purpose of USMLE from the program's inception, the CEUP recommendations triggered an in-depth evaluation of the value of the current program in this regard. For all stages of test design and content development, test committees are asked to consider the importance of the various content areas and testing formats for the decision to grant an independent license for unsupervised practice or for the earlier decision to grant a more limited, resident license. USMLE staff have conducted a series of practice analyses intended to inform these considerations. The first consisted of a survey of new residents and was aimed at collecting information on their clinical responsibilities as they began that new phase of their training.7 The second focused on newly licensed physicians and the key components of the clinical context within which they were practicing as an independent physicians.8

BECAUSE THE SCIENCE AND PRACTICE OF MEDICINE EVOLVES OVER TIME, THE USMLE HAS PERIODICALLY CONDUCTED COMPREHENSIVE, STRATEGIC REVIEWS OF THE PROGRAM.

Results of these efforts, as well as the continued scanning of the education and practice environment by staff and committee members, help maintain the licensure focus of USMLE.

Physician Competencies

For more than a decade, organizations responsible for the specialty certification of physicians have made core physician competencies the focus of their credentialing and assessment process and during this time the undergraduate and graduate educational communities have devoted increasing attention to the competencies that should be mastered and demonstrated at various stages along the training continuum.9 For USMLE, this CEUP recommendation represented a call to be attentive to the competency-related focus that was already pervasive in the house of medicine and to use a similar framework for the design of the licensing examinations. This CEUP recommendation was in part responsible for a significant effort to expand the coding and tracking of USMLE test content according to physician tasks and competencies, and this design dimension, along with others that reflect disease categories and organ systems, is used in developing USMLE examinations.

THE PROGRAM HAS INCREASED TEST CONTENT THAT REQUIRES DEMONSTRATION OF QUANTITATIVE SKILLS, INCLUDING REQUIRING EXAMINEES TO EVALUATE THE QUALITY AND APPLICABILITY OF EVIDENCE PRESENTED IN SCIENTIFIC ABSTRACTS.

Importance of Foundational Medical Science

This CEUP recommendation emphasized what had already been a priority for the USMLE program, to assess knowledge and application of medical science in a way that is clinically relevant and important. For years prior to the CEUP recommendations, test design and development committees had been identifying and gradually reducing questions focusing on the recall of isolated medical facts in favor of test items that were more patient-oriented and required knowledge of the scientific underpinnings of disease in the context of patient care. The CEUP recommendation in this area not only added a sense of urgency in this transition, it also suggested that the scientific foundations of medicine should be more broadly assessed in the USMLE sequence. As a result, test committees have been developing additional test content addressing this area that would be appropriate for assessments related to decisions about independent practice. This material will soon become a more prominent part of the Step 3.

Assessment of Clinical Skills

The CEUP recommendations called for further enhancement of the assessment of clinical skills. To some degree this is being addressed by development and inclusion of multiple choice question formats that prompt examinees to draw upon their knowledge of appropriate skills for a patient encounter and the interpretation of patient history and physical findings in the clinical situation. However, the more direct assessment of clinical skills currently occurs in the Step 2 Clinical Skills (CS) program, which requires examinees to interact with standardized patients who portray patients presenting with a variety of clinical issues. This program was introduced in 2004 and has undergone a series of revisions since that time. Most recently, and coincident with the CEUP recommendations, the Step 2 CS has introduced changes to increase the authenticity of examinee-“patient” interactions, to provide a richer assessment of communication skills, and to require examinees to more clearly demonstrate their data interpretation skills when completing a patient note after the encounter.

Interpreting and Using Clinical Evidence

While the USMLE has always required examinees to demonstrate to some degree their knowledge of biostatistics and epidemiology, the CEUP recommendations called for an increased emphasis on evidence-based medicine throughout all of the examinations. To achieve this, the program has increased test content that requires demonstration of quantitative skills, including requiring examinees to evaluate the quality and applicability of evidence presented in scientific abstracts and in mock pharmaceutical advertisements. Research is underway to evaluate the possibility of requiring future examinees to search for information relevant to a clinical presentation, to evaluate the quality of the information, and to apply it in the management of patients.

Changes to the USMLE Step 3

In addition to the significant work underway as discussed, there are substantive changes that will be introduced to the Step 3 beginning in November 2014. These changes impact not only the content but the format and structure of the Step 3. For examinees, the most apparent change to Step 3 will be the distinct content focus for each of the two days' of testing. Day 1 content has been named as the Step 3 Foundations of Independent Practice (FIP) and will focus on assessing an applicant's knowledge of basic medical and scientific principles essential for effective health care. Content areas covered will include application of foundational sciences; understanding of biostatistics and epidemiology/population health, and interpretation of the medical literature; and application of social sciences, including communication and interpersonal skills, medical ethics, systems-based practice and patient safety. The FIP content will also assess an applicant's knowledge of diagnosis and management, particularly focused on knowledge of history and physical examination, diagnosis, and use of diagnostic studies. The Step 3 FIP will consist solely of multiple-choice questions and will include some of the newer item formats, such as those based on scientific abstracts and pharmaceutical advertisements.

STEP 3 WILL CONTINUE TO BE ADMINISTERED OVER TWO DAYS WITH A TOTAL TESTING TIME COMPARABLE TO ITS CURRENT LENGTH.

Day 2 content has been named Step 3 Advanced Clinical Medicine (ACM) and will focus on assessing an applicant's ability to apply comprehensive knowledge of health and disease in the context of patient management and the evolving manifestation of disease over time. Content areas covered will include assessment of knowledge of diagnosis and management particularly focused on prognosis and outcome, health maintenance and screening, therapeutics, and medical decision-making. Knowledge of history and physical examination, diagnosis, and use of diagnostic studies will also be assessed. This test day will include multiple-choice questions and computer-based case simulations.

While this represents an important change in Step 3 content, there are key aspects of the exam that remain unaltered. Step 3 will continue to be administered over two days with a total testing time comparable to its current length and the exam will continue to consist primarily of multiple choice questions and computer case simulations with a single three-digit numeric score and pass/fail outcome reported. Additionally, the Step 3 will continue its fundamental focus on the knowledge and application of biomedical and clinical sciences necessary for the unsupervised practice of medicine.

As the Federation of State Medical Boards is the registering entity for all Step 3 applicants, operational preparations have been underway for some time to accommodate these changes. In mid-July, the Federation closed registration for those individuals who will sit the Step 3 with the current content. In August, the Federation opened registration for the revised Step 3 examination that will be administered beginning in November 2014. It is anticipated that the first scores under the new Step 3 content will be released in early April 2015.10

While this represents a substantial amount of time before the first release of scores, it is necessary in order to accumulate a sufficient volume of examinees to complete analyses that underlie scoring and reporting. This timeline for the new Step 3 (with administrations beginning in early November and score release in early April) should not disrupt the process leading to decisions made by medical boards and residency programs around July 1 dates. Although such a long interval in the release of Step 3 scores is unusual, it is not unprecedented. Indeed, the timeline for the new Step 3 content in 2014–2015 is similar to that utilized when the Step 3 transitioned from paper-pencil administrations to computer-based testing (CBT) over a decade ago (i.e., first CBT administration of Step 3 in November 1999 with a score release in April 2000).

From the perspective of medical licensing authorities, one other change is important to note for Step 3 beginning with registration for the new exam in August 2014. Since the administration of the first Step 3 examination in 1994, state medical boards have had a direct nexus with the exam through their role in setting any state-specific eligibility criteria for those applicants sitting Step 3 on their behalf. In other words, all Step 3 applicants have been ‘sponsored’ by a specific state medical board who established any time or attempt limits on sitting Step 3, a minimum number of months of graduate medical education, requirement that a license application and fee be on file with the board, etc. This state board sponsorship role for sitting Step 3 will be discontinued when registration commences in August for the new Step 3 content. With the introduction of the restructured Step 3 examination, the USMLE program will no longer require examinees to apply for Step 3 under the eligibility requirements of a specific medical licensing authority.

In making this decision, the USMLE Composite Committee took its guidance from the state medical board community. Staff and members from state medical boards provided feedback to USMLE in multiple venues in 2012 concerning the state board sponsorship role including the FSMB annual meeting, regional meetings of Administrators in Medicine (AIM), the State Board Advisory Panel to USMLE and an FSMB Roundtable webinar. Additionally, a fall 2012 survey of state medical board executive directors found 89% of responding boards with a preference to discontinue board sponsorship for Step 3 in favor of dealing directly with physicians only after they completed the entire USMLE sequence and had met all other requirements for licensure in their jurisdiction. Feedback from these various quarters made it clear that as the administration of USMLE evolved over the past twenty years, it was now preferable for state boards to interact with examinees after completing the entire USMLE sequence and were ready to seek a full, unrestricted licensure in that jurisdiction. State boards preferred to interact with USMLE as one of the multiple credential pieces that licensure applicants presented to the board. Examinees have been advised that removing state board sponsorship as part of the Step 3 examination application does not impact initial medical licensure requirements in the United States and most medical licensing authorities have, and will continue to maintain, specific criteria for completion of the USMLE, such as time and attempt limits.11

THE USMLE PARENT ORGANIZATIONS VIEW THE PARTICIPATION OF STATE BOARD MEMBERS AND STAFF IN THE USMLE AS CRITICAL TO THE PROGRAM'S CONTINUED SUCCESS AND INTEGRITY. STATE MEDICAL BOARDS HAVE HISTORICALLY BEEN A STRONG PRESENCE.

Although this represents a change to how state boards have interacted with the USMLE program and Step 3 in particular, it is important to highlight that state boards will continue to have an important connection with the program through membership on USMLE committees (e.g., the Composite, Budget and Management Committees, item writing committees) and participation in standard setting and advisory panels. The USMLE parent organizations view the participation of state board members and staff in the USMLE as critical to the program's continued success and integrity. State medical boards have historically been a strong presence within the program and continue to serve in critical roles. In 2012–2014 alone, individuals from 34 state medical boards were active participants on USMLE committees, panels and workshops. Clearly, interest in the program remains strong as evidenced by the dozen scheduled attendees set for the program's annual USMLE workshop in fall 2014.

As the USMLE moves deeper into its third decade, the program continues its engagement in strategic enhancements designed to keep the program aligned with the needs of state medical boards, with the realities of American medical education and training, and with advances in assessment science intended to better support licensure decisions and the protection of the American public.