The United States Medical Licensing Examination (USMLE): Maintaining the Integrity of the Examination Process

WHAT IS IRREGULAR BEHAVIOR?

The USMLE program has established rules regarding eligibility for the examinations and rules to govern administration of the examinations to ensure that no examinee, or group of examinees, receives unfair advantage on the examination, whether inadvertently or otherwise. The rules include standard test administration conditions consistent with the principles on which the examinations are developed and scored. For example, examinations are designed to sample knowledge across specified content domains and unauthorized access to examination content prior to testing violates that principle.

In submitting an application to sit any USMLE Step or its component, an individual certifies that the information provided on the application is true and correct, and acknowledges having read, and agreeing to abide by, the policies and procedures described in the current USMLE Bulletin of Information (Bulletin). The Bulletin, which is updated annually, spells out the steps taken to assure the appropriate administration of the USMLE Step examinations and sets forth the policies and procedures under which applicants and examinees apply for and sit a Step or its component. The Bulletin also presents the rules of conduct (see Table 1), which examinees are expected to follow in sitting for any USMLE Step.

So what does the USMLE program mean by “irregular behavior?” The USMLE program defines irregular behavior broadly as “any action by applicants, examinees, potential applicants or others when solicited by an applicant and/or examinee that subverts or attempts to subvert the examination process.” Specific examples of potential irregular behavior include falsifying information; giving, receiving or obtaining unauthorized assistance during the examination; reproduction of examination materials; altering or misrepresenting scores; disruptive or unprofessional behavior, etc. A complete listing of potential actions that could warrant a finding of irregular behavior is provided in the Bulletin.¹

The USMLE program administers more than 100,000 examinations annually. In the overwhelming majority of cases, these Step administrations occur as scheduled and without incident. This stems, in part, from the precautionary measures taken by the program to assure a secure testing environment, e.g., audio and video taping of test administrations; secure handling of test materials, including electronic data encryption; and identity checks by proctoring staff. Additionally, the program provides examinees with a listing of the rules of conduct for testing in both the Bulletin and as part of the introductory materials viewed by examinees prior to testing. In those relatively few instances when irregular behavior is suspected, program staff work through a formal process of review and investigation.

THE INVESTIGATION PROCESS

There are a number of means by which instances of possible irregular behavior are detected by, or brought to the attention of the USMLE program. The two most common means involve the vigilance of program staff and test proctors. Staff at the Federation of State Medical Boards (FSMB), the National Board of Medical Examiners (NBME) or the Educational Commission for Foreign Medical Graduates (ECFMG) may detect discrepancies or inconsistencies in the application or other materials filed by applicants as part of a Step application. Also, on occasion, discrepancies in score reports or transcripts are brought to the attention of the program by staff at medical schools or residency training programs seeking to verify information provided by the student or resident.

Another means of detection stems from test administration reports filed by proctors on site at the testing center. The USMLE Step 1, Step 2 Clinical Knowledge (CK) and Step 3 are administered by computer at Prometric Testing Centers in the United States and around the world. The Step 2 Clinical Skills (CS) is administered at five regional testing sites within the United States. Proctoring staff at all of these sites are responsible for implementing specific test administration criteria involving the physical layout of the test center, verification of candidates’ identity prior to testing and physical proctoring of the examination. Furthermore, proctors are required to file a report with the USMLE program in any instance of an irregularity or deviation in the standard administration of a Step exam. These reports are forwarded to USMLE program staff in Philadelphia. There, an inter-organizational staff committee, comprised of representatives from the FSMB, NBME and ECFMG, meets bimonthly to review the reports. This staff committee is responsible for additional fact gathering and data analysis (when appropriate), and for determining whether there is a reasonable basis to believe an examinee violated the rules of conduct during testing and may be reasonably believed to fall under the definition of irregular behavior.

In some ways, the staff committee performs a function similar to that of a grand jury. The committee determines whether sufficient evidence exists to warrant elevating the matter to a higher level in the adjudicatory process. Cases identified as bearing sufficient evidence of possible irregular behavior are then referred to the USMLE Committee on Irregular Behavior (CIB) for a formal hearing and final determination. Routinely, the examinee’s Step score is withheld pending the outcome of the CIB referral. If the nature of the alleged irregular behavior calls into question the validity of a score that is at or above the minimum passing level, the staff group may also refer the matter to the Committee on Score Validity (CSV.)

Using the person’s last known address, the individual is notified by certified mail of the nature of the suspected irregular behavior. The individual is provided with a copy of the applicable USMLE policy and procedures. They are further advised of the information that initiated the investigation and relevant findings of the investigation.

THE COMMITTEES ON IRREGULAR BEHAVIOR AND SCORE VALIDITY

Much of the work for the USMLE program rests on a structure of physician committees. More than 40 committees and 300 individuals are involved in developing examination content and maintaining the high quality of the USMLE. These volunteers are drawn from all regions of the country, various specialties and academic, regulatory and private practice settings. These individuals offer the USMLE program a national faculty whose expertise is invaluable to the ongoing maintenance of the program.

While not involved in test development, two groups focused upon maintaining the integrity of the USMLE program are the CIB and the CSV. Both groups are appointive committees established by the program’s governing body, the USMLE Composite Committee, for the purpose of assuring the integrity of the examination system and validity of reported scores. The CIB is held responsible for reviewing cases of alleged or suspected irregular behavior that occur during the administration of a Step examination or as a part of the broader examination process. If evidence suggests that the alleged irregular behavior affects score validity, or if unexplained inconsistencies in performance within an examination or between takes of the same examination are detected, the matter may be reviewed by the CSV. If the CSV concurs that no satisfactory explanations exists for the aberrancy in performance, the results of the administration in question can be classified as indeterminate. The score for that administration is withheld and the examinee is advised of options for retaking the examination in order to validate the administration in question. It should be noted, however, that classification of a score as indeterminate does not automatically equate to inappropriate or irregular behavior on the part of the examinee.^1,2

Depending upon the volume of pending cases, these two committees meet approximately four to six times annually. Members of one committee serve as alternates on the other, and current membership includes sitting and former members of the FSMB board of directors, senior executive staff from the ECFMG, the executive directors of the Minnesota and Kansas medical licensing boards and the program director for a pediatric residency program in Pennsylvania.

MEETING PROTOCOL FOR THE CIB

Prior to its meeting, the committees review applicable USMLE policy and procedures and the relevant case materials for all individuals under review. The individuals referred to the CIB/CSV are given an opportunity to provide an explanation for the reports or other information that has been obtained and/or present other relevant information. Individuals are allowed to make personal appearances before the committee to present their case if they so desire. In making a personal appearance, the CIB asks the referred individual to provide sworn testimony with a stenographic or audio recording being made of that portion of the proceedings during which the individual is in attendance. The chair of the CIB introduces the referred individual to the members of the committee and any staff present; a staff member from the office of the USMLE Secretariat then summarizes the case, including any relevant evidence. Referred individuals are allowed to make opening and closing statements on their behalf and, in most cases, they agree to answer questions put to them by members of the committee. Legal counsel sometimes represents individuals referred to the CIB. Those individuals who choose not to make a personal appearance routinely provide the CIB with a written response presenting their explanation of the matter.

PENALTIES IMPOSED BY THE CIB

In general, the outcome of a CIB referral comes down to the committee’s determination of whether an individual engaged in irregular behavior and, if so, whether any sanction, in addition to annotation of the individual’s record, is warranted. The CIB, in some instances, may conclude that the evidence is not sufficient to conclude that the individual engaged in irregular behavior. In other situations, they may conclude the examinee’s violation of the applicable rule was inadvertent and sufficiently de minimus that only a warning letter cautioning against any future infractions is appropriate. When the CIB determines that an examinee engaged in irregular behavior, the annotation (i.e., an explanatory note) of the individual’s score report and transcripts to reflect that determination (as further described below) is the automatic and minimum sanction imposed.

The CIB also determines whether the violation is egregious enough to warrant reporting to the FSMB’s data bank of disciplinary actions. Most state medical boards check this Board Action Data Bank routinely as a standard element of their process for issuing a medical license. Every USMLE transcript carries an annotation indicating whether or not a disciplinary action is on file with the FSMB Board Action Data Bank. Previous actions that warranted inclusion in the FSMB Board Action Data Bank include falsification of medical credentials, falsification of score reports and transcripts, attempts to access unauthorized materials during the test and disruptive behavior during the test.

Additionally, if the CIB finds that the irregular behavior is egregious and/or threatens the integrity of the examination system, the CIB may implement special administrative procedures for future examinations and/or may recommend that the individual involved be barred from future examinations. A recommendation by the CIB to bar an individual from the USMLE is automatically referred to the Composite Committee for its approval, with the bar being in effect in the interim until the next meeting of the committee. Almost all decisions to bar an examinee from future administrations of the USMLE have been time limited to three years. Also, in instances in which the nature of the irregular behavior raises concerns about the validity of the score obtained, e.g., unauthorized assistance during the examination, the CIB may refer the matter for further review focused upon the issue of whether the score will be reported. Whatever the decision of the CIB, a “determination” letter—presenting the committee’s findings and final decision, including any penalties imposed— is drafted by staff at the office of the USMLE Secretariat and forwarded via express courier to the referred individual.

CIB DECISIONS IN 2004

In 2004, the CIB heard 57 cases and in 24 instances found that the referred individual engaged in irregular behavior. Penalties imposed included annotations to the individual’s USMLE transcript, reporting to the FSMB’s Board Action Data Bank and barring an individual from the USMLE for a specified period of time. The most common infractions presented to the committee included procedural violations (e.g., unauthorized possessions such as cell phones in the testing area), security violations (e.g., accessing notes during the exam), falsifying an official document and disruptive behavior. In 16 of the 24 instances in which the CIB found individuals to have engaged in irregular behavior, it imposed the maximum penalties: annotation of the USMLE transcript, a bar from the examination and reporting the CIB’s determination to the FSMB Board Action Data Bank.

THE USMLE TRANSCRIPT

Because the USMLE program believes that medical licensing authorities should possess all relevant information necessary in making a decision to issue a medical license, the USMLE record of an examinee found to have engaged in irregular behavior is clearly annotated. If the action occurred in connection with a specific administration of a Step, the individual’s USMLE score report and transcript will carry the phrase “IRREGULAR BEHAVIOR” next to the reported score. In situations where the infraction occurred outside the context of a specific test administration, the bold cap “IRREGULAR BEHAVIOR” appears at the bottom of the transcript.

Beginning in 2001, the USMLE program provided transcript recipients with even more explicit information on irregular behavior. All instances of irregular behavior occurring on or after Jan. 1, 2001 have resulted in a brief descriptor listed below the comment IRREGULAR BEHAVIOR on the USMLE transcript. Additionally, the USMLE program proactively provides with such transcripts, the determination letter sent to the examinee explaining the finding of irregular behavior and the details on the basis for that finding. If the transcript recipient still has questions, a telephone call can be placed directly to the staff at the office of the USMLE Secretariat for clarification and/or additional documentation can be requested.

CLOSING

Maintaining the integrity of the USMLE is critical if state medical boards are to rely upon the Step examination as an integral part of their decision-making process for licensure. As described above, a great deal of time and effort is devoted to assuring that there is compliance with the rules designed to provide a standardized and fair testing experience for all examinees and to assuring the integrity of the testing program. It is hoped that a finding, reached after careful consideration and with attention to due process, that an individual engaged in conduct that would or could subvert the medical licensing examination process is of relevance to all licensing authorities in assessing the character and fitness of applicants for licensure.