An Assessment of USMLE Examinees Found to Have Engaged in Irregular Behavior, 1992–2006

Demographics

Male examinees are over-represented in the irregular behavior cohort when compared with their historical presence among all USMLE first-taker examinees between 1992–2006. As seen in Tables 1 and 2, male examinees accounted for 66% of the irregular behavior cohort but represented only 58% of USMLE test administrations during the same period. Conversely, female examinees are under-represented. Female examinees from U.S./Canadian medical school programs were the subgroup least represented among the irregular behavior cohort when compared with the overall historical presence as first-takers of USMLE examinations. A predominance of males in the irregular behavior cohort is not entirely unexpected, though published studies on cheating among medical students are limited and have not routinely addressed gender differences.⁷

International medical graduates (IMGs) accounted for 78% of the irregular behavior cohort, a percentage well above their historical presence in 49% of all USMLE test administrations between 1993 and 2007² or in 43% of all first-taker administrations from 1992–2006 (Table 2). Male IMGs comprised the largest subgroup, with 222 individuals, or 51%, of all individuals in the irregular behavior cohort. While a definitive explanation for the disproportionate presence of IMGs is lacking, it seems possible that cultural differences play some role in this regard.^8,9

Another characteristic of the irregular behavior cohort considered was age. Specifically, the study looked for differences in mean age within the cohort and compared with the overall population of USMLE applicants and examinees. Among the irregular behavior cohort, no significant gender differences in mean and median ages were present. However, the mean age of the IMG subgroup (34 years) was higher than that of the U.S./Canadian subgroup (30 years). This may be attributable to timing differences by which IMGs interact with the USMLE. ECFMG internal data shows 70% of all Step 1, Step 2 CK and Step 2 CS applicants in 2008 reported having already graduated from medical school. Thus, most IMGs do not take their first Step examination until after graduation from medical school in their native country—a marked difference from the students of LCME-accredited medical education programs who routinely take Steps 1 and 2 before leaving medical school.¹⁰

The mean age of the U.S./Canadian subgroup appears somewhat higher than what would be anticipated based upon the average age of matriculants to LCME-accredited programs. Data from the Association for American Medical Colleges for the time period 1992–2008 indicates a mean age at matriculation of 23–24 years for both male and female students.¹¹ Because the infractions for the U.S./Canadian subgroup were fairly equally distributed across all three Steps and presuming the sequence by which this group proceeds through the USMLE (i.e., Steps 1–2 prior to graduation; Step 3 during residency training), one might have anticipated a mean age for this subgroup perhaps closer to 27–28 years.

The mean age for the IMG irregular behavior cohort (34 years) seems consistent with what one might anticipate based upon the later start for most IMGs first interaction with the USMLE program. An FSMB query of all IMGs sitting USMLE Step 1 between 1992–2006 provided a mean age of 30 years.

Infractions

Table 3 summarizes the infractions identified by the CIB as their basis for the irregular behavior finding. Timing violations constituted the most common irregular behavior infraction—163 findings in 433 cases. This infraction (specific to the paper-pencil test administrations from 1992–1999) involved marking answers or “bubbling” on the answer sheet after time was called ending an examination or section within an examination. Because scoring for the USMLE was cumulative and guessing was not penalized, examinees had an incentive to complete all items and leave no omissions on the answer sheet.

Setting aside infractions unique to the pencil-paper format utilized until 1999 (e.g., timing violations, looking at another examinee's test book), the most common irregular behavior infractions fall under the general heading of falsification. This might involve providing false information as part of the application for a USMLE Step or falsifying/altering a document (e.g., score report, diploma, document signatures). Nearly one-third of cases (129/433) involved some form of falsification. This is not surprising based upon published studies of academic cheating that identified comparable behaviors, e.g., fake lab or research results, false information on a patient chart.^12,13,14 This is also consistent with reported problems centered upon falsified credentials and/or application materials for exams pre-dating the USMLE, i.e., the Federation Licensing Examination (1968–1994).¹⁵ The next most frequent infraction involved making, possessing or accessing notes during a test administration (58/433).

The transition to computer-based testing in 1999 and broad reach of the Internet created new and different security challenges and potential infractions for irregular behavior. USMLE applicants are prohibited from “any unauthorized reproduction…of examination materials by any means, including the Internet.”² The program actively monitors chat rooms and discussion boards dedicated to USMLE topics and has successfully pursued 21 cases of irregular behavior involving “web posting.” The prevalence and increased sophistication of electronic devices (e.g., cell phones, scanners) pose a continuing potential threat to examination security. These are reflected in many of the irregular behavior cases for “security” and “procedural” violations.

Sanctions Imposed by the Committee on Irregular Behavior: Barring an Individual from USMLE Administrations and Reporting to the FSMB Board Action Data Bank

As noted previously, allegations of possible irregular behavior are generally routed through a staff committee for additional research and/or review. Internal data from the Office of the USMLE Secretariat shows that in recent years (2004–2008) this staff committee reviewed approximately 400 incident reports annually. This committee provides a function akin to a grand jury by reviewing available information to determine whether sufficient evidence exists to warrant case referral to the CIB for formal disposition. Once a case is referred, the CIB conducts a review of the evidence, including any statement from the individual under investigation, and make its determination—either a finding of irregular behavior or a finding of no irregular behavior. While this study is focused on individuals deemed to have engaged in irregular behavior, it should be noted that the CIB issued a finding of no irregular behavior at comparable rates for both U.S. and IMG candidates—39% and 45% respectively for the period 2002–2006.

Records from the Office of the USMLE Secretariat show 703 cases referred to the CIB between 1992 and 2006. The committee found for irregular behavior in 61.6% of the referred cases (433/703). A finding of irregular behavior results in an automatic annotation to the individual's USMLE record. This is a significant action as any USMLE transcript subsequently produced by the program alerts the transcript recipient (e.g., residency program, state medical board) to the irregular behavior finding.

For those individuals found to have engaged in irregular behavior, two further sanctions are possible. The first involves the decision to bar the individual from future USMLE administrations. (See Table 4.)

Nearly one-third of all irregular behavior cases (121/433, or 27.9%) resulted in the imposition of a bar. Most of these bars (84/121, or 69%) were for a period of 1–3 years. In 29 of these cases, the bar was “tolled,” meaning the time-specific bar did not take effect until the individual established, or re-established if applicable, their eligibility for USMLE. In these instances, the actual length of time the individual was precluded from sitting USMLE was likely even longer than the time-specific bar.

A small number of bars (17/121, or 14%) were “indefinite,” i.e., the CIB precluded the individual from sitting USMLE until they met all eligibility criteria and/or until a state medical board, fully apprised of the facts of the case, requested that the individual be able to sit the examination. The average bar imposed by the CIB ran approximately 2 years.

In reviewing the 121 cases for which the CIB imposed a bar, it appears the committee reserved this sanction primarily for cases involving issues of “character.” These cases all appear to have involved premeditation and intent to deceive, e.g., falsifying applications, unauthorized access or dissemination of test materials, altering key documents such as score reports or diplomas. It would seem these actions are viewed by the CIB as inherently more egregious than the procedural violations (e.g., timing violations) characterizing the paper-pencil era. (While timing violations might also be seen as “character” issues, only one of these cases was sufficiently egregious—and with other factors involved—that a bar was imposed).

The decision to limit access to USMLE occurred at a rate fairly close to the overall demographic composition of all examinees deemed by the committee to have engaged in irregular behavior during the 1992–2006 time period. In comparing the demographic breakdown for individuals sanctioned with a bar (Table 5) with the overall demographics of the irregular behavior cohort in Table 1, it appears that IMG males were somewhat more likely to have a bar imposed by the CIB.

Upon a finding of irregular behavior, the CIB then makes a separate decision whether this determination should be reported to the FSMB board action data bank. For the 433 cases of irregular behavior, approximately one-third (142/433, or 32.7%) also resulted in a decision to report their finding to the board action data bank. This sanction ensures that any inquiry by a state medical board to the FSMB board action data bank will result in information on the irregular behavior finding being shared with the inquiring board. It also means that any USMLE transcript will also carry an annotation indicating board action history exists on the individual and a copy of the board action report (with information explaining the basis for the board action) would be included with the transcript.

The decision by the CIB to report a finding of irregular behavior to the FSMB board action data bank (Table 6) occurred at a rate consistent with the overall demographic composition of the irregular behavior cohort (Table 1).

Completing the USMLE Sequence

Fewer than half of the individuals with findings of irregular behavior from 1992 to 2006 were subsequently successful in completing the USMLE sequence (198/433, or 45.7%) by the end of 2008 (Table 7).

This overall percentage is lower than might have been expected, particularly so for U.S./Canadian examinees who otherwise have performed at a high level on the USMLE. First-taker pass rates for students and graduates of LCME-accredited medical school programs have consistently been above 90% throughout the history of the USMLE. When this performance is combined with the fact that the program does not impose any attempt limits upon a USMLE Step, the resulting ultimate pass rate on the USMLE for U.S./Canadian examinees is likely 99% or higher.² Yet among the irregular behavior cohort, only 60% of the U.S./Canadian subgroup have been able to successfully complete the USMLE sequence.

There are several possible explanations for this moderate achievement in successfully completing the USMLE sequence. First is the below average performance of the irregular behavior cohort as first takers of the USMLE (Table 8). This cohort's performance was noticeably below that of all U.S./Canadian and IMG first takers throughout the 1992–2006 period. The U.S./Canadian first-taker pass rate has never fallen below 91% for any USMLE Step over the history of the program; IMG first-taker pass rates have ranged between 47–81% on each Step.² Another consideration is the imposition of a bar in 121 cases. Recall that in 17 of these cases the CIB imposed an indefinite, rather than a time-limited, bar upon the examinee. To date, no state medical board has requested that any of these individuals be provided access to the USMLE. Additionally, the imposition of a time-specific bar seems to significantly lower the likelihood of completing the USMLE sequence. Only 12 of the 121 individuals with time-limited bars successfully completed the USMLE. Thus, the decision to impose a bar upon access to the USMLE appears to significantly reduce the likelihood that subsequently an individual will successfully complete the licensing examination sequence. Another explanation may be the basis behind many findings of irregular behavior, i.e., falsification of an application or key document. In those instances where the falsification is directly related to the individual's eligibility status for a Step examination, the USMLE has required the individual to establish (or re-establish, if applicable) eligibility before the clock begins on the time-limited bar. Finally, for those individuals whose irregular behavior finding dates to more recent years (e.g., 2004–2006), it is possible these individuals are still progressing through the educational/training system toward licensure.

Obtaining a Full, Unrestricted License in a U.S. Jurisdiction

A review of licensure information available on the 433 individuals identified with irregular behavior shows that 37% of these individuals obtained a full, unrestricted U.S. license (Table 9).

There are several possible explanations for these relatively modest percentages. For example, none of the 17 individuals sanctioned with an “indefinite” bar has yet been, or is likely ever to be, licensed by a U.S. jurisdiction. Furthermore, individuals with bars imposed more recently (i.e., 2004–2006) may not have had sufficient time to finish moving through the USMLE sequence and gain/complete sufficient residency training to be eligible for licensure. Additionally, other individuals appear to have diverted from the USMLE pathway entirely. This may reflect pursuit of an alternate career pathway (e.g., seven individuals gained licensure in Canada). Some individuals may have chosen to pursue other professional opportunities or, in the case of some IMGs, they may have returned to, or opted to remain in, their native country.

The U.S./Canadian subgroup was most likely to successfully complete the USMLE sequence and obtain a full, unrestricted license in a U.S. jurisdiction. This examinee subset possesses a strong financial incentive, stemming in part from student indebtedness, pushing them to persist in their progress toward licensure (2006 graduates averaged $130,000 in medical school debt),¹⁶ and, unlike IMGs, there is no option to return to a native country to practice medicine. Additionally, as noted above, this group has demonstrated strong performance on USMLE and other standardized examinations.^17,18,19

State Medical Boards and Examinees with Irregular Behavior

The USMLE transcript serves as the primary means by which the program alerts state medical boards to instances of irregular behavior. State medical boards licensing graduates of medical schools issuing the M.D. degree routinely require an original USMLE transcript as part of their licensing application process. If the CIB delivers a finding of irregular behavior, the official record of the individual is annotated to reflect the committee's finding and any official USMLE transcript subsequently generated carries this annotation alerting the document's recipient. A comment reading “irregular behavior” appears in bold cap lettering on the USMLE transcript along with a brief description of the infraction immediately underneath the irregular behavior comment. For all findings of irregular behavior since 2001, a copy of the determination letter^† from the Office of the USMLE Secretariat also accompanies the transcript.

A second pathway for flagging irregular behavior and bringing it to the attention of a state medical board occurs when the CIB finds for irregular behavior and decides to also report this finding to the FSMB board action data bank. This means that any subsequently produced USMLE transcript carries an annotation at the bottom indicating the individual has board action history on file with the FSMB. A copy of the board action report (listing the basis for the board action) would accompany any USMLE transcript produced by FSMB, NBME, or ECFMG. Any state medical board querying the FSMB board action data bank as part of its licensing processes (standard practice for all boards) would be made aware of the irregular behavior finding by virtue of this history in the FSMB board action data bank.

One of the questions originally prompting this investigation was, “What actions, if any, are taken by state medical boards when they are presented with an individual who has been found previously to have engaged in irregular behavior by the USMLE program?”

In order to answer this question it is first necessary to establish that state medical boards are adequately alerted to the presence of irregular behavior history.

Most state medical boards (47 of 70) issue resident or training licenses. However, only 15 of the 47 boards issuing a resident or training license require passage of USMLE Steps 1–2, which would be verified through a USMLE transcript sent to the board.²⁰ Consequently, state medical boards do not generally become aware of an individual's examination history with irregular behavior until the point when the individual directs FSMB to forward a transcript as part of the application process for a full, unrestricted license. As noted in Table 9, only 161 individuals with irregular behavior obtained a full, unrestricted medical license by the end of 2008. FSMB internal applications for transcript production and records on file with the Office of the USMLE Secretariat were consulted to ascertain whether state medical boards were alerted to the presence of the irregular behavior. This review determined that for 150 out of 161 individuals, the state board issuing the license either received a transcript alerting them to the individual's history of irregular behavior or contacted the Office of the USMLE Secretariat to obtain information concerning the irregular behavior, indicative of their awareness of the irregular behavior infraction prior to license issuance. Incomplete records at FSMB and the Office of the USMLE Secretariat may account for the 11 instances in which it appears the state medical board issued a license without apparent knowledge of the irregular behavior finding.

Another avenue of research looked at actions taken by state medical boards against licensure applicants found previously to have engaged in irregular behavior. A cross check against the FSMB board action data bank on the 433 individuals in this study found only a handful of instances where a state medical board took a subsequent prejudicial action against an individual previously found to have engaged in irregular behavior. Of the 198 individuals who completed the USMLE sequence, five individuals had subsequent action taken against them by a state medical board (5/198, or 2.5%). The actions taken ranged from fine and reprimand to denial of license. This number is likely influenced by the actionable bases upon which a licensing board is empowered to act, e.g., non-criminal matters preceding the candidate's application for licensure may not be actionable.

This study also examined another subset of individuals with irregular behavior who completed the USMLE sequence, i.e., 87 cases “non-administrative” irregular behavior.^‡ In this subset, 12 individuals completed the USMLE sequence with seven subsequently obtaining a full, unrestricted U.S. medical license. For the remaining five individuals, three were denied licensure or withdrew their license application to avoid a reportable denial of licensure by the board; one individual has a license application pending currently; the final individual's location and circumstance is unknown.

Additionally, it should be noted that the ‘official' actions taken by medical boards discussed above do not represent the full extent of inquiry boards may take when presented by a licensure candidate with irregular behavior history. Conversations with executive directors at several state medical boards confirmed that individuals with irregular behavior were required to appear before a board panel or subcommittee to answer questions regarding the nature and basis for the irregular behavior annotation and/or board action history. It appears that an irregular behavior annotation diverts a license applicant from the routine administrative processing of the application.