Through its program of certification, the Educational Commission for Foreign Medical Graduates (ECFMG) is responsible for assessing the readiness of international medical graduates (IMGs) to enter programs of graduate medical education in the United States that are accredited by the Accreditation Council for Graduate Medical Education (ACGME). ECFMG determined that it would be in the interest of the American public to ensure that these graduates seeking to enter US training programs demonstrate clinical skills proficiencies comparable to those of graduating US medical students. To that end, the Clinical Skills Assessment (CSA) was developed to assess these skills.
ECFMG implemented the CSA as an additional examination requirement for ECFMG certification on July 1, 1998. The purpose of the CSA is to ensure that IMGs demonstrate the ability to gather and interpret clinical patient data and communicate effectively at a level comparable to a standard reasonably expected of students graduating from accredited US medical schools. CSA assesses an examinee’s data-gathering skills (medical history and physical examination), interpersonal skills, spoken English proficiency and the ability to compose a written patient note. The incorporation of CSA as an additional requirement for ECFMG certification helps ensure that individuals receiving ECFMG certification possess the necessary basic clinical skills for entry into supervised graduate medical education training in the United States. Since ECFMG certification is a requirement for all IMGs seeking to enter accredited graduate medical education programs in the United States, CSA represents a high-stakes medical performance examination.
HISTORICAL PERSPECTIVE ON CSA
There are approximately 1,600 medical schools worldwide, each with varying educational standards and curricula. The Liaison Committee on Medical Education (LCME) requires that the assessment of clinical skills be part of the overall evaluation of US medical school students. However, in the absence of any single set of international accreditation standards for medical schools throughout the world, and hence, no way to predictably influence curriculum, the only viable alternative was to assess outcomes. This could be done using a standardized assessment of students graduating from those medical schools, the CSA.
The development of CSA spanned 2 decades, consisting of an extensive program of planning and research to develop a prototype that would provide an objective and consistent evaluation of the readiness of international medical graduates to enter graduate medical education (GME) programs in the United States. Pilot projects were conducted around the world, and as a result of those studies, the current CSA model was developed. The current CSA examination was tested in the United States and abroad to compile data and demonstrate its validity and reliability.
DESCRIPTION OF CSA
The CSA is built on an Objective Structured Clinical Examination (OSCE) model, with 10 scorable encounters constituting a test form. The encounter format is standardized, requiring the candidate to elicit a medical history, perform a physical examination, communicate in spoken English with a patient in a clinical setting, and generate a written record of the encounter. In each station, the examinee encounters a unique standardized patient, a lay person recruited and trained to realistically portray a patient with a standardized medical and psychosocial history, and standardized findings on physical examination. Each case has a case-specific checklist containing the elements of medical history and physical examination considered pertinent to that particular case. SPs are trained to recognize appropriate queries and/or physical examination maneuvers, including acceptable equivalents or variants, and to document each checklist item performed by the candidate. SPs also evaluate each candidate’s interpersonal skills and spoken English proficiency. After each encounter, the candidate generates a patient note, in which the pertinent positive and negative elements of history and physical examination are recorded. A differential diagnosis is constructed, and a diagnostic workup plan proposed.
COMPONENTS OF CSA
The CSA consists of 2 separate components: the Integrated Clinical Encounter (ICE) and the Communication Skills (COM) component. The ICE is a combination of the Data Gathering (DG) and Patient Note (PN) scores. The COM component is derived from the evaluation of the examinee’s interpersonal skills and spoken English proficiency.
INTEGRATED CLINICAL ENCOUNTER (ICE)
Data Gathering (DG)
The CSA is designed to assess a candidate’s ability to consider reasonable diagnostic possibilities by presenting a set of common clinical scenarios. By gathering a pertinent medical history and performing a focused physical examination, the examinee demonstrates the ability to collect information unique to each encounter. Using history-taking and physical examination checklists, the SP documents the examinee’s performance in gathering data relevant to that clinical encounter. The SP does not evaluate the examinee’s medical performance, but documents whether an examinee successfully obtained relevant information or correctly performed the case-specific physical examination maneuvers.
Patient Notes (PN)
The purpose of the CSA is to assess not only the candidate’s ability to gather clinical patient data, but also the ability to interpret that data and to communicate in writing to the health care team their findings, interpretations (impressions), and diagnostic plans. The ECFMG CSA, therefore, includes a patient note in which, in addition to listing pertinent positive and negative elements of medical history and physical examination, candidates must evidence interpretation of the data by generating a differential diagnosis and propose a diagnostic workup plan. Specifically trained physicians review and score the patient notes, based on pre-defined criteria, including organization, quality of information, interpretation of data, egregiousness of action, and legibility.
COMMUNICATION SKILLS (COM)
Following each encounter, the SP also will evaluate the examinee’s communication skills in 5 areas:
Interviewing and obtaining information
Counseling and delivering information
Rapport
Personal manner
Spoken English proficiency
SPs undergo extensive and continuous training to rate an examinee’s communication skills.
CSA CASE DEVELOPMENT
Practicing physicians and medical educators write and review cases to ensure that they are fair and valid. Cases are specifically designed to elicit a process of history-taking and physical examination in a clinical encounter that will allow the candidate to demonstrate the ability to pursue various possible diagnoses.
The cases used in CSA represent the types of patients that would typically be encountered during the core clinical clerkships included in the curriculum of LCME-accredited medical schools in the United States. These disciplines include:
Internal Medicine
Surgery
Obstetrics and Gynecology
Pediatrics
Psychiatry
Family Medicine
The cases that make up each administration of CSA reflect a balance of presenting complaints, as well as a diversity of patient age, sex, and ethnicity. There is also a mix of acute, subacute, and chronic problems. On any assessment day, the set of cases will differ from the combination of cases presented the day before or the day after, but each of the cases will have comparable degrees of difficulty.
SCORING AND EXAMINEE PERFORMANCE
CSA is based on a conjunctive scoring model. Successful candidates must separately meet the standards in both components, the ICE and COM. Exceptional performance in one area cannot compensate for substandard performance in the other.
The ICE score is composed of the Data Gathering (DG) score and the Patient Note (PN) score. The 2 scores, DG and PN, are then combined on a weighted basis to generate a single ICE score for each encounter. A candidate’s final DG and PN scores reflect the examinee’s average performance across 10 scored encounters. The COM score is composed of 4 elements of interpersonal skills and 1 element of spoken English. Using scales developed for each element, the SPs assign a score for each of these 5 areas. The COM score for each encounter is the sum of the performance in these 5 scores. The examinee’s final COM score is the average of COM scores received in the 10 scored encounters.
The CSA is a criterion-referenced instrument designed to differentiate ICE candidates around a standard or cut point. Candidates who pass the standards on both the ICE component and COM component receive a “PASS” designation. No further scores or subscores are provided. Candidates who do not achieve the standard in both components of CSA will receive an overall FAIL designation. They will, however, receive additional diagnostics information and will be told whether or not they met the standard in each of the areas used to compute CSA scores. The standard of performance required to pass CSA is reviewed periodically.
CSA pass rates have become fairly consistent at approximately 80% overall for first-time takers. US-citizen IMGs perform somewhat better than non US-citizen IMGs, primarily as a function of better scores on the COM component of CSA.
LESSONS LEARNED
SPs Work
In the ECFMG CSA model, the SPs carry a heavy level of responsibility. They must consistently portray their cases and are subject to repeated physical examination maneuvers. In addition, they play a significant role in scoring and evaluation. They must not only respond and react to history questions and physical examination maneuvers, but must also determine if the question or maneuver was posed or done in a way that would warrant credit on the checklist. Further, they must evaluate the interpersonal skills and spoken English proficiency of each candidate and score those entities.
Fortunately, the ECFMG SPs are up to the task, as demonstrated by a comprehensive quality assurance program that ensures the accuracy of checklist scoring via real time and/or video review, as well as analysis of several data streams that track systematic performance of SPs. Although many Objective Structured Clinical Examinations (OSCE) rely on individuals other than SPs, e.g., physician observers, to score encounters, those responsible for designing high-stakes assessments should be aware that based on ECFMG’s experience, properly trained and motivated SPs can carry a great deal of the load in providing accurate and reliable performance scores, which lead to valid decisions. Finally, it is worth noting that post-CSA questionnaires completed by examinees consistently reflect a very high rating of SPs as credible, providing adequate opportunity for candidates to demonstrate their clinical skills.
Security a Daily Challenge
Those responsible for high-stakes examinations must obviously ensure that the examination is secure and that no candidate has access to information that would provide an unfair advantage. In the area of performance assessment, a challenging issue is what exactly does or does not constitute such information. Does simple knowledge of the content of cases provide any meaningful advantage? Does an examinee need more detailed information, such as the actual checklist items? ECFMG’s position is closer to the latter proposition, although advance knowledge of a specific set of cases that a candidate would be encountering probably would give an unfair opportunity to prepare. Case checklists are kept quite secure, although reality suggests that there are general lists of cases circulating among candidates.
As a practical matter, given the number of cases in the test bank and the generation of a different case form for each administration of the CSA, this information is probably of no more value than access to the table of contents of any general medical textbook. ECFMG does take special precautions, however, to maintain the security of the makeup of individual forms. These are the daily set of cases that will be presented to candidates on any given date. Our belief is that the best assurance of security in a high-stakes examination is to have a large case bank to be able to manipulate test forms so that a candidate is highly unlikely to know what particular set of cases will actually be encountered. This may be particularly challenging for smaller organizations or institutions since this approach depends heavily on a large case bank. Case development, if done with the rigor necessary for a high-stakes examination, can be an expensive and time-consuming endeavor.
Impact of CSA
It has long been a dictum that assessment drives curriculum. Although evidence to date is only anecdotal, ECFMG has been informed that several medical schools located outside of the United States and Canada are altering their curricula, providing special training for CSA, or sending their students to preparation courses. We have also heard that in many schools, students are opting to spend more time in actual clinical settings than before in an attempt to enhance their clinical skills.
Through March 2002, ECFMG had tested more than 24,000 CSA examinees. The high volume and continuous administration of CSA has allowed ECFMG to acquire a great deal of experience in this unique testing environment. This experience clearly demonstrates that a high-stakes CSA can be done fairly and consistently, producing valid decisions regarding examinee’s mastery of a given set of skills. It is especially noteworthy that passing this high-stakes, high-volume examination requires demonstration of acceptable communication skills. For too long, these skills were considered too ill-defined or too elusive to be the subject of high-stakes assessment, but ECFMG’s experience is that with properly developed constructs, behavioral anchors, and proper training, such assessments can be done fairly, accurately, and reproducibly.
References
- 1.Educational Commission for Foreign Medical Graduates . ECFMG 2002 Information Booklet.Philadelphia: Educational Commission for Foreign Medical Graduates, 2001.
- 2.Educational Commission for Foreign Medical Graduates . Clinical Skills Assessment (CSA®) Candidate Orientation Manual.Philadelphia: Educational Commission for Foreign Medical Graduates, 2000.
- 3.Whelan,Gerald P. “Educational Commission for Foreign Medical Graduates: clinical skills assessment prototype.” Medical Teacher 21( 2): 156– 160 ( 1999).
- 4.Whelan,Gerald P. “Educational Commission for Foreign Medical Graduates: lessons learned in a high-stakes, high-volume medical performance examination.” Medical Teacher 22( 3): 293– 296 ( 2000).
