THE YEAR IS 2015.
A computer-based “agent” or “knowbot” from your physician’s office contacts you about your regular health evaluation, asking detailed, interactive questions about your current health, needs and expectations. An appointment is arranged at an outpatient facility a few days later, where your body cells are sampled to identify tumor and disease markers, and evaluate health risks. Whole-body imaging is completed along with a battery of minimally invasive tests.
While computer-based decision aids assist in interpreting your data, your physician coordinates activities of health team members to make judgments from vast amounts of information. For instance, your genome is evaluated in the context of your medical history and lifestyle; relevant genetic and image databases are consulted regarding a miniscule irregularity in your descending colon. Additionally, she identifies resources with information germane to your condition, and reconciles discrepant views in pertinent literature.
Finally, with information in hand the following week, she reviews with the team your identified needs, and arranges a question and answer session tailored to your learning style. Since you are temporarily out of the country, this is conducted using virtual telepresence. You discuss the recent evaluation, including new technologies used, and agree upon next steps for follow-up including recommended changes in lifestyle and medication.
While this scenario seems futuristic and may not be an accurate prediction, requisite technologies are currently available or under development. Clearly, the differences from current medical practice suggest the need for new skills for the physician of this future. What implications might this have for licensure assessment?
BACKGROUND AND INTRODUCTION
The National Board of Medical Examiners (NBME) and the National Board of Osteopathic Medical Examiners provide examinations for those seeking medical licensure in the United States. The process for licensure assessment has undergone considerable change in the last two centuries. In the early 1800s, a state-issued medical license was awarded for passing an examination usually administered by the state medical society. But, “… by 1870, licensing laws had been repealed in virtually every state.”1 Not until 1895 did most states re-introduce a process separate from medical school graduation for examination and licensure of physicians.2 As of 1910, concern about uneven quality of education was still pronounced; state medical board examinations were recognized as the primary means to ensure that standards were maintained in medical education.3 In 1916, motivated by a desire for state licensure reciprocity, the AMA Council on Medical Education and the Federation of State Medical Boards (FSMB) in joint session endorsed the National Board of Medical Examiners (NBME), which had been founded the prior year. In 1935, the National Board of Osteopathic Medical Examiners was founded. The primary purpose for these organizations was to develop medical licensure examinations for state medical boards.
By the 1980s, nearly all US jurisdictions recognized the NBME examination as the licensing exam for graduates of medical schools granting the MD degree in the United States and Canada accredited by the Liaison Committee on Medical Education (LCME). Graduates of international medical schools (and those of some states) were required to pass a variety of other exams, including the Federation Licensing Examination (FLEX), the Visa Qualifying Examination (VQE) and the Foreign Medical Graduate Examination in Medical Sciences (FMGEMS). In 1992, driven by interest in a single common pathway for all physicians seeking licensure in the United States, the United States Medical Licensing Examination (USMLE) was introduced. Students of osteopathic medical schools are eligible to take the USMLE as well as the Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA).
Although these standardized examinations provide valid measures of components of physician competency, they must be supplemented by training, observation and additional forms of assessment during training in accredited medical school and residency programs. Understandably, credentials (e.g., licenses, certification) are under continuing pressure to become time limited, and examinations more comprehensive in what they assess.4,5,6 Recent pressures can be attributed to heightened attention to training inadequacies,7,8,9,10,11 medical errors12 and litigation, as well as publicity surrounding physician disciplinary actions.
Licensure assessment of the physician of the future will also be influenced by trends in demographics, the marketplace and workforce; and by developments in genetic sciences and imaging, measurement and information technologies. This article discusses currently defined physician competencies and the instruments used in their assessment, as well as some trends and competing forces and their potential implications.
PHYSICIAN COMPETENCIES
Physician competencies encompass the knowledge, skills, behaviors and attitudes required for practice. Their assessment and assurance is distributed among those involved with medical education and assessment. Review and redefinition of competencies is periodically undertaken. Recent efforts include CanMeds 2000 Project,13 the Accreditation Commission for Graduate Medical Education’s (ACGME) Outcome Project and the Association of American Medical College’s (AAMC’s) Medical Schools Objectives Project. Similarities among the different competency categories are obvious in Table 1.
Physician Competencies
These definitions provide a structure at a high conceptual level, and guidelines for assessment design. Available assessment methods cover differing degrees of these competencies.
CURRENT ASSESSMENT METHODS
USMLE complements assessment performed by medical schools and residency training programs. USMLE consists of three separate examinations administered over four days, and includes computer-based multiple-choice questions (MCQs) and computer-based case simulations.
Table 2 lists a classification of competencies (loosely based on Table 1) and some examples of assessment tools and where they are used. More detailed discussion follows.
Assessment Methods In Use Or Under Development
MCQs
Multiple choice questions provide highly reliable scores in a day or two of testing, and are widely used in medical schools and other educational settings. Procedures for writing, delivering and scoring high-quality MCQ examinations are well defined. MCQ variations have been developed to optimize measurement information, for example, by increasing the numbers of options, focusing on key decision features,14 or requiring higher order reasoning.15
Computer-based case simulation
“Computer simulation” covers a broad array of applications. Computer-based case simulation (Primum® CCS) in USMLE Step 316 is an interactive format in which, after a short text introduction to the patient, examinees type requests for diagnostic tests, treatments, consultants and procedures without lists of possible actions. The software recognizes more than 12,000 terms representing about 2,500 distinct actions. The examinee manages the patient over simulated time, obtaining results of studies ordered and modifying the treatment plan based on the patient’s evolving course. For each case, the computer records what the examinee did and when. The resultant record is scored automatically by software in which physician policies for management have been modeled.17 Computer-based case simulations measure aspects of decision-making that are distinct from those measured with MCQs.
Standardized patients
Standardized patients (SPs), first described in 1964,18 are individuals who are trained to portray a health care situation in a consistent fashion and evaluate the examinee during the encounter. They are used for assessment of communication, interpersonal, history-taking and physical examination skills. In a sample SP examination, physicians spend 10–15 minutes with the patient and then write a post-encounter note. About 10 patients might be seen in the course of the examination. SPs are in use by the Medical Council of Canada,19 the Province of Quebec,20 and Educational Commission on Foreign Medical Graduates21,22 and contribute to licensing decisions. The NBME has pilot studies under way with the ECFMG to investigate SP use in USMLE within the next few years. While SPs are relatively expensive and logistically complex, they measure aspects of physician behavior that cannot be measured with other formats.
Examples of other tools in use or under development
Oral examinations are used by many specialty boards and can measure not only static knowledge, but also other aspects of clinical performance (e.g., ability to respond to changing information).23 With oral examinations, a trained examiner asks focused questions of the examinee; questioning can be redirected based on the examinee’s response. The format requires extensive training of the examiners for standardization, and can be expensive.
Other simulations have been developed but few have validated scoring systems. Many computer-based systems incorporate video or audio images to provide more realistic simulation. Simulations are also under development to model physician policies and provide feedback on how physicians use information to make decisions (i.e., interactive judgment analysis).24 Simulations using manikins and/or virtual reality also are in use and/or under investigation for instruction and assessment.25
Another class of assessment tools uses actual work samples based on direct observation of the subject by peers, coworkers or patients to assess traits that may not be amenable to other formats, such as aspects of professionalism. For example, surveys are done of faculty who have direct clinical contact with medical students. Surveys benefit from multiple, independent observers, and are relatively inexpensive to conduct. Another technique uses audits of patient charts to assess the quality of record-keeping and decision-making; these also can be used as a basis for interview (chart-stimulated recall).
FORCES OF CHANGE
A number of forces will impact the role of the physician of the future, the competencies required to practice, and therefore the assessment tools needed.
Decision support systems
One of the strongest forces driving change is the electronic digital computer. Moore’s Law, which describes a miniaturization in semiconductor technology by a factor of two every 12 months, has more or less* held true since Gordon Moore made the prediction in 1965, and is likely to continue for the next 15 to 30 years.26 This geometrically advancing power has already resulted in machines that can emulate human decisions in restricted domains. With increasing computer power, medical decision support will become more widely used. Such systems already have been shown to influence physician diagnostic reasoning.27
Mass storage of medical data
Decreasing size and cost of permanent memory will facilitate storage of vast amounts of imaging and genomic information, which in turn will contribute increasingly to diagnosis and treatment. The National Library of Medicine’s Visible Human Project contains digitized MRI, CT and pathologic sections of a man and woman,28 with data sets of about 100 gigabytes (at up to three slices per millimeter). This volume of imaging data will be stored on everyone in the future. Within the coming years, data will be amassed from the approximately 3 billion letters of DNA belonging to each individual. Physicians will have access to this patient data along with tools for managing and understanding them.
Telemedicine
Increasing bandwidth will change communication between patient and physician in a number of dramatic ways. Many physicians already are turning to e-mail to answer questions. Although progressive substitution for face-to-face visits is still unpalatable to many and inappropriate for some patients and circumstances, e-mail can be efficient and time-flexible. This trend reflects society in general and may be accelerated by time pressures imposed by economic forces.
As with miniaturization of microchips, Internet bandwidth is doubling about every 2 years. This suggests that text e-mail soon will be supplemented with or replaced by recorded or real-time video, audio and, eventually, by even more realistic media (e.g., three-dimensional images and sensory feedback) that will facilitate “full-experience” communication with someone at a remote site.
Telemedicine, or patient care at a distance, is one implication of this increased bandwidth.29 It will occur from the comfort of home or other convenient locations. The predicted drops in international telecommunication rates, mirroring the recent drops in domestic US rates, means that not only will the best physicians be available without limit to geography, but also that patients in emerging middle classes from all parts of the world will be seeking their services, as will more affluent aging populations in developed countries.
Telepresence surgery and simulation
Telepresence surgery, in which equipment is controlled at a distance, is another implication. Currently, laparoscopic procedures separate the physician from the patient by a few feet. Remote operations have already been demonstrated.30 Furthermore, with a computer between the physician and patient, procedures can be enhanced by providing to the operator additional information (e.g., projecting three-dimensional images of MRI scans into the operative field), enhancing precision and dexterity, preventing obvious errors (e.g., not allowing the surgeon to nick a major vessel) and correcting for surgeon tremor or patient body movement.31 Enhancement will also be provided by increased sophistication of the distal equipment. Robotic applications have been demonstrated or are under development for biopsying brain lesions, shaping the femur for hip replacement, and for closed-chest heart bypass and ophthalmologic microsurgery.32
A variation of telemedicine and telepresence surgery (in which a person is at the other end of the line) places a simulation at the other end of the line. The development of high fidelity, immersive simulation available locally and across the next generation Internet33 will change the way training occurs.34 Surgeons, for example, will train in simulators that in many if not all aspects, mimic the look and feel of operative procedures with a real patient. Prior to the operation on the real patient, they will practice the operative procedure on a simulation constructed from the patient’s actual body data. After these are used for training, they also will be used for credentialing, certification, and licensing.
Licensing bodies are now being pressed to deal with medical practice across state lines. The Federation of State Medical Boards adopted in 1996 the report “A Model Act to Regulate the Practice of Medicine Across State Lines: An Introduction and Rationale.” The gist of the report defines “a model legislative act which calls for an abbreviated but effective licensure process for physicians who will not be practicing physically within a state's jurisdiction but wish to provide services to patients located within that jurisdiction.” No doubt, pressure will mount in the near future to address similar concerns across national boundaries.
Medical imaging
Advances in medical imaging include not only increased resolution and function of existing modalities (MRI cardiac function imaging, contrast-enhanced MRI for cancers and high resolution three-dimensional ultrasound), but also new techniques. For example, cellular imaging technologies will allow new views of cellular structure and the histopathology of living tissue, allowing virtual biopsies of living tissues. Molecular imaging will be possible with probes or contrast agents that will identify specific receptors.35 These techniques will allow, for example, earlier disease detection, and enhanced and focused treatment.
Potential implications for the practicing physician include increased reliance on imaging for diagnosis.
Genomics
The mapping of the human genome will translate into, among other things, more effective diagnostic and prognostic tools. Sorting out the relevance of the genome and associated proteome for the individual patient will continue to be a major objective of medical science over the coming decades. Tools are under development for molecular diagnosis of genetic and infectious disease and personalized drug therapies. Statistical inferences from analysis of the vast amount of population genomic information will be aided by increased computing power, memory storage and automated means for discovering relationships in the data. Each individual’s genome will be compared with those of the population with known states or outcomes, for instance, to allow the physician to make specific recommendations based on risk factors (e.g., “you have an 80% chance of having a heart attack in the next year if you don’t …”).
Physicians will need to become adept in understanding genomics, statistics, population genetics, and in educating patients about how to interpret probabilistic information and the resultant implications for lifestyle changes.
Computer-based order entry and the electronic medical record
As the computer becomes an intermediary of procedural skills between surgeon and patient, so it too will become an intermediary of cognitive skills between all physicians and patients, facilitating other applications. For example, through computer-based order entry, many types of medication errors might be eliminated.36 Likewise, the computer can provide reminders for screening tests and other management tasks to ensure compliance with practice guidelines.
The electronic medical record will store increasing amounts of codified information. As natural language processing systems become more widely used, the record will contain more useful information from which decision support systems can make recommendations. Additionally, computer-mediated actions taken by health care providers will be entered automatically into the medical record. This record will become as much an accounting of the healthcare provider’s (e.g., individual, team, department of hospital) performance, as it is a view of the patient’s medical history.
Continuing medical education (CME) and available medical information
Instant and inexpensive access through the Internet to medical information in instructional and assessment formats virtually guarantees other developments. For example, online CME will become popular with physicians who are increasingly pressed for time. In addition, the continuing information and knowledge explosion will require that physicians obtain CME to maintain currency. It is possible to envision de facto credentialing by insurance companies, wherein successful completion of CME is related to rates for malpractice premium renewal; or, that third parties, regulating agencies or state medical boards take action that influence the type and quality of CME.
Medical information available to physicians also will be available to consumers. The growing number of Internet sites providing medical information demonstrates the point. This trend is likely to continue as a result of consumerism and effective patient advocacy on the part of large national groups. The next logical step will be the availability of medical decision support systems to consumers, which will further challenge the physician’s role.
Demographic changes and globalization
In developed countries, the percentage of the population over 65 is growing rapidly. This has implication for the types of diseases physicians see, the type of care provided, and the strength of lobbying groups to alter resource allocation. As well, the increasing percentage of minority populations in the United States has implication for physician readiness to care for those with a first language other than English, and for those of different cultures. Additionally, future access by patients anywhere in the world to physicians in the United States, will pressure health services organizations to adapt further to a wider range of languages and cultures.
Another factor revealed by the 2000 US Census is the increase in individuals completing high school and gaining a college education. The higher educational level of the consumer and the ready availability of medical information and decision support systems on the Internet has implications for physicians’ preparedness to respond rapidly to increasingly sophisticated queries.
Economics
Changes in health care over the last 2 decades have resulted in increased financial pressure on many aspects of patient care. Likely results include increasing dissatisfaction by more educated consumers, more aggressive competition from those providing “comparable” services for less money,37,38,39 and displacement of costs to the consumer. Increasing numbers of the uninsured or dissatisfied will seek less expensive alternatives to health care. For the insured, large disparities in coverage by payers will expand the potential for mistrust of and dissatisfaction with the physician and health systems in general. The availability of metrics reflecting quality of care will drive health-care providers to market themselves more aggressively, and make “relevant,” salutary information available to consumers.
In combination, these factors will tax the communication and management skills of the physician.
CONSEQUENCES FOR THE PHYSICIAN
Predictions of implications based on existing trends are fraught with risk. Nonetheless, several future areas of increased emphasis are likely (see Table 3).
Implications For Skills Needed For Physicians In The Future.
Implications for Future Assessment
Although the methods described previously are available now, other technologies are on the horizon that will facilitate new methods targeting competencies required of tomorrow’s doctors. They include high-fidelity simulators, natural language understanding, and the codification of medical practice.
High-fidelity simulators (e.g., virtual reality simulators) encompass a range of technologies that immerse the user in artificial environments and allow the user to interact with that environment. The user can modify this virtual reality by, for example, hand movement. Virtual reality simulators have already been shown to be beneficial for training40 and no doubt will find their way into licensure or certification testing in the future. They are approaching maturity for training and observing skills that include right heart catheterization, endoscopy, colonoscopy, laparoscopic procedures and certain physical examination skills.
Natural language understanding, combined with knowledge bases, will facilitate new testing formats. Some approaches to determining the quality of text samples41 show promise for essay scoring. The ability of computers to understand short phrases would facilitate: oral examinations conducted by intelligent computer agents; simulations in which the computer pretended to be the patient, answering questions as would a human; and even for evaluating the quality of hospital records to render an opinion of the care provided.
The codification of practice through computer-based medical entry and record systems may prove the most practical, seamless and exciting addition to assessment. These systems are likely to be in place to ensure the safety of patients by screening physician orders and providing decision support.42,43 As a result, physicians could be assessed as they practice medicine every day. The appropriateness, cost-effectiveness, and timeliness of care could be tracked over time.
The same advances in computer technology that might increase the “information manager” responsibility for the physician of the future will provide a means for assessing these skills. It is easily conceivable that a physician’s ability to seek and process information could be assessed in a simulated or real scenario. In response to a problem, the physician would be required to retrieve information from disparate sources, evaluate the quality of the information, and decide how to apply it. The computer could track the quality of these steps.
The extension of these points is that instruction, practice and assessment will be seamless. With simulations added, an environment for training would be identical to the real-life setting.
These changes are likely to affect assessment in a number of ways:
Instruments used to test physician competence will become increasingly realistic.
An increasing number of skills, formerly observable only in a real-life setting, will be assessed in the examination setting.
Assessment will be based increasingly on performance resembling real-life activities, and will focus increasingly on more qualitative metrics.
In the examination setting, human examiners and raters will be supplemented progressively by computers.
The distinction between instruction and assessment, and therefore between examination and actual patient care settings, will blur.
As patient records are computerized and computer-based order entry systems are used increasingly, data available from the patient care setting will allow assessment of physician, team, and health system performance.
Additional competencies (professionalism, teamwork) will be assessable in both examination and patient care settings.
CONCLUSION
This article provides a brief background on licensure assessment, discusses some trends that might change the competencies required of physicians over the next decade or two, and lists some possible implications for licensure assessment. Examinations are and will remain important tools for state licensing authorities to meet their obligation to assure that new physicians demonstrate minimally acceptable competency. They are among the most important tools that allow state-based licensure to implement a common “national” standard of medical practice without resorting to a federalized system of licensure.
However, at the present time, licensure examinations provide an incomplete picture of the new physician. Current examinations do not assess some characteristics important in safe and effective practice, requiring that licensing authorities depend on less standardized and objective evidence in granting the initial license. Advances in testing technology will expand the objectively measurable competencies. State licensing authorities and the profession will be faced with the dilemma that more comprehensive testing will require additional investments in assessment, with monetary and time commitments. These enhancements will likely be perceived as burdens and resisted by the profession.
State licensing authorities also are responsible for assuring the ongoing competency of physicians for practice. The notion of periodic reassessment of physicians has been widely discussed but not yet implemented by any licensing authority. It was one of several recommendations by the Institute of Medicine in its report To Err Is Human: Building a Safer Health System. Needs for reassessment will be heightened by the accelerating pace of change in the practice of medicine – telemedicine, imaging, genomics and proteomics, empowered consumers, complementary and alternative medicine – each of which makes new demands on the knowledge and skills of practicing physicians to provide safe and effective patient care. As the ecology of medicine changes and the tools for assessment become more comprehensive, state licensing authorities will find it more and more difficult to ignore the issue of ongoing assessment of competence for practice. Embracing the need for continued competency assessment is likely to be contentious and costly, with state licensing authorities potentially caught on the horns of a cost/effort versus quality/safety dilemma. While technology offers solutions to the assessment needs, it offers little to ease the difficult task of changing our system of evaluation to keep in step with both the practice of medicine and the technology of assessment.
References
- 1.↵LudmererK. Learning to Heal: the development of American medical education. NY: Basic Books; 1985, p. 3.
- 2.↵HubbardJ, LevitE. The National Board of Medical Examiners: the first seventy years. Philadelphia: National Board of Medical Examiners; 1985, p. 284.
- 3.↵FlexnerA. Medical education in the United States: a report to the Carnegie Foundation for the advancement of teaching. NY: Arno Press; 1972(c.1910).
- 4.↵GellhornA. Periodic physician recredentialing. JAMA.1991; 265: 742– 745..
- 5.↵FinestoneA. CME and clinical competence: is it time for mandatory relicensure and recertification? J Contin Educ Health Prof. 1988; 8: 67– 70.
- 6.↵ElsteinA. Beyond multiple-choice questions and essays: the need for a new way to assess clinical competence. Acad Med.1993; 68: 244– 249.
- 7.↵MangioneS. Cardiac auscultatory skills of physicians-in-training: a comparison of three English-speaking countries. Am J Med.2001; 110: 210– 216.
- 8.↵KassebaumD, EaglenR. Shortcomings in the evaluation of students' clinical skills and behaviors in medical school. Acad Med.1999; 74: 842– 849.
- 9.↵GaskinP, OwensS, TalnerN, SandersS, LiJ. Clinical auscultation skills in pediatric residents. Pediatrics.2000; 105: 1184– 1187.
- 10.↵MangioneS, NeimanL. Pulmonary auscultatory skills during training in internal medicine and family practice. Am J Respir Crit Care Med.1999; 159( 4 Pt 1): 1119– 1124.
- 11.↵IssenbergS, FelnerJ, BrownD. Cardiac auscultation skills of physicians in training. JAMA.1997; 278: 1740– 1741.
- 12.↵To Err is Human: building a safer health system. Washington, DC: National Academy Press; 1999.
- 13.↵Skills for the New Millennium: Report of the Societal Needs Working Group. The Royal College of Physicians and Surgeons of Canada's Canadian Medical Education Directions for Specialists 2000 Project. Ottawa, Ontario: The Royal College of Physicians and Surgeons of Canada; September1996. Available at: http://rcpsc.medical.org/english/publications/
- 14.↵PageG, BordageG. The Medical Council of Canada's key features project: a more valid written examination of clinical decision-making skills. Acad Med.1995; 70: 104– 110.
- 15.↵NorciniJ. Examining the Examinations for Licensure and Certification in Medicine. JAMA1994; 272: 713– 714.
- 16.↵ClymanS, MelnickD, ClauserB. Computer-based case simulations from medicine: Assessing skills in patient management: From paper-and-pencil to virtual reality. In:TekianA,McGuireC,McGahieW, eds.Innovative simulations for assessing professional competence.Chicago: University of Illinois Department of Medical Education; 1999: 29– 41.
- 17.↵ClauserB, HarikP, ClymanS. The Generalizability of Scores for a Performance Assessment Scored with a Computer-Automated Scoring System. J Educ Meas.2000; 37: 245– 262.
- 18.↵BarrowsH, AbrahamsonS. The programmed patient: A technique for appraising student performance in clinical neurology. J Med Educ.1964; 39: 802– 805.
- 19.↵Grand'MaisonP, LescopJ, RainsberryP, BrailovskyC. Large-scale use of an objective, structured clinical examination for licensing family physicians. CMAJ.1992; 146: 1735– 1740.
- 20.↵ReznickR, BlackmoreD, DauphineeW, RothmnanA, SmeeS. Large-scale high-stakes testing with an OSCE: report from the Medical Council of Canada. Acad Med.1996; 71(( 1 Suppl)): S19– S21.
- 21.↵WhelanG, McKinleyD, BouletJ, MacraeJ, KamholzS. Validation of the doctor-patient communication component of the Educational Commission for Foreign Medical Graduates Clinical Skills Assessment. Med Educ.2001; 35: 757– 761.
- 22.↵Ben-DavidM, KlassD, BouletJ, . The performance of foreign medical graduates on the National Board of Medical Examiners (NBME) standardized patient examination prototype: a collaborative study of the NBME and the Educational Commission for Foreign Medical Graduates (ECFMG). Med Educ.1999; 33: 439– 446.
- 23.↵MancallE, BashookPG , eds.Assessing Clinical Reasoning: The oral examination and alternative methods. Evanston IL: American Board of Medical Specialties; 1995.
- 24.↵WigtonR. Use of linear models to analyze physicians' decisions. Med Decis Making.1988; 8: 241– 252.
- 25.↵IssenbergS, GordonM, GordonD, SaffordR, HartI. Simulation and new learning technologies. Med Teach.2001; 23: 16– 23.
- 26.↵JoyB. Why the Future Doesn't Need Us. Wired Magazine; April2000. Available at: http://www.wired.com/wired/archive/8.04/joy.html
- 27.↵FriedmanC, ElsteinA, WolfF, . Enhancement of clinicians' diagnostic reasoning by computer-based consultation: a multisite study of 2 systems. JAMA.1999; 282: 1851– 1856.
- 28.↵AckermanM. The Visible Human Project: a resource for education. Acad Med.1999; 74: 667– 670.
- 29.↵AngoodP, SatavaR, DoarnC, MerrellR, Group, TE . Telemedicine at the top of the world: the 1998 and 1999 Everest extreme expeditions. Telemed J E Health.2000; 6: 315– 325.
- 30.↵MarescauxJ, LeroyJ, GagnerM, . Transatlantic robot-assisted telesurgery. Nature.2001; 413( 6854): 379– 380.
- 31.↵SatavaR. Emerging technologies for surgery in the 21st century. Arch Surg.1999; 134: 1197– 1202.
- 32.↵HoweR, MatsuokaY. Robotics for surgery. Annu Rev Biomed Eng.1999; 1: 211– 240.
- 33.↵DevP, HeinrichsW, SrivastavaS. Simulated Learning Environments in Anatomy and Surgery Delivered via the Next Generation Internet. Medinfo.2001; 10( Pt 2): 1014– 1018.
- 34.↵SatavaR, Jones,S. Preparing surgeons for the 21st century. Implications of advanced technologies. Surg Clin North Am.2000; 80: 1353– 1365.
- 35.↵Biomedical Imaging Symposium: Visualizing the Future of Biology and Medicine . Summary Report. Bethesda MD: NIH Bioengineering Consortium (BECON); June25–261999. Available at: http://www.becon.nih.gov/report_19990625.pdf.
- 36.↵BatesD, TeichJ, LeeJ, . The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc.1999; 6: 313– 321.
- 37.↵HookerR, McCaigL. Use of physician assistants and nurse practitioners in primary care, 1995–1999. Health Aff (Millwood).2001; 20: 231– 238.
- 38.↵CounselmanF, GraffeoC, HillJ. Patient satisfaction with physician assistants (PAs) in an ED fast track. Am J Emerg Med.2000; 18: 661– 665.
- 39.↵CashB, SchoenfeldP, RansohoffD. Licensure, use, and training of paramedical personnel to perform screening flexible sigmoidoscopy. Gastrointest Endosc.1999; 49: 163– 169.
- 40.↵TuggyM. Virtual reality flexible sigmoidoscopy simulator training: impact on resident performance. J Am Board Fam Pract.1998; 11: 426– 433.
- 41.↵LandauerT, DumaisS. A solution to Plato's problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review.1997; 104: 211– 240.
- 42.↵KupermanG, TeichJ, GandhiT, BatesD. Patient safety and computerized medication ordering at Brigham and Women's Hospital. Jt Comm J Qual Improv.2001; 27: 509– 521.
- 43.↵MurffH, KannryJ. Physician satisfaction with two order entry systems. J Am Med Inform Assoc.2001; 8: 499– 509.
