Home | Mission | Working Relationships | Position Papers | Publications | Clerkship Administrator Resources

Guidebook for Clerkship Directors | Annual Meetings | Member Organization Links | LCME ED-2 | Contact Us

Chapter 9 : Directing a Clerkship Over Geographically Separated Sites

Co-Lead Authors:
Paul A. Hemmer, MD and Nancy Schindler, MD

Co-Authors:
Mary C. Ciotti, MD

Download Adobe PDF version of chapter	Printed version of Guidebook is available.
Sections on this page:
Introduction LCME Standards Selecting Sites Administrative Structure of a Multi-site Clerkship Curriculum for Multi-site Clerkships Assessment of Students at Separate Sites Formative (Feedback) and Summative Assessment at Separate Sites Issues of Faculty and Resident Development Evaluating a Multi-site Clinical Clerkship Program Issues of Communication Conclusion	TABLE 1: Selected LCME Standards Pertinent to Clerkships with Geographically Separate Sites TABLE 2: Resource Audit Checklist TABLE 3: Memorandum of Agreement between Medical School and Site TABLE 4: Topics for Site Director Orientation TABLE 5: Topics for Faculty Orientation
<Chapter 8: Faculty Development <Return to Table of Contents	Chapter 10: Working with Students with Difficulties >

Introduction

More than two decades ago, Irby1 summarized the key features needed to run a clinical clerkship with geographically separated sites: "…a cohesive multi-site clinical clerkship is made possible by a core curriculum which includes a comprehensive evaluation system, faculty development activities, and administrative support." We hope this chapter helps illuminate these features. It is common for medical schools to use geographically separate sites to accomplish their educational mission. In some cases, this may mean sites across a city or state and in other cases it may mean clerkships in different states or separated by many miles.

Separate sites can offer many advantages to faculty and students, including diversity of patients and practice settings, exposure to a wider scope of medical conditions, and a broader spectrum of faculty. However, maintaining comparable educational experiences and equitable, consistent evaluation for students at separate sites is a challenge. Separate sites may mean different sites within a single city or separated by hundreds (or thousands) of miles. However, many of the separate site challenges, such as maintaining consistency, also apply to different services within the same hospital!

Some of these challenges are addressed in this chapter, with suggestions and potential solutions. Which solutions will work best may vary depending on the unique situation, including the structural organization of the geographic sites, the proximity of sites, the roles of the individual site directors, the allocation of resources from the medical school, and the student body. The authors hope to set forth some ideas and suggestions to help the clerkship director of a geographically diverse clerkship to address the challenges before him or her. We will begin with a review of Liaison Committee for Medical Education (LCME) standards specifically addressing geographically separate sites, then address issues of administration including selecting sites, administrative structure, and issues of curriculum. Finally, we will discuss issues of evaluation and feedback and suggestions for improving communication among sites. (return to top)

LCME Standards

The LCME standards for accreditation of medical schools address five areas: Institutional Setting (IS), Educational Program for the MD degree (ED), Medical Students (MS), Faculty (FA), and Educational Resources (ER).2 A number of these standards specifically relate to geographically separated sites (Table 1). Each clerkship should be in full compliance with these standards, and clerkship directors considering geographically separate sites must develop, implement, and evaluate sites with the LCME standards in mind.

Perhaps the most important standards for the clerkship director that specifically relate to geographically separate sites are ED-8 and ED-41. ED-8 requires that clerkship sites must offer comparable (but not identical) educational experiences and that evaluation methods should be standardized across sites. Student assessment methods, criteria, and weighting should be uniform. ED-41 says that the faculty must be functionally integrated. Functional integration can be accomplished using a variety of techniques, including meetings, electronic communication, site visits by the clerkship director, and sharing of evaluation data.

The clerkship director should also be familiar with ED-42, MS-37, ER-5 and ER-7, all of which refer to facilities needed at each geographically separate site, and ER-8, which says required clerkships should be performed where residents are in training. In community settings it is understood, however, that attendings may provide teaching and supervision without residents.

A number of other standards are specific to geographically separate sites; however, they are related more to the administrative structure of the medical school than to specific clerkships and they will not be discussed here. Where pertinent, further discussions of the above standards will follow in this chapter (See also Chapter 15, The Clerkship Director and the Accreditation Process). (return to top)

Selecting Sites

The clerkship director should determine whether multiple clerkship sites are needed to achieve the program’s educational goals and objectives, and assess the role of each site in achieving them. There is a continuum of geographically separate rotation sites. At one end is a model where students receive their entire clerkship experience at a separate site that has some central administration. The other end of the continuum includes a location where students spend only a portion of their clerkship time and the majority of administration, didactic education, and structured evaluation occur in a central location. Regardless of the model, the key to successful separate sites is consistency. The expectations, experiences, and evaluation of all students must be consistent and equitable. It is also important to have a well-established organization chart with clearly delineated lines of authority and a commitment to a central institution.

Audit Resources

Clerkship directors considering separate sites should audit their available resources. This resource audit should begin with identification of possible sites and medical school resources for administration of the clerkship. Distant sites create a more complex clerkship structure and require more administrative and travel funds. Accessibility and cost of travel for students and the clerkship director must be considered. Ultimately, a budget will need to be created to determine how medical school funds allocated to the clerkship will be distributed among faculty, site directors, clerkship director, sites, and the central institution.

All sites should also undergo a resource audit before they are included in the educational program. Sites used to provide the entire clerkship education must be thoroughly evaluated to ensure that they have the necessary resources and faculties to accomplish all of the clerkship’s goals and objectives. Each clerkship objective should be matched with teaching/learning activities and the available instructional resources. For sites where students will spend only a portion of the clerkship time, the clerkship director needs to ensure that the resources are appropriate for the objectives to be achieved at that site. Limitation in the learning opportunities offered at a site may be offset by the complementary experiences offered at other sites to ensure that all the required elements are covered. A checklist for performing the resource audit is included in Table 2. The resource audit should include a visit to the site.

Visit the Site

The clerkship director should visit the site and meet with the local faculty members to explain the program’s educational goals and objectives. The meeting should include:

• Reviewing the knowledge objectives, clinical experience objectives, and teaching objectives of the clerkship
• Assessing the commitment of the faculty to meeting the objectives of the proposed program
• Evaluating the support of the hospital administration for the program
• Assessing the available resources and facilities.

This ‘faculty meeting’ is an important opportunity to assess faculty ‘buy-in’ and identify potential obstacles to successful integration of the site. (return to top)

Obtain Buy-in from Local Physicians and Staff

If educational activities already exist at the site, such as resident conferences or grand rounds, the schedules and format of conferences should be reviewed and evaluated. The need for additional student-focused sessions should be addressed. If residents are going to be involved in the teaching of medical students, the clerkship director should meet with the program director and chief residents to assess their commitment and ability, and to obtain insights regarding the educational milieu from the perspective of trainees. The program director must commit to the residents’ teaching and to their accountability for teaching and evaluating students. If nursing or allied health personnel are to be involved with incorporating students into the clinical setting, it is important that they be included in this initial assessment process as well.

Assess Resources

The number of patients and the variety of clinical problems they present should be assessed to ensure there are sufficient clinical opportunities for student involvement to meet the goals and objectives of the rotation. Both the clinical and educational facilities should be evaluated to determine whether they would provide an environment conducive to learning. Inpatient units, outpatient facilities, call room space, libraries, computer access, and available audiovisual support should all be assessed. In addition, availability of individual beepers, parking, public transportation, housing close to the medical center, and safety issues need to be assessed.

The level and quality of administrative support should also be assessed. The level of support needed will depend on the level of autonomy of the site. However, some administrative support in the form of a coordinator will be essential to meet the day-to-day needs of medical students. A site coordinator may need to prepare teaching schedules, assign lockers and meal tickets, schedule examinations, and handle other logistics in connection with the educational experience. If a residency is present at the site, the administrative responsibilities may be handled by the residency coordinator. In some cases, the residency coordinator may already be overwhelmed with responsibilities and a separate coordinator will need to be identified.

This visit with the faculty and potential site director is an opportunity to identify obstacles to achieving the goals of the clerkship. It is also an opportunity to assess the need and desire for faculty development and how to deliver it most effectively (See also Chapter 8, Faculty Development). If the central administration can offer incentives, this is also the time to find which ones would be most helpful or appreciated by the staff involved. This initial process is very important to building a strong and sound collegial working relationship.

Develop a Memorandum of Understanding

Once the rotation site is determined to be appropriate, a memorandum of understanding should be generated by the medical school administration and signed by appropriate persons from the medical school and the site (LCME Standard ER-9). The document should detail expectations from the perspectives of both the school and the site, the process to be used to evaluate the success of the program-site relationship, the planned mechanism for correcting problems, and the time frame encompassed by the agreement. Table 3 lists components that might be included in such an agreement.

Identify a Site Director

The clerkship director should identify a qualified faculty member to serve as site director at the proposed location. The site director will be responsible for the student program and should serve as a principal educator and mentor for the students at the site. This individual should have credibility within the department and institution, and direct or delegate authority to implement programs and effect change. Educational expertise is helpful, but a major commitment to teaching and adequate protected time to perform the job is most important. Expectations of the site director should be explicitly described in a job description, which should also delineate his or her involvement with curriculum development, teaching responsibilities, and role in student evaluation. This job description should also describe any compensation that will be provided to the site director, and the time frame of the appointment.
(return to top)

Administrative Structure of a Multi-site Clerkship

The administrative structure of any clerkship should facilitate the achievement of the desired educational goals and objectives. The structure will vary from discipline to discipline and from institution to institution, based on the specific education model, the culture of the institution, and the available resources. Compared to single-site clerkships, a multi-site clerkship generally requires a more complex administrative structure for management and coordination of various activities. Sufficient resources to support the clerkship must be committed by the department and the medical school (See also Chapter 2, Educational Administration and Leadership).

Responsibilities of Clerkship Director

The central clerkship director should be responsible for communicating with all site directors and for assuring that each site is functioning within the overall goals and objectives of the clerkship. He or she is responsible for overall clerkship evaluation and for assuring that no individual site is falling below the acceptable standard of performance. The clerkship director is responsible for ensuring that the clerkship and each individual site are in compliance with LCME standards. The level of responsibility assigned to site directors may vary based on site distance, autonomy, and resources. All assessment and evaluation data should be collected and submitted to the central director for evaluation and ultimate distribution. The central director should be responsible for final grade assignment and communication of grades to the medical school. The clerkship director will generally be responsible for the majority of administrative issues, and a full-time administrative assistant or coordinator will usually be required.

Responsibilities of Site Directors

Responsibilities of site directors vary. They may function fairly autonomously to the point of coordinating all site-specific educational activities, schedules, administration of student assessments (exams), collection and assimilation of faculty evaluation of students, and administration of clerkship evaluations. Even if the site director is responsible for all of the above, the central clerkship director and site director communicate and coordinate effectively. Assessment and evaluation data should be submitted to the central director for collection and for overall clerkship evaluation. In other situations, site directors may have more limited responsibility and assessment and evaluation may be administered through the central director. These specific responsibilities will be determined by the structure of the clerkship. It is imperative that they be clearly delineated in the policies and procedures of the clerkship and that all site directors undergo an initial orientation to the clerkship (see Table 4 for a checklist for Site Director Orientation).

In most cases, the site director will be responsible for communicating with local faculty and will need to orient new faculty (See Table 5). The degree of administrative support needed for site directors will depend on their responsibilities.

Communication among all sites needs to be frequent and multi-directional.  There should be regular scheduled communication between site directors and the clerkship director, which may take place by phone, electronically, or in person.  Unscheduled communication when problems arise should also be encouraged. Meeting periodically with all site directors and administrative staff is an essential aspect of consistency and it develops a sense of community across a geographically separated clerkship. The clerkship director can accomplish this by visiting all sites on a regular basis or having quarterly or annual meetings of all site directors and administrators at a central location. This meeting is an opportunity to review program evaluation data, discuss curricular and policy changes, come to mutually acceptable agreements on changes, hold faculty development sessions, and further develop community. (return to top)

Curriculum for Multi-site Clerkships

The importance of and rationale for establishing a standardized curriculum across the geographic separate sites needs to be clearly understood by the site directors and the faculty. Standardization of the curriculum helps keep faculty instruction consistent at all locations, reduces the risks of varying educational content due to different perceptions of individual faculty regarding the importance of various topics or skills, and helps ensure consistency in learning outcomes and student expectations regardless of rotation site.

Goals and Objectives

Goals and objectives serve as a blueprint for planning the teaching and learning activities and the performance evaluation process. Goals and objectives should be uniform across sites. Many professional societies have published goals and objectives for the medical student in their respective disciplines. These can be adopted as written or modified based on faculty input. Employing established objectives is simple (relatively speaking!) and efficient. Usually the objectives are developed and peer reviewed by a team of physicians in a discipline. The group development and peer review process enhance content validity.

Implementation Plan for Each Site

The next step is determining the level of competency that each student should achieve on the clerkship. Faculty at each site should be involved in defining and revising objectives and determining how the objective will be met at the site. This will increase the familiarity of the faculty with the objectives.
 
Following the acceptance of uniform goals and objectives across sites, each site director should be requested to develop an instructional plan. The format of how the student will meet these objectives should be delineated at each clerkship site. This plan should outline the teaching and learning methods to be used to address the objective, when in the rotation they will be addressed, and how achievement of the objective will be documented. This documentation is best accomplished with a log system that is uniform at all sites. The log system is discussed in greater detail below.

Even with a common curriculum, the specific teaching strategies used to achieve individual objectives may vary from site to site, depending upon patient availability, instructor experience, support systems, and other factors. For example, certain didactic teaching sessions may need to be added to or deleted from a site depending upon the volume and mix of patients. The site director should be able to determine if the objectives are achievable at the site. If the objectives cannot be met at the site, alternate learning modes should be developed, such as CD-ROM, Web-based interactive, or travel to alternate sites for clinical experience.

Strategies to Deliver a Uniform Core Curriculum

Core Content: Teaching Conferences

Students Return to the Primary Campus for Daily Conferences

Consider a centrally conducted teaching program (e.g., conferences or tutorials) attended by students from various sites. Such a program avoids the need for duplication of faculty effort and conserves resources. However, the feasibility depends on the geographic proximity of sites and the time students would spend commuting.

In the surgery clerkship at MCP Hahnemann School of Medicine, students rotating at various sites in the Philadelphia region commute daily to a central location to participate in tutorials that are conducted by faculty and address core content areas. Due to limited resources, these tutorials would be difficult to replicate at all rotation sites.

Students Return to the Primary Campus for a Concentrated Series of Conferences

An alternative approach is to concentrate teaching sessions within a given period of time during the clerkship. For example, family practice clerks at the Southern Illinois University School of Medicine remain at their assigned site until the last week of the clerkship, and at that time they return to the parent site for core instruction, to present a researched topic, and to take the final exam.

Students at the University of California, Davis (UCD) Obstetrics and Gynecology (OBG) clerkship return to central campus one day a week for problem-based learning with UCD faculty. This time can also be used to review write-ups and establish mentoring relationships with central faculty.

Students Remain Off Site and Use Distance Learning Modalities

Distance education is a sound, creative, nontraditional approach that uses a variety of media in a wide range of settings. The media include television, computers, computer-assisted instruction packages, electronic mail systems, bulletin boards, interactive video discs, audiocassettes, and print. Independent study modules that use one or more of these media could be developed for students to use at various sites.

Standardized patients trained at the medical school could travel to each site to participate in student teaching and evaluation activities. Videoconferencing is another alternative to accomplish a combined conference without commuting, but requires fairly sophisticated information technology.
 
The virtual clerkship using video conferencing, blackboard or Web board resources, discussion boards, and case modules is another method to decrease need for duplication at sites and unnecessary travel for students. Self-assessment in the form of periodic quizzes can be used as a formative assessment of knowledge levels.

Students Participate in Complete Conference Series at Multiple Sites

The OBG clerkships at the University of Washington and Michigan State University are autonomous and provide a complete didactic/problem-based learning module at each site. The curriculum to be delivered is determined by a consensus of the central administration and faculty of all sites.

Core Content: Clinical Experiences

It is important to have consistent clinical experiences, feedback, and evaluation in addition to consistent didactic experiences. LCME Standard ED-2 requires defining students’ required clinical experiences and how their competency will be assessed. There should be alternative learning experiences for students who do not see specified numbers of patients or specific diagnoses, such as standardized patients or case modules. Virtual problem-based learning using patient information and real-time data are other opportunities to augment learning experiences.  (return to top)

Assessment of Students at Separate Sites

This section focuses on assessment and evaluation across a clerkship with geographically separate sites. (See also Chapter 6 , Evaluation and Grading of Students).

Issues Related to Consistency of Assessment/Evaluation

The majority of clinical clerkships use a variety of evaluation methods, including descriptive evaluations of students by teachers ("subjective" evaluations or clinical performance evaluations), and one or more examinations.3-7 Evaluation methods and implementation must be consistent across all clerkship sites. For example, all teachers should evaluate students from the same frame of reference and document their observations in similar ways. It may be easier to gain consistency in descriptive (narrative) evaluations if a common paper or Web-based form is used at all sites. Similarly, common methods of documenting direct observation of trainees could be used, using encounter cards,8 the mini-Clinical Examination Exercise (CEX),9,10 or other methods.11-13 These evaluation forms and methods communicate goals to clinical teachers, emphasize expectations, and outline minimal proficiency standards. The timing of the assessments should be the same across all teaching sites, ideally at the midpoint and end of each clerkship rotation. Timely collection of faculty evaluations of students is the responsibility of the site director, and may be enhanced through direct meetings or the use of electronic evaluations with automated reminders.

Examinations

Where and how examinations will be administered must be determined. If the teaching site is far from the parent campus, the clerkship director must decide whether to bring all students back to the main campus for final examinations or give the examinations at the local clerkship site. If examinations will be administered at multiple sites, arrangements must be made to ensure timely delivery of the examinations to the site and their return for scoring, that there is someone to administer the examinations, and that the examinations and the testing setting are secure. There may be an opportunity for students to share confidential testing information if sites cross time zones.

Components and Assignment of Grades

The clerkship director is responsible for determining the components that will be used to determine the final grade and the relative weight of each component. This may involve determining whether:

• Each component of the clerkship will have similar or different weights
• Differing sites carry similar or different weights and,
• Students evaluations based on experiences later in the clerkship should contribute more to the final grade determination.

The decisions must be clearly communicated to the students and to the individual clerkship sites and site directors. A decision must be made regarding who will be responsible for determining the final grade. Final grades could be determined by the clerkship director , a grading committee at the central site, or the faculty/director at the separate sites. The faculty at separate sites will play a key role in providing narrative and other evaluations even if they do not determine the students’ final grade.

Record Keeping of Student Evaluations and Grades

We strongly advocate for a central, secure location for collection of the students' clerkship information (e.g., evaluation forms, examination scores), such as a locked file cabinet, password-protected computer, or secure Web site. Individual clerkship sites may need to keep copies of the records generated locally until the student's grade is complete.

Suggested Ways of Achieving Consistency of Assessment

Group (team) Meetings to Evaluate Students at Separate Sites

Descriptive evaluation is an essential feature of student evaluation and is an opportunity for faculty development. Electronic or hard copy forms should be short and include behavioral descriptors for performance levels within each evaluation domain (See also Chapter 6, Evaluation and Grading of Students). Meeting with the teachers to discuss the students' performance is an effective and "low-tech" way to improve teachers' evaluations of students.14-21 A common evaluation form and method for evaluation, including group evaluation meetings, have been central features of system that yielded consistent assessment over 10 years in a clerkship with sites as far as 6000 miles from the medical school.22 Group evaluation meetings, including attending physicians and residents, may occur during resident conference time, which sends a clear message that the activity is important.17

Centralizing Evaluation at the Medical School

Another method is to remove some of the responsibility of evaluation from the separate teaching sites and place it centrally at the medical school. For example, students taking the primary care clerkship at Penn State are distributed throughout rural Pennsylvania.23 Students must complete several requirements, including a learning objectives assessment, a family project, and a clinical reasoning exercise. Each is reviewed, scored, and graded by core clerkship faculty at the medical school. A process that centralizes evaluation and grading may be particularly important when students are sent to a large number of sites that that many not have a specific site director.
 
Regardless of the evaluation methods you choose, the clerkship site directors should be an integral part of the clerkship decision-making process.1,14,22,24 Site directors who feel ownership of the clerkship, including the evaluation process, are likely to understand clerkship goals and expectations, and effectively monitor their site. This may be particularly important for clerkship sites that host students from more than one medical school, where different clerkships may have different expectations of students and teachers.25 (return to top)

Formative (Feedback) and Summative Assessment at Separate Sites

Ensuring students receive feedback at multiple sites of a clinical clerkship is a challenge. Students should understand their performance in the context of clerkship goals and expectations, know their areas of strength and deficiency, and have a plan for improvement before they leave a clerkship site. The site director is responsible for ensuring they receive effective feedback. All students should receive specific feedback and suggestions for improvement. However, it is particularly important that students who are not meeting expectations are given explicit and documented feedback on their performance by the site director (See also Chapter 6, Evaluation and Grading of Students, section on Failing Grades).

Pangaro21 has outlined characteristics of credible formative and summative feedback to students (Chapter 6). The AAMC expects clerkships and departments to improve the formative feedback given to students during clerkships.26 All faculty at all sites musts become skilled at providing and documenting feedback. Some students do not recognize feedback when it is given27 and few believe they have received enough. However, the process of feedback, formative and summative, can be ubiquitous and more consistent across clerkship sites.

Video Teleconferencing

In an OB-GYN clerkship, video teleconferences have been used to perform mid- and end-of-rotation evaluation, and generate comments for feedback.28 Video teleconferences are held at the mid-point of the 6-week clerkship and involve the individual site directors and the overall clerkship director . The directors discuss students currently on the clerkship, and generate mid-term feedback. They also discuss the students who completed the clerkship 3 weeks earlier, and generate final, summative evaluations. The clerkship director should guide the site directors in their evaluation discussions and provide faculty development concurrently. By holding the conferences at the mid-point of the rotation, they emphasize the critical importance of formative feedback.

Face-to-face Formal Evaluation Sessions

Face-to-face meetings between the clerkship director or  site director and the teachers currently working with the students have been used successfully at several institutions.14-19,29 Formal evaluation sessions are regularly scheduled meetings held throughout a clerkship, at all clerkship sites, to generate formative and summative evaluations and feedback and to provide faculty development.  During the sessions, the clerkship director can help the teachers decide how to provide feedback and what to say to students. Direct communication with faculty facilitates communication and does not require the technical support necessary for teleconferencing (See also Chapter 6, Evaluation and Grading of Students).

Other Methods of Providing Feedback at Clerkship Sites

Medical students can be trained to solicit specific feedback from their teachers.30 Mini-CEX (clinical evaluation exercise) and clinical encounter cards can be used to formalize feedback after direct observation.8-13 The clerkship director can send reminder notices to students and faculty about seeking and providing feedback.31 Workshops for teachers about giving feedback have been shown to change behavior. In one study that used pre- and post-intervention audio-taped encounters between students and teachers, teachers at one institution improved their feedback to students.32,33  (return to top)

Issues of Faculty and Resident Development

This section highlights areas of particular importance for a clerkship director with multiple, geographically separate sites. Chapter 8, Faculty Development, and several review articles provide detailed information.34-37

Site Directors

The importance of choosing, developing, and mentoring clerkship site directors cannot be overestimated. These individuals are central to the success of the clerkship, including ensuring effective feedback and consistent assessment, evaluation, curriculum, and experiences. Site directors should participate in a thorough orientation (Table 4). The clerkship director should create a master repository (electronic or paper) of information for each site director. The repository should include clerkship objectives, materials to be distributed during the clerkship, copies of commonly used forms, expectations of and for members of the teaching faculty, job descriptions, key phone numbers (including Office of Student Affairs at the medical school), and landmark educational articles (See also Chapter 2, Educational Administration and Leadership).

The clerkship director and site directors share the responsibility for leading the faculty. All site directors should be an integral part of the clerkship policy and planning process, which can also include reviewing and grading marginal students.24,25,28,38 Involvement in key clerkship functions helps site directors acquire a thorough understanding of the clerkship and helps ensure consistency across sites. Site directors should be encouraged to join and participate in their relevant national clerkship directors' organizations and their meetings. All site directors should be urged to join the Group on Educational Affairs of the AAMC, and seek opportunities to attend and/or lead medical education-related workshops and lectures. Clerkship directors can play a key role in advancing the reputation and recognition of their site directors, certainly on an institutional, and even regional, level (See also Chapter 8, Faculty Development and Chapter 12, Career Development for Medical Student Clinical Educators).

Faculty—New and Existing

Faculty involved in the teaching program should understand the expectations of their role and should receive orientation from the site director (Table 5). When a new site is added to the clerkship, the clerkship director should assess the need for faculty development in key areas, such as providing feedback, student assessment, and teaching in ambulatory settings. Ongoing faculty development should occur periodically, based on any problems or deficiencies identified in faculty or program evaluations. Evaluation sessions15 have been estimated to provide 2 to 4 hours of faculty development annually for faculty and 6 to 8 hours for residents. Teachers should expect that they can express their views and provide feedback about the clerkship to the site director and/or the clerkship director, and that the clerkship director will help them grow and improve in their teaching skills. "One thing seems certain though—effective faculty development involves listening to faculty about their concerns, ideas and aspirations and then encouraging active involvement and engagement in the educational process." 39 Both new and experienced teachers should also seek to participate in local, regional, or national medical education courses.

Residents

Housestaff play a central role in clerkship teaching programs and the need to help them develop as teachers is well recognized.26,40-42  While it may be beyond the scope of the clerkship director to provide faculty development for residents at all teaching sites, knowing and documenting what programs are in place at the local residency level is essential. Residents should be included in clerkship faculty development whenever possible. In this way, clerkship directors can aid residency directors in meeting the need to provide residents with teaching skills.

Responding to Problems

Invariably, problems will arise in the relationship between students and teachers.  The clerkship director must have confidence in the site director to be the first person to address the concerns of the student and/or teacher and to keep the clerkship director informed. Students must feel safe in discussing their concerns about a teacher, and clerkship directors need to be prepared to discount any teacher's evaluation that is not based on clerkship expectations, or is based on factors other than the student's performance. Similarly, concerns about a student should be addressed promptly, and often provide an important "teachable moment" for the faculty. Prompt and documented feedback from the site or clerkship director to the student or teacher is necessary.

Benefits to Community-based Faculty and Residents

Offering faculty appointments to those physicians who regularly teach is an important gesture, and often confers tangible benefits such as tuition benefits, library access (electronic and/or "brick and mortar" use) or discounts at bookstores. Invitations to workshops at the medical school may be offered if faculty are close enough to attend. Teaching awards, with or without a monetary award, are simple methods to recognize outstanding contributions by teachers and can be established easily. Residents should be recognized through similar mechanisms. (return to top)

Evaluating a Multi-site Clinical Clerkship Program

The LCME requires comparable educational experiences for students who rotate at different clinical sites during a clinical clerkship.2 Standard ED-2 stipulates that clerkship directors must specify the disease states or conditions that all students are expected to encounter during the clerkship, including the numbers of patients with the problems and the level of student responsibility. Furthermore, clerkship directors must be able to make adjustments for students to meet educational objectives. These requirements necessitate the use of some type of clerkship log.

Using Clinical Logs to Track Students’ Experiences at Various Sites

Clerkship logs have been used for decades to track students’ clinical experiences. A comprehensive review of their development, use, and usefulness will be published.43 (See also Chapter 4, Technology in Clerkship Education and Chapter 7, Evaluation of the Clerkship: Clinical Teachers and Program). Paper logs and computer readable cards were the primary modalities used in the past, but Internet-based or personal digital assistant (PDA) logs are gaining popularity. Electronic logs can be viewed in "real-time" during the clerkship and used to provide feedback to individual students and for site evaluation (e.g., compare diagnoses seen at different clerkship sites). Logs are useful but have limitations, including accuracy of data entry (mostly under-reporting problems seen and procedures performed), compliance, cost of developing and implementing, and clerkship director time to review and monitor.

Decisions about using a “homegrown” log system, purchasing software, or adopting a system developed at another medical school are discussed in Chapter 4, Technology Clerkship Education. Decisions about the items to track may be made by the individual clerkship director or collaboration among all clerkship directors at the school. However, the same system and same requirements should apply to all clerkship sites.

We suggest dividing the experiences students are expected to log into essential and desirable categories. This approach can help improve student compliance by limiting the amount of effort it takes to complete the logbook and ease of analysis by limiting the variables necessary to review. Essential logbook content might include student identification, graduating year, clerkship, rotation number (time of the year), clerkship site name, inpatient or ambulatory rotation, patient age, patient gender, problems (e.g., specific diagnoses, problems, or symptoms grouped by relevant organ system), level of student responsibility (e.g., participated in care, teaching case, observation only), type of patient (e.g., real or simulated), and procedures performed. Desirable content might include distinguishing primary and secondary problems, and time spent on specific activities such as reading, in lectures, or in the operating room.
 
Ideally, the clerkship director, site director, or other faculty should review the student's logbook entries during the clerkship to provide feedback on the problems seen and outline a plan to address any core problems the student has not yet seen. An example of an Internet-based clerkship logbook can be found at http://cweblog.usuhs.mil. The log data can be used to ensure that objectives are being met at each site and to identify any areas of deficiency specific to a given site. (return to top)

Developing Comprehensive Program Evaluation of a Multi-site Clerkship

A comprehensive evaluation system is imperative for all clerkship, but especially for clerkships with multiple sites. The results have been used for evaluating student performance, determining whether student achievement is affected by site, identifying problem areas, and facilitating communication among faculty. Information obtained from such an evaluation system not only documents student performance, but also serves as a stimulus for improving the academic program and teaching effectiveness of the clerkship.1  (See also Chapter 7, Evaluation of the Clerkship: Clinical Teachers and Program.)

One of the greatest challenges of directing a multi-site clinical clerkship is ensuring that students receive comparable experiences at all sites so they are not harmed or benefited by performing their clerkship at a particular site. Inter-site consistency is important regardless of whether the sites are at different hospitals or practices in the same city or are spread over a large geographic area.22

Impact of Site and Clerkship Length on Students’ Performance

Many studies of multi-site clerkships have assessed short-term outcome measures of student performance, such as examination results. Some studies have found no differences when comparing student examination performance based on whether they were assigned to community or academic health center teaching sites,43,44 while others found that the site can make a difference. Many studies have found that the site of the clerkship had no impact on the students' final clerkship grade,47 faculty ratings of students’ final examination performance,48-52 or even subsequent ratings as PGY-1s.50 However, performance differences may be identified that lead to educational interventions to achieve consistency across clerkship sites.52 

Length of clerkships at various sites may also impact students’ performance. One study of a widely distributed OB-GYN clerkship at the University of Washington demonstrated that ratings of student presentations, clinical performance and patient write-ups were lower at sites with 4-week than 6-week rotations.1 (return to top)

Experiences, Activities, Patient Problems Encountered

Often there are differences in the number of patients seen at different sites, the types of problems seen, the number of procedures performed, and feedback received.53 There may be differences in the illness severity for a given problem,54 or in the differences in student responsibility and teaching styles among sites.55 Studies report significant differences in how students spend their time at different clerkship sites, including total time invested in learning, in patient care, in observing or participating in care, and in doing patient-related work such as charting.56,57 Other studies have reported smaller differences in activities and problems encountered across clerkship sites.58,59 Overall, however, similarities in experiences tended to outweigh statistical differences The educational impact on students was unclear (e.g., no differences in examination scores) and the information revealed the experiences were comparable.53-59 

Role of Student Critiques

Student critiques of the clerkship provide a critical means to assess the success of the clerkship.60 Students should be asked to evaluate the clerkship (program evaluation) and individual faculty (See also Chapter 7, Evaluation of the Clerkship: Clinical Teachers and Program). Program evaluation questions should include how well clerkship goals and objectives were specified, how well they were met, the quality of the clerkship experience, mandatory clerkship activities (e.g., lectures, written work requirements), and areas for improvement. Individual teacher evaluation should include the faculty and residents with whom they worked. The evaluations provide feedback to the clerkship director on the clerkship, specific sites, and individual teachers at each site.
 
The evaluations can be paper or electronic. Paper-based evaluations are easy to administer and circulate among individual clerkship site leaders for review and comment before they are returned to the clerkship director and individual teachers. However, they can be time intensive because the data must be entered into a database for effective longitudinal tracking of trends and ratings. Computerized evaluations allow rapid analysis of the information, plotting of trends, and creating summary data. Electronic systems require student access to the computerized database, (Web- or PDA-based). Eight to 20 student critiques are generally needed to achieve a dependable assessment of a given teacher.61,62

Faculty Critiques

The teachers who participate in the clerkship should also evaluate both the students and the clerkship. The clerkship director should assess the clerkship within the context of the entire medical school curriculum and use the results for determining areas of strength and weakness.63

Programmatic Evaluation: What to Collect and How to Analyze?

Clerkship directors need to decide what information they should collect and monitor about the students' performance and the teaching program at each site (See also Chapter 7, Evaluation of the Clerkship: Clinical Teachers and Program).

Durning et al. 22 described a framework of data to collect before, during, and after the clerkship.

Before-the-Clerkship Data:
Essential before information may include the students' pre-clerkship grade point average, USMLE Step 1 scores, end-of-second-year OSCE, or a test given on the first day of the clerkship.64 The before information is critical because much of the variance in student performance on the clerkship may explained by knowing "what they bring" to the clerkship.22,65 Statistical analysis of clerkship data should control for baseline student characteristics.
 
During-the-Clerkship Data:
Essential during information might include ratings or recommendations provided by the teachers, scores on within-clerkship examinations or required exercises, the clerkship site, and the rotation location in the academic year.

After-the-Clerkship Data:
Essential after information should include short, intermediate, and longer-term outcomes of importance to the clerkship. Data might include final examination scores (e.g., NBME Subject Tests, oral exams, internal exams, OSCEs), final recommendations from teachers, the students' final clerkship grade, performance on USMLE Step 2 examinations, or internship ratings.38,66 The information collected can be readily entered into a spreadsheet or directly into a commercially available statistical software program.
 
Analysis of the Data:
The data may be analyzed in multiple ways. You may need assistance, depending on your skills. Data analysis using descriptive statistics (e.g., mean, median, modes, standard deviations), or simple graphic plots of the data may uncover trends or differences across sites.67 One can look for differences across sites using analysis of variance, chi-square, or similar statistical methods. Linear or logistic regression models are powerful ways to explore whether a clerkship site, or any other independent variable, has an impact on the outcome of interest (for example, examination scores).

Ideally, the clerkship site does not contribute to the variance in outcomes, so it should not matter where the student performs the clerkship.22

Evaluation Summary

Important programmatic evaluation data can come from routinely collected data, including student experiences (based on clerkship logbook entries); student performance (e.g., grades, examinations); and critiques completed by students and/or faculty. The information should be analyzed and reviewed on an annual basis. Some information should be reviewed more frequently, such as reviewing logbooks and student critiques after each clerkship rotation. Other data, such as examination scores, might be best reviewed at the mid-point and end of the year. The results of the analysis should be shared and discussed with all who have an interest in the clerkship program. At a minimum, this includes the department chair and vice-chair for education, and the clerkship site directors. (return to top)

Issues of Communication

Clear lines of communication are particularly important in a multi-site clerkship. There must be clearly delineated, multi-directional lines of communication that allow students, faculty, site directors, and the clerkship director to communicate whenever needed.

Communication with Students

If possible, the clerkship director should conduct a combined orientation for all students from all sites. This could be done at a central location or by video conferencing from individual sites. Goals, objectives and grading policies should be discussed. This will help students to understand that these are uniform aspects of the clerkship. Students should be told the appropriate line of communication if they have a problem (personal or with a resident, faculty, site director, or clerkship director). They should be given permission to contact the clerkship director if they have a concern that is not addressed at another level. For prolonged clerkships, the clerkship director should make additional contact with students midway through the clerkship to solicit comments or questions. This contact can be made electronically or at a site visit.

Communication with Faculty

The clerkship director should have at least bi-yearly contact with faculty at site visits. In the intervening period, there should be an open channel of communication by which faculty can raise concerns or suggestions. Electronic list serves or bulletin boards can be used to update all faculty or to provide faculty development. A periodic clerkship newsletter can be used to disseminate successes, areas to work on, and evaluation data. The site faculty will communicate mostly with the site director, but they should also be encouraged to communicate with the clerkship director as needed. Site directors should meet with faculty regularly. clerkship directors and site directors need to have regularly scheduled meetings to cover changes or updates in curriculum, faculty development, student and program evaluation and clerkship response to evaluation data. In addition to regularly scheduled meetings, informal contact about specific issues or student concerns will be needed. (return to top)

Conclusion

Clerkships at geographically separate sites create opportunities and challenges. Effective implementation requires a clearly defined structure that is understood by all participants. Select sites and site directors carefully and communicate frequently, effectively, and multi-directionally. Site directors and community-based faculty should actively participate in improving the clerkship so they feel a sense of “ownership” of the clerkship. The clerkship director must functionally integrate site directors, faculty, and students, and monitor issues of curriculum, feedback, student assessment, and program evaluation continuously to ensure comparable, equitable learning and evaluation for all students at all sites. Geographically separate sites create opportunities for exemplary student education, faculty development, and educational scholarship.

(return to top)

Table 1

Selected LCME Standards Pertinent to Clerkships with
Geographically Separate Sites

From: Functions and Structure of A Medical School: Standards for Accreditation of Medical Education Programs Leading to the MD Degree
Complete standards available at: http://www.lcme.org

*Hint: Click Back button on browser to return to previous spot in text
OR (return to top)

Table 2

Resource Audit Checklist

*Hint: Click Back button on browser to return to previous spot in text
OR (return to top)

Table 3

Memorandum of Agreement between Medical School and Site

*Hint: Click Back button on browser to return to previous spot in text
OR (return to top)

Table 4

Topics for Site Director Orientation

*Hint: Click Back button on browser to return to previous spot in text
OR (return to top

Table 5

Topics for Faculty Orientation

*Hint: Click Back button on browser to return to previous spot in text
OR (return to top)

 

References

1. Irby DM, Vontver LA, Stenchever MA.  Evaluation of student performance in a multi-site clinical clerkship. Am J Obstet Gynecol. 1980;136:1020-3.
2. Functions and Structure of a Medical School: Standards for Accreditation of Medical Education Programs Leading to the M.D. Degree. Liaison Committee on Medical Education. October 2004. Available at www.lcme.org/pubs.htm Last accessed 30 April 2005.
3. Zahn CM, Nalesnik SW, Armstrong AY, Satin AJ, Haffner WH. Variation in medical student grading criteria: a survey of clerkships in obstetrics and gynecology. Am J Obstet Gynecol. 2004 May;190(5):1388-93.
4. Magarian GJ, Mazur DJ. Evaluation of students in medicine clerkships. Acad Med. 1990; 65:341-5.
5. Hemmer PA, Szauter K, Allbritton TA, Elnicki DM. Internal Medicine Clerkship Directors’ Use of and Opinions about Examinations. Teach Learn Med.  2002;14(4):229-35.
6. Kumar A, Gera R, Shah G, Godambe S, Kallen DJ. Student evaluation practices in pediatric clerkships: a survey of the medical schools in the United States and Canada. Clin Pediatr. 2004;43(8):729-35.
7. Schwiebert LP. How required US family medicine clerkships and preceptorships evaluate medical students. Fam Med.1996;28:559-64.
8. Hatala R, Norman GR. In-training evaluation during an internal medicine clerkship. Acad Med. 1999;74(10 Suppl):S118-20.
9. Kogan JR, Bellini LM, Shea JA. Feasibility, reliability, and validity of the mini-clinical evaluation exercise (mCEX) in a medicine core clerkship. Acad Med. 2003;78(10 Suppl):S33-5.
10. Hogan JR, Bellini LM, Shea JA. Implementation of the mini-CEX to evaluate medical students' clinical skills. Acad Med. 2002;77(11):1156-7.
11. Hasnain M, Connell KJ, Downing SM, Olthoff A, Yudkowsky R. Toward meaningful evaluation of clinical competence: the role of direct observation in clerkship ratings. Acad Med. 2004;79(10 Suppl):S21-4.
12. Howley LD, Wilson WG. Direct observation of students during clerkship rotations: a multiyear descriptive study. Acad Med. 2004;79(3):276-80.
13. Holmboe ES. Faculty and the observation of trainees' clinical skills: problems and opportunities. Acad Med. 2004;79(1):16-22.
14. Jacobson MJ, Sherman L, Perlman I, Lefferts R, Soroff H. Clerkship site and duration: Do they influence student performance? Surgery. 1986;100(2):306-10.
15. Hemmer PA, Pangaro L. Using Formal Evaluation Sessions for case-based faculty development during clinical clerkships. Acad Med. 2000;75:1216-1221.
16. Battistone MJ, Pendleton R, Milne C, Sande MA, Hemmer PA, Shomaker TS. Global Descriptive Evaluations are More Responsive Than Global Numeric Ratings in Detecting Student Progress During the Inpatient Portion of an Internal Medicine Clerkship. Acad Med. 2001;76:pS105-7.
17. Battistone MJ, Milne C, Sande MA, Pangaro LN, Hemmer PA, Shomaker TS.  The Feasibility and Acceptability of Implementing Formal Evaluation Sessions and Using Descriptive Vocabulary to Assess Student Performance on a Clinical Clerkship. Teach Learn Med. 2002;14:5-10.
18. Ogburn T, Espey E. The R-I-M-E method for evaluation of medical students on an obstetrics and gynecology clerkship. Am J Obstet Gynecol. 2003;189(3):666-9.
19. Albritton TA, Fincher RM, Work JA. Group evaluation of student performance in a clerkship. Acad Med.1996;71(5):551-2.
20. Holmboe ES, Hawkins RE, Huot SJ. Effects of training in direct observation of medical residents' clinical competence: a randomized trial. Ann Intern Med. 2004 Jun 1;140(11):874-81.
21. Pangaro L. A New Vocabulary and Other Innovations for Improving Descriptive In-training Evaluations. Acad Med. 1999;74:1203-1207.
22. Durning SJ, Pangaro LN, Denton GD, Hemmer PA, Wimmer A, Grau T, Gaglione MA, Moores L. Intersite Consistency as a Measurement of Programmatic Evaluation in a Medicine Clerkship with Multiple, Geographically Separated Sites. Acad Med. 2003;78(10):S36-8.
23. Kaplan KM, Masters PA, Leong SL, Wallen BA, Lederer SE, Hawkins AH, Jones RL. Evaluating students on an interdisciplinary primary care clerkship at the Pennsylvania State University College of Medicine. Acad Med. 1999;74(1 Suppl):S67-9.
24. Parenti CM. A Process for Identifying Marginal Performers among Students in a Clerkship. Acad Med. 1993;68:575-7.
25. Durning SJ, Pangaro LN, Sweet J, Wong RY, Sealey ML, Nardino R, Alper E, Hogan K, Hemmer PA. Clerkship Sharing on Inpatient Internal Medicine Rotations: An Emerging Clerkship Model. Teach Learn Med. 2005;17(1):49-55.
26. Nutter D, Whitcomb M. The AAMC Project on the Clinical Education of Medical Students. Association of American Colleges, Washington, DC (2001). 
27. Irby DM. What Clinical Teachers Need to Know. Acad Med. 1994;69:333-42.
28. Browne MW, Hines JF, Satin AJ, Haffner WHJ. Videoteleconferencing for Administration of a Multisite Obstetrics and Gynecology Core Clerkship.  Obstet Gynecol. 2000;95:461-3.
29. Noel G. A system for evaluating and counseling marginal students during clinical clerkships. J Med Educ. 1987; 62:353-5.
30. Frellsen SL, Riddle J. ASAP: Teaching Medical Students to Solicit and Receive Effective Feedback from their Residents. Presented at Clerkship Directors in Internal Medicine National Meeting, October 2004.  Available at, www.im.org/AAIM/Meetings/PastMeetings/2004/CDIM/Wksp104-Frellsen,Sandra-Feedback.pdf. Last accessed 30 April 2005.
31. Kogan JR, Bellini LM, Shea JA. Have You Had Your Feedback Today? Acad Med. 2000;75(10):1041.
32. Salerno SM, Jackson JL, O'Malley PG. Interactive faculty development seminars improve the quality of written feedback in ambulatory teaching. J Gen Int Med. 2003;18(10):831-4.
33. Salerno SM, O'Malley PG, Pangaro LN, Wheeler GA, Moores LK, Jackson JL. Faculty development seminars based on the one-minute preceptor improve feedback in the ambulatory setting. J Gen Int Med. 2002;17(10):779-87
34. Griffith CH. Evidenced-based Educational Practic: The Case for Faculty Development in Teaching. Am J Med. 2000;109:749-752.
35. Reid A, Stritter FT, Arndt JE. Assessment of faculty development program outcomes. Fam Med. 1997;29:242-7.
36. Wilkerson L, Irby DM. Strategies for improving teaching practices: a comprehensive approach to faculty development. Acad Med. 1998;73:387-396.
37. Meurer N, Morzinski JA. Published literature on faculty development programs. Fam Med. 1997;29:248-50.
38. Lavin B, Pangaro L. Internship ratings as a validity outcome measure for an evaluation process to identify inadequate clerkship performance. Acad. Med. 1998; 73:998-1002.
39. Bligh J. Faculty Development, Editorial. Med Educ. 2005;39:120-121.
40. Bordley DR, Litzelman DK. Preparing residents to become more effective teachers: a priority for internal medicine. Am J Med. 2000;109(8):693-6.
41. Busari JO, Scherpbier AJ, van der Vleuten CP, Essed GG. The Perceptions of Attending Doctors of the Role of Residents as Teachers of Undergraduate Clinical Students. Med Educ. 2003;37(3):241-7.
42. Zabar S, Haley K, Stevens DL, Kalet A, Schwartz MD, Pearlman E, Brenner J, Kachur EK, Lipkin M. Measuring the competence of residents as teachers. J Gen Int Med. 2004;19(5 Pt 2):530-3.
43. Gerald D Denton GD, DeMott C, Pangaro LN, Hemmer PA. Narrative Review: Use of Student-Generated Logbooks in Undergraduate Medical Education.  Teach Learn Med. In Press. 
44. White CB, Thomas AM. Students Assigned to Community Practices for Their Pediatric Clerkship Perform as Well or Better on Written Examinations As Students Assigned to Academic Medical Centers. Teach Learn Med. 2004;16(3):250-4.
45. Williams M, Ambrose M, Carlin AM, Tyburski JG, Steffes CP. Evaluation of Academic and Community Surgery Clerkships at a Midwestern Medical School. J Surg Research. 2004;116:11-3.
46. Nackman GB, Sutyak J, Lowry SF, Rettie C. Predictors of Educational Outcome: Factors Impacting Performance on a Standardized Clinical Evaluation. J Surg Research. 2002;106:314-318.
47. Whalen JP, Moses VK. The Effect on Grades of the Timing and Site of Third-year Internal Medicine Clerkships. Acad Med. 1990;65:708-9.
48. Prislin MD, Fitzpatrick CF, Radeck S. A Comparison of Family Medicine Clerkship Student Performance Across Multiple Teaching Sites. Fam Med. 1998;30(4):279-82.
49. Baciewicz FA, Arent L, Weaver M, Yeastings R, Thomford NR. Influence of Clerkship Structure and Timing on Individual Student Performance. Am J Surg. 1990;159:265-8.
50. Talbert LM, Stritter FT, Riddle MH. A Clinical Clerkship in a Multicenter setting: An Appraisal. J Reprod Med. 1981;26(8):421-4.
51. Irigoyen MM, Kurth RJ, Schmidt HJ. Learning Primary Care in Medical School: Does Specialty or Geographic Location of the Teaching Site Make a Difference? Am J Med. 1999;106:561-4.
52. Puyear DA, Miller DA. The effect of clerkship site on medical student examination grades. J Med Educ. 1990;24:239-42.
53. Carney PA, Eliassen MS, Pipas CF, Genereaux SH, Neirenberg DW. Ambulatory Care Education: How Do Academic Medical Centers, Affiliated Residency Teaching Sites, and Community-Based Practices Compare? Acad Med. 2004;79:69-77.
54. Markham FW, Rattner S, Hojat M, Louis DZ, Rabinowitz C, Gonnella JS. Evaluations of medical students' clinical experiences in a family medicine clerkship: differences in patient encounters by disease severity in different clerkship sites. Fam Med. 2002;34(6):451-4.
55. Prislin MD, Fitzpatrick C, Munzing T, Oravetz P, Radecki S. Ambulatory primary care medical education in managed care and non-manged care settings. Fam Med. 1996;28(7):478-83.
56. Lawrence SL, Mark DH, Fulkerson PK, Cohen R. Students' time allocation in a required third-year ambulatory care clerkship. Acad Med. 1994;69:60-61.
57. Calhoun JG, Davis WK, Erlandson EE, Maxim BR. A multisite comparison of student activities in the surgery clerkship. Surgery. 1982;91(6):622-7.
58. Bornsztein B, Julian TM. Quantifying Clinical Activity in a Multi-Site Clerkship in Obstetrics and Gynecology. Obstet Gynecol. 1991;78(5):869-872.
59. Friedman CP, Stritter FT, Talbert LM. A Systematic Comparison of Teaching Hospital and Remote-site Clinical Education. J Med Educ. 1978;53:565-573.
60. Burke WM, Williams JA, Fenner DE, Hammound MM. The Obstetrics and Gyncecology Clerkship: Building a Better Model from Past Experience. Am J Obstet Gynecol. 2004;191(5):1771-6.
61. Mazor K, Clauser B, Cohen A, Alper E, Pugnaire M. The dependability of students' ratings of preceptors. Acad Med.1999;74(10 Suppl):S19-21.
62. Elnicki DM, Ainsworth MA, Magarian GL, Pangaro LN. Evaluation Techniques for the Internal Medicine Clerkship: A CDIM Commentary. Am J Med. 1994;97:I-vi.
63. DaRosa DA, Niehaus AH, Markwell SJ. Assessment of a surgery clerkship's performance evaluation system. Am J Surg. 2000;179(2):145-9.
64. Denton GD, Durning SJ, Wimmer AP, Pangaro LN, Hemmer PA. Is a faculty developed pretest equivalent to pre-third year GPA or USMLE step 1 as a predictor of third-year internal medicine clerkship outcomes? Teach Learn Med. 2004;16(4):329-32.
65. Ferguson KJ, Kreiter CD. Using a Longitudinal Database to Assess the Validity of Preceptors' Ratings of Clerkshp Performance. Adv Health Sci Educ. 2004;9:39-46.
66. Hemmer PA, Pangaro L. Natural history of knowledge deficiencies following clerkships. Acad Med. 2002;77: 350-353.
67. Strand DA, Vontver LA, Stenchever MA, Hadac RR, Irby DM.  Longitudinal Analysis of a Multisite Clerkship. Ob Gynec. 1985;66(5):723-6.

(return to top)

<Chapter 8: Faculty Development

<Return to Table of Contents