Home | Mission | Working Relationships | Position Papers | Publications | Clerkship Administrator Resources

Guidebook for Clerkship Directors | Annual Meetings | Member Organization Links | LCME ED-2 | Contact Us

Chapter 4 : Technology in Clerkship Education

Lead Author:
Richard P. Usatine, MD

Co-Authors:
Bryan Bognar, MD, Kevin O’Brien, MD, Patricia A. Carney, PhD, Heidi Chumley, MD,
John J. Orriola, M.A., M.Ed., A.H.I.P., Catherine Florio Pipas, MD

Download Adobe PDF version of chapter	Printed version of Guidebook is available.
Sections on this page:
Courseware and Distance Learning, Evidence for electronic learning, simulations PDA Logbooks Evaluation Reference Resources/Information Retrieval	TABLE 1: Comparison of Palm OS and Pocket PC Devices TABLE 2: Recommended PDA Programs TABLE 3: Student Reported Frequency of Handheld Computer Use in a Third-Year Primary Care Clerkship TABLE 4: Annual Cost of Paper-based and Handheld Computer-based Documentation and Feedback Systems TABLE 5: Comparison of Palm OS and Pocket PC Devices TABLE 6: Bibliographic Resources RESOURCE: Online Education Resources and Organizational Websites ADDENDUM: Examples of Web-based learning cases available on-line
<Chapter 3: Creating a Clerkship Curriculum <Return to Table of Contents	Chapter 5: Instructional Methods and Strategies >

Courseware and Distance Learning, Evidence for Electronic Learning, Simulations

Kevin O’Brien, MD and Heidi Chumley, MD

Technology Explosion

Over the past 10 years, much has changed in the area of technology in the clerkship. This field has literally exploded. The original Handbook for Clerkship Directors discussed what is a URL, how to navigate the Internet, using e-mail and working with Microsoft Office® applications. Today, technology in the clerkship means so much more than these original building blocks. Use of information technology is no longer a mandate from educators1 or professional bodies,2 but a vital part of the everyday practice of medical education. E-mail has become one of the prime communication methods for clerkship directors to contact their students and faculty, and many, if not most, clerkships have their own unique Web sites. Technology using databases and software management has revolutionized the management of the clerkship, facilitating dissemination of course schedules and lectures to students, creating course documents, receiving evaluations from faculty and students, administering surveys to faculty and learners, administering examinations, and streamlining the grading process, etc. Thankfully, the 25-pound, three-ring binder or clerkship syllabus that was typically distributed at orientation has been replaced by the CD/DVD/clerkship Web site.

Technology has also been used to enhance teaching and learning with on-line reference Web sites, PowerPoint®, and even streaming video. This has enabled learners to achieve the same didactic learning exposures even when they are located at geographically distinct sites. Because learning can occur at any time and any place, students are no longer dependent upon having the library open to access important clinical information. Ten years ago, personal digital assistants or PDAs were unknown. Today, PDAs assist students with patient care and drug prescribing, clinical decision making, evidence-based medical care, maintaining patient logbooks, and a variety of other tasks at the point-of-care and beyond. 

Simulation of patient care scenarios is also an exciting area that technology is only just beginning to address. Many schools currently run patient simulation laboratories. Highly technical human mannequins with realistic pulmonary, cardiovascular, and pharmacological models are used to simulate a variety of traumatic, cardiac, and airway emergencies. These mannequins will respond to a number of invasive procedures and more than 70 different medications. The centers are typically equipped with the latest technology in cardiovascular monitoring, airway management, and advanced cardiac life support. Most are also equipped with comprehensive multimedia systems to capture the learner’s performance or provide live interactive feeds to a variety of audiences. These simulation centers are often run by the Department of Anesthesiology, but are available to a wide variety of learners, including medical residents, medical students, physician extenders, and nursing students.  

Technology also has dramatically improved documentation of patient care through creation of electronic health records and use of electronic prescribing platforms. Many homegrown and proprietary systems have been developed and have helped to streamline patient care and improve patient care documentation.  No longer do students struggle with deciphering the penmanship of an attending or another provider, determining what medications the patient is taking, or the uncertainty of potential drug interactions and medication allergies. This information is right at their fingertips and is readily accessible and legible, permitting more efficient delivery of care and reducing medical errors and misinterpretation. While information technology in clerkship education has exploded these past 10 years, the possibilities of advancement in education in the coming 10 years seem limitless. 

Communication with Learners

Geographic dispersion of teaching sites is a common practice at many medical schools in the United States.  Clerkship directors often cannot meet directly with each student, resident, and faculty member. Instead, e-mail has become the predominant method for clerkship directors to communicate with these groups.  Orientation and regularly scheduled meetings with learners and faculty still occur; however, the day–to-day administration of the clerkship and the communication that it entails has been dramatically facilitated by the advent of e-mail. Similarly, list serves allow clerkship directors to transmit information to a large number of students, residents, and faculty with a few key strokes. 

Clerkship administrators and directors previously spent much of their time on the phone or in person answering individual student questions.  Electronic dissemination of clerkship information has led to more efficient and timely delivery of information, which has allowed both clerkship directors and their clerkship coordinators to devote more time to other administrative duties, course improvement, and patient care. Last-minute schedule changes no longer require verbal contact with all parties involved, thanks to e-mail technology.

Clerkship Web Sites

Web sites using both Intranet and Internet technology have allowed clinical departments and clerkships to place vital and frequently referenced information in one central repository.  Development of a clerkship Web site is considered one of the key elements in the 360 degree planning for a clerkship.3  Current data suggest that 29% to 72% of clinical clerkships are using clerkship Web sites4,5 and Web-based education is also being used in postgraduate training programs.6,7 Furthermore, some schools have used a centralized Web site for all of the clinical clerkships to improve dissemination of information, because many students are off campus even during their core clerkships.8  The learner no longer has to wait until orientation to learn about the fundamental features of the learning experience. Course requirements, conferences, faculty information, and clerkship policies can be viewed easily and displayed by anyone from virtually any location. Some clerkship web sites even contain a "Frequently Asked Questions" section to allow students to easily access information about the clerkship thereby minimizing calls to the coordinator's office.
Similarly, frequently asked questions are now automatically and independently answered by the learner, who simply looks up information on the course’s Web site. Issues regarding patient care assignments, grading, evaluations, and examinations can be posted in one centralized electronic location, which enables the user to access the information anytime. For confidentiality and security purposes, password protection is a built-in feature of many Web sites; however, it is not used universally.4

Course Web sites are composed of two main formats, homegrown and proprietary. The advantages of homegrown developed sites include personalized development and cost savings. The major disadvantages with homegrown sites are the information technology (IT) support required to develop and maintain the site and the lack of integration with other clerkships or sites inside and outside of the college. One interesting solution to this dilemma is for schools to use medical students as “understudies” to faculty members to develop on-line curriculum applications.9 
Advantages of proprietary sites include rapid implementation, complete integration between educators and learners, and IT support. In traditional university settings, access for hundreds of students and faculty members can be created in a matter of days to weeks. 

Using network learning environments, students and teachers can learn, connect, and share educational materials and resources from practically any location and at any time. Several institutions are using this new technology to improve collaborative efforts across many clinical departments and among medical schools,10-12 improving integration of knowledge in the clinical curriculum.  Learning can occur in both the traditional and digital settings. Learning can be Web-based or Web-enhanced.  Students can have virtual study groups or review on-line learning resources at their convenience.

Courseware

Courseware allows course directors to create course sites easily without knowing how to program for the Internet. Well-known examples include Blackboard and WebCT. The most up-to-date clinical information can be placed easily on the clerkship Web site at virtually anytime. Rather than printing out the course syllabus yearly or by each clerkship cycle, clerkship directors are now able to continually update and improve the clerkship’s courseware. For example, cutting-edge articles, updated PowerPoint® talks, and streaming videos can be quickly and efficiently uploaded onto the clerkship Web site or disseminated via e-mail to the clerkship’s constituents. Data suggest that most, if not all, clerkship Web sites are updated as frequently as every 6 months.2 Today’s groundbreaking study published in a major journal does not have to wait until tomorrow (or potentially longer) to reach the interested learner or faculty member.

Distance Learning Technologies

One school that is truly on the cutting edge of this Web-enhanced educational technology is the Michigan State University College of Human Medicine (http://www.med-training.org/im/). The college functions at six sites located across the state, including the Upper Peninsula.  Its solution to this geographic challenge is quite impressive.  The best faculty have been filmed giving lectures to residents and students. The lectures are available in both audio and video format, depending on the speed of the user’s Web browser.  Learners can listen to streaming audio while simultaneously viewing the PowerPoint® slides.  Using DVD technology, these lectures can be viewed via streaming video over the Internet.  Moreover, DVD technology allows the lectures to be broken down into their individual components, which enables learners to review key concepts or difficult-to-grasp topics quickly and efficiently without having to view the entire lecture.
(return to top)

Web-based and Computer-Assisted Learning

Reports of Web-based learning began (WBL) appearing in the medical education literature in 1992.21  Building on the themes developed in computer-assisted learning, proponents have suggested that digital learning is superior to text-based, lecture, and traditional educational methods. Reasons cited include learner control of the content, timing, and place of learning;22 enhancing learning efficiency and cognitive reasoning,23,24 and cost savings.25 Technical aspects of Web-based learning provide several educational advantages, including universal accessibility, ease in updating content, and the capability of hyperlink functions that permit rapid cross-referencing with other learning resources.26 These advances in WBL fit within the paradigm of constructivist learning theory, allowing learners to search out and create their own knowledge bases. 

Evidence about Web-based Teaching and Student Learning

Few studies address whether WBL can enhance or stimulate learning. Currently, results are mixed, but higher level students may be more likely to search for answers instead of asking an instructor in a WBL course.27  One well-designed randomized controlled trial comparing WBL with text-based learning reported that learners achieved equivalent test scores with shorter study times using WBL materials (27 minutes versus 38.5 minutes). The authors calculated efficiency scores (median improvement in score per hour with 95% confidence interval) of 8.6 (7.1–11.7) for WBL and 6.7 (5.9–8.1) for text-based learning (p = .04).28

Learner Reaction and Recent Trends

Learners enjoy WBL, plan to use it again,14,29,30 and prefer it to continuing medical education conferences, lectures, video, audiotapes, journals, or text-books.28,31-33  There are positive and negative reports of student satisfaction with the asynchronous interactions with instructors and peers typical of WBL;27,30,34,35 however, it remains unclear if asynchronous communication benefits or adversely affects learning. While accessibility, ease of navigation, and Web design are important features to learners, the main predictor of satisfaction with WBL is download speed, and students report that this is as important as content.36 When download speed is inadequate, it is often reported to be the major disadvantage and most significant barrier to learning.31,37-39

Download time has led several institutions to use CD-ROM technology in an attempt to improve student satisfaction. Since the late 1990s, medical students at the University of South Florida on the internal medicine clerkship receive a CD-ROM during orientation that is a mirror image of the Web site posted on the university’s secure educational server. While the CD contains much information on the basic mechanics of the course, it also contains many teaching files and PowerPoint® talks, some as large as 25 MB. Students can quickly and easily obtain important information without having to depend on the availability of high-speed Internet technology.

WBL appears to be an important adjunct to traditional learning formats. Studies have shown that 70% of WBL students also choose to attend lectures.40  WBL students also feel that it allows them the ability to independently research questions that they might have traditionally asked during lecture,27 thus promoting the independent and reflective learning that is the ultimate goal of adult learning theory. Furthermore, WBL students  also value features inherent to the traditional lecture format. Sound educational format and design appear to matter most to students, rather than the technology, communication, or course management that WBL provides.41

A systematic review of WBL in medical, dental, and nursing education, conducted by Chumley-Jones et al.,42 has helped clarify the uses of WBL as described above, but also identifies several areas that need to be assessed in a systematic fashion prior to widespread and universal implementation. In well-designed research studies, Web-based learning appears compatible with, but not superior to, other educational methodologies in the area of assessment of knowledge gains.28,31,32,40,42-44  It remains unclear whether WBL improves learning efficiency or reduces the costs associated with medical education. Further study is needed to determine if lower distribution costs offset the technical support costs or lessen costs associated with faculty time.42,45,46 Most learners welcome WBL, yet more research is needed to determine its advantages and disadvantages as compared to more traditional methodologies. In the meantime, WBL will continue to serve as a valuable addition to the educational armamentarium at most, if not all, medical schools.

Standards and Quality

As WBL continues to expand into clerkship education, two general principles need to be kept in mind—standardization and quality. Standardized Web-design and curriculum delivery are key elements not only for student learning, but also for assessment purposes regarding quality and utility. A minimum set of core essentials should be addressed, such as instructor information, schedules, course requirements and policies, suggested learning strategies, learning resources, learning goals and objectives, and syllabi.4 Pilot studies and evaluation programs have been designed to review these online medical sites and could serve as valuable resources as institutions implement more WBL curricula.4,47,48  Furthermore, curricula need to be developed so that we can teach our students how to properly use the Internet for learning opportunities and improving patient care.49  Students and educators need to be able to assess quality of the information obtained, search and filter through vast amounts of information efficiently and rapidly, and even counsel their patients on appropriate use of World Wide Web health information resources.49  Lastly, dissemination of information to learners through well-designed Web sites, CD/DVD, and other computer-based technology can serve as an important stepping stone for clinician educators in their promotion and career advancement.50

An exciting new technology in clerkship education is helping to reinvigorate traditional teaching methods. The use of interactive, anonymous voting systems with question-and-answer sessions during lectures is shifting learning from passive to active and replacing the old metaphor of “transmission of knowledge” with the new metaphor of “dialogue.” This format promotes cognitive and critical reasoning skills, analogous to problem-based learning theory.51 To engage active participation during the lecture, the audience responds anonymously, using wireless key pads.  A computer-assisted program displays the distribution of answers.52  This interactive teaching format promotes group learning and provides instantaneous feedback to the teacher and the audience.52  The anonymous response system translates learning into a non-threatening mode.53  Medical educators have found that this teaching modality facilitates probabilistic thinking, improved consideration of alternative options, improved recognition of misconceptions, and more honest communication by respondents.53  It appears that this new technology can enhance the Socratic method of teaching, which practices interactive learning. (return to top)

Simulation-based Medical Education

Low-technology simulators and standardized patients have been used for quite some time in undergraduate medical education.54 On the other hand, high-fidelity patient simulators have been used predominantly in high-risk training in anesthesia,55,56 but are now gaining more widespread acceptance in general medical training,57-61 at both the undergraduate62 and graduate levels.63,64  Recent work has demonstrated that simulation-based medical education  improves outcomes for real internal medicine patients.63

Previously, these simulation models have been used as methods to promote safety and reduce medical error in high-risk, clinically uncommon scenarios;54,56,63 however, more recent data suggest62 that this teaching modality can be used to promote reflective and comparative analysis among core internal medicine concepts. The basic tenets of adult learning theory suggest that learning occurs through a series of sequential steps of cognitive analysis.65,66  Initial experiences are retained as a memory and incorporated with existing knowledge to provide general meaning or understanding of the event.  Repetition of meaningful experiences and preparation prior to a planned learning endeavor appear to improve learning; however, the highest form of learning may arguably take place when it occurs with reflection and cognitive analysis.62

McMahon et al.62 studied students on the internal medicine clerkship who participated in a simulated patient learning environment dealing with the pathophysiologic concepts of coronary ischemia and hypoxemia. While students in each scenario were able to accurately obtain a history and perform an appropriate physical examination, they consistently demonstrated errors in application of knowledge within the clinical circumstance. Three related clinical scenarios in each concept allowed students to compare and contrast similarities in presentation and pathophysiology, but more importantly, they learned how this translates into nuances in the clinical management of each disorder. Presenting these cases within a short time span optimized the comparative analysis and avoided the temporal disconnect between similar cases that often occurs in real-life clinical education.62  In conclusion, we are only just beginning to realize the many benefits that simulation-based medical education provides. It will most likely continue to grow as an invaluable tool in the assessment of clinical skills, competency, and patient safety in the undergraduate, graduate, and continuing education/recertification settings. (return to top)

Incorporating PDAs into your Clerkship

Richard P. Usatine, MD

Introduction

With the explosion of information in medicine, it is not possible for the unaided brain to keep up with all the new drugs and studies that are released yearly. PDAs (personal digital assistants) or hand-held computers can be the aid that we need to manage the explosion of information. 

Why Recommend/require PDAs in Your Clerkship?

1. PDAs can enhance clinical learning by providing medical guidelines, textbooks, and databases at the point of care.  Medical students often carry many small books in their coat pockets and large textbooks in their backpacks, but these may not be readily available at the point of care. The printed textbooks go out of date quickly, and searching these textbooks is cumbersome. The medical information contained in PDAs is easy to update and search. The PDA may hold enormous amounts of data and still fit in the pocket, ready for use at the point of care.

2. PDAs can enhance the practice of medicine and decrease the risk of errors.67 PDA software allows students to:

• Look up accurate drug doses
• Check for drug interactions
• Find toxicology information
• Access multiple drug databases
• Calculate pediatric drug doses
• Check immunization schedules
• Calculate obstetrical information, including Bishop Scores and estimated due dates
• Access the ACLS guidelines

3. PDAs can help students use clinical decision-making tools and formulas such as:

• Preoperative clearance risk calculations using the Goldman criteria
• Deciding need for hospitalization in community-acquired pneumonia
• Determining Aa gradients and BMI

4. Clerkship directors can give students clerkship schedules, assignments, and syllabi as files they can use on their PDAs.

Should the PDA be Required in Your School/Clerkship?

The advantage of requiring PDAs is that students are then eligible for financial aid to purchase them. Bulk-purchasing prices for students and faculty can sometimes be negotiated. PDAs can then be used in the curriculum and their use can be tested fairly. Some students will complain about a requirement that costs money, and some students don’t like computers. It is easier to require the PDA in your clerkship if it is a school-wide requirement as well. Lending PDAs to students is one way that allows all students to participate in PDA learning when a school requirement is not in place.

When to Use the PDA for Students

The use of the PDA is may be emphasized throughout the clerkship in clinical settings, during didactic sessions, in the testing process, and for clinical logs.

• Clinical setting – inpatient and outpatient
• Didactic sessions – problem solving
• Testing
• Log clinical experiences

Clinical setting

Whether the student is in the inpatient or outpatient setting, PDA programs can help them answer their clinical questions and practice better medicine. Some clerkship specialties have non-commercial PDA reference sites; a few examples are shown below:

Many of the commercial sites can be searched based on specialty to find programs that will help students on those clerkships. See Table 2 for ideas about free software for your clerkships.

Didactic sessions

Students can use their PDAs to find answers to many clinical questions. For example, they can diagnose sore throats, using clinical prediction rules and pattern recognition. Students in our clerkship didactic session on pharyngitis are prompted to use their PDAs to calculate pretest probabilities of strep pharyngitis, using the McIsaac modification of the Centor criteria. In a polypharmacy portion, students review case studies analyzing potential drug interactions using a PDA-based drug database, such as ePocrates.

Testing

Students are more likely to work at learning to use specific programs on their PDAs if they know that they will be tested on it. Dr. Tom Agresta at the University of Connecticut School of Medicine tests his family medicine clerkship students on the use of InfoRetriever, Drug database, Calculator, and Pediatric Oral Health Resources. These questions are part of an open Internet end-of-clerkship exam. At UTHSCSA, we have 12 questions on the final family medicine clerkship exam that are open PDA questions. After those questions are completed, the students must put their PDAs away. Students need to load on to their PDA and be familiar with Shots 2005, ePocrates or a comparable drug database, and Archimedes or a comparable medical calculator.

Examples of OSCE stations that test students’ skills with the PDA:

An elderly woman with HTN, diabetes, hyperlipidemia, insomnia, and depression complains of increasing fatigue and is worried that it is from her medications. The patient asks if the medications may be causing problems with each other. The student needs to check for drug interactions and adverse effects on the PDA. The student needs to apply a critical analysis of polypharmacy to this case.
Problem solving with the PDA can be tested on the written final exam. The station evaluates the student’s ability to use the PDA in a patient encounter to determine whether he or she can maintain excellent physician-patient rapport while using information technology tools.

Sore throat/ clinical decision making with the PDA:

A patient comes in with signs and symptoms of a sore throat that might be viral or bacterial. She believes she has a strep throat and wants antibiotics. The student needs to take a focused history and perform a focused PE. The student is prompted to collect these data and use a clinical-decision-making tool (MedRules or InfoRetriever) on the PDA and to talk to the patient about the likelihood of a strep throat.

Clinical Logs

PDAs can be useful for collecting log data.69-71 If the software is well-written and the ability to download the information to the central database is smooth, this can be a very effective way to collect information on your students’ clinical experiences. Whether a log is paper-based, Web-based, or PDA-based, it will be extra work for the students. Giving students meaningful feedback on their log data will help make this work more acceptable to them. If the data can be used to enhance their future clinical experiences by making adjustments in what they see and do, that is even better. See section on Logbooks. (return to top)

Faculty Development to Facilitate PDA Use by Students

You can provide these resources for your faculty:

• Buy PDAs for paid and volunteer faculty
• Provide sessions on how best to use the PDA
• Develop Web sites as references for the PDA

Faculty may then choose to:

• learn to use a PDA along with the students and share free software by sending new information to each other
• lead their students in PDA use and show them that it is never too late to learn new ways to practice medicine
• avoid PDAs, but be thankful when a smart student has a vast database of information at his or her fingertips that improves the quality of patient care and decreases common drug prescribing errors.

Here are some examples of using the hand-held computer to teach in the clinical setting:

• You've come to the end of a patient encounter and you are starting to write a prescription. You're not sure about the dose of the medicine, so you look it up in your hand-held computer. However, before you get your hand-held computer out of your pocket and turned on, the student has already found the correct dosage in his/her hand-held computer. It is okay to lose the hand-held computer duel because it helps the student feel useful and you get the correct answer very rapidly.
• It is time to write the antibiotic prescription for a child and you forget the formula to calculate the dose by weight. This time you ask the student to look up the dosing formula and to calculate the dose based on the child’s weight. The student gets practice with calculating doses and you may then provide some practical pointers on how to write the prescription.
• The student is seeing a child that may need immunizations. There is no immunization chart in the room but you have the newest version of “Shots” in your hand-held computer. You either show the student this on your hand-held computer or ask the student to look it up on his or her own hand-held computer.  If the student has a hand-held computer, but not the software, you can send him or her this free software or show where the software can be downloaded from the Internet. In the spirit of letting the student solve the problem, you ask him or her to analyze the immunization history and chart and then make a recommendation.
• A student comes out of the exam room after seeing an elderly woman whose children are worried that she may have Alzheimer's disease. You ask him if he has done the mini-mental status exam and the student replies that he could not remember the whole exam. You take out your hand-held computer and tap on the icon to reveal the mini-mental status exam in full. Not only is the exam there on your screen, it also scores automatically every time you tap a box. There are line-by-line instructions on how to give the exam if the student has forgotten how. 72

How to Help Students and Faculty Choose Their PDA

Which operating system is best for your students and faculty?

While there are other operating systems for handheld devices, Palm OS and Pocket PC account for the vast majority of the PDA market. Devices using the Palm operating system (OS) are the most popular in the medical field. As the Pocket PC devices have become more affordable, their use has risen. However, some free medical software is written for the Palm OS, but no Pocket PC versions are available. A number of physicians have produced their own applications and offer them free to others to be downloaded over the Internet. One explanation for this is that it is easier to write programs for Palm OS than for Pocket PC. Most of the large, commercially available medical programs are now available in both formats.

New software is now available to run Palm OS programs on the Pocket PC. This makes the Pocket PC more appealing as a medical assistant because the Pocket PC user can now take advantage of free Palm OS programs such as Eponyms, MedRules and Menstat.

StyleTap™ Platform runs most applications for the Palm OS® platform on Microsoft Windows Mobile™ Pocket PC handhelds. It supports most of the thousands of application programs written for the Palm OS® 5.2 platform and earlier versions. Applications written for Palm OS® platform show up as native Windows Mobile™-based Pocket PC programs and operate in the same way. On the latest Windows Mobile™-based Pocket PC devices, StyleTap™ Platform takes full advantage of high density VGA screens for maximum readability.
Price:   $29.95 (US)
Size:  1.3 MB
OS:  Windows Mobile™ 5 for Pocket PC, 2003 (including Second Edition and full VGA support), Pocket PC 2002 or Pocket PC 2000

Palm OS devices had the advantage of having a longer battery life, but this is changing. One has to regularly charge both types of devices, because it is possible to lose all your files and information if the battery runs out completely. Some PDAs have backup systems or batteries that prevent this from happening.

For a long time, the Pocket PC devices were much more powerful than the Palm OS devices in processor speed and internal memory capability. This is no longer true. Both types of devices come with slots to add memory. Memory cards come in a number of forms, but they all work in a similar fashion to allow more data and program storage on your PDA. The cards are needed to store and run some of the best programs. See Table 1 for a short comparison.

The built-in digital cameras are a fun addition to some of the new PDAs. You can use these cameras to document medical findings in your EMR or print the photo for your paper charts. If you want to combine your PDA with a phone, consider the Treo. Get a PDA with a color backlit screen regardless of the operating system.

Web sites can help you review the prices and features when you are choosing the exact product to buy. We suggest looking at the Palm and Sony sites for Palm OS devices. Hewlett-Packard, Toshiba, and Dell make excellent Pocket PC devices. Dell currently has the best prices for the Pocket PC. You can use Froogle to compare prices. (return to top)

Software

Instead of investing a lot of money in commercial software, students can start with inexpensive or free programs that are useful and authoritative.73

Software that Comes with a PDA

Any new PDA will come with all the software you need to keep track of your schedule and contacts, but not the software you need to practice and teach better medicine. The program ePocrates has driven the demand for the PDA most among physicians and medical students. ePocrates is a very powerful drug database that originally was free and now costs $59 per year for the full database, including tables, drug choices for infectious disease, and alternative medicines. A stripped-down free version still exists for the Palm OS and contains valuable information, including medication dosing, indications, and costs. If your hospital uses Micromedex, a free PDA version for Palm OS and Pocket PC is available. Other drug databases are available for purchase.

Data Driven Programs

You will want more than a good drug database to be a better doctor and teacher. The unaided mind can no longer handle the vast amount of medical data. Good, free, data-driven programs that you can download from the Internet include the most recent CDC guidelines for STDs, Shots 2004, and MedRules. Shots 2004 includes the 2004 childhood immunization schedule, 2004 catch-up schedule, and the 2004 adult immunization schedule with medical conditions. MedRules is an application containing useful clinical prediction rules based on evidence from the medical literature.

The Seventh Report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure (JNC 7) is available in multiple free PDA versions. Also, the National Guidelines Clearinghouse now has a feature that allows you to download the summary from most guidelines to your computer, then install it on your PDA. Palm reader is helpful for reading the free documents from the National Guideline Clearinghouse: http://www.palmdigitalmedia.com/products/palmreader/free .

The Antibiotic Guide, by Johns Hopkins University, is quite similar to ePocrates ID. Perhaps the most important difference is that the Antibiotic Guide is available free of charge, while ePocrates ID comes only with the versions of ePocrates that cost money. The Antibiotic Guide is updated continuously and allows the user to access antimicrobial treatment guidelines indexed by bug, drug or organ system. Typical entrees include information on drug class, indication, and spectrum of activity, as well as dosing, price, pregnancy risk, drug interactions, and adverse reactions. In conjunction with the free version of ePocrates, students are well-prepared to meet the prescription challenges presented by typical patients.

Another free program is Merck Medicus, powered by Unbound Medicine, available at http://www.unboundmedicine.com/mmedicus.This is a suite of programs. The first is the Merck Manual with its concise synopsis of etiology, pathology, diagnosis, laboratory findings, and treatment. It also comes with a handbook of diagnostic tests. The third feature is an asynchronous information portal. A check box labeled “more info” follows many entries in the Merck Manual and in the handbook of diagnostic tests. Click the box, and the next time you perform a hot sync, related articles, pictures, and graphics are retrieved from the Internet for viewing on your PC. Another feature of the information portal allows you to subscribe to Reuters medical news service, with refreshed content every time you hot sync. Yet another service offers daily “mini-abstracts” from hundreds of available Medline journals. After you read the mini–abstract, you can click “more info” and the next time you hot sync you will be taken to your Personal Library page at Merck Medicus, where you can download and even store articles and information that you have selected.

Diagnosaurus, another free offering from Unbound Medicine, has quickly become a favorite of many students. This little gem indexes over 1000 differential diagnoses, and with its help students not only learn to look good on rounds, they also learn to make important connections in the clinical setting. One of the stumbling blocks for medical students in the early clinical years is premature closure. Excited by making a connection to information learned in the preclinical years, students often misapprehend a clinical situation when they recognize just one or two key elements. Hearing hoof beats, they immediately recognize a zebra. Precisely because their clinical experience is so limited, junior medical students can recognize only a small number of clinical patterns. This handy index, cross-referenced by organ system, symptom, and disease, can help students defer premature closure by suggesting likely alternative diagnoses.

MentSTAT is a real mind saver. After a good night’s sleep, most students and clinicians can pass the Folstein MMSE, but few can administer it to patients correctly from memory. This one-screen program can guide even the post-call and bleary-eyed through flawless administration of the exam. For pediatric drug calculations, which are a common source of medical errors, Peds Omnibus will lend a hand. This free utility also calculates formula/nutrition requirements and IV fluids.

FileZ is a free program that no palm user should be without. This file manager allows you to display and transfer files on your Palm OS device. You can edit file attributes like “back-up” or “hidden,” move files between RAM and memory cards, and beam files to another user via the built-in infra red data port.

iSilo is a document reader that supports hypertext and embedded graphics. It can be purchased for $20 and comes with a free companion program for document conversion called iSilo-X. In a single step, any HTML document can be converted and displayed on iSilo and the hot links will all remain functional. By simply “clicking” on the table of contents, you can jump to the corresponding section of the text, allowing easy navigation within even the largest documents.

Literally thousands of iSilo medical documents are available on the Web. Most of them are free. Many have embedded full-color graphics and photographs – spanning topics from rash diagnosis to congenital heart disease. A quick Google search will turn up many sites, but one of the best collections can be found at http://www.meistermed.com/

At meistermed.com, Dr. Andrew Schechtman makes great use of hypertext links in iSilo with free programs like LyteMeister and AsthmaMeister. LyteMeister provides a guided algorithm that helps students learn to diagnose and manage common electrolyte disturbances. AsthmaMeister provides a guided decision tree to help manage acute and chronic asthma. The newest is called DermMeister, a dermatology photo atlas featuring more than 500 high-resolution photos of 66 common skin disorders. Other “meisters” include LipidMeister and PapMeister and a meister for endocarditis prophylaxis.

After free registration at Apprisor http://www.apprisor.com/, you can download a wide variety of abstracts and documents, including prevention guidelines from the American Heart Association, the American College of Physicians, Colorado Clinical Guidelines Collaborative, JNC-7, NCEP ATP III guidelines, and many more. Apprisor is a free document viewer created from an earlier version of iSilo. Therefore, all of these documents are also viewable on iSilo.

Using iSilo-X, you can easily make your own iSilo documents. Use Microsoft Word to save your document in HTML format, then drag-and-drop the file onto iSilo-X. With one click of the mouse, your document is instantly converted to iSilo format and placed in your palm install directory. Perform a hot sync and you’re ready to go. Meistermed.com has a brief tutorial for inserting hot links into your own documents with professional results. For example, we converted our department’s family medicine third-year clerkship manual into an iSilo document, complete with hot links. Now students can view the manual on their palm pilots.

Web-clipper places a clickable button on your Internet Explorer toolbar that converts any Web site into an iSilo document with graphics and hot links preserved, if you choose to have them,.You can specify the ‘depth’ of conversion and iSilo-X will follow all of the hot links on that Web page and convert every linked Web page as well. Set as many layers of depth as you like and iSilo-X will trace each page, and the pages linked to that page, and so on, and return a single document for your palm pilot with its entire hot link structure intact.

PDAs play many important roles in medicine today. In the clinic and on the wards they provide instant access to the most current and sophisticated medical references. PDAs foster enthusiasm for learning and create a wealth of teachable moments. Whether you are a seasoned professional or just beginning to use PDAs in medicine, a few well-written, inexpensive programs can help you teach students and residents how to practice smarter medicine, and how to deliver a higher quality of patient care. (return to top)

Wireless issues

We do not need wireless networks in the medical school/hospitals/clinics to make PDAs valuable tools. Most software is installed by downloading it from the Internet and using a synchronization cradle that is plugged into the USB port of the computer. This software does not depend upon wireless access to run.

Wireless is the relay of data via radio waves rather than copper wire. The following section is a short primer for deciphering the wireless alphabet soup.74

PAN = personal area network that is short range (a matter of feet). Examples include infrared communications between two devices and Bluetooth technology.
LAN = local area network. Usually consists of a set of computers connected to a central server. An example would be a physician’s office with PCs hard wired together.
WLAN = wireless local area network. Essentially the same as LAN except information is transmitted between networked devices via radio signals rather than hardwire.
WAN = wide area network. The WAN is often the DSL, cable modem, or cell phone network.
Wi-Fi =wireless using the standard 802.11 technology that allows your laptop or PDA to connect to the Internet. PDAs work with each of these types of networks.

Some new Palm OS devices contain an integrated 802.11b wireless card for Wi-Fi, like the Palm Tungsten C. Other PDAs have expansion slots that can accommodate an add-on wireless card. More PDAs will have integrated Wi-Fi and Bluetooth technology over time. Wi-Fi devices must be within range of a wired access point for Internet connectivity. These PDAs can be prompted to search for a wireless network and connect to it automatically. This could be a wireless network in your hospital, office, home, or “public” hotspot (public wireless network access point). Once you are connected, you can surf the Internet. Special PDA-friendly Internet sites send the PDA smaller, more readable pages. Some Web sites have special pages that detect when you are connecting with a PDA and will transmit news, weather, and entertainment information in a readable format for the small screen.

If you want to set up a wireless network at home or in your office for your PDA and laptop, the following site is useful: http://palmtops.about.com/library/weekly/aa033103a1.htm

One company, Patient Keeper, uses a product that allows one to wirelessly “sync” to a server to download a complete list of inpatients. This list can include recent labs, consultation reports, radiology reports, and medication allergies. This technology can contribute to point-of-care medicine and remain HIPAA compliant. This system is also available in the Pocket PC environment.  

Bluetooth technology is a second wireless option. Bluetooth is used to transmit signals from one electronic device to another within the same room. A number of Palm OS and Pocket PC models contain Bluetooth technology. These models can communicate with Bluetooth-enabled cellular telephones. This allows the user to access the  Internet from any location that has adequate appropriate cell phone service. Unlike WiFi, Bluetooth does not need a “wired” access point for one enabled device to communicate with another. Bluetooth has been used to send faculty didactic lecture notes from a single handheld device to an entire intern class with just a few key strokes and in a couple of seconds. Bluetooth also can be used to transmit a print order to a Bluetooth-enabled printer.

Conclusion

The PDA is becoming as important as the stethoscope in the practice of medicine. Medical schools should not leave the learning process of using the PDA entirely to student interest and chance. Clerkship directors can help students take advantage of the vast resources available by including PDA programs in the didactics and encouraging students and faculty to learn from each other. (return to top)

Patient Logbooks and Clinical Documentation Systems

Catherine Florio Pipas, MD, Patricia A. Carney, PhD

A comprehensive documentation and feedback system can provide an essential infrastructure for evaluating the educational process and enhancing teaching and learning in core clerkships. The Liaison Committee on Medical Education (LCME), the accrediting body for US medical schools, now requires the following information from medical schools. A clinical documentation system will help achieve this:75

1) Curriculum must clearly list core competencies and how they are evaluated (ED-1A).
2) The objectives for clinical education must include quantified criteria for the types of patients (real or simulated), the level of student responsibility, and the appropriate clinical settings needed for the objectives to be met (ED-2).
3) There must be comparable educational experiences and equivalent methods of evaluation across all alternative instructional sites, within a given discipline (ED-8).
4) Medical school faculty must establish a system for the evaluation of student achievement throughout medical school that employs a variety of measures of knowledge, skills, behaviors, and attitudes (ED-26).
5) Directors of all courses and clerkships must design and implement a system of formative and summative evaluation of student achievement in each course and clerkship (ED-30).

Medical schools and residencies nationally are increasingly using different types of documentation systems to track learners’ clinical experiences.76-78 At Dartmouth Medical School, we initially developed a system for our multidisciplinary primary care clerkship (family medicine, ambulatory internal medicine, ambulatory pediatrics) that documents students' clinical and educational experiences and provides feedback designed to enhance clinical training, using a timely data reporting system. The five critical components of the system are as follows:

• a valid, reliable, and feasible data collection instrument
• orienting and providing ongoing support for student and faculty users
• generating and distributing timely feedback reports to students, preceptors and clerkship directors
• adequate financial and technical support
• creating a database design that allows for overall evaluation of educational outcomes.79

We have generated and distributed approximately 150 peer comparison reports of clinical teaching experiences to students, preceptors and course directors over the past year in formats that are easy to interpret and use to individualize learning. In a recent publication,79 we present cost estimate comparisons of paper-based and PDA-based system development and maintenance, which range from $36,987 to $53,780 for development and maintenance of the paper-based system and $50,337 to $109,308 for the PDA-based system.79.The following pages represent our experience with both paper and computerized documentation systems, from development to ongoing maintenance, with a particular focus on data uses. (return to top)

Getting Started

Gaining Support

Departmental, multi-departmental, or institutional funding and technical support is a challenge for both start-up and maintenance phases of system development.  We depended entirely on grant support to initially develop and expand our system until it became institutionalized. We began with a paper-based system in 1997, which was used in a single clerkship. It was completely supported with grant funding, costing less than $50,000, though it expanded in 1998-1999 into all primary care ambulatory clerkships. At this time, it was still grant funded (with costs closer to $100,000 since becoming a PDA–based system). In 2003, further enhancements were made to expand the system to inpatient clerkships and make it Web-based (device neutral). At this point, the institution funded the system costs. Without clearly demonstrated feasibility and usefulness, the institution would not have undertaken this endeavor. It is similar to a Phase I clinical trial where proof of principal is tested. The initial costs for expansion to all clerkships and to the Web were $150,000.

Obtaining Consensus on Content

Although the challenge of consensus grew proportionately with the number of courses and faculty using the system, the benefits included:

• having similar information and comparable databases on all clerkships
• having a comprehensive look at all of our precepting sites
• having an institutional view of the educational experience rather than a single clerkship view
• students mastering the system would have only one system to learn rather than a separate system to learn for each clerkship.

We were fortunate to be able to build on the format from the initial Family Medicine Clerkship (which allowed that group to continue assessing their data).  The two primary sections of the system were maintained. The Common Problem Tally Section documents common primary care problems, symptoms, and diagnoses.80 The Teaching and Learning Section obtains information on the educational process, including demographic information on patients, students, and preceptors (age, gender); type of health care visit (acute, chronic, acute exacerbation of chronic condition, and health maintenance);

• physical exam elements performed
• type of learning in history taking and physical exam (student observed preceptor, preceptor observed student, student performed unobserved, or both participated)
• amount of feedback received from preceptor on clinical skills
• preceptor teaching content (disease, diagnosis, and management). 

A comprehensive list was developed of organ-based topics that covered all clerkship learning objectives, the six core competencies (ACGME), and educational processes generic to all clerkships: Motivating Users.

Obtaining “buy in” from all parties involved is crucial to systems development.  This is an ongoing challenge that requires time and continuous communication about the identifiable benefits to all user groups. Many benefits can only be realized after the system has been established and data can be displayed and shared. This provides an indication of the usability of information collected for enhancement of the educational experience. An active and responsive feedback loop for our students, their preceptors, and clerkship directors is a critical feature of the system. In some cases, the clinical applications on the PDAs (e.g., drug formularies and EBM tools) have provided the incentives needed for students to comply with data collection. (return to top)

System Development and Implementation: Five Critical Components

Determining Validity & Reliability

Any instrument developed for evaluation or research purposes should undergo testing prior to its use. This is critical for students and faculty to trust the data they are collecting and to ensure that the results generated are accurate, reliable, and meaningful. Validity can be addressed using several approaches, but at a minimum content and construct validity must be addressed. Content validity refers to the congruency between the item being collected and the objective of the evaluation. In an educational setting, this would be the linkage between the curriculum being taught or materials assigned as self-study and the educational experiences aligned with those educational activities. If an item is not taught either in the curriculum or through self-study, it should not appear on the evaluation instrument. Construct validity is often addressed by having a panel of experts review the literature and/or curricular content and come to consensus on the linkage between educational objectives and items to be measured. 

Construct validity refers to the ability of the evaluation instrument itself or the setting in which it is used to capture those items deemed appropriate after content validity has been addressed. For example, if part of the curriculum includes assessing a learner’s ability to do spinal taps, but the setting does not provide the opportunity to see or do spinal taps, it is a threat to construct validity of the instrument to include spinal taps on it. Pilot testing and subsequent refinement of an instrument can address construct validity.

Reliability refers to the reproducibility of an instrument or its ability to capture information similarly across several students or settings. Establishing reliability can be accomplished several ways. Test-Retest methods may work in some settings, where you determine if the responder answers the same way at two different times. Alpha coefficients can also be calculated to determine the strength of one or more variables in reflecting a specific domain, such as the several questions that can be asked to take an accurate history. Another approach includes undertaking an inter-rater reliability assessment where two different raters observing the same event score what happens and kappa coefficients are calculated to determine the likelihood of similar scoring when the analysis accounts for correct scores that could have happened by chance. 

We used a panel of educational experts and 1 year of pilot testing to address construct and content validity; checks were conducted prior to assessing reliability. We used inter-rater techniques to determine reliability where we had faculty observers watch and score several different student-patient-preceptor interactions. We obtained an overall kappa of 0.68, indicating good agreement or reproducibility between the independent observers scoring the same encounter.81

Orientation and Ongoing Support for Users
  
A second key component of our system involves ensuring that learners and their preceptors are oriented to and understand how to use the system. To orient students, a 90-minute overview of the hardware and documentation system is provided during clerkship orientation. At this session, students learn how to enter encounter data and then practice this skill using a sample clinical case, enabling faculty to assess student comprehension and skill with using both the handheld computer and ClinEdDoc.™. The ClinEdDoc™ codebook, which is included in each student’s clerkship manual, is carefully reviewed.  Sample summary data reports are shared with students to demonstrate the importance of accuracy and encounter-recording compliance. Students also are introduced to other clinical applications (licensed, shareware, and freeware) installed on the handheld computers, including evidence-based medicine tools, prevention guidelines, clinical texts and a pharmacological database..

Preceptors are oriented to the documentation system by first reviewing peer comparison reports (reports that illustrate how individual preceptors work with their students compared to all students working with preceptors in the database).  This primarily occurs during site visits made by course directors. Although they do not receive formal orientation to ClinEdDoc™ software, preceptors are coached on how to use the reports to adjust learning experiences that will meet both student and clerkship goals.

Data review sessions provide ongoing support for the system. They occur during weekly clerkship didactic sessions, which are attended by all students whose clinical placements are within driving distance of our school. During the weekly sessions, students download data from their handheld computers onto a designated desktop computer. The data are then cleaned and stored in relational databases for report production and for subsequent analytic purposes. ClinEdDoc™ includes a summary function that displays, on the handheld computer’s screen, a total for each item in the Common Problem Tally Section based on encounters recorded to date. Because we can quickly view an individual student's data using this function, it aids in the identification and correction of coding problems. The summary function does not display Teaching & Learning Section data, but it allows clerkship directors and students to view a scaled down version of the Common Problem Tally Section report at anytime.  These 15- to 30-minute troubleshooting sessions enhance data quality by allowing clerkship directors and trained faculty and staff to answer questions about the documentation system or the handheld computer, and to identify data entry or coding errors that can then be corrected immediately.

Report Review Session

We have found that distributing the reports alone is not enough to stimulate an adequate discussion between the clerkship director and student and between the student and his or her preceptor. Coaching in a report review session is needed to ensure that students and preceptors use the reports to revise their educational plans. The clerkship director and project director spend 30 minutes reviewing these monthly reports with the students and assisting each student in setting goals for the current or subsequent clerkship. We have also noted improved encounter-recording compliance and accuracy with the initial thorough orientation and ongoing weekly support.

Timely Report Generation and Distribution

The system uses the information collected to provide summary data during the clerkship to students, preceptors, and clerkships directors. Timely reports are necessary to activate changes in students' clinical experiences while they are still participating in the clerkships. To accomplish this, reports are generated monthly for students, preceptors, and course directors. Considerable effort was spent designing a report format that is readable, comprehensive, and provides value for students, preceptors, and course directors.

Students receive weekly formal paper-based reports and an overall summary of the block during the last week of the clerkship. The clerkship director uses these reports to help students set individual educational goals and identify specific areas of focus, as mentioned above. Preceptors receive ongoing support in the use of reports generated through the documentation system during site visits and faculty development workshops. Annual site reports are generated and dispersed to preceptors and clerkship directors.

Adequate Financial and Technical Support

Financial and technical support is critical in the development of both the paper-based and the computer-based systems. Initially, this came from educational grant support. Currently, our institution supports a portion of the maintenance activities required, with the remainder being supported by educational grants.  Using data collected for scholarly activities (e.g., national presentations, publications, and educational grant submissions) assisted in obtaining institutional support. Day-to-day operations of the system require a master's level part-time faculty person to purchase, distribute and maintain handheld computers, load software, troubleshoot, assist the clerkship director in conducting student orientation and ongoing coaching, and collect and manage data, including report generation.
 
Evaluation and Educational Outcomes

The system has been used to generate data for scholarly evaluative purposes.  Three recent peer-reviewed papers describe analyses conducted with data from this system, which have allowed for a better understanding of this complex learning environment.80,82,83 To use these data for analytic purposes requires deliberate relational database design, construction, and testing, which we conducted during the pilot phases of instrument and system development. It also requires careful data management and quality control. Database specifications were developed based on instruments being pilot tested during the development phase. The design was adjusted for each revision in the instrument. Field names and variable responses as outlined in the codebook were included in the design, and the databases were constructed using FileMaker Pro.™ Quality control activities involved reviewing data that were either hand entered or computer downloaded for internal consistency. Coding mechanisms for handling missing data were designed. During database testing, a data file was constructed from the database and exported into a statistical package for test analyses until the data file construction was deemed suitable for univariate and multivariate analyses. 

Two database design experts and statistical analyst collaborated to construct and test the database and data file creation. These carefully constructed resources, the Teaching and Learning Databases, have great potential for use in the study of clinical education. For instance, we know that students are often asked to see walk-in patients with acute problems,82 which may not provide the diversity of conditions and procedures that will suit the students’ learning needs for the clerkship. We also know that the gender match of the student/preceptor dyad appears to influence clinical teaching experiences.83 Our comparison of how clinical experiences complement each other in family medicine, ambulatory medicine, and ambulatory pediatrics is described elsewhere 80 and has allowed us to define what is possible in an integrated ambulatory clerkship. Given its larger volume of data (8-week vs. 4-week clerkship, additional year of data collected), the Family Medicine Department has been able to define reasonable expectations for clinical experiences across all of its teaching sites (Figure 1 and Figure 2). We aim to develop similar clinical expectations for internal medicine and pediatrics, based on data collected with the documentation system. Once these expectations have been developed in all three disciplines, we hope to propose global primary care expectations and to set goals for the integrated clerkship.

We plan to use this important resource to explore factors that influence ambulatory teaching and learning, such as whether a comprehensive documentation and feedback system actually leads to improved student performance. We plan to do this by linking our students’ logged experiences to other educational outcomes, such as our Interdisciplinary primary care clerkship, clinical skills exam data, course specific exams, clerkship grades, and USMLE Step II test scores. 

Another important aspect of evaluation is student usage and usefulness.(Table 3). At the end of each 16-week IPCC block, students complete a survey to assess the overall frequency of handheld computer use and detailed information on the usefulness of the clinical software programs using a 5-point Likert scale where 1= not at all useful and 5=very useful. 

Report Formats

By 2005, we had over 28,000 student encounters with patients and preceptors in the Teaching and Learning Databases, which represent 6 years of ambulatory clerkship data in family medicine and three years of ambulatory internal medicine from this data. Figure 1 illustrates the report format for Common Problem Tally Section data 3 weeks into the Family Medicine Clerkship.  As indicated here, achieving experience with OB checks, patients with syncope, arrhythmias, Type I diabetes, dementia, cardiovascular accidents, and developmental delays in children have been a challenge for this sample student.. Figure 2 outlines the report format for Teaching and Learning Section data. As indicated, each report includes a summary of the individual student’s clinical or teaching and learning experiences to date with comparisons to the current course average (aggregate of all data collected at the point the reports were generated). In this example, the student is receiving less feedback on clinical skills, but more teaching than expected. This student is also not working as independently as expected compared to his/her peers in conducting physical examinations. Reports in the Family Medicine Clerkship additionally include a comparison to course expectations for each item (see Figure 1 and Figure 2) to assist learners in identifying their individual learning goals. 

Figure 3 illustrates the annual report format for clerkship directors. This report presents data derived from the Teaching and Learning Section by clinical sites with comparisons to the clerkship goals and aggregate data on all sites combined. As illustrated, only two of the 11 sites (18%) achieve the proportion of continuity visits we would like to achieve in the Family Medicine Clerkship.  Figure 4 illustrates a customized report format currently being evaluated in one of our clinical courses.

Table 3 shows self-reported frequency of handheld computer use by students in six 16-week blocks of IPCC from October 1999 through June 2001. These data are based on 79 surveys administered at the end of IPCC to assess usefulness of the system (response rate = 86%). Students reported using their handheld computers an average of 8.6 times a day (range of 0-30), with the highest use being data entry for the documentation system (mean 4.0, range 0-20). The supplemental clinical applications provided on the handheld were used an average of 3.8 times per day (range, 0-18). The three applications rated most useful by students were the drug database (mean = 4.25); the immunization guide (mean = 2.7); and the list of developmental milestones (mean = 2.7). 

Table 4 outlines the costs associated with development and maintenance of both paper-based and a handheld computer-based documentation and feedback system. While expenses of the paper system are 57% of the computer system's expenses ($90,767 versus $159,645), when equipment and programming is excluded from the analysis, the costs are much more comparable ($90,767 versus $125,695). (return to top)

Overall Benefits of the Logbook System

Benefits for Students

The unique emphasis of our system on data use encourages students to take responsibility for their own learning needs and promotes faculty development among clerkship directors and ambulatory clerkship preceptors. Data-driven conversations about clinical training align the educational objectives with teaching/learning approaches, enabling students to meet individual learning goals. The formal reports encourage this communication. Multidisciplinary learning can also be enhanced by reviewing experiences in all three ambulatory specialties. This allows students to pursuit complementary clinical opportunities as they move from one core clerkship to another. 

Anecdotal student feedback regarding ClinEdDoc™ has been very positive. In report review sessions, students have stated that the reports help them to identify areas for self-directed learning; select appropriate patients; plan for electives; focus clinical goals for skill practice; request feedback from preceptors; and promote discussions with preceptors and course directors about their development as clinicians. 

Benefits to Preceptors/Teaching Faculty

Clinical Documentation System reports have the added value of providing substantial and individualized content to review during site visits with preceptors.80,82,83  Preceptors anecdotally report valuing the opportunity to gauge their own performance using annual comparison data of their individual practices to aggregated data from all of other clerkship teaching sites.   

Benefits for Course Directors

Clerkship directors play an active role in students’ clinical experiences by addressing specific strengths and weaknesses identified in the data when it is most useful to both teachers and learners. The data provide clear indications of when supplementary experiences are needed to round out a student’s clinical experiences. For example, exposing students to obstetrical training in family medicine is becoming more of a challenge as fewer family medicine preceptors are doing obstetrics. However, we can both explore other opportunities to provide these experiences or ensure that these are obtained in the OB/GYN Clerkship.  Ongoing review of data stratified by teaching site allows for increased uniformity in meeting clerkship goals.  

Benefits for the Institutions

Our system assists with accreditation processes by providing a comprehensive and uniform look at all clinical courses and precepting sites to identify potential redundancy as well areas of the curriculum in need of enhancement. (return to top)

Ongoing Challenges

Though we are very enthusiastic about the system we have created, ongoing challenges exist in many critical components of the system. Striking a cost/benefit balance is a constant issue for any institution faced with vital innovations, such as a handheld-computer-based documentation and feedback system. A new requirement that students entering Dartmouth Medical School purchase their own laptops has enabled the laptop purchase to be considered as part of each student's financial aid arrangements. We also are challenged by rapidly evolving technology in handheld computer hardware and software and a growing number of students who own their own handheld computers, which may or may not be compatible with our operating systems. We are working with the medical school’s developing computer support office in hopes that support for the handheld computer documentation system will be provided through its office.  This is a new area of educational technology and institutional resources are just being established.

Challenges exist in educational evaluation designs, such as observational designs, that do not allow us to draw strong conclusions from our data. Designs that include random assignment to educational interventions with comparison groups are much more robust, but difficult to implement in educational settings.  The generalizability of our findings may also be limited because we do not know the applicability of our system to other multi-disciplinary courses. To address this, we must expand the use of our system to other educational settings. We have begun to provide the paper-based instrument to other schools by request.  Additional dissemination plans include expanding to other programs internally at our school as well as nationally. 

Conclusion

In conclusion, we believe a comprehensive documentation and feedback system allows us to better understand and enhance community-based medical education. Although we know that feedback is critical for both learners and teachers, there is much to learn about how to accomplish this in a way that ensures educational success for our students and both our on-site and community-based faculty.

Lessons Learned

You can never have enough expertise, time, and money; however, two areas stand out as critically important to success. The first is that effort be made early on to design a valid and reliable instrument and corresponding database that will add to the long-term research value of the project. Communication and collaboration with all users throughout the span of the project can enhance buy in and promote successful institutionalization. (return to top)

Evaluations

Bryan Bognar, MD

Evaluations are an important and required part of any clerkship. A number of LCME educational standards necessitate the collection of program and student evaluative data (ED-27, ED-28, ED-30, ED-32, ED-47). The need to document and quantify program effectiveness at all levels is now the norm. For years, programs were able to rely upon “paper and pencil” systems, and in some situations, this may still be entirely appropriate. Like many aspects of healthcare, programs are increasingly leveraging electronic systems to facilitate and enhance the evaluation process. 

Evaluations commonly used within a clerkship will likely fall into one of the following categories:

• Student clinical performance evaluations
• Student evaluations of their preceptors (residents and faculty)
• Course/clerkship evaluations
• Ad hoc evaluations (e.g., specific lectures or conferences)

Electronic evaluation systems can run the spectrum from simple and cheap to sophisticated and expensive. The major differences lie in the ease of survey creation, distribution, and data management. If you are just getting started, perhaps the best advice is to research whether your institution already licenses this type of software. The challenge then becomes adapting what is available to your specific needs. The crudest of systems would be to send your target audience a direct e-mail or an e-mail attachment. This is certainly more efficient than chasing paper through snail mail but still requires labor intensive data entry and analysis once responses are received. Until moving to a propriety system, we used a homegrown system using Web-based forms for several years. The major advantages lie in ease of distribution and consistency in data received, but there was still much work on the back end in terms of data manipulation and analysis. Commercial systems typically offer a method for creating, sending, and managing (e.g., reminders) evaluations/surveys. In addition, data collection, storage, and analysis are simplified, secure, and streamlined. Courseware such as Blackboard and WebCT offer built-in evaluation and surveying components.  There are other freestanding survey tools available as well, such as Flashlight Online (www.tltgroup.org/flashlightonline.htm). (return to top)

Computer Based Testing

Online and computer based testing has become an integral component of many programs. The United States Medical Licensure Examination (USMLE) is now delivered in this format. Most courseware offers online testing components. 

Many institutions have developed homegrown systems with varying degrees of sophistication. Important limitations of these systems often surround their ability to support multiple users at once (scalability) and general security issues. Some of the key features to consider when evaluating an online testing system are summarized in Table 5. Whether you are using a homegrown system or a propriety system, the ability to support graphics (e.g., radiographs, EKGs, and other images) and other multimedia has proven valuable and added to the quality of the exam items.

HIPPA

A comprehensive review of the Health Insurance Portability and Accountability Act of 1966 (HIPPA) is clearly beyond the purview of this guidebook. The Office of Civil Rights (OCR) within the Department of Health and Human Services has an official web site detailing and summarizing key aspects of the privacy rule, www.hhs.gov/ocr/hipaa/ . This online resource also contains general educational materials and a useful FAQ section.

It would be important to emphasize that clerkship directors should make their students aware of key aspects of maintaining privacy and confidentiality of personal health information (PHI). This would include both electronic and written PHI. Good habits begin early and should be role modeled in the clinics and wards. Students are fond of keeping "cheat sheet" and crib notes on patients they are following. They should be reminded that patient lists, trackers and any similar documentation should be treated as PHI. Most schools now require HIPPA training for students at the beginning of their clinical rotations. (return to top)

Reference Resources/Information Retrieval

John J. Orriola, MA, MEd, AHIP

Reference resources are quite numerous and varied as to formats and media. Classification of resources is helpful for the purpose of discussion. The first major classifications are print and online. Though there has been a sizeable increase in the availability of online or full text material, it is important to realize that many resources, i.e., specific journals, books, CD-ROMs, etc., are not available online for a number of reasons, principally, the library in a particular health sciences center does not have a subscription to the item or does not own it. A secondary reason is that most journal literature has not been digitized. A rule of thumb is that if the article is from 1995 or prior, the chances of online availability diminish considerably. It is important that we communicate this to our students, so they become aware of the importance of using their library catalog to determine the availability and format, whether online or in print of the material they seek. As publishers digitize journal issues backwards, the threshold moves accordingly. There doesn’t seem to be a rush, however to place those backfiles online. The movement is certainly from current to future.

The next classification scheme is one that separates reference materials according to their primary use, regardless of media. Reference resources may fit into one of the following broad classes:

• Bibliographic
• EBM
• General reference

Bibliographic resources

These are used to identify or retrieve bibliographic citations. This is the main purpose. This is an important distinction, as we don’t want to expect more than the tool can provide. The best example of this class of tool is Medline, whether we use PubMed, Ovid, or another provider. Medline’s purpose is to indicate to the searcher the availability of literature in a subject matter or to identify a journal article by using other access points or fields such as Author, Journal Name, Article Title, and many others. This is essentially the raw literature, the source of foreground information. Of course, most health science center libraries offer customized PubMed, with built-in links to full text articles. This however is a function of the library, not PubMed. See Table 6 for a listing of the primary bibliographic databases.

EBM resources

These filter literature to meet specific evidence focused criteria. The main purpose here is to draw from the raw literature, the bibliographic resources, through the use of filtering or limiting, the literature that best addresses a particular question or situation. Good filtering retrieves the literature from which we draw evidence. An example of this class of resource is the Cochrane library. There are many others which can be seen in Table 6. EBM resources may be further stratified. This will be addressed later.

General reference resources

These are resources that we use to find definitions, guidelines, basic, and background information. These resources include all books, atlases, software, directories, handbooks, etc., regardless of media, and software, CD ROMs, videos, and so forth. The best single access source for general reference resources is the local academic health sciences center Online Public Access Catalog or OPAC, more commonly known as the library catalog. Google can and should be used for web searching inasmuch as it is the “catalog” of the web. However, to maximize effectiveness and precision of Google searching, the Advanced Google Search interface must be used. This interface is located just to the right of the Google text entry box.

There are several big names in the general reference resources publishing business such as MD Consult and Stat!Ref, These are also known as aggregated databases. Though they possess both bibliographic and EBM (to some extent) capabilities, their best use is as general reference resources. It is important to note that at the author’s academic health sciences library, each resource within these aggregated databases, is individually catalogued, In this manner, the searcher interfaces with one central archive, the library catalog, for any item that is contained within MD Consult, Stat!Ref, or any other aggregator. The alternative would be to search each database individually to find a reference resource, e.g., a core textbook such as Merritt’s Neurology. If we consider that a library might have numerous aggregated databases, it is quite useful to only have to manage a single search interface to identify the whereabouts of a core textbook or a journal. It would be practical to contact the library to determine if they utilize this system of cataloguing. (return to top)

EBM Hierarchical Classification

The purpose of classifying is to set the stage for directing the search in the most expedient manner. Reference resources have grown in number and will continue to do so. EBM resources have experienced huge growth themselves. More importantly, EBM resources have evolved as well. 10 years ago, Cochrane was a very small database and Pubmed as we know it didn’t exist. The proliferation of databases that actually filter the literature themselves, the advent of the Pubmed clinical Queries, which is the secondary PubMed interface and literally dozens of EBM resources, have prompted the use of models to distinguish among these many resources. Though there are several, below is a hierarchical pyramidal model, developed by Brian Haynes, MD,89 which provides a relative utility perspective to assist the searcher in selecting the best database for the purpose at hand. The model can be even more useful when customized by correlating pyramid level to actual resources to which the specific academic medical library subscribes or utilizes.

Computerized Decision Support Systems (CDSSs), as envisioned by Dr. Haynes in his article are still in the making (existing CDSSs however, can have a positive impact on practitioner performance 90). There are a number of tools beginning to approach the ideal, namely databases such as Uptodate 91, FirstConsult 92, InfoRetriever 93, and Clinical Evidence. Guidelines are considered by some to be systems themselves 94; therefore guidelines databases such as the National Guidelines Clearinghouse are included in this level. The Synopses level is represented by the ACP Journal Club and Evidence Based Medicine, both available from Ovid. Syntheses can be found in the Cochrane Library while Studies can be retrieved by utilizing PubMed Clinical Queries. The previous named databases are examples. There are others as indicated in Table 6.

A final classification is warranted before negotiating the tabular data. Though not of major significance in terms of the quality or value of resources, a division into free and fee is useful to identify those tools which are and will be readily available regardless of the future affiliations of graduates and faculty, as they are available to anyone, anywhere. Table 6 is selectively comprehensive but not exhaustive. There are many sites that were not included because they were too new, did not originate from sources that were sufficiently well established, sources seemed to be in transition or the resource appeared on the scene after this was written. In future editions, others will be added while some will be removed.

(return to top)

Figure 1

*Hint: Click Back button on browser to return to previous spot in text
OR (return to top)

Table 2

Recommended PDA Programs

CATEGORY	Recommended PDA Programs (Richard Usatine, MD. Updated 2005)	PALM	POCKET PC	COST $
Clinical Decision Making	MedRules  http://pbrain.hypermart.net/medrules.html InfoRetriever (includes 5 minute CC) http://www.infopoems.com/infopoems/downloads.cfm	x x	x	Free Free
Document Readers	Acrobat Reader   http://www.adobe.com/products/acrobat/readermain.html	x	x	Free
	iSil, www.Isilo.com (allows free use of many great programs – see dermmeister.com)	x	x	$20
Medication References	ePocrates Rx Pro, www.epocrates.com ($30 with UTHSCSA discount) ePocrates Rx,  www.epocrates.com	x x	x	$30 Free
	Antibiotic Assistant & Pain Management Assistant from TheraDoc – free through Merck site	x	x	Free
	Mobile MicroMedex    http://www.library.uthscsa.edu/internet/pda.cfm?CatID=2	x	x	Free
Evidence Based Med Reference	ebm2go, www.ebm2go.com/	x	x	Free
Palm File Management	FileZ, www.Freewarepalm.com/utilities/filez.shtml	x		Free
Growth Chart	StatGrowthChart, www.statcoder.com/	x	x	Free
ID Guide	Johns Hopkins Antibiotic Guide, http://hopkins-abxguide.org/	x	x	Free
Immunization References	Shots 2005, www.immunizationed.org/	x	x	Free
Medical Calculators	MedCalc, http://medcalc.med-ia.net/	x		Free
	MedMath, http://smi-web.stanford.edu/people/pcheng/medmath/index.html	x		Free
	Archimedes, www.skyscape.com/	x	x	Free
Medical Dictionaries	Dorland's, www.doctorsgadgets.com/software/medicaldictionaries/dorlands.htm	x	x
	Stedman's, www.skyscape.com/	x	x
	Taber’s, www.skyscape.com/	x	x
Tools for the Ward	Diagnosaurus, www.diagnosaurus.com	x		Free
MMSE	MentSTAT   http://goldenratiodesign.com/	x		Free
Pediatric Decision Support	Peds Calc, Peds Omnibus – Free from www.Fppda.com	x		Free
Pregnancy Calculator	My OB Wheel, www.fppda.com/timob.html code 598e24	x		Free
	*Also available free on MedCalc	x
Nutrition	DietCalc, http://pbrain.hypermart.net/medfiles.html	x		Free
Reference Tools	*May be found at: Skyscape, Handheld Med, Pbrain
	Merck Medicus:  Merck Manual, Pocket Guide to Diagnostic Tests, MEDLINE and Reuters Medical News, http://merckmedicus.com/pp/us/hcp/hcp_mobile_medicus.jsp?t=H0030	x	x	Free
	PalmEKG, www.pdacortex.com/PalmEKG_Download.htm	x	x	Free
Surgery	http://www.meistermed.com/isilodepot/isilo_depot_index_subj.htm#Surgery/Trauma