Preventive Medicine 30, 244–251 (2000) doi:10.1006/pmed.1999.0624, available online at http://www.idealibrary.com on
A Randomized Controlled Trial of a Clinic-Based Support Staff Intervention to Increase the Rate of Fecal Occult Blood Test Ordering1 Nancy J. Thompson, Ph.D.,*,2 Edward J. Boyko, M.D., M.P.H.,†,‡ Jason A. Dominitz, M.D., M.H.S.,‡,§ Donald W. Belcher, M.D.‡,¶ Brian B. Chesebro, B.A.,| Linda M. Stephens, R.N., B.S.N., M.N.,** and Michael K. Chapko, Ph.D.‡,†† *Department of Community and Behavioral Health, College of Public Health, University of Iowa, Iowa City, Iowa 52242; †Epidemiologic Research and Information Center (ERIC) and Medicine Service, §Gastroenterology Section, ¶ General Internal Medicine Section, **Ambulatory Care, and ††Health Services Research and Development Service, VA Puget Sound Health Care System, Seattle Division, Seattle, Washington 98108; ‡University of Washington, Seattle, Washington 98195; and |School of Medicine, Johns Hopkins University, Baltimore, Maryland 21218
Background. Colorectal cancer is the second most common fatal malignancy in the United States. Early detection using fecal occult blood tests has been shown to reduce mortality, but these tests are underutilized among those eligible for this screening. Attempts to increase use of fecal occult blood tests in eligible populations have focused on the provider, patient, or system. But none have examined whether a support-staff intervention is effective in achieving this aim. We therefore conducted a randomized controlled trial to test the impact of authorizing support staff to order fecal occult blood tests in a general internal medicine clinic organized into four teams. Methods. A total of 1,109 patients were included in the study, 545 of whom were in the two teams randomized to treatment. Univariate and multivariate regression analyses were used to evaluate the impact of the intervention. Results. The intervention resulted in significantly more fecal occult blood test ordering in the treatment group than in the control group for all patients (52% vs 15%, P , 0.001). Treatment fecal occult blood test cards were returned as frequently as the control cards for all patients (44% vs 48%, P 5 0.571). Conclusion. Delegation of selected screening tasks to support staff can enhance patient access to preventive care. q 2000 American Health Foundation and Academic Press
1 We gratefully acknowledge monies provided by the VA Puget Sound Health Care System to conduct this project. 2 To whom reprint requests should be addressed. Fax: (319) 3359200. E-mail: [email protected]
Key Words: allied health personnel; colorectal neoplasms; mass screening; occult blood; practice guidelines; preventive medicine; randomized controlled trials; reminder systems.
Colorectal cancer is the second most common fatal malignancy in the United States. It has been estimated that during 1999 approximately 129,400 new cases will be diagnosed and 56,000 persons will die from the disease . The United States Department of Veterans Affairs (VA)  reported that 4% of veterans who were users of this medical system had either stomach or colon cancer. Survival in colorectal cancer is related to the extent of disease at the time of diagnosis. Among individuals with colorectal cancer, those diagnosed at an advanced stage have a projected 5-year survival rate of 7% in contrast with a survival rate of 92% for individuals detected at an early stage . Use of the fecal occult blood test (FOBT) as a screening test for colorectal cancer has been proven to decrease colorectal cancer mortality [4–7]. Further, policy makers have concluded that this test is cost effective [8,9]. The VA recommends that all veterans 50 years and older receive an annual FOBT or a sigmoidoscopy with unspecified periodicity . However, FOBT order rates are less than optimal [11–13]. The VA’s goal is to have 50% adherence for fecal occult blood testing by fiscal year 2000 . Within the VA, the 1996 national survey of users indicates that 33% of the males and 29% of the
0091-7435/00 $35.00 Copyright q 2000 by American Health Foundation and Academic Press All rights of reproduction in any form reserved.
FECAL OCCULT BLOOD TEST ORDER RATE
females over age 50 reported that they had received an order for FOBT within the previous year. Comparable data for the Seattle Veteran’s Affairs Medical Center were 28% for males and 19% for females . Other data , based on abstracted medical reports for care provided in each VAMC in fiscal year 1995, indicate that the mean rate of screening for colorectal cancer (FOBT or sigmoidoscopy) is 36% with a range of from 18 to 58%. Provider time is often cited as a barrier to more optimal delivery of preventive services within the context of primary care [14–16]. Yet time is rarely addressed by interventions aimed at improving FOBT order rates. One study that included an intervention addressing provider time was conducted by Belcher . In a large randomized clinical trial over a 5-year period he and his colleagues assessed the effectiveness of three methods of delivering preventive services: (a) a brochure advising patients to ask for preventive services; (b) a program for physicians including education, motivation, a flowsheet indicating recommended activities, and periodic assessment of performance; and (c) the establishment of the separate clinic staffed by nurse practitioners and devoted to screening, health counseling, and coordinating follow-up care. Providers of patients receiving the brochure placed an average of 2% less orders than the controls. Providers receiving the second intervention placed an average of 2% more orders than the controls. However, providers involved in the separate clinic placed an average of 58% more orders than the controls. Unfortunately the Health Promotion Clinic did not continue after the study for administrative reasons. Many interventions within primary care clinics have been shown to improve FOBT order rates. Several randomized controlled trials have demonstrated the effectiveness of patient-specific reminders to providers [18– 20]. These studies indicated that the treatment group will place 17–33% more orders than the control group when reminders are introduced. The study of Tierney et al.  also indicated that performance feedback  led the treatment providers to place approximately 13% more orders than the control group. We projected that an intervention that addressed the barrier of time within the context of a primary care clinic would yield order rate improvements greater than those achieved by reminders to primary care providers, more on a par with those reported by Belcher. To address this barrier within the primary care clinic we chose to invoke the concept of delegation, redefining the extent of human resources allocated to the delivery of preventive services. At the Seattle VAMC, delegation of FOBT ordering meant involving personnel other than the physicians and nurse practitioners who were designated as providers. Staff personnel in the GIM primary care clinic included registered nurses, licensed practical
nurses (LPNs), and desk clerks. LPNs were chosen because they routinely interacted with patients (to assess blood pressure, weight, and need for various preventive services) prior to each patient’s appointment with the primary provider. In summary, we were interested in seeing whether an intervention addressing the time barrier via delegation to nonproviders would substantially improve order rates.
The objective of this study was to determine the impact of having LPNs order FOBTs on the rate of ordering and returning these cards. The intervention was in addition to the existing practice of having orders placed by physicians and nurse practitioners. The design was a firm-randomized controlled trial . The study setting was the General Internal Medicine Clinic of the VA Puget Sound Health Care System, Seattle Division. This clinic is divided into four teams within two firms. Each firm occupies its own adjacent hallway in the clinic area. Staff includes physicians who have faculty appointments at the University of Washington, internal medicine residents and fellows from this same educational institution, and nurse practitioners. Each of these practitioners functions as a primary provider to a panel of assigned patients and continues to follow the patient throughout his/her training in the case of trainees or indefinitely in the case of staff. This firm system was initially established on the basis of random assignment of both patient and providers to various teams. At the current time, assignment is not totally random. However, ongoing analysis of team comparability has shown that the firms are equivalent with regard to patients and providers. The impact of the intervention was measured by comparing ordering and returning in these two hallways, one of which allowed LPNs to place the FOBT order and one of which did not. The sample included the first clinic visit only of all 50 to 69-year-old patients scheduled to see a primary provider between January 27 and March 31, 1998. In preparation for each day’s clinic, we generated lists of the study patients classified as treatment or control patients and as eligible or ineligible for a FOBT based on review of computerized laboratory files and medical records. Only the LPNs in the treatment firm received lists indicating whether patients were eligible for an FOBT. The criteria for classification as eligible included the following: not having had an FOBT within the calendar year preceding the appointment date and not having had either a flexible sigmoidoscopy or a colonoscopy in the prior 5 years. As the designated patients arrived, they completed a survey called the Health Promotion Screening Form
THOMPSON ET AL.
that included items relevant to the delivery of preventive services including colorectal cancer screening. Subsequently the LPNs proceeded to collect vital sign information, review the survey, and provide various preventive care services. To determine a patient’s eligibility for a FOBT, the LPN was to use information from both the researcher-generated lists and the patient’s Health Promotion Screening Form. Seven hundred fifteen patients were determined to be eligible on the basis of computerized laboratory files and medical records. Of these, 270 would have been declared ineligible based on their responses to the Health Promotion Screening Form alone. Once the LPN determined that the patient was eligible for a FOBT, she discussed the option with the patient including the purpose of the test, the technique of obtaining the samples, and the possibility that there could be further testing if the findings were positive. The patient was to have the opportunity to refuse the test. The LPN ordered the test by marking the clinic’s encounter form. This form was then submitted to the patient’s primary provider. The protocol allowed the primary providers to countermand the LPN order if the provider felt it was inappropriate. Patients for whom the test was ordered received the FOBT kit as they left the clinic from a clinic receptionist. Data on FOBT “ordering” and “returning” were collected for two separate time periods (January–March 1997 and January–March 1998). Data from the earlier time period, before the intervention was implemented, were used to establish baseline information on each variable. Information about the 1997 orders was acquired from the General Internal Medicine Clinic’s encounter forms. Information about the 1998 orders was acquired from three sources: the Health Promotion Screening form, the General Internal Medicine’s encounter forms, and the VA Medical Center’s centralized computer database. For both time periods, information on returning was acquired from the VA Medical Center’s centralized computer database, allowing a 90-day window of time to elapse before the cards were designated as returned or not returned. All FOBT cards are returned to this central laboratory for processing. While we did not explicitly determine whether any of the patients may have died prior to the return cut-off date, all patients for whom an order had been placed were located in the center’s computer database. Therefore it was assumed that no patient died prior to the cut-off date. In addition to information on orders and returns, other data about the system were obtained including the following: type of orderer, countermanding, impact on other LPN services, barriers to ordering, volume of patients, and primary care provider. Other data about the patients included test refusals and demographics. x 2 analysis was used to assess associations between categorical variables and to compare proportions, while the Kaplan–Meier curve was used to examine data on
time to return of the FOBT cards by the patient . Multivariate logistic and Cox regression analyses were conducted to assess the association between independent variables, such as treatment assignment and test orderer characteristics, and whether a test order was placed or the time until return of the FOBT cards. Given the randomization of clinical care teams to the treatment or control groups, the subjects within each team were not strictly independent observations. To account for the nonindependence of observations within teams, the Huber/White/sandwich robust variance estimator of STATA 5.0  was used for tests of significance and generation of confidence intervals . The analysis regarding ordering was based on the 715 individuals who were eligible for an order, while the analysis regarding time to return of the FOBT cards was based on the 371 individuals for whom a test was ordered. The study methods were approved by the University of Washington’s Subcommittee on Human Studies on December 26, 1997. RESULTS
Data were available for 1,109 (545 treatment, 564 control) patients. This number was derived from 1,734 appointments with primary providers in the General Internal Medicine Clinic between January 27 and March 31, l998. Of the 1,734 appointments, 35% did not occur due to cancellations or failure to show. Of the 1,123 study patients who presented to the clinic and met the inclusion criteria (approximately 10% of all the patients processed by the LPNs during the study period) 14 (1.2%) were excluded from the analysis. Eight of the fourteen were excluded because they were inadvertently triaged by the wrong-firm LPNs. The final sample of 1,109 individuals was 98% male, 80% Caucasian, and with a mean age of 60.26 years. Table 1 presents the distribution of the sample in both the treatment and the control teams. Analyses were conducted to establish the comparability of the treatment and control teams. One year prior to the study time period (January 27–March 31, 1997), the treatment and control teams were comparable in terms of both FOBT ordering and returns for 50 to 69year-old patients (Table 2). Furthermore, data acquired during the study period indicated that the teams were comparable in terms of patients’ eligibility for various preventive services including the FOBT, as well as other patient characteristics. For example the treatment and control patients were comparable with regard to age and gender. Patients were found to be different with regard to the number of diagnoses as of the appointment (treatment mean 5.55, control mean 3.57, P , 0.001) while the system was found to be different with regard to the distribution of provider type among the appointments. Staff physicians in the treatment teams covered
FECAL OCCULT BLOOD TEST ORDER RATE
TABLE 1 Distribution of the Sample by Treatment Assignment Treatment patients (n 5 545)
Control patients (n 5 564)
60.4/5.9 534 (98%) 5.6/2.80 173 (32%) 361 (66%) 184 (34%) 107 (20%) 52 (10%) 25 (5%)
60.1/6.1 552 (98%) 3.6/1.6 253 (45%) 354 (63%) 210 (37%) 103 (18%) 72 (13%) 35 (6%)
0.436 0.898 ,0.001 ,0.001 0.227 0.227 0.071 0.199 0.396
Age (mean/SD) Male (No./%) No. diagnoses (mean/SD) Appointments covered by staff physicians (No./%) Eligible (n/%) Ineligible (n/%) Prior FOBT alone Prior flex sig or colonoscopy alone Othera a
Other combinations of the criteria used to determine eligibility for the test.
a smaller percentage of the appointments than their counterparts in the control teams (32% vs 45%, P , 0.001) (Table 1). Overall, during the study period, the treatment teams, where the orders were made primarily by the LPNs, ordered significantly more FOBTs than the control teams for all patients (52% vs 15%, P , 0.001) as well as for eligible patients (72% vs 19%, P , 0.001). Also, the treatment firm FOBT cards were returned as frequently as the control cards for all patients (44% vs 48%, P 5 0.571) as well as for eligible patients (46% vs 43%, P 5 0.605). And further, of the 167 patients who returned cards, 14 or 8% (9% treatment, 7% control) had one or more positive readings that were conveyed to the primary providers for use in decisions about subsequent care (Table 2). In analyses that take into account differences between firms in number of diagnoses and provider coverage, the strength of the association between the intervention and a higher frequency of FOBT ordering diminished somewhat, but nevertheless
TABLE 2 Intervention Impact—Univariate Analysis Treatment (%) Control (%) P value baselinea/ baselinea/ baselinea/ interventionb interventionb interventionb Orders All patients Eligible patients Returns All patients Eligible patients Percentage positivec a
47/44 NA/46 6/9
52/48 NA/43 3/7
0.546/0.571 NA/0.605 0.555/0.777
Baseline time period, 1 year prior to the intervention period. Intervention time period, 44 days between January and March 1998. c At least one positive reading. While the kit was designed to collect three samples from each patient, operationally a patient was defined as needing follow-up if at least one of the samples was positive. b
remains quite strong and statistically significant (Table 3). As some tests were ordered for ineligible patients, it is important to determine how the LPNs compared with the primary providers in orders placed for ineligible patients. Of the 263 orders for which information was available on both patient eligibility and orderer, 15/ 239 or 6% of the LPN orders were placed for ineligible patients, while 9/24 or 38% of the MD/NP orders were placed for ineligible patients. The associated P value for this distribution of orders is ,0.001. While the study protocol included patient refusal and provider countermanding features, neither was used extensively. Of the treatment team patients eligible for the test, 3% (11/361) refused to have the order placed by the LPN. Of all orders known to have been placed by the treatment team LPNs, primary providers countermanded 6% (17/263). Further, only 3 of these countermands aborted orders placed by LPNs for patients ineligible for a FOBT. Finally, we found that ordering FOBTs did not have a negative impact on delivery of other LPN prevention activities. During the study period, the LPNs were expected to provide three other preventive services: order cholesterol tests and administer shots for protection from pneumonia and tetanus. The treatment LPNs actually implemented these tasks at a higher frequency than the control LPNs (cholesterol tests 17% vs 4%, P 5 0.003; pneumonia shots 15% vs 9%, P 5 0.192; tetanus shots 65% vs 14%, P , 0.001). The average time to return the FOBT cards was 22 days, with 95% of the returns occurring between 6 and 38 days of the patient having been issued the kit. Compared with the previous year, the rates at which FOBT cards were returned by the patient were slightly lower in both the treatment (3% decrease) and the control (4% decrease) teams. For the study period, the intervention did not yield a noticeable difference in return rates by treatment assignment based on both x2 test and survival analysis (Table 2 and Fig. 1). In analyses that
THOMPSON ET AL.
TABLE 3 Odds Ratios Comparing Treatment and Control Groups on Ordering the FOBT in Univariate and Multivariate Regression Models Model
Variables in the model
95% Confidence interval
1 2 3 4
No covariates No. diagnosesa No. diagnosesa, provider coverageb No. of diagnosesa, provider coverageb, age, gender
10.83 8.89 8.80 8.87
9.12–12.86 6.71–11.77 6.62–11.71 6.72–11.70
a Regression analysis indicated that number of diagnoses is a significant predictor of ordering. There is a 15% increase in the odds of having an order placed for each additional diagnosis at the time of the appointment. b Provider coverage refers to the proportion of appointments covered by various types of physicians (staff, fellows, residents) and nurse practitioners.
take into account differences between firms in number of diagnoses, provider coverage, age, gender, and orderer, the association between the intervention and return of the FOBT cards again was still not significant (Table 4). Within the treatment firm, the difference in the return proportion for orders placed by LPNs and the rate for orders placed by primary providers was not statistically significant (LPNs 99/246, 40%, versus primary providers 17/30, 57%, P 5 0.085).
This is the only study known to us of an intervention based on the ordering of FOBTs by support staff provided with a reminder system that identified patients eligible for the test. Specifically LPNs, working in an existing clinic structure, were given clear authority to order this test and information from patient records to assist them in identifying patients for whom the test should be ordered. Physicians and nurse practitioners could order the FOBT in either the treatment or the
FIG. 1. Survival analysis return. Black line, treatment group; gray line, control group.
FECAL OCCULT BLOOD TEST ORDER RATE
TABLE 4 Hazard Ratios Comparing Treatment and Control Groups on Returning the FOBT in Univariate and Multivariate Regression Models Model 1 2 3 4 5 6
Variables in the model No covariates No. diagnoses No. diagnoses, provider coveragea No. diagnoses, provider coverage,a ageb Orderer Orderer, age,b gender
95% Confidence interval
0.92 1.32 1.39
0.59–1.44 0.66–2.69 0.66–2.92
a Provider coverage refers to the proportion of appointments covered by various types of physicians (staff, fellows, residents) and nurse practitioners. b Regression analysis indicated that age is a significant predictor of return. There is a 4% increase in the odds of returning the cards with each additional year of age.
control firms and were motivated to do so to comply with VA guidelines for colorectal cancer screening. Primary providers in the treatment firm had the option of countermanding LPN orders they deemed inappropriate. The broad implication of this study’s findings is that delegation, supported by use of decision-support algorithms, can dramatically increase the rate at which a preventive service is offered to patients without decrement in the rate at which the patient returns the FOBT samples. The finding that primary providers placed a larger percentage of orders for ineligible patients should be considered in the light of the following facts. First, while the support staff had lists that specified eligible patients, the providers did not. During the appointment the providers could have looked up information about prior FOBTs on a computer, but they did not have access to computerized information regarding prior endoscopic procedures. They did have access to the charts that had endoscopy reports, and the patients could answer questions for them. Further, providers may have ordered FOBTs for reasons other than screening. The major limitations of this study include the following: use of a firm system for a randomized controlled trial, contamination, and the possibility of changes other than the intervention in the treatment teams. While this study design is not a true randomized controlled trial, it is similar to what Campbell and Stanley  termed a “quasi-experimental design.” Even though the two arms of the study are not randomized, team randomization produced very similar groups of patients. Cebul  provides a rationale for the use of firm systems as a means of evaluating medical interventions. Firm randomization is considered a viable
alternative to true randomization if the following conditions exist: (a) preintervention firm equivalence on all important parameters, (b) truly random selection of the treatment and control groups, (c) an assurance that a “steady state” exists, and (d) control for cross-team contamination. Curley et al.  demonstrate the use of a firm system for a randomized controlled trial. Firm equivalence is important in ensuring that differences in ordering and returning attributed to the intervention could not be attributed to characteristics of the firms that are unequally distributed. The treatment and control teams were found to be comparable in ordering and returning for a period of time equal to the study period 1 year prior to the study. In addition, patients seen by the teams were found to be comparable in terms of numerous other variables (demographic, health status, health care utilization, and satisfaction with health care). The fact that the relationship between ordering and the intervention was diminished only slightly even when differences between firms in number of diagnoses was taken into account is interesting since the treatment patients had more diagnoses than the control patients and number of diagnoses was a significant predictor of ordering. Others who have studied this relationship have found that a compromised health state is a barrier for ordering [14,28,29]. We are unable to provide an explanation for this finding at this time. The fact that four teams are located in two adjacent hallways could have led to contamination, with all or some of the components of the intervention being implemented in the control team as well as the treatment team. Since the study established a fourfold difference by treatment assignment, it seems unlikely that contamination is a major limitation, since, if it had been present, it would have served to diminish differences associated with the intervention. The findings may be attributed to some change, other than the intervention, that occurred during study period. However, we are unaware of any change that may have uniquely affected the treatment teams. The generalizability of this study is limited in several ways. Two limitations are as follows (a) the use of only one site and (b) the possibility of replicating the study elsewhere. Replication of the intervention would depend on the feasibility of assigning this responsibility to LPNs and on their ability to provide preventive services on the basis of standing orders. The fact that the treatment LPNs provided all four preventive services at higher frequency than the control LPNs may indicate that the basic relationship between ordering and delegation was subject to effect modification based on the individual LPN’s willingness to implement standing orders. Subsequent testing of this type of intervention should implement a study designed to measure such a variable.
THOMPSON ET AL.
Furthermore, other sites may not have computerized patient records necessary to generate the eligibility lists. Finally, replication of the findings could be affected by a variety of barriers to the intervention. Barriers identified on the basis of observation and interviews at the study site involved both the desk clerks and the LPNs. Occasionally the desk clerks did not identify patients as study patients and did not have enough time to dispense the kit as the patient checked out. Barriers involving the LPNs included the following: insufficient time for presenting the triage package, incomplete acquisition and/or processing of information regarding eligibility for the FOBT, infrequent disclosure of ramifications of positive FOBT findings, and periodic lack of communication with the providers regarding intent to order. Additional research is needed to confirm and enhance these results. Subsequent studies of this intervention should place more emphasis on improving the validity of the “eligibility” classification and on the patient’s right to refuse the test. During the times LPNs were observed interacting with patients, the LPNs rarely offered a clear opportunity for the patient to refuse the test. While giving more opportunity to refuse the test is likely to reduce the order rate, it is consistent with the recognition that good return rates are associated with the patient’s having made a commitment to participate in the recommended preventive behavior . Naturally, the generalizability could be improved if similar findings were replicated in other VA centers as well as non-VA primary care clinics. Finally, subsequent studies should be conducted over a long enough period of time so that it is worthwhile following up on the disposition of those with positive findings. Based on the distribution of patients in this project, that would mean processing approximately 1,700 patients in the treatment firm, which could take 4.5 to 5 months. The VA’s goal is to have 50% adherence for fecal occult blood testing by fiscal year 2000 . This intervention increased the general internal medicine clinic’s rate to 72% for eligible patients, indicating that it has the potential to enable this center to surpass this goal. ACKNOWLEDGMENTS Special mention goes to the four licensed practical nurses and four desk clerks who made the study possible. They are, in alphabetical order: Jo Bodurian, Terry Cummings, Veronica Falter, Betty Johnson, Mary Ann Murphy, Aveleen Quenga, Barbara Tague, and Carldine VanAllen. REFERENCES 1. American Cancer Society. Cancer facts and figures, 1999 Atlanta, GA: American Cancer Society, 1999. 2. U.S. Department of Veterans Affairs. Survey of medical system users: final report. Washington, DC: Department of Veterans Affairs, Assistant Secretary for Finance and Planning, Office of Planning Management and Analysis, 1990. [OPMA-M-043-90-2]
3. American Cancer Society. Cancer facts and figures, 1998. Atlanta, GA: American Cancer Society, 1998. 4. Hardcastle JD, Chamberlain JO, Robinson MHE, Moss SM, Amar SS, Balfour TW, et al. Randomized controlled trial of faecal occult blood screening for colorectal cancer. Lancet 1996; 348:1472–7. 5. Kronborg O, Fenger C, Olsen J, Jorgensen OD, Sondergaard O. Randomized study of screening for colorectal cancer with faecaloccult-blood test. Lancet 1996;348:1467–71. 6. Mandel JS, Bond JH, Church TR, Snover DC, Bradley GM, Schuman LM, et al. Reducing mortality from colorectal cancer by screening for fecal occult blood. N Engl J Med 1993;328:1365–71. 7. Winawer SJ, Fletcher RH, Miller L, Godlee F, Stolar MH, Mulrow CD, et al. Colorectal cancer screening: clinical guidelines and rationale. Gastroenterology 1997;112:594–642. 8. Liberman DA. Cost-effectiveness model for colon cancer screening. Gastroenterology 1995;109:1781–90. 9. Wagner JL, Tunis S, Brown M, Ching A, Almeida R. Costeffectiveness of colorectal cancer screening in average-risk adults. In: Young G, Levin B, editors. Prevention and early detection of colorectal cancer. London: Saunders, 1996:321–56. 10. U.S. Department of Veterans Affairs. Health promotion and disease prevention program. In: Veterans affairs handbook 1101.8. Washington, DC: Veterans Health Administration, 1996. [(RCN) 10-066] 11. National Center for Health Promotion. Preventive medicine program. Washington DC: Veterans Health Administration Office of Primary/Ambulatory Care, 1996. [Information Bulletin IB 11-89] 12. L. Branch, Associate Director, National Center for Health Promotion, VAMC, Durham, NC. [Personal communication, June, 1997] 13. U.S. Department of Veterans Affairs. Chronic disease index and prevention index report. Washington, DC: Department of Veterans Affairs, 1996. [Unpublished annual report] 14. Belcher DW, Berg AO, Inui TS. Practical approaches to providing better preventive care: are physicians a problem or a solution? Am J Prev Med 1988; 4(4 Suppl): 27–48. 15. Litzelman DK, Dittus RS, Dame MA, Tierney WM. Barriers to preventive care. Clin Res 1990;38(2):717A. 16. U.S. Public Health Service. Put prevention into practice. Implementing preventive care. Am Fam Physician 1994;50(1):103–8. 17. Belcher DW. Implementing preventive services. Arch Intern Med 1990; 150: 2533–41. 18. McDonald CJ, Hui SL, Smith DM, Tierney WM, Cohen SJ, Weinberger M, et al. Reminders to physicians from an introspective computer medical record: a 2-year randomized trial. Ann Intern Med 1984;100:130–8. 19. McPhee SJ, Bird JA, Jenkins CN, Fordham D. Promoting cancer screening. A randomized controlled trial of three interventions. Arch Intern Med 1989;149:1866–70. 20. Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician performance versus immediate reminders to perform preventive care: effects on physician compliance. Med Care 1986; 24:659–67. 21. Hulscher MEJL, Wensing M, Grol RPTM, van der Weijden T, van Weel C. Interventions to improve the delivery of preventive services in primary care. Am J Public Health 1999;89(5):737–46. 22. Cebul RD. Randomized, controlled trials using the Metro Firm System. Med Care 1991; 29:JS9-18. 23. SPSS Inc. SPSS 7.5 for Windows. Chicago, IL. 24. Stata Corp. STATA. College Station, TX. 25. White H. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 1980;48:817–30.
FECAL OCCULT BLOOD TEST ORDER RATE 26. Campbell DT, Stanley JC. Experimental and quasi-experimental designs for research. Chicago: Rand McNally, 1963: 34. 27. Curley C, McEachern JE, Speroff T. A firm trial of interdisciplinary rounds on the inpatient medical wards. Med Care 1998; 36:AS4-12. 28. Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring
physicians to respond to computerized reminders improves their compliance with preventive care protocols. J Gen Intern Med 1993;8(6):311–17. 29. Fontana SA, Baumann LC, Heilberg C, Love RR. The delivery of preventive services in primary care practices according to chronic disease status. Am J Public Health 1977; 87(7):1190–96.