Finding #2: Each of the programs studied in Special Strategies offered clear strengths, yet even when visiting sites that were multiply nominated as exemplars, we often found great variance in both implementation levels and effects.
The strengths of the various Special Strategies have been well described by their various developers. Some of those strengths are summarized in Appendix A. While some programs offer complementary components, in no sense could the programs be measured on a simple "effectiveness" scale. For example, Reading Recovery is intended to ensure initial reading and writing skills for targeted groups of first graders, while the Coalition of Essential Schools strives to instill higher levels of thoughtful knowledge and skills to teens.
By analogy, aspirin, penicillin, and chemotherapy all represent extraordinary advances in modern medicines ability to treat human suffering. But no common scale for determining which drug is "better" or "more effective" makes sense. Penicillin provides no relief to a headache, chemotherapy would not assist a patient with pneumonia, and aspirin cures no form of cancer. Each of the three is a "miracle drug" at treating a specific problem.
A quick examination of the overview descriptions of the 25 longitudinal sites in Special Strategies (see Appendix C) makes some of the context issues clear. Within Special Strategies there were schools in which under 25% of students receiving free or reduced price lunches (a common measure of poverty), and schools in which 99% of all students received free or reduced price lunch. There were schools in which over 80% of all students were of European extraction, schools in which over 90% of students were Hispanic, schools in which 100% of students were African-American, and a school in which 100% of the students were Native American. It is implausible that, even under the best of implementation conditions, one single reform type would be the "best" solution for each of these schools.
The Comer School Development Program, Success for All, Reading Recovery, and the other designs may each prove valuable in helping some schools improve, but none of the developers claim that their program can "fix" all schools or students. Some of the nominated-as-exemplary schools in Special Strategies were clearly losing their race to educate all children.
Figure 8 makes this point clearly. The Prospects background data and the
programs being implemented are the same as in Figure 7. Again, the Special Strategies
schools have student populations among which 80+% of students receive free lunch, and 90+%
are members of minority groups. However, the School Development Program was never embraced
by the principal at Comer SDP-B, and the district did not actively intervene to ensure
that the components of the program were working as intended. As a result, major components
of the School Development Program existed in name only. Similarly, at Success for All-B,
the districts support for the program was at best lukewarm. The principal who had brought
the program to the school became a district administrator, and the new principal was
selected without being informed that SFA-B was a Success-for-All school. When
district-administered test scores for the entire school did not rise in her first year at
the school, the principal dismantled the program during year two of Special Strategies.
Replacement ideas proved ineffective and reading comprehension scores fell. The school
partially re-implemented Success for All during year three.
Students in both schools increased their academic achievement gains marginally more than the Prospects averages for students attending similarly high-poverty schools. At Comer SDP-B, the CTBS Reading Comprehension NCE for students with pretests below the national average grew from 18.6 to 28.8 NCEs (N=33). At SFA-B student scores rose from a mean of 28.6 NCEs in the fall of 1990 to 34.9 NCEs in the spring of 1993 (N=20). These gains are respectable but far from the very large gains in Comer SDP-A and SFA-A. At no point did these groups of students surpass the high poverty comparison group, much less the national average.
The importance of implementation is amplified in Figure 9. This figure
presents data from 10 schools, a high- and a low-implementation school for each of five
strategies. Three of the five programs were implemented in first through third grades (the
Comer School Development Program, Success for All, extended year schoolwide projects), and
two in third through fifth grades (urban schoolwide projects and CCC). Test scores for
students in the five strong-implementation schools were averaged to produce the data
points marked with squares in Figure 9. Test scores for the weaker implementation schools
were averaged and are represented by circles on Figure 9. All data are mean CTBS reading
comprehension scores for all students with pretests below the 50th percentile (e.g.
students who were theoretically eligible for Chapter 1 services). Each school was given
equal weight. Figure 9 indicates that in the schools in which the reforms were not well
implemented or integrated, the mean student reading rose from a fall 1990 mean NCE of 29.7
(16th percentile) to a spring of 1993 mean NCE of 30.5 (18th percentile) for a gain of 1.4
NCEs. In effect, these schools, each using a seemingly promising program for improving the
academic standing of disadvantaged students, produced from very marginal to no meaningful
gains.
The five other schools each implemented one of the same set of promising programs. Initially low achieving students in these schools began with CTBS reading comprehension scores that were comparable to the beginning scores of students in the low-implementing schools (mean fall 1990 NCE = 29.2, or the 16th percentile). Yet over three years, the average initially-low achieving student in the higher implementation schools raised their reading comprehension score by 10.8 NCEs (spring 1993 mean NCE = 40.0, or the 32nd percentile).
Whereas Figures 7 and 8 present clear case studies of the effects of strong implementations of solid programs, Figure 9 provides strong evidence of the level of program implementation across five diverse programs. In Finding 4, evidence will be presented that the match of a particular program to a particular site and set of needs can matter a great deal; however, Figures 7, 8, and 9 clearly indicate that the specific program chosen only matters if the program is reasonably rigorously implemented. Replicating findings from several previous studies, Special Strategies found that level of implementation of the various programs varied greatly among sites, and that, regardless of program type, low levels of program implementation were associated with small to nonexistent three-year academic gains.
