Finding #4: While the literatures on most programs were promising, norm-referenced achievement measures of effects within the limited Special Strategies samples were more mixed. In general, promising programs that concentrated their efforts in the early grades tended to obtain larger achievement gains from students placed at risk than did programs spreading resources more evenly over the elementary grades or in secondary schools. Within the schools observed during first through third grades, students in schools using externally developed designs tended to achieve greater academic gains than did students in locally developed programs. Students in schools working with whole school reform tended to achieve greater gains than did students in schools attempting various pull-out programs. None of the secondary schools achieved stable implementation across the full school, and, perhaps as a result, none produced a pattern of achievement gains.
The literature reviews conducted on each of the ten promising programs provided statements, and in most instances, quantitative studies, suggesting that each of the programs could work under some circumstances. (See Appendix A of this overview, and the Final Report.) Given that the sites followed through three years by Special Strategies researchers were nominated as exemplars of their program or design types, the research teams initial expectation was that academic achievement gains would be nearly universally positive. This expectation was not realized.
The background characteristics of the longitudinal Special Strategies schools varied greatly. Variables beyond the control of the school and program, such as regional recessions affecting several communities, city-wide riots affecting two others, and levels of drug infestation in the communities surrounding the schools, all had impact on schools abilities to implement programs and to obtain desired effects on students achievements. Given that Special Strategies research teams gathered longitudinal data in only two schools per program type, that programs had diverse goals, and that there were often powerful contextual variables affecting outcomes, the following data should be examined with caution.
Using the Reading Comprehension subtest of the Comprehensive Test of Basic Skills, Figures 10a through 10d provide mean reading comprehension scores over three years for initially low achieving students attending the pairs of schools nominated as exemplars within the various programs. A second level of analysis, including Figures 11 through 13, provides data broken by several cross-program and context variables.
Figure 10a provides background data on the Prospects first grade cohorts average reading comprehension scores through spring of third grade for students attending very high poverty schools (e.g., 75+% free lunch) and with initial test scores below the 50th percentile. As with Figures 7 and 8, there data are shown on the line defining the shaded area of the figure. Figure 10a also provides data averaged across the pairs of schools participating in the first-through-third grade whole-school change programs. Two of three early intervention whole-school reform efforts obtained clear CTBS Reading Comprehension gains relative to the national comparison group, with the third moving at approximately the national rate.
Figure 10b provides data on means for Special Strategies and Prospects students involved in programs targeted to specific within-school sub-populations. CTBS Reading Comprehension Prospects first grade data are on the sample of students (presented through third grade) who received Chapter 1 services during at least first grade. The Prospects data are from schools in which at least 75% of students received free or reduced price lunches (shaded area). Figure 10b indicates that, while students in one group were able to stay consistently above the national mean, in none of the three adjunct programs did the average of the two participating schools targeted groups of students achieve reading comprehension gains that clearly surpassed the national average Chapter 1 gain over three years.
Figure 10c provides mean CTBS Reading Comprehension scores over time for
reforms that were followed from the fall of grade three through the spring of grade five.8 Only one of four reforms produced above average reading
comprehension gains for initially low achieving students.
Figure 10d examines reading comprehension data from the four high school
(grades 9-11) projects on which CTBS data were available. The Coalition of Essential
Schools (CES) was the only reform followed in the ninth through eleventh grade range. No Prospects
control data were gathered across this grade range. The two urban schools are presented as
one group, and the two rural as a second. Neither the urban nor the suburban/rural pair of
schools achieved CTBS Reading Comprehension progress over the four cycles of testing.
Rather, a moderate drop in scores among initially low achieving students was typical
within these nominated as exemplary schools.
While three of the six elementary-level whole-school reform programs studied in Special Strategies clearly exhibited multi-year reading comprehension gains for initially low-achieving students; on average, the other schools involved in elementary grades targeted (pull out) programs did not produce long term reading comprehension gains for their students. Initially low achieving students in the high school reform did not obtain mean reading gains.
The research teams initial assumption that in the main we would be tracking exemplary success across the great majority of sites did not prove accurate. A discussion of these patterns across not just programs but schools contexts follows.
Three years of Special Strategies observations produced dozens of stories of individuals and whole schools struggling valiantly to improve the academic lives of students living in poverty. Yet by the end of Special Strategies, some of the considerable efforts made by educational professionals seemed to be reaping greater rates of academic growth than others. Effects of strong versus weak or inconsistent implementations were discussed in Finding #2. Three additional achievement-related findings are discussed in the following paragraphs.
Within Special Strategies it appeared that programs concentrating scarce resources early in students careers, before a pattern of failure could set in, were more successful than schools at which the intervention began later.9 Figure 11 provides an overview of CTBS reading comprehension data over three years for the different Special Strategies starting-grade cohorts.
Academic gains as measured by mean Reading Comprehension scores on the CTBS were greater for the full range of students and for students beginning the study with achievement scores below the 50th percentile at schools that concentrated their improvement resources in the early grades. By concentrating efforts in the primary grades, the first grade programs had raised mean achievement levels for whole cohorts to near the national average by the end of grade three (mean NCE = 44.6 for all students, and 40.0 for students with beginning scores below the 50th percentile). These represented achievement gains of over eight NCEs for the whole cohort, and over 11 NCEs for students who began the study with reading levels below the national average.
By contrast, the cohorts followed in grades three through five were achieving at lower levels at the spring of third grade (Mean NCE = 36.2 for all students, and 25.9 for initially low achieving students). The schools followed third through fifth grade were unable to produce stable patterns of reading gains (mean NCE change = -4.0 for all students, and -2.4 for students with initially low academic achievement levels).
All of the secondary schools participating in Special Strategies were involved in the Coalition of Essential Schools. As noted earlier, none was able to sustain whole school implementation of CES. While qualitative data noted many examples of individual teachers instructional efforts and students declarations of the value of the program, none of the schools was able to produce a pattern of academic gain on the CTBS Reading Comprehension or Mathematics tests. It should be remembered that Theodore Sizer, founder of the Coalition, has repeatedly expressed the view that demonstration of knowledge rather than norm-referenced test scores are the desirable measures of the impact of the Coalition of Essential Schools.
Given limited numbers of schools implementing any specific program type and the small number of Special Strategies programs and schools in the third- and ninth-grade samples, further achievement outcome analyses in this overview are limited to the first-grade cohort and its 11 schools.10 As Figure 12 indicates, the interventions concentrated in the primary grades tended to produce positive effects for whole schools, and large positive effects for students whose pretests were lowest. This was a positive finding. Among those schools, students attending schools that were attempting whole school reforms tended to receive a greater three year academic benefit. Analyses of mathematics gains produced similarly large gain scores, again with differences favoring whole school reforms.
Following the two dimensional differentiation among program types from Figure 1, Figure 13 presents mean gains in reading comprehension for first- through third-grade students attending Special Strategies schools. The data reaffirm that most of these students experienced academic gains. Figure 13 adds data suggesting that schools using externally developed designs were generally achieving greater academic gains for their students. Schools often drew unifying strength and focus from their associations with external, reform-focused organizations. It was often the case that the tighter the connection, the greater the schools ability to sustain the reform. Both Comer SDP-A and SFA-A in Figure 7 had strong histories of connections to the respective program development teams. The Reading Recovery program had similarly strong ties to Reading Recovery teachers at their sites.
In retrospect, it does not seem surprising that the two schools in Figure 7 produced
unusually large gains in reading. Each involved a relatively strong implementation, began
early, involved virtually the whole faculty, and sustained connections to a focused
national reform effort.
8 In one of two computer assisted
instruction sites, CAI services were provided to all students in the school. The other
provided targeted,
pull-out services in a computer lab. However, the schools are presented
together in Figure 11.
9 Other studies, involving
programs not studied in Special Strategies, provide evidence of positive effects in the
middle school years, e.g.,
Pogrow, 1990; Stevens & Durkin, 1992; MacIver & Plank, 1996.
However, the general trend favoring early intervention found in Special
Strategies appears to be reflective of the larger field (Slavin, Karweit,
& Wasik, 1994).
10 Analyses including individual
programs and schools are available in the Special Strategies Final
Report, Ch. 15-18.
