mdrc

Building Knowledge
To improve Social policy

Sustained positive effects on graduation rates

PolICY

BRIEF

produced by new york city’s Small public High Schools of choice

January
2012

D

By Howard S. Bloom and Rebecca Unterman

uring the past decade, New
York City undertook a districtwide high school reform that is perhaps unprecedented in its scope, scale, and pace.
Between fall 2002 and fall 2008, the school district closed 23 large failing high schools
(with graduation rates below 45 percent),1 opened 216 new small high schools (with different missions, structures, and student selection criteria), and implemented a centralized high school admissions process that assigns over 90 percent of the roughly
80,000 incoming ninth-graders each year based on their school preferences.
At the heart of this reform are 123 small, academically nonselective, public high schools. Each with approximately 100 students per grade in grades 9 through 12, these schools were created to serve some of the district’s most disadvantaged students and are located mainly in neighborhoods where large failing high schools had been closed.2 Hence, they provide a realistic choice for students with widely varying academic backgrounds. MDRC researchers call them “small schools of choice” (SSCs) because of their small size and the fact that they do not screen students based on their academic backgrounds.3
In June 2010, MDRC, a nonprofit, nonpartisan research organization, released a report on the effectiveness of 105 of the 123 new SSCs, which demonstrated demonstrated that they are markedly

improving academic progress and substantially improving graduation prospects, particularly for disadvantaged students. This report included data for over 21,000 students from four cohorts who entered ninth grade between fall
2005 and fall 2008.4 Findings in the report were based on a series of randomized lotteries that were used when SSCs had more applicants than seats, in order to determine who got offered an opportunity to enroll in an SSC and who did not. SSC lotteries are a little-known byproduct of the current district-wide high school application processing system (HSAPS).5
These lotteries make it possible to rigorously estimate the effects of enrolling in an SSC versus enrolling in some other
New York City public high school, based on a comparison of subsequent outcomes for lottery winners and lottery losers and a procedure that accounts for who among them enroll in an SSC.6
In a large sample, like that used for the
MDRC study, lottery winners and lottery losers are the same, on average, in all ways before they enter high school.
Consequently, it is valid to attribute any differences in their future academic outcomes to their access to an SSC.
Because students who lose an SSC lottery attend over 200 high schools that vary widely in their size, age, structure, academic programs, and effectiveness, the MDRC report judged SSCs against the

MDrC PoliCy Brief

overall effectiveness of a diverse group of other high schools.7 The results released in
2010 indicated that, on average, the 105 SSCs studied increased student progress toward graduation during their first three years of high school and increased students’ four-year graduation rates.

resources during start-up.8 By integrating a demanding and comprehensive academic curriculum, personal attention to student academic progress, and real-world experiences with community partners, the new small schools intended to prepare students for both college and career.

Given data that were available at the time, the first MDRC report could only follow the first student cohort (that entered in fall
2005) through four years of high school to examine its graduation rates. This policy brief extends the analysis by a year, which adds information on high school graduation rates for the 2006 cohort and provides a fifth year of follow-up for the 2005 cohort. This information makes it possible to address the following three questions:
• Was the positive average effect of SSCs on four-year graduation rates for the study’s first student cohort sustained through the second cohort?
• Was this positive average effect sustained across subgroups of students with different prior academic proficiency, family income, race/ethnicity, and gender?
• Was the average four-year graduation effect sustained after five years?

In other cities, small high schools are often fashioned by reconfiguring large existing schools into smaller units in the same buildings with the same teachers and students; in contrast, the typical SSC was created largely
“from scratch” with a principal, teachers, and students who were new to the school. At the same time, many SSCs are located with other
SSCs in buildings that previously housed a large public high school that was closed.

The answer to all three of these questions is yes.

2

w H aT a r e S S c s ?
SSCs were developed through a competitive proposal process that was designed to ensure that school founders met specified conditions and to stimulate innovative ideas from a range of stakeholders and institutions.
SSCs emphasize academic rigor and strong and sustained personal relationships among students and faculty. In addition, most were founded with community partners who offer students relevant learning opportunities inside and outside the classroom and provide school faculty with additional staffing support and

SSCs are not just small in size. They are purposefully organized around smaller educational units that are designed to give students a better chance of being known by their teachers and other adults in the building.
For example, many SSCs have created special student advisory structures that provide common planning time for teachers to discuss their students’ progress and problems.
SSCs received start-up funding from philanthropic organizations plus technical assistance and policy support from the New
York City Department of Education and intermediary organizations that were often experienced in managing the launching of new schools — which helped to facilitate school leadership development, staff hiring, and program start-up. Most SSCs began with only a ninth-grade class, adding a subsequent grade each year. Finally, SSCs received special allowances with respect to serving English language learners and special education students during their first two years of start-up.9
The creation of SSCs was supported by a consortium of funders, led by the Bill &

January 2012

Melinda Gates Foundation, the Carnegie
Corporation of New York, and the open
Society Institute. SSCs were implemented in collaboration with the United Federation of Teachers and the Council of School
Supervisors and Administrators.10 Unlike charter schools, which typically operate independently of the school district in which they are located, SSCs are directly responsible to the New York City Department of Education.

wHy STudy SScs?
Almost all major urban school districts in the
U.S. and all but four states have attempted to create new small schools or attempted to transform large schools into campuses with “small learning communities.”11 This movement had its beginnings in the 1960s, when urban educators and community organizations began to implement smaller school structures as an alternative to the large high schools that dominated the landscape, many of which were failing. By the late 1990s, small schools had emerged as a national reform strategy championed by such groups as the Coalition of Essential Schools and the
National Association of Secondary School
Principals. This strategy proliferated through district- and foundation-led initiatives in major cities like Chicago, Philadelphia, Boston, oakland, and New York. In 2000, the Bill &
Melinda Gates Foundation launched a national campaign to improve failing urban high schools, with small schools as a key strategy.12
The common theory of change underlying both small schools and small learning communities specifies that smaller educational units promote stronger relationships among students, among adults, and between students and adults.
These relationships, in turn, are expected to increase student engagement and help teachers respond to students’ academic and social needs, thereby increasing students’

academic achievement.13 Prior research on small schools has produced mixed results about their effectiveness. Some studies have found that they improve student outcomes, such as increasing levels of academic achievement and lowering rates of school dropout.14 In addition, some studies suggest that these effects are most pronounced for disadvantaged students.15 on the other hand, a major study of a large number of small schools initiatives suggests that they do not improve student academic outcomes appreciably.16 In all of these cases however, the nonexperimental nature of the research limits its ability to support causal conclusions.
MDRC’s findings about SSCs are relevant to current federal policy on high school reform, particularly the U. S. Department of
Education’s School Improvement Grants
(SIGs) for failing schools. Reforms funded by
SIGs include school transformation, school restart, school closing, and school turnaround.
SSCs straddle several of these categories since they are typically replacements for schools that have closed and they operate as regular public schools. However, it is important to recognize that SSCs represent far more than just changes in school size and structure.
They also represent innovative ways to use these structural changes to leverage human, financial, and curricular resources.
The coupling of small schools with a system of choice for all entering ninthgraders also has relevance for the national debate about charter schools. on the one hand, SSCs share many similarities with charters — they are small and personalized; most began with the help of intermediary sponsors that resemble charter management organizations; the students and teachers are there by choice; and the curricula are demanding. on the other hand, SSCs are overseen by the Department of Education

3

MDrC PoliCy Brief

Table 1. estimated average effects of SSCs on Four-Year Graduation Rates: Student Cohorts 1 and 2
TargeT
SSc enrolleeS conTrol group counTerparTS

67.9

59.3

8.6 **

0.000

local diploma granted

19.9

19.6

0.3

0.406

regents diploma granted

41.5

34.9

6.5 *

0.041

advanced regents diploma granted

6.2

4.7

1.6

0.154

passed math regents at 75 or higher

23.3

22.5

0.8

0.682

passed english regents at 75 or higher

37.3

29.7

7.6 **

0.002

ouTcome (%)

eSTimaTed eFFecT p-value For eSTimaTed eFFecT

graduaTion graduaTed From HigH ScHool

college readineSS

NoTES: There are 13,064 student observations for cohorts one and two combined. There are no statistically significant differences between estimated effects for the two cohorts. Because of rounding error and the complex calculations involved, results by diploma type do not sum exactly to those for overall graduation rates.

with unionized teachers and principals; the schools are nonselective and open to students regardless of academic proficiency; and the school choice process was designed to engage all entering ninth-graders, not just the most motivated parents and children that might seek out charter schools.

How mucH do SScs
Help STudenTS? previous Study Findings: Because students became part of the study sample in four annual cohorts, MDRC’s first report on the effects of SSCs was based on four years of follow-up for the first cohort of entering ninth-graders, three years of follow-up for the second, two years of follow-up for the third, and one year of follow-up for the fourth cohort.

4

The report found similar positive effects on progress toward graduation in ninth grade for subgroups of students who varied in terms of their reading and math proficiency before entering high school, their eligibility for free or reduced-price lunches, and their race/ethnicity and gender.
At two and three years after students entered high school, the study found that SSCs increased their progress toward graduation by roughly the same amount as they did during the first year. And at four years after students entered high school, the study found that
SSCs increased high school graduation rates by 6.8 percentage points (to 68.7 percent for target SSC enrollees from 61.9 percent for their control group counterparts).

Based on data for all four cohorts, the report found that SSCs increased the percentage of students who were on track17 toward graduation in ninth grade by 10.0 percentage points (to 58.5 percent for “target SSC enrollees” from 48.5 percent for their control group counterparts).18

new Findings on graduation and college readiness for Two Student cohorts: Findings in Table 1 indicate that SSC effects on four-year high school graduation rates are sustained through a second cohort of entering ninth-graders. Enrolling in an SSC instead of another type of New York City public high

January 2012

school increased average four-year graduation rates for the first two cohorts combined by 8.6 percentage points (to 67.9 percent for target
SSC enrollees from 59.3 percent for their control group counterparts).19 Almost all of this graduation effect reflects an increase in receipt of New York State Regents diplomas.
For this type of diploma, students must pass a minimum of five Regents examinations
(English language Arts, Mathematics,
Science, Global History, and American
History) with a score of 65 points or above and must pass all courses that are required by the state. None of the SSC graduation effect appears to reflect an increase in receipt of local diplomas, which have less stringent standards for scores on Regents examinations and are being phased out by New York State for the graduating class of 2012. likewise, little if any of the SSC graduation effect reflects an increase in receipt of Advanced
Regents diplomas, which have the most stringent standards and are received by very few students who apply to SSCs.
Table 1 also indicates that enrolling in an
SSC increased students’ college readiness in English but not in math, as measured by the percentage who scored 75 points or higher on the corresponding New York State
Regents examinations.20 This threshold is used by the City University of New York to exempt students from taking remedial courses in these subjects. new Subgroup Findings on graduation for Two Student cohorts: Adding a second cohort of students to the analysis of SSC effects on four-year graduation rates increased the sample size by enough to make it possible to examine how these effects vary across subgroups of students with diverse backgrounds. Findings in Table
2 indicate that SSCs increased four-year high school graduation rates appreciably for all

subgroups examined that had samples that were large enough to support findings.21
Consider the results for subgroups defined by students’ prior academic proficiency, as measured by their eighth-grade state test scores in reading and math. New York State reports this proficiency in four levels. levels 1 and 2 (did not meet or partially met standards) represent student performance that is below grade level. levels 3 and 4 (fully met standards or met standards with distinction) represent performance that is at or above grade level.
Since very few sample members score in the top category, no findings are reported for it.
The findings indicate that SSCs increased fouryear graduation rates appreciably for students in levels 1, 2, and 3 of reading and math prior proficiency. Although specific estimates vary somewhat
(from 7.7 to 11.3 percentage points for reading and from 8.6 to 11.4 percentage points for math), all of the effects are appreciable and statistically significant, indicating that every group benefited substantially from attending an SSC. For example the 11.4 percentage point gain in high school graduation rates produced by SSCs for students with a level 1 eighth-grade math proficiency score implies a 33 percent increase (11.4/34.5) in graduation rates for this subgroup of students with especially weak math preparation. However, there is no clear pattern to the variation in results across proficiency subgroups; so while the impacts are statistically significant, the reported differences between subgroups are not statistically significant.
Therefore, one should not conclude, for instance, that SSCs worked differently for level 1 students than for level 2 students, even though the point estimates differ.
These subgroup findings by academic proficiency are especially striking given the dramatic differences that exist in the

5

MDrC PoliCy Brief

Table 2. estimated effects of SSCs on Four-Year Graduation Rates for Student Subgroups: Student Cohorts 1 and 2
STudenT cHaracTeriSTic

TargeT
SSc
enrolleeS

conTrol group counTerparTS

eSTimaTed eFFecT p-value For eSTimaTed eFFecT

8TH-grade reading proFiciency not meeting State learning Standards (level 1)

37.3

29.6

7.7 *

0.044

partially meeting State learning Standards (level 2)

66.7

55.5

11.2 **