Search Publications | The Evaluation Exchange | FINE Network | OST Database

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

A Profile of the Evaluation of Citizen Schools

Program Description

Overview	Citizen Schools (CS) operates a national network of apprenticeship programs for middle school students that connects adult volunteers to youth in hands-on afterschool learning projects. The CS programs are designed to help youth develop academic and leadership skills needed to succeed in school, get into college, and become leaders in their careers and in their communities.
Start Date	1995
Scope	national
Type	afterschool
Location	urban
Setting	public school

Participants	middle school students (grades 6–8)
Number of Sites/Grantees	30 sites in 2006–2007 nationwide; 37 sites nationwide (2009–2010)
Number Served	approximately 3,000 in 2006–2007 nationwide; approximately 4,000 nationwide (2009–2010)
Components	Twice a week after school, CS youth participate in apprenticeships, which consist of hands-on learning projects led by volunteer Citizen Teachers. Apprentices work collaboratively in small groups to build solar cars, litigate mock trials, publish children's books, and so on. On other weekday afternoons, CS staff lead structured afterschool activities to enhance school success of youth, working on homework and building organizational and study skills to help youth navigate their education through high school, graduation, and beyond. Each semester culminates in “WOW!”—a public presentation of the CS participants’ projects. In 2001–2002, CS launched its 8th Grade Academy program in Boston for eighth graders who began CS in a prior year. In addition to offering activities similar to those at other CS campuses, 8th Grade Academy is intended to help youth apply to and succeed in competitive high schools and to introduce them to the college application process through experiential learning activities that build academic and life skills and give youth access to coaches, technology, internships, and other educational programs. Each participant is assigned a writing coach (typically a local lawyer). In 2004, CS launched an alumni program to support Academy graduates during the high school transition period.
Funding Level	approximately $20.9 million in 2009–2010
Funding Sources	Major funders include ArcLight Capital Partners, The Atlantic Philanthropies, Bank of America, Josh & Anita Bekenstein, Edna McConnell Clark Foundation, The John S. and James L. Knight Foundation, The Koogle Foundation, The Lovett-Woodsum Foundation, The Picower Foundation, The Samberg Family Foundation, Skoll Foundation, AmeriCorps, State of North Carolina Department of Social Servies, Boston Public Schools, Duke Endowment, Houston Independent School District, Nellie Mae Education Foundation, and many other individual, corporate, and philanthropic donors.

Evaluation

Overview	A 7-year evaluation (2001–2008) focused on the program experiences and outcomes of 5 cohorts of participants at the Boston site.
Evaluators	Policy Studies Associates, Inc.
Evaluations Profiled	Evidence from Two Student Cohorts on the Use of Community Resources to Promote Youth Development: Phase II Report Putting Students on a Pathway to Academic and Social Success: Phase III Findings Preparing Students in the Middle Grades to Succeed in High School: Findings from Phase IV Progress Toward High School Graduation: Youth Outcomes in Boston Achieving High School Graduation: Citizen Schools’ Youth Outcomes in Boston
Evaluations Planned	Citizen Schools is proceeding with a new evaluation plan in which Abt Associates Inc. and Public/Private Ventures will serve as external evaluators.
Report Availability	Fabiano, L., Espino, J., & Reisner, E. R., with Pearson, L. M. (2003). Citizen Schools: Using community resources to promote youth development. Phase I Report of the Citizen Schools evaluation. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2003.pdf Espino, J., Fabiano, L., & Pearson, L. M. (with Kirkwood K. P., Afolabi, K., & Pasatta, K.). (2004). Citizen Schools: Evidence from two student cohorts on the use of community resources to promote youth development. Phase II report of the Citizen Schools evaluation. Washington, DC: Policy Studies Associates. Fabiano, L., Pearson, L. M., & Williams, I. J. (2005). Putting students on a pathway to academic and social success: Phase III findings of the Citizen Schools evaluation. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2005.pdf Fabiano, L., Pearson, L. M., Reisner, E. R., & Williams, I. J. (2006). Preparing students in the middle grades to succeed in high school: Findings from Phase IV of the Citizen Schools evaluation. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2006.pdf Pearson, L. M., Vile, J. D., & Reisner, E. R. (2008). Establishing a foundation for progress toward high school graduation: Findings from Phase V of the Citizen Schools Evaluation. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2008.pdf Vile, J. D., Arcaira, E., & Reisner, E. R. (2009). Progress toward high school graduation: Citizen Schools’ youth outcomes in Boston. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2009.pdf Arcaira, E., Vile, J. D., & Reisner, E. R. (2010). Achieving high school graduation: Citizen Schools’ youth outcomes in Boston. Washington, DC: Policy Studies Associates. Available at: www.policystudies.com/studies/?download=79&id=39

Contacts

Evaluation	Erikson Arcaira Research Associate Policy Studies Associates, Inc. 1718 Connecticut Avenue, NW, Suite 400 Washington, DC 20009 Tel: 202-939-5343 Fax: 202-939-5732 Email: earcaira@policystudies.com
Program	Michael Kubiak Director of Research and Evaluation Citizen Schools 308 Congress Street, 5th Floor Boston, MA 02210 Tel: 617-695-2300 ext. 132 Fax: 617-695-2367 Email: michaelkubiak@citizenschools.org
Profile Updated	May 9, 2011

Evaluation 5: Achieving High School Graduation: Citizen Schools’ Youth Outcomes in Boston

Evaluation Description

Evaluation Purpose	To explore whether former 8th Grade Academy participants achieve positive academic outcomes and successfully complete high school.
Evaluation Design	Quasi-Experimental: Data were collected on 8th Grade Academy youth who attended Boston Public Schools (BPS) in one of five cohorts: Year 1 (2001–2002) through Year 5 (2005–2006). Each participant was matched to 3 nonparticipants who attended BPS, based on five core matching variables: gender, race, grade in school, free or reduced-price lunch (FRPL) eligibility, and grade 4 Massachusetts Comprehensive Assessment System (MCAS) test scores in math and English language arts (ELA). When possible, youth were also matched on school attended and bilingual/special education status. With the exception of constants like grade 4 MCAS scores, these data were based on the year prior to each youth’s participation in the evaluation. Each participant’s 3 matches were ranked based on how similar they were to the participant on the matching criteria. Participants were compared to their most similar nonparticipant. Matched nonparticipants and participants were similar on all matching criteria. A participant’s matched nonparticipant sometimes changed from year to year; the second or third most similar match sometimes had to be used when a matched nonparticipant left the BPS system or was not promoted on schedule. Between Year 1 and Year 5, 448 youth participated in 8th Grade Academy and had parental consent to be included in the evaluation; 48 participated in Year 1, 85 in Year 2, 118 in Year 3, 103 in Year 4, and 94 in Year 5. To be included in the analyses, youth had to be promoted to grade 9 at the end of the year in which they participated in 8th Grade Academy; 97% of the sample did so. Grade 9 promotion did not vary significantly by class year, nor between participants and nonparticipants. Analyses focused on former 8th Grade Academy youth as they transitioned to high school and progressed through grade 12.
Data Collection Methods	Secondary Source/Data Review: BPS and four Boston-area charter schools provided data on student achievement (MCAS scores and course grades), demographics, and school performance (e.g., attendance, high school choices). For the measure of high school choices, BPS high schools were classified as top-tier (referred to in previous evaluations as high quality) or non-top-tier (referred to in previous evaluations as medium or low quality). CS encouraged 8th Grade Academy graduates to attend top-tier schools. In classifying schools, CS considered school size, MCAS scores, percent of graduates attending college or postsecondary training, attendance, dropout rates, grade 9 retention, reputation, and relationship with CS (since an existing partnership would help CS track participants and support them through high school). Test/Assessments: MCAS is a standardized test required by Massachusetts public schools. Results are reported as scaled scores and performance levels, defined as follows: Advanced, has a comprehensive and in-depth understanding of rigorous subject matter and provides sophisticated solutions to complex problems; Proficient, has a solid understanding of challenging subject matter and solves a wide variety of problems; Needs Improvement, has a partial understanding of subject matter and solves some simple problems; and Warning, has a minimal understanding of subject matter and does not solve simple problems.
Data Collection Timeframe	Data covered the 2001–2002 academic year (Year 1) through the 2007–2008 academic year (Year 7).

Findings:
Summative/Outcome Findings

Academic

Former 8th Grade Academy participants were significantly more likely to select a top-tier high school than were matched nonparticipants (59% vs. 28%, p < .001).

Former 8th Grade Academy participants were significantly more likely than nonparticipants to persist in a top-tier high school in grades 11 (67% vs. 49%, p < .05) and 12 (58% vs. 32%, p < .05). In addition, former participants who enrolled in a top-tier high school in 9th grade were more likely to persist in a top-tier high school in 10th grade than were their matched counterparts (85% vs. 75%), although this difference was not significant,

Looking across all 4 years, former participants were significantly more likely than nonparticipants to select a top-tier high school and persist in such a school for all 4 years of high school (31% vs. 9%, p < .001).

Former participants attended school at significantly higher rates than did matched nonparticipants in every grade of high school. Compared to matched nonparticipants, former participants attended an average of 9 more days in 9th grade (p < .001), 5 more days in 10th grade (p < .05), 13 more days in 11th grade (p < .001), and 9 more days in 12th grade (p < .01).

Former 8th Grade Academy participants and matched nonparticipants were suspended at similar rates at all grade levels. No significant difference was found in the suspension rates of former participants and nonparticipants at any grade level.

Former participants had significantly higher pass rates in math courses than did matched nonparticipants in grade 9 (74% vs. 65%, p < .05), grade 10 (79% vs. 64%, p < .001), and grade 11 (78% vs. 68%, p < .05). The groups passed at similar rates in grade 12.

Former participants had significantly higher rates of earning As and Bs in math courses than did matched nonparticipants in grade 9 (33% vs. 23%, p < .05) and grade 10 (38% vs. 26%, p < .01). Rates were similar in grade 11. Participants outperformed nonparticipants in grade 12, with 33% vs. 23% earning As and Bs, respectively (though this difference was not statistically significant).

Former participants passed their ELA courses significantly more often than matched nonparticipants in grade 9 (82% vs. 69%, p < .01) and grade 12 (88% vs. 66%, p < .01). Pass rates were similar between the two groups in grades 10 and 11.

While ninth grade participants had a higher percentage of students earning As and Bs in their ELA courses, the difference was not statistically significant. Participants had higher rates of As and Bs in ELA courses in grade 10 (44% vs. 28%, p < .001). While 12th grade participants had a higher percentage of students earning As and Bs than matched nonparticipants (40% vs. 30%), the difference was not statistically significant.

Participants and matched nonparticipants passed their math and MCAS examinations at similar rates for both ELA (89% vs. 86%) and math (94% vs. 91).

Former participants earned Proficient and Advanced levels on the MCAS at significantly higher rates than their matched peers (53% vs. 44% in math and 50% vs. 41% in ELA, p < .05 for both).

A significantly higher percentage of former participants were promoted on time from 9th to 10th grade than were matched nonparticipants (83% vs. 76%, p < .01) On time promotion rates to 11th and 12th grades were not significantly different between the two groups.

A significantly higher percentage of former participants were on track to graduate (as defined by passing on time from 9th to 10th grade without having failed a math or ELA course) at the beginning of 10th grade, as compared to their matched peers (63% vs. 51%, p < .01).

A significantly higher percentage of former participants graduated on time compared to matched nonparticipants (73% vs. 59%, p < .05).

A Profile of the Evaluation of Citizen Schools

Profile Information